Application of the RoboSurgeon System in Medical Care

Fig. 10.1

RoboSurgeon system. Three dimensional (3D)

There are basically two different types of HMDs: the non-see-through type and the see-through type. We can select the type of HMD based on the situation. We have applied binocular non-see-through-type HMDs, HMM-3000MT (designed for medical use) as a surgical monitor for surgeon and assistant in endoscopic surgery and other medical procedures and HMZ-T2 or HMZ-T3W (designed for consumer use) for examinee, both of which have a binocular display composed of 0.7-inch organic light-emitting diode screens (resolution: 1,280 × 720 pixels) for each eye (Sony Corporation). While the immersive view with sharp and high-contrast image quality provided by the non-see-though type HMD is advantageous for surgeons, scrub nurses need to have a comprehensive view of what is happening in the operation room. Scrub nurses who attend endoscopic surgery use a see-through-type HMD (AiRScouter^®, Brother Industries Ltd.). This is a monocular-type HMD, which provides the wearer with a transparent image in front of the eye without impacting the field of vision through a high-definition liquid crystal panel with a resolution of 800 × 600 pixels.

Here we introduce the application of the HMD in transurethral endoscopic surgery, prostate biopsy, brachytherapy, ultrasound scanning, and telemedicine. We also show the utility of HMD for the assistants in the da Vinci Surgical System^® (Intuitive Surgical) and for scrub nurses.

10.2 Ureteral Stent Placement and Nephroureteroscopic Examination

Ureteral stent placement and nephroureteroscopic examination are carried out through the cystoscopic or nephroureteroscopic view under the guidance of fluoroscopy. The operators heretofore performed these procedures using a separate monitor for each image. However, these multiple forms of information can be seamlessly integrated using multiplexers, and the composite image is displayed on the HMD. Applying the RoboSurgeon system as a personal integrated imaging monitor provided through the HMD is advantageous for these procedures.

The application of the RoboSurgeon system in ureteral stent placement and nephroureteroscopic examination is shown in Fig. 10.2 [12–14]. The fluoroscopic image, the endoscopic image, the patient’s vital signs, and the view from a head-mounted camera attached to the HMD were connected to a high-definition HMD through a multiplexer (VMI-40MD, Sony Corporation, or VPM-H1, MEDIA Corporation). These integrated images were displayed on the HMD using a four-split screen or a picture-in-picture feature (Fig. 10.3). By using this HMD, all the participants could simultaneously and continuously monitor the endoscopic and fluoroscopic view in an ergonomically natural position during the procedures. When wearing the HMD device, the operator can use direct vision by moving the angle of sight downward. However, the view obtained from the camera attached to the HMD device allows the operator to avoid glancing aside.

Fig. 10.2

Scene from ureteroscopic examination in which the operator and assistant use a head-mounted display (HMD). The operator and assistant simultaneously share the composite image displayed on the HMD

Fig. 10.3

(a) The multiple information streams were integrated on the multiplexer using a four-split screen technique. (b–d) Captured image of displayed integrated image data during (b) ureteral stent placement and (c, d) biopsy under ureteroscopy. In these figures, each type of input information is color coded

Tip Each HMD wearer can select the input information displayed on the HMD.

10.3 Transurethral Resection of the Prostate

Conventionally, transurethral resection of the prostate (TURP) has been performed under endoscopic vision only. However, we can use the rectum as a “window” by which monitor the surgical field using transrectal ultrasonography (TRUS) in a simple manner. TRUS provides useful information to identify the landmarks for TURP, including the contour of the prostate, the shape of the protruding adenoma in the bladder, and the shape of the apex. Applying TRUS as an intraoperative navigation for TURP is a way to improve treatment efficacy and to minimize the possible risks, including severe bleeding, perforation of the prostatic capsule, and injury to the urethral sphincter.

The imaging information obtained from the cystoscope, the TRUS, the video camera attached to the HMD, and the patient’s vital monitor were split using an imaging splitter (400-VGA003, Sanwa Supply Incorporated), and the images were output into two multiplexers. The composite images integrated on the multiplexer were displayed on the HMDs. The lead surgeon and the assistants could simultaneously and continuously monitor the integrated multiple images through the HMD (Fig. 10.4). By each team member wearing an HMD, the lead surgeon and the assistants share the same information but can independently arrange their image arrays as they see fit, depending upon their roles and the steps of the surgery, and they could do so in an ergonomically efficient posture (Figs. 10.5 and 10.6) [15, 16].

Fig. 10.4

Scene from transurethral resection of the prostate under the guidance of transrectal ultrasonography, in which the operator and assistant use a head-mounted display (HMD). Both the operator and the assistant wear an HMD

Fig. 10.5

Each head-mounted display (HMD) wearer can arrange the images displayed in the HMD, based on the engaging procedure. Captured image of displayed data through HMD set (a) an endoscopic view and (b) an ultrasonographic view as a primary image during transurethral resection of the prostate

Fig. 10.6

The composite image of an ultrasonographic view and video view attached on the head-mounted display using a picture-in-picture feature during the evacuation of resected tissues

Tip Depending upon the roles and steps of the surgery, each HMD wearer can select the array of the integrated image information displayed on the HMD.

10.4 3D Image System for Transurethral Surgery

In accordance with the growing availability of high-definition binocular 3D endoscopes, 3D imaging techniques are increasingly used in minimum invasive surgeries. However, transurethral surgeries are still performed under two-dimensional (2D) vision, because of the technical difficulties in introducing 3D technology to transurethral surgery, the double vision induced by the large parallax of binocular endoscope at close range, and the need for thinner optical scopes.

The first practical system that converts conventional 2D endoscopic images to 3D images in real time was developed in cooperation with our efforts (Fig. 10.7). This system uses 2D endoscopic images as the only input for the algorithm, and it can be combined with any conventional 2D endoscope. The combination of this system with the 3D-HMD system (HMM3000-MT) allows the operator to perform transurethral surgery using 3D vision on the HMD (Fig. 10.8).

Fig. 10.7

The conventional two-dimensional endoscopic image is converted through the imaging converter in real time. The processed three-dimensional (3D) image is output into the head-mounted display

Fig. 10.8

Scene from transurethral surgery, in which the operator monitors the processed three-dimensional (3D) image using the head-mounted display. White arrow, 3D image display; white frame arrow, conventional two-dimensional image display

10.5 Prostate Biopsy and Prostate Brachytherapy

Prostate biopsy and brachytherapy are performed mainly under TRUS guidance. Recently, the excellent diagnostic ability of magnetic resonance imaging (MRI)-guided prostate biopsy has been reported, and MRI/TRUS fusion prostate biopsy has become a frontline technique for the diagnosis of prostate cancer. In brachytherapy, real-time dosimetric planning and fluoroscopic information are used for precise seed implantation. Furthermore, customized seed implantation including hemi-ablation or quadrant ablation has begun to be introduced as a minimized invasive treatment [17–20]. Similarly as in the aforementioned transurethral procedures, a personal imaging display to monitor the multiple information streams at the same time is beneficial for the operator in these procedures. The applications of the HMD in prostate biopsy sampling and seed implantation in prostate brachytherapy are shown in Figs. 10.9, 10.10, and 10.11.

Fig. 10.9

Head-mounted display (HMD) use for prostate biopsy. The operator can use a composite image displayed through the HMD and direct vision by moving the angle of sight downward. The image displayed through the examiner’s HMD was split using an imaging splitter. The examinee can share the same image by wearing the HMD displaying the split image

Fig. 10.10

The composite image viewed by the operator. Red arrow, magnetic resonance imaging/transrectal ultrasonography fusion; green arrow, planned biopsy scheme; yellow arrow, vital signs monitor; blue arrow, view of video camera attached to the head-mounted display

< div class='tao-gold-member'>

Only gold members can continue reading. Log In or Register to continue