Cognitive Surgical Robotics

Introduction
This survey on cognitive surgical robotics shall provide an overview over the existing, planned and future application of computer aided surgical systems. Therefore the existing systems will be examined and described with their capabilities and limitations. In addition to the already available systems, the current research is described. Here the targeted improvements and achievements are pointed out. Based on this set of information, the future applications and scenarios of the use of robotics in surgery will be developed.

Explanatory Note
The following document reviews the application of robotics in surgery from the technical point of view. Therefore the current applications and research topics are only examined with respect to their technical properties. The future scenarios and applications are develop on this set of technical data. There are certainly problems with ethical, legal or even technological points, but those are not considered here. Any limitation through those fields would limit the outcome of this think tank document.

The Development History of Medical Robots
The development of medical robots started in the late 1980s when the PROBOT – a robotic system to assist in transurethral prostatectomies developed by Imperial College in London – was introduced. In 1992, RoboDoc revolutionized orthopaedic surgery. The first laparoscopic camera holder approved by the FDA (U.S. Food and Drug Administration), became the AESOP 1000 (the "Automated Endoscopic System for Optimal Positioning" established by Computer Motion, Santa Barbara, CA, USA.) in 1994. This system was enhanced with voice control in 1996 and with seven degrees of freedom in 1998. In this year telerobotics or telepresence to robotic surgery became commercially available: The ZEUS-System – an FDA cleared medical robot designed to assist in surgery, originally produced by the American robotics company Computer Motion. It was discontinued in 2003, following the merger of Computer Motion with its rival Intuitive Surgical. –, which was equipped with one AESOP arm and two surgical arms with four degrees of freedom. During the telepresence surgery, "the surgeon (the master) sits on a console which is at a distance from the robot (the slave) which operates on the patient" p.144. At that time the Defence Advanced Research Projects Agency (DARPA) for the U.S. military wanted to have a system, which allows a surgeon to operate from a Mobile Advanced Surgical Hospital (MASH) via robotic arms in an armoured vehicle (entitled the Medical Forward Area Surgical Team (MEDFAST)) at a safe distance, about 10-35 km, on the battlefront. In 1994 the first remote telesurgical procedure using a wireless microwave connection between a MASH test and a MEDFAST vehicle was performed and fired the starting pistol for Intuitive Surgical: The development of the da-Vinci was initialized p.142-145. Although a fistful of systems exist, providing interaction between surgeon and machine, the development of robotic surgery is still in its infancy. Nowadays the most robots are in research yet or high-cost-products. Although nearly 75% of surgeons declared in a survey that systems more than $500,000 would not be profitable for them p.24S. For that Mr. Troccaz stated, that a proof of clinical upgrading using a medical robot is needed by contrast with normal robotic applications. It is not satisfactory if the robot is able to cut precise or to tie a knot! "'One must prove with quantitative data that the system has a clinical advantage over other existing techniques' p.2" For example using a robot reduce costs or time needed to stay in hospital, bring less complications and so forth. In fact demonstrating those advantages need accurate tests with significant patients groups consistent with applicable law and ethical codes, which could maybe economical uncommonly since a couple of years is necessary for evaluating expected a long-term advantages.

Therefore, because the development of medical robots is a challenging task which occupy a lot of resources in time and capital, cooperation between experts from medical, scientific and industrial fields is needed since the beginning of the project "in order to find cost-effective, innovative technical solutions that are compatible with a clinical use and that may lead to significant breakthroughs." p.2

The da-Vinci Robotic System


... is a surgical robotic system which is cleared by the FDA for surgical fields in Laparoscopic, Thoracoscopic, Prostatectomy, Cardiotomy, Revascularization, Urology, Gynaecology, Paediatric, Transoral Otolaryngology p.141, p.3 and was developed in the middle of the nineties by Intuitive Surgical (Sunnyvale, CA, USA). For the future Intuitive Surgical aim for extending da-Vinci to surgical fields like Urology, Gynaecology, Cardiothoracic, General, Transoral Surgery p.3. The da-Vinci offers a 10x magnification, tremor filtration, stereoscopic vision and motion scaling p.141. On 31 March 2012 2,226 Systems were installed in over 1,785 hospitals worldwide and approximately 360,000 procedures were performed in 2011, which is up 29% from 2010 p.3.

The robot has a master console to control the robotic arms and an imaging system (see Figure 1). For safety reasons the system is only active an infrared sensor confirms that the head of the surgeon is placed into the console to view the 3D imaging system. The first da-Vinci robot had one arm for the camera and two for surgical instruments (a description of the principle of operation for the old three-armed da-Vinci is given by Ballantyne p.1392), but the current one has an extra arm for surgical instruments. By manipulating the robots arms, the system downscales their motion given by the surgeon's movement from 3:1 or 5:1 and also eliminates hand tremor (see Figure 2).



To imitate the surgeon's hand, the surgical instruments are equipped with seven degrees of freedom and two degrees of axial rotation. The instruments are attached to the arms of the robot by special ports (10 mm for the binocular robotic camera and 8 mm for the instruments). Hence, while the da-Vinci monitors a location near the trocar tip instead of tracking the instruments tip as the ZEUS did, the robot offers a binocular endoscopic vision for the surgeon console, which creates a truly 3D experience. This is realized by synchronizing images of two 5-mm scopes inside the 12-mm telescope which are projected onto two screens inside the console p.145, p.845, p.45. A lot of applications for the da-Vinci surgical system and there experiences are desribed in the book "Medical Robotics", edited by Vanja Bozovic. In Table 1 the advantages and disadvantages of the da-Vinci surgical system are summarized.

Compensation of Patient Movement
Differently to systems for movement-compensation in surgery, like described in, there are already existing systems in non-invasive healthcare: Here the focus is on radio-therapy. Because irradiation plays an important role for the treatment of cancer and while the efficacy depends on both accuracy of radiation and avoid damaging the surrounding healthy area, compensate respiration is absolutely necessary. For example a spherical tumour of radius 1 cm will need an 8-fold increase in treated volume, which primarily involves healthy tissue. Furthermore breathing will expand the dose in a 27-fold increase. You can see that an accurate method for compensating respiration has clinical relevance. Therefore "the exact spatial position of the tumor must be determined in real-time" and "the radiation source must be moved in space to follow a prescribed motion with high precision" p.2.

Usually a medical LINAC (Linar Accelerator) is used to irradiate the patient. Old systems respected respiration while switch off the LINAC during the target is outside a predefined window, which increase the treatment time. Another problem was the resulting inaccuracy. A new solution is, to move the activated beam in order to follow the tumours movement. The problem thereby is, that the up and down breathing-movement of patients surface does not confer on tumours movement. In fact, the tumour maybe will move in left/rigt direction. Thus a series of images from infrared and X-ray sensors is used from which compensation of respiratory motion is possible for a robotic arm moving the beam source (see Figure 3) pp.4f.



The CyberKnife Robotic Radiosurgery System is such a system for the treatment of tumours anywhere in the head, spine, lung, prostate, liver and pancreas. Since 2001 the CyberKnife System is cleared by the FDA to treat tumours. Until December 2010, "more than 100,000 patients have been treated and 244 systems are installed worldwide" FAQ.

Surgery Guard
A risk during a surgery is, that the surgeon leaves tools in the body behind, which are not supposed to be left behind there. Today the hospitals try to minimize the risk, in order to reduce the complications caused by such errors. The current ways to handle the risk are weighing or counting of the used tools. Both types of error management inherit the human error problem, because they are done by hand and this can lead to mistakes.



Newer attempts are using RFID (Radio-Frequency Identification) chips to identify the different tools. A sample system is the SmartSponge System. A central part is shown in Figure 4. It's a bucket, where the nurse has to register the sponges provided for the surgery. The system automatically tracks the sponges and remembers everything put back into the bucket. The nurse can see immediately, if anything is missing. The search after missing parts is supported by a hand-held antenna, which allows the surgeon, to check, if the missing sponge is still inside the body.

Organ Inspection
To be able to inspect the inner organs, endoscopes are used. They are inserted via natural orifices or an incision of the skin. Usually they are flexible and can be therefore moved around in the body. But some sections of the colon are difficult to reach, that why the capsule endoscope was developed. Such a capsule is shown in Figure 5. As you can see, it’s basically an oversize pill, which contains flash lights and one or two cameras.



Before the procedure, the patient has to empty is stomach and bowel. This means, he or she has to stop eating 24 hours before the procedure. During the inspection, the patient has to wear a receiver, which collects the taken images. The images are downloaded from the receiver and reviewed in sequence. The off-line diagnosis can take up to two hours and the results depend on the experience of the reviewer. Also the passive locomotion of the capsule limits its uses. The position is not detectable and controllable and also the lighting of the images may be wrong because of the complete passivity of the system. But on the positive side, the complete bowel can be inspected. This inspection can detect sources of illnesses, which are undetectable for the conventional endoscopy.

RoboDoc


As stated in the introduction, the RoboDoc was invented in 1992 by Integrated Surgical Systems, Sacramento, CA, USA, but is currently marketed by Curexo Technology Corporation following their takeover of Integrated Surgical Systems in 2007 and performed about 24,000 procedures until June 2010 p.143. The system was approved by FDA for Total Hip Arthroplasty (THA) in 2008 and waits for clearance Total Knee Arthroplasty (TKA) FAQ. It is used for clinical practice in Germany since 1994 p.445. In THA surgery, the ball-and-socket joint and a portion of the femur are replaced by a prosthetic device consisting of an acetabula cup and femoral stem. The job of RoboDoc is to mill a cavity in the patient's femur for the implant. This procedure is classified by the following steps (see p.1624): The RoboDoc System consists of three components: This construction separates components that must be sterile and near to the operating table (1) from the ones that needn't to be near the patient (2,3) (see Figure 6) p.1624.
 * 1) This step isn't necessary anymore, since a pin-less method exist p.28! To create a geometric frame of reference for preoperative planning and for the robots orientation during the surgery, two or three titanium locator pins were implanted in the patient's femur.
 * 2) A Computed Tomography (CT) scan of the femur is taken.
 * 3) The CT data are transformed into 3D-images of the femur by the ORTHODOC Preoperative Planning Workstation. Now the surgeon plans the surgery by manipulating the digital model in combination with implant models, to find a suitable prosthesis and its position. These output become the input to the surgical robot.
 * 4) In the operating room the femur of the patient is fixed to the robot by a femoral fixator.
 * 5) The robot translates the surgeon's placement of the cavity from the preoperative plan.
 * 6) The robot starts to machine the cavity according to planed position.
 * 1) The surgical robot
 * 2) The Operating Room (OR) Display
 * 3) The Control Cabinet



The femoral fixator of the surgical robot, as can be seen in Figure 6, prevents motion of the femur during operation. To increase safety, the bone motion monitor (BMM), signals the Control Cabinet to stop the cutting process, if the bone movement reaches threshold of >2mm for longer than one second. The surgeon got the force to control the robot by PAUSE and STOP buttons. First one halts robot motion and turn off the cutter. Last one removes power from the robot motors and cutting tool. The data of the BMM and the pushing of the safety buttons are checked by the safety flags of the Safety Processor (SP). These safety system software is executed every 18,2 milliseconds (medium timescale) by the Real-Time Loop (RTL) of the Motion Control System (MCS), which coordinates data flow between MCS, Axis Control Cards (ACCs) and the SP (see Figure 7) pp.1625-1628.

Altogether it is necessary to recognize, that the usage of the RoboDoc System holds some risks (see Table 2). The medical service of health fund in Germany (MDS - Medizinischer Dienst der Spitzenverbände der Krankenkassen e.V.) commissioned a study for the method of robotic milling on femur in 2004. As an outcome they stated RoboDoc to be still an experimental procedure and advised to educate patients about the risks p.7. Therefore we list some advantages and disadvantages of the system in Table 2.

The Acrobot System


The Active Constraint Robot (Acrobot), was developed in the mechanical engineering department at the Imperial College in London and released in 2004 after 16 years of research p.2. It is "the world's first hands-on robot for unicompartmental knee replacement surgery" p.28 and is also used for hip replacements. At 2008 almost 200 patients were operated by this system, which costs about ₤70,000 (~$109,969) p.2. The Acrobot is a so called active robotic system, like RoboDoc or CASPAR (Computer-Assisted Surgical Planning And Robotics, developed by orto MAQUET GmbH & Co. KG, Rastatt, Germany. In 2001 U.R.S. Universal Robot Systems Holding GmbH, Rastatt, Germany, take over the company and the robot.), but differs in use. The last ones move a milling tool automatically following the preoperative plan. The Acrobot never moves a surgical tool! Instead it is designed to increase step by step the resistance of the tool, held by the robot, when it comes close to a pre-defined boundary. This allows the surgeon to move the surgical tool freely in the safe region, but prevent milling outwards the favoured area p.445.

Acrobot uses volumetric image-based navigation, which has been successfully used in Total Hip Arthroplasty (THA), Total Knee Arthroplasty (TKA) and other applications p.443. For that, in the pre-operative phase a first CT scan is taken. Interactive software allows then to build 3D models form the given data. The surgeon chooses a suitable prosthesis and placed it in the model. From that, the software generates the sufficient operation area (RI+RII) (see Figure 9) p.5. As said above the robot checks the movement of the surgical tool to prevent a failure by the surgeon's movement. Therefore Acrobot allows free movement inside the pre-defined safe region (RI). The force normal to the boundary, i.e. movement in boundary direction, is regulated down in pieces. At the boundary of the safe region, the robot doesn't allow further motion over the boundary (RIII). Note that only the movement in a direction toward the boundary is limited. Motion along or away from is still allowed p.7.



Surgical Knowledge in computer-assited Surgery Systems


Experiments show, that there is a lack in Computer-Assisted Surgery (CAS) systems: While the complex surgical reality often requires exchange about current practices or surgical state of the art. Current systems features just a base formal knowledge of surgery, or none at all. In fact they don't provide any possibility to get informed about the necessary information. Therefore data-flows and workflows aren't often correctly guided. This could cause high costs or even clinical failures. Because of that it is essential to make explicit surgical know-how and associated scenarios available in CAS systems.



A design of such a methodology was derived from knowledge modelling and cognitive systems engineering (see pp.3-5):
 * 1) Definition of the modelling objective (i.e., aim of modelling approach).
 * 2) Definition of the surgical work domain to be modelled (i.e., universe of discourse).
 * 3) Definition of an ontology for the work domain which involves identification of concepts and relationships describing the surgical work domain, choice of a formalism for representing concepts and relationships (see Figure 11), implementation of the formalized ontology into prototype software, and test of the prototype for testing previous steps.
 * 4) Data Acquisition which consists in describing surgical cases using the formalized ontology.
 * 5) Visualization and browsing of descriptions (see Figure 10).
 * 6) Analysis by knowledge extraction from these descriptions which involves distinction between predicted and predictive parameters, data pre-processing, and data mining.
 * 7) Evaluation of generated knowledge.

The developers expect for example an optimized work flow, when an expert system is used. This helps the surgery team to perform the operation in less time, which then reduces the stress for the patient and even saves the hospital money.

Another usage may be an improved diagnosis through the usage of such systems. This can be reached, by filling in the symptoms of the patient. Base of this information the system calculates the most plausible illnesses. But it can also provide very rare illnesses that are possible with the given indications. A major problem may the infection with multiple sicknesses. The modelling has then to allow the pairing of symptoms and sickness based to the given information. Therefore the system needs to have a vast base on which it can rely on.

Improvement of existing Techniques
Presenting the da-Vinci system in The da-Vinci Robotic System, Table 1 is listing the advantages and disadvantages of the system. The most common complaints are about the missing haptic feedback, the size and the price of the system. The first two points are improved by SOFIE (Surgeon's Operating Force-feedback Interface Eindhoven) and the DLR MiroSurge System. An example setup of SOFIE is shown in Figure 12. Both systems are providing a force-feedback interface with less space for the robot consumed.



Another drawback of the da-Vinci system is, that the tools available are mostly specialized for the use in cardical surgery pp.1394f. This limits the benefits of the system for other surgery uses. A new set of tools was developed by the DLR, the MICA (Minimally-invasive, computer-assisted) tools. They are supposed to be more flexible and versatile in robotic surgery.



To make a robotic surgeon accessible for all and not only for the richest institutions, Blake Hannaford and his colleagues at the University of Washington, in Seattle, developed an open-source robo-surgeon, called Raven as shown in Figure 13. "The Raven Surgical Robot is a 7-DOF cable-actuated surgical manipulator designed for use in either MIS or open surgery". In comparison with the da-Vinci, the Raven is a bargain, because it costs only $250,000. Furthermore the software of the robot is compatible with the open source robotics coding platform ROS (Robot Operating System – http://www.ros.org/wiki). The concept of the Linux-based operating system enables anyone to modify the original code in order to find and fix bugs. This and also by releasing a graphical simulation of the Raven which can be used to test the control system virtually, creates a way for researchers to experiment and collaborate. At present the system has not yet been approved so that it is restricted to operations on animals or human cadavers for the moment. There is also another, legal, problem:

Intuitive Surgical held exclusive field-of-use license for more than 870 patents as well as more than 990 pending patent applications at 31 March 2011 Company Profile. From 7 April 2011 to 16 August 2012 they got at least 119 patents more. Those relate to a number of important aspects like surgeon console, electromechanical arms, vision system, endoscope positioning system and EndoWrist instruments. This obviously hinders the development of medical systems by other companies. To accommodate someone with the development of its own technologies Intuitive Surgical cultivate relationships to industry leaders like IBM Corporation, Massachusetts Institute of Technology (MIT), Johns Hopkins University (JHU) and Heartport, Inc. (Johnson & Johnson) Company Profile. Because of such a huge market power, they could prevent anybody to enter the market and to commercialize enhanced systems.

There are also attempts to increase the usage of the tools. Right now, the tools are imitating the natural movement of the human arm and hand. This approach is very easy to implement and controllable by a surgeon. But it also limits the usage of the robot/tool combination. Therefore new tools need to be developed and are in development Appendix B. They are aiming at an increased number of degrees of freedom. In this way, new surgery types can be performed and even current applications could be simplified, if for example a tool change is not longer needed. But this sets the requirements for a more complex control system, which helps the surgeon in controlling the additional degrees of freedom. Here a cognitive system in combination with a control loop may come into usage in order to estimate the next command and optimize the robots movement and paths.

Compensation of Organ Movement


Currently there are efforts to compensate the movement of organs during robotic surgery. The expected benefit is that the organ has not to be stopped to be able to perform a high precision surgery. Also especially for heart surgery the invention of the heart-lung bypass machine was a milestone in medical history, but the technology is not without risks : The stopping and restarting of the heart, with the intermediate support of the blood circulation by the machine causes problems, e.g. blood contact with extrinsic surfaces, inevitable blood clotting attenuation, typical generalized inflammation reaction, hypoxia (insufficient oxygen delivery to tissues), lowered blood pressure, irregular heart rhythms, or body temperature that is too warm or too cold. Obviously this slows down the healing afterwards and may lead to neurological complications.



Therefore a system is in research which compensates the movement of the heart, respectively the breathing movement. This will make the heart-lung bypass machine become obsolete. The condition of the motion compensation is, that a robot is used to perform the surgery. Then the system tracks the motion of the beating heart and estimates, which movement will be done next. With this information the system can move camera and tool above the heart in such a way, which provides the surgeon the impression, that the hearth underneath his instrument is not moving at all. But in reality the computer is moving the surgical tool in exactly the same way, as the organ moves. By doing so, the also moved camera provides every-time the same image. The basic setup of such a system is shown in Figure 14. First evaluations of the approach were investigated by and.

A first system, "although the robot hasn't yet entered commercial use", which is evaluating the possibility of compensating the movement of the heart is the MiroSurge (shown in Figure 15), developed by the German Aerospace Center (DLR). The system is composed of three independent, 7 DOF each, robotic arms.



To avoid failures during surgery the system has also to be able to compensate movements, in which for example the camera is masked by the cutting instruments. The system could either controlled by a master console or by a teleoperator. The surgeon gets both the video stream and the measured forces displayed at his console, so one can also feel what one is doing. Figure 16 shows an overview over the MiroSurge-procedure, including pre- and postoperative steps.

In fact the system "is several years away from being used in an operating" is tested at the moment on an artificial heart and will be tested on an animal in the next step. Related systems are also in development at Imperial College London, UK, and at Harvard University and Children's Hospital Boston, USA. At Harvard University for example, mechanical engineers using the Raven Surgical Robot, as described in Improvements of existing techniques, to operate on a beating-heart. For that the robot also compensates the movement of the heart so that a surgeon can operate as if the organ doesn't move.

Improved Organ Inspection
The organ inspection with capsules of Organ Inspection is very limited, but it has also given huge advantages compared to the conventional inspection. One disadvantage is that the surface of the bowel can't be seen completely. To solve this problem, RF Corporation has developed the Sayaka system. They modified the conventional and proven capsule in such a way, that the camera is pointing to the side and can be rotated freely. The aimed construction of the capsule is shown in Figure 17. Caused by the rotation, the image sequence corresponded to a spiral throughout the bowel. The increased energy consumption is compensated by a wireless energy transmission to the capsule. The images are stitched afterwards to a giant overview map, which allows the physician to see problematic areas in one view.



Another attempt to improve the capsule endoscopy is the automated image processing. This aim at a significant reduction of the time needed for the review of the images. Therefore the images have to be preprocessed. This can be done, by classifying the images thought different markers. For example, blood in the bowel has to marked and pointed out. The first research trials in this area are very promising and point towards a major reduction of the needed time to review the results of the capsule endoscopy.

A third problem for the classic capsule endoscope is, that it can't move own its own and it can't be moved from the outside. This can be solved by using a magnetic capsule. Then the operator can use a hand-held device, which manipulates the capsule. With real time video streaming, the operator can access new regions or hover above critical areas. This works also, if the stomach is filled up with water. The water filling helps to enlarge the stomach and makes therefore hidden areas visible. The problem with hand-held devices is, that it's only as precise, as the operator. A additional improvement can be gathered, when the capsule is controlled through a computer system and a static magnet field generator. Such a system allows also the precise localization of the capsule, within the magnetic field and therefore in the patient. An even further improvement can be reached, when the capsule itself can manipulate its own magnetic field. Then it can handle smaller steps and a more precise movement.

To increase the uptime of the capsule, the energy could also be transported wireless. This helps keeping the capsule small, while its uptime can be theoretical infinite. Additionally the support of extra tools may be reachable. Such tools may be sensors, drug delivery, etc..

Automated Movements of an additional robotical surgical Tool
The minimal invasive surgery has its disadvantages in the accessibility of organs and the degrees of freedom of the surgical tools. There are trials to minimise the drawbacks by using a robotic assistance. It shall handle additional tools or the camera. The movement of the additional tool shall be automatized, so that the surgeon has not to care about the planning and executing of the movement. E. Bauzano, V.F. Muñoz, and I. Garcia-Morales describe a possible solution to do so. They are using the concept of Figure 18(a).

The surgeon takes the two hand-held tools to the right. While the system is monitoring the position of those, it calculates the position of the end-effector within the body of the patient. If a move command is given, the system navigates on its own to the target position at the surgeon's tool. It avoids also collisions during its movement.

The concept was realized with a technology demonstrator and tested on a patient simulator. The results where as expected: The path planning and execution is working and even a moving surgeon tool was avoided.

Another use of automated movement may be the centering of the captured image of the camera on the current working position of the surgeon. Therefore the camera has to be automated, and the tools within the captured image detected. If the detection is stable enough, a robust centering can be established. In the case, that the tracking of a tool was lost, it can be reinitialized, by estimating the position in the image through the encoder values of the robot. This gives a rough estimation of the position inside the camera image, but it's sufficient to re-lock the tracking. With additional colour markers, different tools can be successfully distinguished and tracked. This ability can be seen as a precusor of SLAM (Simultaneous Localization and Mapping) for surgical robots.

Tie Knots in Heart Surgery




In minimally invasive surgery, the surgeon has to tie knots with the tools placed inside the patient. This is quite a challenge. Usually a knot can be tight within a few seconds, but under minimally invasive surgery the procedure can extend up to three minutes per knot. Therefore a system is required, which helps the surgeon to tie knots and reduce the time of surgery for the patient. The basic sequence for tithing knots is shown in Figure 19.

The sequence was learned from human example. Therefore a neural network with a long-term storage was used. After only a few tries, to follow the human example, it was capable of performing the basic steps. But the needle has still to be applied by the surgeon. Figure 20 shows the first knot tied by the Raven Surgical Robot (see Improvement of existing Techniques for details about Raven).

Operation planning and Realisation in orthopaedic Surgery
Even though the RoboDoc was denounced as an experimental procedure, because of many errors. There are still efforts to establish more and better robotic support in the orthopaedic Surgery. The possible gains through the computer aided treatment are supposed to be huge. These advantages can provide the patients and hospitals huge benefits.

One big problem was that the robot itself is bigger, than the human hand. This leads to a large wound, to allow the robot to access the region of interest. With a revisited design and the success of the miniaturization during the last years, a smaller generation of such a robot type should be developed. The next problem was, that the robot was not able to detect, when reality differs from the prepared plan. To solve this, the computer system needs to have a live monitoring of the patient and the robot itself. With an X-ray monitoring also hidden areas can be made visible. These new information should allow the system to compensate the differences between the plan and the reality.

Additionally a super-vision by the surgeon should be applied. This gives the surgeon the opportunity to disengage the robot at all time. And with a reduction of the treated area in a single run the surgeon can monitor easily the movements of the robot. As an example for a more human controlled orthopedic surgery robot the Acrobot, may be taken This system supports the surgeon and helps him to avoid failures, but the system never operates by itself. In fact the surgeon operates and is controlled by the robot.

The Cardio-Arm


The classic hearth surgery opens the thorax, in order to be able to access the heart. This procedure leads to a huge wound, with the usual drawbacks of it. A newer attempt the access the blood vessels around the heart, is by inserting a catheter into a vein of the leg and following it up to the hearth. This reduces the wounds for the patient, but limits also the use to intra blood vessel operations. To get around of this problem, researcher at the University of Pittsburgh developed the Cardio-Arm. A prototype realisation of the system is shown in Figure 21.

The idea behind the system is that though a small incision in the patients skin, the hearth can be access completely. Therefore the Cardio-Arm is a highly articulated system. It's controlled by an operator, which commands the tip of the arm. The rest will follow it on exactly the way, the tip took. This behaviour helps preventing of harming the surrounding tissues of the heart. If the Arm has reached its target, it can be seen through the Arms vision system. Additionally tools can be pushed through channels in the arm. Because the Arm leads the tools to the operation area, similar tools as for the vein catheter method can be used. But here, the tools are outside of the blood vessel, which increases the possible uses of this surgery method significant. And it also reduces the wounds needed to access the heart.

Future Applications
The following sections are supposed to provide an overview, what is technical possible or thinkable. Therefore the advantages over the current systems and research areas are pointed out. With this information, the future technology is sketched out. Or, if the future application is similar to a scenario, it will be described. Technical limitations or detail problems are not considered here. But a possible solution for the most significant problems will be given.

The critical demands for this survey are the use of cognition in surgical robotics. Therefore the robots have to have situation awareness and a intuitive human-machine interaction. If the system does not interact physically during the surgery, but if it surveys the surgery, an expert-system may come into usage.

Advanced Robot Control
As stated in Improvement of existing Techniques, there are attempts to improve the currently available systems, such as the da-Vinci system. But these basic technologies may be develop even further in the future.

A first addition may be more degrees of freedom. Right now, the systems are trying to imitate the human hand and its degrees of freedom. This makes the handling for the surgeon easy, when he isn't supported by a computer system, which can handle more degrees of freedom. But it also limits the areas, the robotic tools can reach. There are certainly surgeries or techniques, where an additional degree of freedom may be very helpful. This more complex system has therefore then be controlled by a computer, which handles the degrees of freedom. For example, to takes care about, that the entry point into the body of the patient is not ripped open. But this means also, that to current handling devices as shown in Figure 2 have to be changed.

Another enhancement may be the addition of self-awareness. The current generation of robots don't know, where it's other parts are and therefore frequent collisions are a major problem. A self-aware robot could plan its path even better. This should lead to collision avoidance, through improved path planning. In combination with more degrees of freedom, the reconfiguration needed to perform a surgery can be reduced.

Robotic and automated single Port minimal invasive Surgery
Current systems, such as the da-Vinci system, are using the minimally invasive surgery as their base technique. But there are attempts to reduce the incision in the skin of the patient even more. This can be achieved, by using the single port surgery. A principle working sequence is shown in. The most outer layer of the skin is only sliced once and all needed tools are inserted through this incision. Therefore, the surgeon has only a small space to move the tools in. This requires a good coordination and imagination, to be able to perform the surgery.

With an on-going miniaturisation, the robotic surgery should be able to perform such surgeries too. As shown in Improvements of existing Techinques, the first steps are done, by minimizing the space consumed by the robot itself. This allows a side by side placement of the tools, hold by the robot. A sophisticated computer system may support the surgeon by handling the tools. This is especially useful, when the tools are crossed and the surgeon doesn't have to bother about it. The use of the single port surgery may be even more extended, if the advanced control algorithms of Advanced Robot Control are used. With the additional degrees of freedom most areas of the abdomen may be accessible from a single incision.

Navigation Assistance based on Vision and Knowledge


During a minimal invasive surgery, the surgeon has to rely currently on his own preception and imagination about the position of the tools in the body of the patient. The field of view of the used camera might also be very limited. Therefore the orientation can be difficult.

If the tools and the camera of the surgeon can be localised by a computer system, it may also be able to calculate the real position of the end-effectors within the patient's body. First research implementations are described in Automated movements of surgical tools. The systems are capable of tracking the tools in laboratory experiments. It needs to have situation awareness at all times. And in addition to that, the localisation has to be stable and robust. Therefore, it has to know, how the tools look like and how the patient is placed. This combined information may be provided as additional visual information on the surgeon's video screen. A sample implementation of such an assistance system is shown in Figure 22. Additional information about the surgical tools can be gathered by using ultra sonic images. This may help, even if there are not sufficient visual markers to orientate. Additionally the visual perception of the camera inside the patient's body can come to use. There are first steps, to estimate the position if the tool relative to the surgeon, only based on the camera images. This helps, to circumvent additional sensors and sensing devices in the operation room. Therefore the preparation time can be reduced and even the number of possible errors.

Even when the tool and the camera have reached the targeted position, the assistance system might help the surgeon. For example, it may help the surgeon in the distinction between similar looking tissues or marking the relevant areas. This can therefore reduce the amount of surgical errors, when the system is able to warn the surgeon, that he differs from the predefined targets of the surgery. This can be done, by flashing up a warning light in the surgeon's vision system.

There is currently some research in this area, but nothing aims right now for an implementation in robotic (supported) surgery. Therefore this area is considered here as a future application.

Auto-guided Path finding
As described in Automated movements of surgical tools, there is on-going research, to enable surgical robots the automated movement via online commands of the surgeon. In combination of the technology used to assist the navigation within the patient's body (see Navigation Assistance based on Vision and Knowledge), the robots may be able to move autonomously throughout the patient's body.

Therefore the cognitive surgical system has to be able to determine its position relative to the patient. It also may use previously acquired data about the patient. Those data sets might be harvested from pre-operation examinations. The data can be gathered by using CT, MRI, ultra-sonic imaging, X-ray or similar imaging resources. These data has to be completed with the information about the relative position of the image with respect to the patient's body. Now the system can take the data to its advantage and build a first patient map to navigate in. During the surgery SLAM (Simultaneous Localization and Mapping) is needed to be able to locate and navigate within the body. Because of the position, the patient is hold in, the inner organs might have moved in comparison to the pre-operation images.

Based on this information and a live sensor surveillance of the patient (e.g. with ultra-sonic images, X-ray computed tomography or magnet resonance imaging) the system navigates to the target area or organ. Because of the available information about the patient and the concurrent sensing of him, the system is able to determine the best path from the entrance port to the target. After reaching the target, the surgeon takes over control and performs the actual surgery. When he is finished, the robot retracts on its own from the patient body and the entrance port can be closed.

Advanced Surgery Guard
The Surgery Guard Systems have to be fed by humans. But in a hectic situation in the operation room, the registration might be overwritten, in order to help the surgeon immediately. Therefore systems are needed, which take over the registration of tools on their own.

In Surgery Guard the Smart-Sponge System was presented. It uses RFID Chips to identify and track surgical sponges used. But a drawback is, that the nurse has to register all sponges before they are handed over to the surgeon. An addition to the system may be a RFID antenna, which is focused onto the surgical table. Because of the limited area of "attention" for the RFID system, it may be able to register and track the tools on its own. The assumption that every tool within the monitored area is close or in the patient body can be made.

Another attempt to solve this problem is to monitor visually the table and preparation area of the surgeon. Therefore the system has to be able to distinguish between tools only based on visual data. If the visible area for the system covers the complete table, it can't be tricked. And the tool identification could also be able to detect, which tools where actually inserted into the patient's body and which one is supposed to stay in there. The basic principles for the visual surveillance were already explored. This can be adapted to track a limited set of tools on the surgeon's table.

Similar to Navigation Assistance based on Vision and Knowledge, first research in this area was done. But here the progress is not enough now, to talk about actual research in big project.

Automated Nurse
While the surgery guard is a passive system, the automated nurse active helps the surgeon. It supplies the surgeon with the needed tools. Therefore the system has to know, which operation is planned and what tools are necessary for this type. During the surgery, it monitors the surgeon's table and provides the tools needed next. It also removes used tools. By doing so, the function of the surgery guard, as pointed out in Advanced Surgery Guard, can also be implemented. It may also force the surgeon to look after a missing tool with denial of the tools to close the wound.

Additional communication between the surgeon and the robot may be done by gestures. This helps to keep the robotic nurse as clean as possible and therefore it reduces cost. But also the communication can be very stable and intuitive, if the gestures are choosen properly. Basic research to command a robotic nurse by gestures was already done and Medical systems and assistive technologies. But the progress is still too small to talk about actual research with the aim to commercialize the system.

Automated Anaesthesia
To be able to do an automated anaesthesia, the system has to be filled with the necessary information about the patient (e.g. weight, drugs taken, planned surgery). In addition to this information, it monitors the patient continuously. Here it takes the information amongst other things about the blood pressure, heart rate and oxygen saturation.

The peripheral venous catheter has to be applied by a nurse. After the insertion, the anaesthesia can be conducted. An indication, of how deep the anaesthesia is, can be taken e.g. from the BIS (Bispectral index). This is a dimensionless number that indicates the level of awareness of a person. Based on the number a sophisticated control-loop could be established to maintain a sufficient deep anaesthesia during the surgery. A high-level cognition might support the control-loop in adapting to the "class" of patient to be handled.

Automated Organ Inspection
Today there are systems available, which allow the doctor to inspect the complete colon. Therefore he uses a camera holding pill, as described in Organ Inspection. This method is very "primitive" and as stated in Improved Organ Inspection, there is current research to improve the results of this examination method even further.

The primary usage of the capsule is still the visual inspection of the stomach and bowel. But it may make sense, to think about an active interaction with the environment there. Therefore sensors and actuators should be carried along with the capsule. The additional options are increasing naturally the power consumption of the complete system. This reduces with the current battery technology and the same outline of the capsule the life span of a single inspection run. To circumvent this problem, the energy could be transmitted wireless with the help of an external magnet field and receiver coils in the capsule. The external energy supply enables even more possibilities, which are not realistic, through energy constraints of the current batteries. For example, the capsule could be modified into a small worm, which then is able to propel it-self and carry multiple sensors and actuators.

But all this modifications are bound to the inside of the organs. There are also cases, in which the physician needs to see the outside of an organ or the organ can't be access over the classic capsule endoscopy. For a robot that travels outside the organs but inside the abdomen, it's most important not to harm the tissues around him. Here a worm-like movement could be used. This would also allow a squeezing through small opening, because the outer skin of the robot is flexible. If it then is inserted through a natural orifice, the entrance port of the robot should not be visible after the operation.

With an even further going miniaturisation of the systems, it may also be possible to insert the robot into blood vessels. But here, it has to make sure, that the vessel is not jammed by accident. This could harm the patient very hard.

(Semi-)Automated Surgery
The implementation of one or more of the following systems may lead to a semi-automated surgery:
 * automated Path Finding,
 * advanced Surgery Guard,
 * automated Nurse,
 * automated Anaesthesia,
 * automated Organ Inspection.

When combined the systems have to interact with each other and with the surgeon. This turns each stand-alone system into "only" a subsystem of the automated surgery. To be able to synchronize the systems, the communication interfaces have to be standardized. Only with a stable communication and synchronized operations, the systems can really support the surgeon in his task to help the patient.

Centorobotics
With the assumption, that the automated organ inspection and the path finding with a patients of Auto-guided path Finding are working, it is possible to think about an even more complex system: "Centrorobotics". To be able to implement such a system, it is neccessary to make additionally progress in the field of miniaturisation, artificial intelligence and energy management.

The basic idea is to decouple the tools and vision system inside the patient from the outside. This would give you the advantage, that you can minimized the incision needed to perform a surgery even more, compared to single port surgery. Even if the incision has to be the same, the applied stress to it can be reduced, because it is possible to insert the tools one after another. They don't have to fit at the same time through the incision made.

A "centorobot" is supposed to be not bigger than a Euro-Coin (see for example Figure 23). A swarm of centorobots (imagine a fistful) will be fed through an opening (cut by a surgeon) near the application area. Their on board propellant system – a system similar to the blob-bot of Automated Organ Inspection may be suitable – allows them to move towards the targeted area. When the targeted area is reached, the robots are able to interact with the environment through their carried tools, like for example scissors, lasers or cameras.

In a first generation of the system, the surgeon may control the robots. He commands therefore general tasks, like clamp this blood vessel here. The cognition system of the robots evaluates then, which robot can handle the task at best and fastest, so that an immediate response is guaranteed. After the surgery is completed, the surgeon commands the robots to return to the incision, where they are collected and prepared for their next mission.

A second generation of the system may handle basic task for a surgery on its own. Therefore the robots have to be able to navigate on their own. They may use SLAM for this task. Even the results of Auto-guided Path finding may come into usage here. Additionally the robots have to have situation awareness every-time, so they make to right decision at the right time.

A third generation may go even a step further. For some task the power of a single small robot may not be enough. To get around this problem, the robots are equipped with a docking mechanism. This allows the robot to assemble in the patient's body to a bigger mechanism, while the incision is still very small.

Portable, automated Health Check
A portable scanner, which maybe in parts could also be used from non-highly qualified personal, measures the heat of a patient, the structure of his bones, the texture of his organs, the exhalation of liquids, his weight and his size, etc. and compares the data with previous stored data about the healthy patient. For that, the scanner could use non-contact sensors, like an infra-red thermometer and also contact sensors like a breath temperature sensor, a strain gauge, a spirometer, magneto optical technology etc. This information can be completed by the patient himself. He may add his date of birth, gender, known diseases etc.

This set of data can be used to calculate the basic health of the patient. For example, it should be easy to calculate the patient's weight ratio and similar values to characterize the patient's health. This information may be summarized and hand over to the doctor, who has now a starting point for his examinations. So he can ask more specific questions or precise the therapy needed to cure the patient.



A more sophisticated system may evaluate the data on it's know. Therefore an expert system is used. Its base information set are the healthy values of a set of similar patients. Similar means, patients who are at the same age, have the same weight, body height and history of diseases. This helps the system to narrow the possibilities, which problem the patient may have. Such narrowing of the possibilities may be done by colour marking the irregular values and promote those to the doctor. In "simple" cases the system may also be able to do the diagnosis on its own.

If the physical system is small and light enough to install it at home or to carry it on expeditions, it may also help in urgent situations. For example such a scanner could be developed from the currently existing Magneto Optical Technology (MOT) device (see Figure 24). With a connection to a doctor, e.g. in a hospital, he might be able to assist in the emergency.

World wide Telesurgery
Most hospitals in the world have to be equipped with "reception-slave-robots", which send necessary information about a patient live to a "master-robot". From there, a specialized surgeon could operate the patient. This helps patients who must be operated in a complex way as only few specialists in the world could do. In this way, only "normal" surgeons or medical personal need on-site while the expert is in another part of the world.

The major problem is a reliable, stable, save and minimum-lag connection. With the current network technologies such a connection can be established. The first transatlantic telesurgery, the so-called "Lindbergh-Operation", was successfully performed on the 7th of September in 2001. For this gall bladder surgery the ZEUS (see The da-Vinci Robotic System) robot was used. The surgeons commanded the robot from New York, while the patient was located in the hospital of Strasbourg. For a future economical use, the connections needed have to be cheaper and the system has to be installed at every location. Therefore interoperability between different surgical robotic systems should be a target for future developments.

Another application of telesurgery may be the support of space mission in the case of an medical emergency. Right now, a sufficient connection can only be established in low-earth-orbit (e.g. the orbit of the international space station). But with a sophisticated communication system, a telesurgery may be suitable also in orbit around the moon or even on its surface. For a long term stay, this opportunity may be a necessity to support the astronauts which have no fast option to return to earth. The MIRO-System, as shown in Improvements of existing Techniques, is intended to be used in space. Therefore the first developments are made to reach this target.

In-situ Surgery
While the telesurgery, as described in World Wide Telesurgery, is available in hospitals, it may also be necessary to have support in remote locations. Therefore a system is needed, which is capable of performing a surgery under extreme circumstances. These may be a moving vehicle, in which the operation is performed or support for a surgery team, which is usually too small to perform such an operation.

The first application may be for example a movement compensation of the vehicle movement, e.g. a helicopter or a car. A similar solution is used on cruise ships, to stabilize pool tables. Imagine the patient and the surgery team located on the stabilized table. Here they have a safe surrounding, so they can perform the surgery as usual. This allows to surgeon to concentrate only on the surgery and not additionally on keeping his own balance. The compensation of the external movement can be supported by a knowledge base, which even improves the compensation in such a way, that the relevant areas are at most standing still.



The second case may be more often the case. During a major accident an immediate surgery may be necessary, but the surgery team is far away and won't reach the patient in time or vice versa. In this case, the doctor on-site has to be supported. This can be done by a partial autonomous surgery system, for example a similar system as stated in (Semi-)Automated Surgery. It helps the surgeon during the surgery or at least until sufficient support is available.

Even a combination of both of the cases above may be suitable for certain situations. Therefore already existing systems for compensation of patients movement, e.g. the Cyberknife (see Compensation of Patient Movement), or for compensation of organ movements (see Compensation of Organ Movement) could be upgraded. But before such systems can be used in reality, they have to be tested and verified. The key elements to make this application real, are a lightweight construction, minimal space consumption, fast and intuitive handling. If these targets are reached, the systems can be simply installed in ordinary emergency vehicles and used by any doctor available.

A more advanced scenario is described in Figure 25, where a robot demonstrates surgery under water. The model is set in a fixed environment, but it’s easy to image the robots use in an unfixed submarine or in space.

Trauma-Pod
A Trama-Pod (TP) is a highly integrated and automated operation room. The basic idea is, to have in remote location or under extreme situation (e.g. in war, after a devastation earthquake, etc.) a possibility to perform surgeries. There were first tries to establish such techniques. These attempts are a combination of the automated nurse and telesurgery. Right now this is a assembly of various system, which takes a huge amount of space to be able to perform theoretically a surgery. There has to be done a lot of work to minimize the amount of space consumed and automate the surgery, which has to be performed.

In the future, you may add the functionality of the portable health check, the automated anaesthesia and the in-situ surgery to the system. If you integrate all those functions into a comparable small system, you end up with a Trauma-Pod as shown in. The system is capable of checking the patient for injuries and applying the right medication to help out. This may be done in remote spot automatically, to ensure a fast reaction time of the system. Therefore, a sophisticated intelligence is necessary, which makes the right conclusions at the appropriate time. For extended support, there should be every time a possibility for a human doctor to monitor or even to take over the operation. And in the case that the system encounters any sort of error, the human backup is mandatory to ensure a safe handling of the patient.

Conclusion
The currently available systems for robotic surgery rely on the surgeon as the master control instance. Therefore the human error cannot be excluded, neither reduced. As described in Currently ongoing research and developement, the research areas in this area are looking for opportunities to support the surgeon with his work. This will lead to a set of support systems, which may help the surgeon in critical situations. But there are no crucial attempts to take over the control from the surgeon. Those systems are considered to be the future targets, as stated in Future Applications. That's while surgeons are forced to expand their technical knowledge of today. Obviously "three-dimensional transformations, co-ordinate systems and error propagation are all familiar concepts to the scientists" p.29, but unnatural for medical trained persons. But while robotic systems need this information, the usage of surgical robotic systems schould only require a basic understanding for such technical devices. Also the complexity in usage, for example the long set-up times, has to be reduced in order to increase the usability of such systems. In fact reducing the general requirements for the surgeon to use a surgical robot will make their application more common.

Future systems have to consider the 7SSRs (Seven surgical robot risks) always during the execution of their tasks. They are listed in Table 4. Mainly the concern lays about the safety of the patient, which leads to a extreme reliable, safe and robust robotic system.

In the case of used cognition in one of the future systems, the following questions have to be stated and answered properly: Only, if you can answer those questions appropriate, a system should be tested at a human patient. And if these tests in combination with the theoretical verification of the system are satisfying, the aim may be set to commercialize the system.
 * How can be ensured, that the system is safe and knows its borders? And how can be guaranteed, that if the system reaches its borders, it hands over the control to the surgeon?
 * How can self-learning systems be validated?
 * How to verify complex systems working on an even more complex and non-deterministic system, known as the human body?