During my 5 years as a PhD student at the Robotics Institute at Carnegie Mellon University, the focus of my research was on intelligent microsurgical instruments. Specifically, I design and develop computer vision and control system algorithms for Micron, a fully handheld tool, to suppress tremor and provide helpful behaviors. The key advantage to Micron is that it is small and compact with motors between the handle grip and the end-effector, which allows the tip to move semi-independently of the hand motion. To aid the surgeon with guidance, I have proposed a number of vision-based control behaviors that aid vitreoretinal surgeons during retinal procedures. I focus on combining realtime computer vision algorithms such as feature tracking, stereo, 3D reconstruction, tip tracking, anatomical analysis, etc. with control approaches such as state estimation with Kalman Filters, feedforward controllers, virtual fixtures, etc. In phantom or ex vivo porcine eyes, I have successfully demonstrated that Micron can aid three common retinal procedures: laser photocoagulation, membrane peeling, and vessel cannulation.

Micron: Fully Handheld Surgical Robot (TRO’12)

We describe the design and performance of a handheld actively stabilized tool to increase accuracy in microsurgery or other precision manipulation. It removes involuntary motion, such as tremor, by the actuation of the tip to counteract the effect of the undesired handle motion. The key components are a 3-degree-of-freedom (DOF) piezoelectric manipulator that has a 400-μm range of motion, 1-N force capability, and bandwidth over 100 Hz, and an optical position-measurement subsystem that acquires the tool pose with 4-μm resolution at 2000 samples/s. A control system using these components attenuates hand motion by at least 15 dB (a fivefold reduction). By the consideration of the effect of the frequency response of Micron on the human visual feedback loop, we have developed a filter that reduces unintentional motion, yet preserves the intuitive eye–hand coordination. We evaluated the effectiveness of Micron by measuring the accuracy of the human/machine system in three simple manipulation tasks. Handheld testing by three eye surgeons and three nonsurgeons showed a reduction in the position error of between 32% and 52%, depending on the error metric.

Position-Based Virtual Fixtures (ICRA’11, TRO’12)

Performing micromanipulation and delicate operations in sub-mm workspaces is difficult because of destabilizing tremor and imprecise targeting. Robotic aid combined with filtering techniques that suppress tremor frequency bands increases performance. However, if knowledge of the operator’s goals is available, virtual fixtures have been shown to improve performance even more. In this paper, we derive a position-based virtual fixtures framework for active handheld micromanipulators. For applicability in surgical environments, the fixtures are generated in real-time from microscope video during the procedure. Additionally, we develop motion scaling behavior around virtual fixtures as a simple and direct extension to our formulation. We demonstrate that virtual fixtures outperform tremor cancellation algorithms on a set of artificial but medically relevant tasks: hold still, circle tracing, move-and-hold, and vein tracing (p < 0.05).

Membrane Peeling (ICRA’12)

Peeling delicate retinal membranes, which are often less than 5 μm thick, is one of the most challenging retinal surgeries. Preventing rips and tears caused by tremor and excessive force can decrease injury and reduce the need for follow up surgeries. We propose the use of a fully handheld microsurgical robot and vision-based virtual fixtures to enforce helpful constraints on the motion of the tool. Our key contribution is a vision-only system to reduce and limit forces during vitreoretinal surgery: no force feedback is used in the control system. Using stereo vision and tracking algorithms, the robot activates motion-scaled behavior as the tip reaches the surface, providing finer control during the critical step of engaging the membrane edge. A hard virtual fixture just below the surface limits the total downward force that can be applied. Furthermore, velocity limiting during the peeling helps the surgeon maintain a smooth, constant force while lifting and delaminating the membrane. On a phantom consisting of plastic wrap stretched across a rubber slide, we demonstrate our approach reduces maximum force by 40-70%.

Retinal Vessel Cannulation (EMBC’10)

Cannulation of small retinal vessels is often prohibitively difficult for surgeons, since physiological tremor often exceeds the narrow diameter of the vessel (40-120 μm). Using an active handheld micromanipulator, we introduce an image-guided robotic system that reduces tremor and provides smooth, scaled motion during the procedure. The micromanipulator assists the surgeon during the approach, puncture, and injection stages of the procedure by tracking the pipette and anatomy viewed under the microscope. In experiments performed ex vivo by an experienced retinal surgeon on 40-60 μm vessels in porcine eyes, the success rate was 29% (2/7) without the aid of the system and 63% (5/8) with the aid of the system.

Kalman Filter + Feedforward Control (IROS’11)

Active compensation of physiological tremor for handheld micromanipulators depends on fast control and actuation responses. Because of real-world latencies, real-time compensation is usually not completely effective at eliminating unwanted hand motion. By modeling tremor, more effective cancellation is possible by anticipating future hand motion. We propose a feedforward control strategy that utilizes tremor velocity from a state-estimating Kalman filter. We demonstrate that estimating hand motion in a feedforward controller overcomes real-world latencies in micromanipulator actuation. In hold-still tasks with a fully handheld micromanipulator, the proposed feedforward approach improves tremor rejection by over 50%.

Retinal Laser Photocoagulation (EMBC’09, LMS’10)

In laser retinal photocoagulation, hundreds of dot-like burns are applied. We introduce a robot-assisted technique to enhance the accuracy and reduce the tedium of the procedure. Laser burn locations are overlaid on preoperative retinal images using common patterns such as grids. A stereo camera/monitor setup registers and displays the planned burn locations overlaid on real-time video. Using an active handheld micromanipulator, a 7×7 grid of burns spaced 650 μm apart is applied to both paper slides and porcine retina in vitro using 30 ms laser pulses at 532 nm. Two scenarios were tested: unaided, in which the micromanipulator is inert and the laser fires at a fixed frequency, and aided, in which the micromanipulator actively targets burn locations and the laser fires automatically upon target acquisition. Error is defined as the distance from the center of the observed burn mark to the preoperatively selected target location. An experienced retinal surgeon performed trials with and without robotic assistance, on both paper slides and porcine retina in vitro. Robotic assistance can increase the accuracy of laser photocoagulation while reducing the duration of the operation.

Visual Servoing with Micron (ICRA’09)

In microsurgery, a surgeon often deals with anatomical structures of sizes that are close to the limit of the human hand accuracy. Robotic assistants can help to push beyond the current state of practice by integrating imaging and robot-assisted tools. This paper demonstrates control of a handheld tremor reduction micromanipulator with visual servo techniques, aiding the operator by providing three behaviors: snap-to, motion-scaling, and standoff-regulation. A stereo camera setup viewing the workspace under high magnification tracks the tip of the micromanipulator and the desired target object being manipulated. Individual behaviors activate in task-specific situations when the micromanipulator tip is in the vicinity of the target. We show that the snap-to behavior can reach and maintain a position at a target with an accuracy of 17.5 ± 0.4µm Root Mean Squared Error (RMSE) distance between the tip and target. Scaling the operator’s motions and preventing unwanted contact with non-target objects also provides a larger margin of safety.