WO2015158756A1

WO2015158756A1 - Method and device for estimating an optimal pivot point

Info

Publication number: WO2015158756A1
Application number: PCT/EP2015/058137
Authority: WO
Inventors: Benoit Rosa; Emmanuel VANDER POORTEN
Original assignee: Katholieke Universiteit Leuven
Current assignee: Katholieke Universiteit Leuven
Priority date: 2014-04-16
Filing date: 2015-04-15
Publication date: 2015-10-22
Anticipated expiration: 2016-10-16
Also published as: GB201406821D0

Abstract

Methods and devices for determining the position of an optimal pivot point for handling instruments in minimally invasive surgery, comprises providing at least two ex-vivo images of an instrument's motion in time and extracting substantially parallel instrument lines from said images being separated by a same distance to determine the pivot point.

Description

METHOD AND DEVICE FOR ESTIMATING AN OPTIMAL PIVOT POINT

Field of the invention

The present invention provides methods and devices for estimating a pivot point, more specifically an optimal pivot point (OPP), for instruments in minimally invasive surgery (MIS). Background of the invention

The use of robotic assistance in surgery is rising, especially in laparoscopic procedures. The surgeon typically inserts a camera and small instruments (diameter 3 to 12 mm) inside the body of the patient through a cannula (commonly called "trocar") at an incision point in order to perform the surgery in a minimally invasive way. This type of surgery has been found to be beneficial to patients, with low recovery time and morbidity rate. However, the movements of the instruments are constrained due to the trocar point, which makes it more difficult for the surgeon to operate with respect to an open procedure. Several robotic devices have been designed to help the surgeon performing laparoscopic operations. An important requirement is to respect the motion constraint imposed by the trocar point, and therefore limit the stress upon the body wall. Passive pivoting approaches make use of a two degrees of freedom passive joint at the end of the robot. This approach ensures that the instrument freely rotates and aligns with the trocar point. However, control issues might arise when the exact position of the insertion point is unknown to the controller.

Software approaches (soft-RCM) make use of a redundant robotic arm and an appropriate control software that ensures that the instrument passes through the trocar point at all times. This approach has the significant drawback that a control failure could result in high forces applied to the patient's body. Finally, mechanical approaches use constrained kinematic structures to force the instrument to rotate around the incision point, either using a local approach (LCM) or a remote center of motion (RCM). All abovementioned methods require a good alignment with the incision point, either for precise instrument control or for keeping the interaction forces with the body wall and the instruments below acceptable levels. Nevertheless, few efforts have been conducted so far to determine the location of the entry-port (incision point) into the body. Also, since the body wall typically has a non-negligible depth, an optimal pivot point (OPP) will likely not coincide with the outer body surface. Such a point can be defined as a point along the axis of the incision - or at least close to the axis - at a specific depth and pose relative to the incision for which interaction forces upon the body wall when instruments pivot around this point will be minimal with respect to some relevant metric, where the metric to be minimized could be for example the average force amplitude, the maximal force or stress on the tissue over a period of time, the root mean square of the forces or other relevant metrics. To the best knowledge of the authors, no prior work has been conducted to determine the location of this optimal pivot point. Without loss of generality the focus here is on aligning robots with mechanical RCM mechanisms. Compared to passive, LCM or soft-RCM, mechanical RCM mechanisms allow more precise control, but require additional care to align the RCM with the trocar point. In current practice, the alignment of the RCM for these (and soft-RCM) types of solutions is done manually/visually. Many systems are not foreseeing any specific procedure for this purpose. Some systems provide the surgeon with a specific calibration instrument in order to place the RCM at a fixed distance under the patient's skin. In the case of the da Vinci® robot (Intuitive Surgical, USA), the most commonly used robot implementing a mechanical RCM, dedicated cannulas with markers are employed to visually align the RCM with respect to the body wall. According to the Intuitive Surgical prostatectomy procedure guide (PN 871403 Rev. A 4/06) "the instrument arm remote center is indicated by the thick, center black band on the instrument cannula. The thin black band located 1cm from the instrument cannula tip should be used as the insertion depth reference and should be located at the peritoneum". This approach therefore uses a fixed distance between the thin black band placed at the peritoneum and the robotic arm's RCM. It has been shown to be a practical approach as no increased prevalence of trocar-site hernias has been reported for robot-assisted interventions. However it suffers from several drawbacks. First, endoscopic vision is needed to place the cannula marker at the peritoneum, making this approach unsuitable for alignment of the RCM for the endoscopic port itself. A second endoscopic port could in principle be used for this purpose, but given that such ports are typically larger than instrument ports, this would call for larger incisions and could increase the risk of post- procedure complications. Second, this procedure relies on the approximation that placing the RCM at a fixed distance from the peritoneal wall minimizes the forces applied on the patient. While this approximation might be acceptable in laparoscopic procedures where the flexibility of the skin can compensate for misalignment of the RCM, it becomes more problematic for thoracic procedures— because of the rigidity of the ribs— or in single incision procedures. In the latter case, the method using a marker on the cannula is not applicable since the endoscope and the instruments pass through the same incision and it is therefore not possible to see the cannula in the endoscopic images during insertion. Moreover, it has been shown that misalignment of the RCM in single incision procedures might lead to high torques on the instruments and possible loss of the pneumoperitoneum. Finally, severely obese patients have a significantly thicker body wall which might induce larger forces, making correct RCM alignment even more important. As the prevalence of obesity is increasing in industrialized countries, this argument will likely only gain in importance. In summary, from above reasoning it is clear that current practice has its limitations. Simply aligning the robot's RCM with the incision point at the outer surface is not an option as it's optimal position is often significantly lower than the incision point due to the thickness of the wall which ranges from a few to more than 80 mm depending on the location of the incision and the patient's morphology. The heterogeneous nature of the wall further complicates the task, making methods that rely solely on measuring wall thickness suboptimal.

Only a limited number of studies dealt so far with tracking of the pivot point for robot-assisted surgery. Therefore, prior art can be divided into two categories: marker- or sensor-based methods and image-based methods. Passive approaches avoid applying large forces on the patient by making use of a passive linkage between the robotic holder and the instrument. However, the fulcrum effect around the pivot point makes control in Cartesian space imprecise. Ortmaier et al., in "Cartesian control issues for minimally invasive robot surgery", Intelligent robots and systems (2000), pp. 565-571 , therefore developed a method using encoders on the passive joints to estimate an OPP position. By moving the instrument for a few seconds and recording the joint positions, one is able to recover the position of the OPP as the minimum of the squared distances to the instrument positions. This method is precise and quick but needs special encoders on the free joints. Moreover, it is not suitable for LCM or mechanical RCM approaches.

It is known from Wilson et al "Evaluating Remote Centers of Motion for Minimally Invasive Surgical Robots by Computer Vision", 2010 IEEE/ASME International Conference on advanced Intelligent Mechatronics, pp. 1413-1418 (DOI 10.1 109/AIM.2010.5695924) to use two-view computer vision to estimate the position of an RCM. The centerline of a cylindrical instrument is computed based on pre-image (plane passing through the line and the optical center of the camera) and co-image (orthogonal complement to the pre-image) representations. Vectors from the co-images of the two edge lines of the instrument are averaged to give a co-image vector of the centerline. The RCM is computed from the vectors of the cylindrical instrument centreline at multiple poses. However, the method relies on the assumption that instrument's edge lines can be extracted with high precision beforehand, such as by putting a white stripe on the instrument shaft and placing the instrument on a black background to ensure a high image intensity gradient at the instrument's edges, which in real-life surgical situations is far from trivial.

Other prior art methods use endoscopic vision. For these methods it is assumed that the endoscopic camera is calibrated beforehand. This is a serious limitation in real practice because zoom and focus are often changed during surgical operations, turning the calibration data invalid. Recalibrating the camera parameters during the operation is not an option, as this would require special instrumentation (i.e. calibration pattern) and is time consuming. Moreover, previous knowledge of an instrument property is required.

Therefore there is a need for improved methods and devices for determining an optimal pivot point.

Summary of the invention

It is an object of the present invention to provide a method and device for estimating an advantageously optimal pivot point (OPP).

It is an advantage of embodiments of the present invention to provide a good, e.g. improved, method, which is an image-based method for determining a pivot point, more specifically an optimal pivot point, that furthermore allows one to align the robot RCM with the OPP and advantageously limit the interaction forces and stresses applied upon the patients skin during robot-assisted MIS.

In a first aspect the invention provides methods for determining a pivot point, preferably an optimal pivot point, the method comprising:

-moving an instrument about a pivot point;

-providing ex-vivo images (N) of said movement in time;

-determining the pivot point in the ex-vivo images, characterized in that said determining comprises following steps: · extracting straight lines from the ex-vivo images, referred to as arbitrary straight lines, resulting in extracted arbitrary straight lines

• filtering said extracted arbitrary straight lines, wherein said filtering comprises keeping at least two pairs of substantially parallel arbitrary straight lines and classifying said substantially parallel arbitrary straight lines in instrument or background lines; - estimating or determining the pivot point by fitting said instrument lines based on a statistical scheme.

Advantageously, at least three ex-vivo images (N) are provided.

In a further aspect the invention provides methods for determining an advantageously optimal pivot point, the method comprising:

- moving an instrument about a pivot point;

- providing at least two ex-vivo images (N) of said movement in time;

- extracting straight lines from said ex-vivo images;

- filtering said straight lines, wherein said filtering comprises: · obtaining at least one pair of substantially parallel straight lines from a first ex-vivo image, resulting in a first set of at least one pair of substantially parallel straight lines;

• obtaining at least one pair of substantially parallel straight lines from a second ex- vivo image, resulting in a second set of at least one pair of substantially parallel straight lines; · comparing said first and second set of substantially parallel straight lines, comprising classifying said substantially parallel straight lines in pairs of substantially parallel instrument lines and background lines;

- determining the pivot point based on at least two pairs of substantially parallel instrument lines. The substantially parallel straight lines forming each of said pairs of substantially parallel instrument lines are separated by a distance.

The straight lines are advantageously extracted from the ex-vivo images by image analysis without pre-knowledge of the instrument's position, orientation and dimension. The straight lines are advantageously extracted through an edge detection algorithm. The extracted straight lines can hence be referred to as arbitrary straight lines.

The at least two ex-vivo images hence advantageously refer to images of different poses of the instrument about the pivot point. The pivot point is advantageously determined based on at least two pairs of substantially parallel instrument lines originating from different ex-vivo images, i.e. at least two pairs of the substantially parallel instrument lines refer to a different pose of the instrument (about the pivot point).

The term substantially parallel can refer to parallel with a tolerance of +/- 2 degrees or less, advantageously +/- 1 degree or less, advantageously +/- 0.5 degrees or less. When the instrument is moved in an opening, such as an incision, in a soft, resilient, flexible or similar material, such as the human skin, freely and gently, by letting the incision act as a pivot point, methods of the invention allow to obtain an optimal pivot point, i.e. procuring minimal stress to the incision. The instrument is advantageously moved in an opening having dimensions able to self-support the instrument during motion. A following step can comprise moving an apparatus comprising a remote centre of motion such that the remote centre of motion is positioned at the pivot point.

Preferably said instrument lines are dynamic lines, such as lines which are not parallel between different ex-vivo images.

Advantageously, the method further may comprise the step of obtaining at least one pair of substantially parallel straight lines from one or more additional ex-vivo images, resulting in one or more additional sets of (at least one pair of) substantially parallel straight lines, and further comparing said additional sets of substantially parallel straight lines. Advantageously, the step of comparing comprises classifying the substantially parallel straight lines of the one or more additional sets in pairs of the substantially parallel instrument lines and the background lines.

In comparing the substantially parallel straight lines, pairs of the substantially parallel straight lines having lines which are parallel in at least three images, such as at least three subsequent images, are considered background lines.

Advantageously, the pivot point is determined based on an adequate combination of two or more of the first, second and additional sets of pairs of substantially parallel straight lines.

In further preferred embodiments filtering the extracted arbitrary straight lines is performed statistically. Advantageously, extracting arbitrary straight lines is performed using a Hough transform.

Embodiments of the invention provide a method wherein the ex-vivo images (N) are provided by at least one camera, preferably at least two cameras, placed statically at least 30 cm from the pivot point. When at least two cameras are used, it is advantageous to place the cameras such that each camera images the instrument under a different orientation. Instead of using two fixed cameras, it is possible to use one camera which is sequentially placed at at least two different locations, or a stereo camera can be used.

Advantageously, when at least two cameras are used (or, ex-vivo images of the instrument are obtained from different orientations with respect to a position of the instrument, e.g. different viewing angles), the pivot point is determined from the ex-vivo images of each camera/viewing orientation independently, which results in a two dimensional location of the pivot point, advantageously in an image plane of the camera. Methods may comprise a step of determining the location of the pivot point in three dimensional space by triangulation of the two dimensional locations of the pivot point obtained from different viewing orientations/different cameras.

Advantageously, the distance (d) is determined by the dimensions of said instrument, such as a distance between edges of the instrument. The instrument is advantageously elongate and advantageously comprises straight edges extending along a longitudinal axis. It will be convenient to note that the instrument advantageously comprises edges which are parallel, and (substantially) straight over at least a portion along the edge. Advantageously, but not necessarily, the instrument has (right) cylindrical shape. Other shapes, such as (right) prismatic with polygonal base, are possible. The instrument can be part of a larger, e.g. longer assembly comprising flexible parts. The instrument hence can refer to an advantageously rigid end, such as a distal end, of a generally flexible assembly, such as a catheter.

In preferred embodiments determining the pivot point is performed by a RANSAC fitting.

Advantageously, moving the instrument about the pivot point is performed by making the instrument follow a conical trajectory. The conical trajectory has an apex of the conical trajectory advantageously located in proximity of the pivot point. Letting the instrument perform a conical trajectory allows to view the instrument in extreme positions regardless of the viewing orientation from which the camera(s) are observing.

Advantageously, the instrument is a surgical instrument or a surgical instrument rigidly attached to a passive instrument holder or an instrument held by a surgical robot or a surgical robot.

Embodiments of the present invention further comprise monitoring the quality of movement of the instrument around a pivot point. Preferably, monitoring the quality of movement comprises measuring and/or evaluating a parameter related to the interaction between the instrument and the environment during a period of time. The environment can be a material body forming an opening in or at which the pivot point is to be determined. The material body advantageously supports the instrument during motion. The material body can be a patient's body. Advantageously, the method foresees measuring means to estimate the quality or optimality of the estimated pivot point, wherein the measuring means may be a camera, a force sensor measuring the interaction force with the body wall, or any other sensor or combination of sensors. An optimality of the pivot point can be calculated by evaluating a metric during a span of time of suitable length, the metric being advantageously representative of the interaction between the instrument and the environment as described above. Such metric can be for example an average force amplitude, a maximum force or stress applied on tissue, an average tissue load, the displacement or rotation of the tissue in the region close by the incision or any other relevant metric.

Advantageously, a method according to the present invention further may comprise the step of determining a location of the remote center of motion of a RCM system, such as a robot or manipulator, in free space (not in the incision point) and aligning the remote center of motion to the determined pivot point. The RCM system may hold an instrument for e.g. performing a surgical operation. Advantageously, a method according to the invention, may be applied twice or more, both to determine the location of the pivot point, and to determine the location of the remote center of motion of the mechanism/system. Applying this method twice may allow inferring the relative displacement that is to be undertaken by the mechanism to align the remote center of motion with the pivot point. After which, if considered convenient the above method could be applied to align the remote center of motion to the optimal pivot point. Preferably, aligning the instrument with the determined optimal pivot point enables the instrument's axis to maximally pass through the determined optimal pivot point which in the case that the instrument is held by a mechanism with any kind of remote center of motion this alignment would be done by making that remote center of motion maximally coincide with the determined optimal pivot point. It will be convenient to note that if the instrument is fixed to the RCM apparatus when determining the pivot point, the RCM should be unlocked and free to move (in any direction). Hence, according to aspects of the invention, a three-step procedure is proposed for aligning a remote centre of motion of an RCM apparatus/system to an optimal pivot point. In a first step, a pivot point, advantageously an optimal pivot point is determined as described in the above methods, in e.g. an incision in a patient with an instrument (which e.g. can be held by hand, or by a free-to-move manipulator, such as an RCM manipulator). In a second step, the remote centre of motion of the RCM apparatus/system is determined in free space, outside of the incision. This can be done using methods as described in the present invention, such as by attaching an instrument to the RCM apparatus/system and applying methods according to the present invention to the instrument. Alternatively, other methods could be used, such as by tracking markers. In a third step, once both the position of the optimal pivot point and of the RCM are determined, the remote centre of motion of the RCM apparatus/system is aligned on the (optimal) pivot point.

In further preferred embodiments a method is provided further comprising a calibrating step, wherein said calibrating step comprises calibrating the determined optimal pivot point at regular intervals in time. Preferred embodiments of the present invention may comprise measuring means to confirm the validity of the identified optimal pivot point at regular intervals in time, whether such measuring means consists of a camera that is pointed at the region of the optimal pivot point and detects abnormal deformation of the tissue which could be explained by the estimated optimal pivot point no longer being valid, or whether such measuring means would measure the interaction force between the instrument being moved about the earlier optimal pivot point and where such forces would be higher than expected which could be explained by the estimated optimal pivot point no longer being valid or whether the state of the estimated optimal pivot point no longer being valid is measured by any other means.

Advantageously the method according to embodiments of the invention provides signaling means, which is foreseen to inform when a level of optimality of an identified optimal pivot point is lower or higher than a certain threshold. The signaling means advantageously informs the operator, assistant, colleague or system about the optimal or oppositely of the suboptimal nature of the currently estimated pivot point as to give advice that the pivot point identification procedure can be halted in case a sufficient quality is reached or that oppositely the identification procedure and the instrument motion should be continued or repeated in order to reach an estimated pivot point of sufficient optimal quality. Preferably said signaling means is foreseen to inform when the validity of a pivot point is lower or higher than a certain threshold. The signaling means may warn the operator, assistant, colleague or system that the quality of the current pivot point is to be verified and/or advises to conduct the procedure (again) to determine the pivot point.

Advantageously, the method further comprise providing feedback to a user. Preferably providing feedback can be done visually and/or by using sounds and/or by using a display and/or by using a laser based system and/or by providing kinesthetic, tactile and/or vibro- tactile feedback. In further preferred embodiments providing feedback may comprise guiding a user to the determined (optimal) pivot point. Preferably specific guidance be it visual, haptic (kinaesthetic, tactile or vibrotactile), auditory or any other is provided to a user or operator, assistant, colleague or system about the manner to perform the movement of the instrument to identify the optimal pivot point so as to improve the convergence of the pivot point estimation allowing faster reaching of a pivot point with pre-defined level of optimality.

Advantageously, means for providing feedback to a user may comprise a user interface that provides guidance information to the user in order to align a device with the optimal pivot point. Such guidance could simplify motion in the preferred direction and prevent or even prohibit motion in any other direction. The user interface could display this information directly on the device that is being aligned or could employ any other means that is found convenient. For example in case an actuated robot is holding the instrument and is to be aligned, this robot could be programmed to generate low resistive forces in the preferred direction compared to high resistive forces in any other direction so that the user could steer the robot intuitively relying on haptic feedback to perform the alignment. Alternatively, tactile or vibro-tactile or other types of feedback could also be supplied to the user.

Advantageously, moving an instrument about a pivot point to determine or estimate the location of the pivot point may be performed by a surgical robot that performs this procedure in an automated, safe and gentle way and wherein the instrument motion is programmed so as to allow good and reliable estimation of the pivot point while at the same time ensuring minimal interaction with the tissue and wherein this ensuring of minimal interaction with the tissue is done for example by directly measuring the interaction forces while executing the instrument motion or by observing the motion of the surrounding tissue while moving the instrument or by any other means.

In further preferred embodiments, the method further provides means to estimate physiologically-induced motion patterns of the tissue in the area of the pivot point and in the absence or in the presence of an instrument, wherein such physiologically-induced motion patterns are for example induced by heartbeat or breathing and a method wherein this physiologically-induced motion is superimposed in an appropriate manner upon the location of the pivot point so that the location of the pivot point is dynamically altered in a manner that is synchronized with the physiological phenomenon, wherein an instrument holding device is programmed so as to follow the dynamically altered pivot point location. As a result of incorporating such estimation in the determination of the OPP according to the present invention, the OPP is calibrated and advantageously a correct optimal pivot point is provided at all time. Where in embodiments of the invention the term "ex-vivo image" is used, the latter relates to images taken outside an organism, e.g. the patient's body, and thus for instance of the surgical scene, e.g. the movement of an instrument outside of the organism. As a result ex- vivo images exclude endoscopic images taken from within a patient's body. Advantageously, at least one camera is used to provide the ex-vivo images. Preferably any camera which can provide line (and/or possibly depth) information can be used. For instance a stereo-camera, a time-of-flight camera or camera's using structured light information can be used. In further embodiments of the invention, increasing the number of camera's could further help improve the accuracy and reliability of the OPP estimation as it would allow be more robust against occlusions of the view of a single camera. But the use of multiple cameras should be outweighed against their space occupancy. Alternative manners to increase reliability would report quality metrics, would project the estimated OPP in an Augmented or Virtual Reality display, or would use any other means to communicate the estimated OPP to the user for confirmation. In alternative embodiments a system comprising lasers, e.g. a laser pointing or laser line generating systems can be used to provide feedback to a user, instead of for instance systems based on displays or guidance by emitting sounds and commands. Such laser pointing or laser line generating systems can be used in methods or assemblies according to aspects of the present invention as follows: - determine the pivot point according to methods of the invention;

- determine the robot's RCM position. This could be done by detailed identification of the robot's forward kinematic chain or by for instance using image information, for instance using the same method as for the OPP when the robot holds an instrument and moves around its RCM. An alternative could be to use one or several specific markers that are placed at known positions on the robot itself as is implemented on the robot prototype that is used in the experiments reported in this text.

- determine the position of the incision at the outer skin surface, for instance by using 3D reconstruction with cameras.

- compute the 3D vector (u) from the pivot point to the position of the incision at the outer skin surface.

- mount one or more lasers or sets of lasers on the robot or any convenient location in the operating theatre, wherein this laser or lasers are set (for instance using motors to control the angle of the laser beam or controlling mirrors to deflect the orientation of the laser beam) to point towards the point RCM plus 3D vector u. This allows the user for aligning the projected laser point with the point at the outer surface of the incision. When this is the case, the RCM of the robot is aligned with the OPP. This principle is schematically illustrated in Fig. 5.

In an alternative embodiment a convenient point located at a position v from the robot's RCM is aligned with a point at a similar relative displacement v = u + r from the OPP to align the robot's RCM with the OPP. The vector can for example be conveniently selected to match the dimensions of a cannula, so that the offset introduced by the cannula is conveniently incorporated, but any other convenient offset could be envisioned.

Advantageously, an instrument is moved according to an arbitrary trajectory. For instance as soon as there are sufficiently large angles between images of the different instrument poses, a pivot point, more specifically an OPP, can be reliably extracted. The angles of the instrument poses are preferably spanning over at least ττ/6 radians in image space. In embodiments of the present invention a conic trajectory is suggested, however it is merely provided as an example in order to explain the algorithm according to embodiments of the present invention.

Advantageously, the instrument is moved about a pivot point, wherein the instrument extends through the pivot point.

Advantageously, the instrument is preferably a surgical instrument which can be coupled to a robotic arm. Embodiments of the present invention provide an improved method for estimating the position of a pivot point, and more preferably an optimal pivot point, for handling instruments in minimally invasive surgery. Such knowledge is of particular importance for robotic-assisted surgery where robots need to rotate precisely around a specific point in space in order to minimize trauma to the patient's body wall and to achieve good position control. For this purpose Remote Center of Motion (RCM) mechanisms are commonly used, where the RCM point is typically manually and/or visually aligned. Also for redundant robotic arms where software approaches can be employed to program instrument motion about a specific point in space (soft-RCM). The location of such a specific point in space is preferably taught appropriately to the robot controller, which can be done manually and/or visually. Approaches using force sensors have also been used in the art: wherein the robot is programmed to move so that forces at the incision, which are measured by the force sensor, are regulated towards zero. Since integrating a force sensor in the tip of a surgical instrument is a difficult task, such methods typically use an external force sensor. If the instrument gets in contact with the environment, the forces measured at the force sensor then are the sum of the forces at the incision and the forces at the tip. In that case, this approach becomes imprecise.

If not positioned appropriately this misalignment might lead to intolerably high forces on the patient's body wall, increasing risk for post-operative complications. Moreover, the misalignment might lead to high interaction forces between the instrument and the patient's body wall which might cause damage to the instruments, which simply deform the instrument and as such complicate precise position or force control of the instrument within the body cavity or which simply deform or displace the body wall which ultimately might also deform or displace the anatomical structure that forms the target of the instrument action and as such complicate precise control over this anatomical structure.

A good example of the latter is for example given by vitreoretinal microsurgery on the eye. In vitreoretinal procedures micro-surgeons typically introduce instruments into the eye through the pars plana to work at the opposite inner side of the eye-ball namely at the retina. The eye behaves as a spherical joint that is loosely hinged by a plurality of eye muscles in the eye socket. Also the target area, i.e. the retina is directly connected to the incision point on the eye-wall. If instruments do not precisely move around the pivot inside the incision of the eye wall even low interaction forces upon this wall might cause the eye to rotate and as such complicate precision task on the back of the eye as all actions need to be accomplished upon a moving target in such case. Similar problems can appear in other surgical domains as well.

Moreover, embodiments of the present invention provide the use of computer vision and a lightweight calibration procedure to estimate a pivot point, more specifically the optimal pivot point. At least one, for instance one or two pre-calibrated cameras, for example stereo cameras, looking at the surgical scenes can be used hereto. The surgeon is asked to make some short pivoting movements with an instrument of his choice passing through the insertion point while camera images are being recorded. The physical properties of an instrument rotating around a pivot point are exploited in a RANSAC scheme in order to robustly estimate the position of the OPP in the image planes, said position which can then be used to provide guidance to align the robot's RCM. Advantageously, triangulation is used to estimate its position in 3D. Experiments carried out, on a specially designed mockup and described further, show that the position of the pivot point is estimated by a method according to embodiments of the present invention, with an average error less than 1.85 mm using two webcams placed at a distance between 30cm and 1 m from the scene and thus providing ex-vivo images. The whole procedure advantageously takes only a few seconds.

As a result, embodiments of the present invention provide a novel method which is cost- effective and reliable. Moreover, it can be used within a visual servoing approach in order to automatically or semi-automatically place the RCM point. Alternatively results can be simply displayed on a screen to provide guidance to the surgeon. In other embodiments an image- guided alignment method can be provided as well.

In a second aspect, the present invention provides a computer program product for, if implemented on a control unit, performing a method according to aspects of the present invention.

In a third aspect, the present invention provides a data carrier storing a computer program product according to the second aspect of the present invention. The term "data carrier" is equal to the terms "carrier medium" or "computer readable medium", and refers to any medium that participates in providing instructions to a processor for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks, such as a storage device which is part of mass storage. Volatile media include dynamic memory such as RAM. Common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, or any other magnetic medium, a CD- ROM, any other optical medium, punch cards, paper tapes, any other physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described hereafter, or any other medium from which a computer can read. Various forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution. For example, the instructions may initially be carried on a magnetic disk of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector coupled to a bus can receive the data carried in the infra-red signal and place the data on the bus. The bus carries data to main memory, from which a processor retrieves and executes the instructions. The instructions received by main memory may optionally be stored on a storage device either before or after execution by a processor. The instructions can also be transmitted via a carrier wave in a network, such as a LAN, a WAN or the internet. Transmission media can take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications. Transmission media include coaxial cables, copper wire and fibre optics, including the wires that form a bus within a computer.

In a fourth aspect, the present invention provides in transmission of a computer program product according to the second aspect of the present invention over a network.

In a fifth aspect, the present invention provides assemblies for determining a pivot point, such as an optimal pivot point, the device comprising:

- an instrument adapted to be moved about a pivot point:

- at least an image providing means, wherein said image providing means is located outside a patient's body;

- processing means, said processing means adapted to perform a method according to embodiments of the present invention;

Preferably said instrument is provided in a previously provided incision in a patient's body.

Advantageously, the assemblyfurther may comprise monitoring means, said monitoring means adapted to determine the quality of movement of the instrument around a pivot point.

In further preferred embodiments the assembly further may comprise means for providing feedback to a user.

Embodiments of the present invention provide assemblies which are used as a module for and/or part of a robot for minimal invasive surgery. Particular and preferred aspects of the invention are set out in the accompanying independent and dependent claims.

Methods of performing minimally invasive surgery (MIS) are also described. Methods comprise making an incision in a patient's body and determining a pivot point at the incision according to any one of the methods described above. Methods of performing MIS can comprise moving an apparatus comprising a remote centre of motion to position the remote centre of motion at the pivot point. Methods can comprise attaching a surgical instrument to the apparatus such that it pivots on the remote centre of motion, and performing a surgical intervention with the surgical instrument.

Advantageously, the instrument is inserted in a previously provided incision in a patient's body. These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter, which are non limiting.

Brief description of the drawings

Fig. 1 schematically illustrates a perspective projection camera model employed. The 3D double cone C described by the moving instrument projects onto a set of lines L on the image plane 3. The apex of the cone C coincides with the pivot point PP and projects onto a point PI (Pivot point in Image) on the image 3. Examples of instrument centerlines 1 are shown and their projection 2 in the image 3 as well. Reference 4 denotes the camera's focal point. Figs. 2 A-D illustrate a RANSAC implementation according to aspects of the invention and applied on data samples and their representation in the image plane(s) of the camera(s). In particular Fig. 2 A illustrates the extraction of a tentative model from two sets 20, 20' of substantially parallel lines; Fig. 2 B illustrates a check for consensus with data, wherein the outlier lines 21 , 22 are provided as dashed lines and the inlier lines 20, 20' as solid lines; Fig. 2 C illustrates the best model found after the k iterations with all sets 20, 20', 20", 20"' of substantially parallel inlier lines and finally Fig. 2 D illustrates refining the Pivot point in Image (PI), wherein the tentative estimate of PI (i.e. Pl^b) is provided by a circle and the refined estimate of PI (i.e. ΡΊ is provided by a cross, by using all the inliers and only them).

Fig. 3 A shows the first experimental setup used as an example to validate a method according to embodiments of the invention. Fig. 3 B illustrates mapping with the CAD model using the transform 77 and the fiducial markers 12 and the position of the three insertion points 1 1 in the CAD model. Fig. 3 C illustrates a reprojection of PI in the image plane using the inverse transform 77^"1.

Figs. 3 D-F represent images of the instrument 15 in different poses about the insertion point 1 1 .

Fig. 4 illustrates the results of the estimation of the PP according to embodiments of the invention, wherein said estimation is performed on a first experimental setup after removing failed detections using the epipolar constraint. The solid dots 12 represent fiducial markers positions that allow computing the validation frame (wireframe plan with an inclination) and the location of the insertion point (represented by the circles 1 1 ). The points 10, surrounded by the circles 1 1 , are the estimated PP using the method according to aspects of the invention. Fig. 5 illustrates an embodiment of the present invention of providing feedback to a user, e.g. a surgeon, wherein a laser pointing assembly is provided. One or more lasers 53 or sets of lasers are mounted on a RCM robot 51 or any convenient location in the operating theatre. The robot 51 is mounted on a base 52 and advantageously holds the instrument 55. This laser 53 or lasers are set (for instance using motors to control the angle of the laser beam or controlling mirrors to deflect the orientation of the laser beam) to point towards the point 54 obtained by adding the RCM with a 3D vector u. This 3D vector u can be computed by computing the pivot point PP according to methods of the invention, by determining a point P on the outer surface of an entry port (e.g. an incision in a body wall 50) in which the instrument 55 is to be inserted, wherein the 3D vector u is the vector pointing from PP to P. Then, the user preferably aligns the projected laser point 54 with P. When this is the case, the RCM of the robot is aligned with the optimal pivot point PP.

Figs. 6 A-C represent images of different steps of line extraction, pruning and PI estimation on overlayed images using a RANSAC algorithm on a 5 mm black instrument on obstructed background. In Fig. 6A lines are extracted using a Hough algorithm. Fig. 6B shows the result after line pruning, which retains sets of substantially parallel lines. Each light coloured line in Fig. 6B has an associated dark coloured line which is substantially parallel. Fig. 6C shows the best estimate 16 after application of a RANSAC algorithm on the lines of Fig. 6B.

The drawings are only schematic and are non-limiting. In the drawings, the size of some of the elements may be exaggerated and not drawn on scale for illustrative purposes. Any reference signs in the claims shall not be construed as limiting the scope. In the different drawings, the same reference signs refer to the same or analogous elements.

Detailed description of preferred embodiments

The present invention will be described with respect to particular embodiments and with reference to certain drawings but the invention is not limited thereto but only by the claims. Where the term "comprising" is used in the present description and claims, it does not exclude other elements or steps. Where an indefinite or definite article is used when referring to a singular noun e.g. "a" or "an", "the", this includes a plural of that noun unless something else is specifically stated. The term "comprising", used in the claims, should not be interpreted as being restricted to the means listed thereafter; it does not exclude other elements or steps. Thus, the scope of the expression "a device comprising means A and B" should not be limited to devices consisting only of components A and B. It means that with respect to the present invention, the only relevant components of the device are A and B. Furthermore, the terms first, second, third and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily for describing a sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments of the invention described herein are capable of operation in other sequences than described or illustrated herein. Moreover, the terms top, bottom, over, under and the like in the description and the claims are used for descriptive purposes and not necessarily for describing relative positions. It is to be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments of the invention described herein are capable of operation in other orientations than described or illustrated herein.

In the drawings, like reference numerals indicate like features; and, a reference numeral appearing in more than one figure refers to the same element. The drawings and the following detailed descriptions show specific embodiments of a method or device for estimating a pivot point. As indicated above, marker- and sensor-based methods are powerful, but require special instrumentation in order to be used. Therefore, they are not generally applicable. Image- based methods, like embodiments of the present invention, seem to be more versatile, as they advantageously only require a simple affordable camera or an endoscope camera which is present anyway and could be fixedly mounted ex-vivo, neither of which needing to be sterilized for the purpose of this invention.

Existing image-based methods, known in the art, focus mainly on instrument tracking. Hence, they rather concentrate on the various challenges associated with processing in-vivo images, such as low and non-homogeneous illumination, specular reflections and/or moving organs. It is well-known that using in vivo laparoscopic images is difficult because of the need of calibrating the camera parameters. Therefore, embodiments of the present invention provide a method requiring preferably ex-vivo images, i.e. outside the patient. In such case a pre-calibrated camera can be used. For a large depth of view camera, the camera can be positioned stationary or statically at a fairly large distance and advantageously at least 30 cm from the surgical scene, such as 30 cm from the instrument, for instance on the endoscopic tower.

Furthermore, as real-time instrument tracking remains challenging and error-prone, the present invention advantageously does not rely on instrument tracking but only on the physical properties of a pivot point. Moreover when the instrument is moving freely, the centerline of the instrument always points to the pivot point. Embodiments of a method according to the present invention comprise extracting straight lines from camera images and preferably uses a RANSAC scheme to differentiate the inliers -instrument lines providing information about the location of a pivot point— from the outliers — other straight lines detected in the field of view and that are not related to the location of a pivot point, preferably an optimal pivot point.

Estimation of the (optimal) pivot point from camera images using aspects of the present invention

A pivot point is advantageously static. For a certain workable and practical time period, advantageously no major adjustments are made to the surgical scene and the location of the (optimal) pivot point is assumed to be static. This assumption is in line with the principle of a free-joint system described by Ortmaier et al cited above who estimate the PP-location for optimal control of his passive RCM-robot. Note that Ortmaier et al relies on dedicated instruments and sensors embedded into the robot to estimate a pivot point, which the present invention does not rely on. However if the pivot point is not static, aspects of the present invention provide methods to calibrate said PP in time and thus provide an optimal pivot point at all times.

Advantageously, in methods according to the present invention moving an instrument about a pivot point comprises that an operator, for instance a surgeon moves said instrument, advantageously freely and gently in an incision with or without cannula, without a specific surgical task in mind, which will move the instrument about the optimal pivot point. The instrument is advantageously moved about the incision (or pivot point) applying minimal torque to the instrument.

As shown in Fig. 1 , moving the instrument is advantageously performed such that the instrument follows a conical trajectory, i.e. the instrument is moved about a pivot point PP such that the envelope of the positions of the instrument's centerlines 1 is conical. During such free motion, the instrument preferably describes a double cone C in 3D space. The apex of the cones preferably coincides with the PP and instrument centerlines will intersect with this point.

Advantageously, a perspective projection camera model is used. The latter comprises that images captured by an ex-vivo camera can be modelled by a perspective projection model. The camera can be calibrated beforehand or manufactured specifically in order to avoid image distortion. As a result, in case a cone trajectory is used, C will project into the image plane 3 forming a set of lines L that intersect on the projection of the PP into the image plane 3. This point will be referred to as a Pivot point in Image (PI) in the following (see for instance Fig.1 ).

An algorithm according to aspects of the invention to extract a PI from ex-vivo camera images is provided in the following. Contrarily to existing methods known in the art, the proposed invention advantageously does not rely on explicit tracking of the instrument throughout the images. Rather, a statistical scheme is preferably adopted. Since the surgeon moves the instrument freely, an adequately sized set of N ex-vivo images (for instance 100 to 200 images) with different positions of the instrument is acquired. These ex-vivo images, can advantageously go through the following processing pipeline: Line extraction step

Detecting straight lines, for instance by a Canny edge detection step, wherein straight lines are extracted from within the ex-vivo images by looking at their representations in Hough Space. While being computationally costly, this operation can be performed online during image acquisition at a rate of for instance 30 fps. Alternative to a Canny edge detection, other image-based methods known in the art can be applied here to detect straight lines, for instance based on hue-characteristics of the image.

The following approach is advantageously used to tailor the method and extract relevant lines (for instance the instrument's edges) from the ex-vivo image. Since the instrument can be considered to be a long straight cylinder, it is likely that its edges will be among the longest lines that can be found in the images. The longer the lines, the larger the number of intersections or votes in Hough space that can be associated to such a line. Unfortunately, since there is no guarantee that the instrument edge lines are in reality the longest straight lines in the image (longer straight lines could come from other instruments, background lines from the operating table, lines from the ceiling and so on), rather than searching for the line with maximal amount of intersections, an appropriate threshold value a is employed and all lines with more votes than a will be withheld. Starting from a predetermined value, a is adjusted until the average number of detected lines M in the first, for instance 20, images falls in a predetermined range, for instance between 20 and 100.

The angle precision for parallel line extraction algorithms, such as the Hough algorithm, is preferably half a degree or less. In other embodiments the angle precision can be chosen such that one balances the desirable features which one wants to achieve. For instance when considering a smaller angle precision, one should take a longer computational time in account and possible confusion (i.e. if for instance we have a 0.01 degree precision, we could detect a plurality of lines that are separated by 0.01 degrees that come from an edge of the instrument). As a result, the angle precision is preferably set by taking into account the type of line that is to be detecting (width of the edge), the image noise and the distance between the camera and the instrument. Line pruning step, whereby substantially parallel lines are retained

Among the extracted lines are the instrument edge lines and other background lines that should be eliminated. From the assumption that instruments have a diameter of at least a few mm, which corresponds to the dimensions of the instrument, and that the camera is located preferably at least 30 cm and preferably somewhere between 30cm and 1 m away from the surgical scene (e.g. from the instrument), instrument lines are considered substantially parallel within preferably half a degree precision as indicated above. For the avoidance of doubt, this assumption would not be applicable when endoscopic images, thus in-vivo images, would be used, as these in-vivo images are taken at a closer distance (< 30cm) from the instruments. Possibly, lines from the instrument don't project on the at least one camera into substantially parallel lines, because of the projective nature of the camera. But given the size of the instrument (in the range of at least a few mm, preferably between a range of a few mm and a few cm) and the distance of the camera (in the range of at least tens of cm), we assume they are. The tolerance is therefore dependent on the parameters of the camera, the distance to the scene and the instrument size.

Therefore, for each ex-vivo image, only lines that come in parallel pairs are kept as possible candidate instrument (edge) lines. Advantageously, if a line has parallel lines in more than three subsequent images, it is discarded as it is considered to be a background line and not an instrument line. This is done to exclude static lines that belong to the background. This also avoids over-fitting when the instrument appears static in too many images.

PI estimation

The pivot point in Image (PI) is estimated, advantageously based on iteratively checking an instantaneous estimate of the PI for consensus with data. RANSAC, a statistical method for fitting data to a model in the presence of a large number of outliers as illustrated by Fischler et al in "Random sample consensus: a paradigm for model fitting with application to image analysis and automated" (1981 ), is preferably adopted to estimate the position of the PI, but methods such as GroupSac (by Ni et al in Computer Vision, 2009 IEEE 12th International Conference on (pp. 2193-2200), betaSac (by Meier et al in British Machine Vision Conference 2010) or any other statistical methods for fitting data known in the art could be applied as well, such as M-estimators and Least Median of Squares (LmedS).

RANSAC relies on the random selection of data for making a tentative model. The method iteratively checks tentative models for consensus with the data in order to determine the best model, i.e. the model that has the greatest number of inliers within the data. Since this is a statistical scheme there is no guarantee that the best model which is found is actually accurate. The process advantageously ends when the likelihood of finding a better model is less than a selected threshold η. The number k of iterations that are needed therefore is given by: fc = log¾l¾-wⁿ) (Eq. 1 ) where p = 1 - η is the confidence in the model, w is the probability that a measurement is an inlier, and n is the number of data that is needed for establishing the model, as described by Fischler et al.

A possible implementation of PI estimation mehtods, such as RANSAC, according to embodiments of the invention is depicted in Figs. 2 A-D. Each iteration preferably comprises following three steps:

- A guess PI⁹ is made from two randomly selected pairs 20, 20' of parallel lines (Fig. 2A). PI⁹ can be computed as the point in image-space that minimizes the distance to the four lines. Because an identical instrument is being observed, the pair of lines must have a same distance d between them.

- Consensus is checked. Among all data (excluding the pair of lines used in the previous step), sets 20, 20' of substantially parallel lines at a same distance d (and associated tolerance ε) from each other and that each lie at a same distance d/2 (with a tolerance £2) from PI⁹ are counted as inliers. Sets of parallel lines that are far away from the possible PI (set 21 ) and/or that have a different distance between them (set 22) are considered outliers (see Fig 2 B). The total number of inliers N⁹ for the current guess is counted, ε and £2 can be typically set as the standard deviation of measurement noise for the data. In the examples described below 4 pixels were used for both.

- The model is then preferably compared to the best model to date. If N⁹ > N^b, where N^b is the number of inliers of the best model previously found, the current model is kept as the best model : Pl^b = PI⁹ and N^b = N⁹ (Fig. 2C). After k iterations, the final result is Pl^b, with N^b pair of lines considered as inliers. A last computational step can also be applied to minimize the sum of squared distances to the set of inliers in order to refine the estimated position of PI, PI :

where L_s is a line belonging to the set of inliers and dist is the Euclidean distance operator.

Algorithms according to aspects of the invention such as the one described above advantageously provide the location of the projection of the pivot point in the image plane of at least one camera. For the intended application, knowledge of the PP location in 3D (three- dimensional) space is more convenient. The above method can be expanded towards 3D and metric distance values as follows.

Preferably, a stereo-camera is used to estimate the PP position in 3D. Such a stereo-camera can be simply made by for instance fixing together two planar image cameras, such as webcams. In order to be able to estimate 3D positions with the stereo-camera, intrinsic (i.e. perspective projection model and distortion parameters) and extrinsic (i.e. rotation and translation between the cameras) parameters are preferably calibrated. Similarly to the previous situation with a single camera, it is assumed here that this calibration is preferably done only once: the cameras are fixed relative to each other and focus and zoom are not changed. The extrinsic parameters are then expressed in the reference frame of the stereo- camera and metric positions are estimated in this frame. Instead of using multi-view geometry to recover the instrument pose in 3D for each image, algorithms according to aspects of the invention are preferably used on the images of the two cameras independently in order to estimate the position of the PI in both images planes. The position of the PP is then estimated by simple triangulation. This is beneficial for several reasons.

First, live estimation of the instrument pose in 3D is computationally expensive and requires perfect synchronization of the acquisition between both cameras. This is not always possible with cheap solutions such as webcams where low-level control or frame acquisition triggering is not always available. Second, the proposed method allows checking for errors in the algorithm, such as the RANSAC scheme, estimate. Indeed, RANSAC is a statistical algorithm and the guarantee of estimating the good model is only statistical and depends on the number of iterations (see equation 1 ). By running the algorithm separately on images from the two cameras, one obtains two estimates of the projection of the PP; one in the left and one in the right image plane: Pf and ΡΓ, respectively. If Pf and Ρ indeed correspond to the projection of the PP in the images, the epipolar constraint can be expressed as (PI')^TF(Pr) = 0, where F is the Fundamental matrix. If, on the contrary, the algorithm did not converge to the good solution for the left image for instance, PI¹ will not be the projection of the PP in the image and (Pf)^TF(Pl^r) > 0. In practice, due to noise, the epipolar constraint is almost never fulfilled and a threshold £3D is used so that if (Pf)^JF(P\^r) < £3D, the epipolar constraint is considered to be satisfied. Note that the nature of epipolar geometry is such that it maps a point in an image with a line in the other. Therefore, it can be safely assumed that if (PI')^TF(Pr) > £3D the points are not the projection of the same 3D point. The inverse is however not true. Nevertheless, this constraint allows to check the validity of the output of the two algorithms.

Validation on an experimental bench according to examples of the application of a method according to the present invention

This section describes the implementation details and experimental validation of a method according to embodiments of the invention by providing examples that were conducted on an artificial abdominal wall.

Two different experimental setups have been used in order to investigate the accuracy and the robustness of embodiments of the present invention. The first experimental setup used to check the accuracy of the image processing algorithm in detecting a pivot point is depicted in Figs 3A-3C. The setup is designed to provide a fixed and precisely known pivot point in space. Hereto, a rigid metallic frame 13 is clamped to a table. A validation plate, here a wooden plate 14, is laser cut so that a 5 mm, a 8 mm and a 10 mm hole 1 1 are located at well-known positions with respect to four fiducial markers 12. The different holes 1 1 are used as insertion point. This design forces the inserted instrument 15 to rotate about a pivoting point 1 1 , similar to a real surgical practice, but without the variability induced by the compliant body wall, as shown in Figs. 3D-F. During the experiments, the positions of the fiducial markers 12 are extracted from the images. The markers are used to compute the position of the insertion point 1 1 . The perspective transformation 77 between the quadrangle formed by the fiducial markers in the image plane and the square they form in the CAD model of the wooden plate (see Figs. 3 B and C) is used to this end. The 3D position of the fiducial markers can then be computed by simple triangulation.

The second setup (not shown) is designed to investigate the precision of the pivot detection algorithm. The experimental setup is constructed to simulate more closely the behaviour of a real body wall. The same mockup base is used. The wooden plate is now replaced by a flexible foam material simulating the body wall (referred to as pelvi trainer). The stiffness of the used material is comparable to the one of the abdominal wall, but it is here homogeneous (i.e. does not reproduce the layered structure of the body wall). A trocar is inserted through an incision and instruments are inserted in it. In this case, the actual position of the 77 in the images is not known. Given the limited set of experiments that are being conducted of the assumption of a static OPP, is considered justifiable. The setup then gives insight in the robustness (precision) of the method. At the same time the setup can be used to test the validity of the assumption that if a surgeon or any other staff moves an instrument freely and gently in the trocar point, without a specific surgical task in mind, he/she will move the instrument about the optimal pivot point. If this assumption is true and the users limit forces exerted on the flexible body wall, the inter-users variability should be low.

For all experiments provided in the examples, at least one, for instance one or two Logitech C920 USB webcams are used depending on the context (mono- or stereo-camera). Images are acquired using OpenCV, from the Open Source Computer Vision Library http://opencv.org, with a resolution of 640x480 pixels. Camera calibration is preferably done using the calibrateCamera and stereoCalibrate functions of OpenCV. A 18 x13 checkerboard pattern with squares of 12.0mm may be used. For better accuracy, the intrinsic parameters of each camera are first calibrated independently. Afterwards the extrinsic parameters for the stereo-camera are calibrated. Calibration is considered successful when the average reprojection error falls below 1 pixel. For each experiment, the protocol was as follows:

(1 ) the mockup is prepared, either with the rigid frame or with the pelvi trainer;

(2) the camera or stereo-camera is positioned at a distance of at least 30cm to 1 m from the surgical scene, providing ex-vivo images;

(3) the user picks an instrument 15 of his choice; (4) the user inserts the instrument 15 in the according keyhole 1 1 , which refers to an incision, and moves it around while ex-vivo images are recorded by the camera;

(5) RANSAC estimation of ΡΊ is performed and displayed on the screen (along with ground truth information for the first experimental setup).

For each user, experiments are preferably performed both on a clean background (mockup placed in front of a surgical drape) and on an obstructed background representing a worst- case scenario (laboratory with bright and sharp objects as well as people moving in the background - see Figs. 6A-C). Concerning the RANSAC algorithm, a value of 0.99 preferably is used for the confidence p regarding the obtained model, n is equal to two because two sets of parallel lines are used for establishing a model, w is determined by w = 0.5 i (Eq. 3) where N is the average number of parallel lines found per image. A factor 0.5 is used for accounting for the fact that in some images the set of parallel lines does not contain the edges of the instrument.

In the section above it was discussed how to set the threshold a of the Hough method in order to return an average of Ni lines in the images. Ni is preferably between 20 and 100. In a worst-case scenario where all those lines would be parallel and then not filtered in the second step of the algorithm, the absolute average maximum number of substantial parallel lines N in the ex-vivo images would then be equal to

Using Eq. (1 ) and (3) the number k of iterations in the worst-case scenario is then 184212. In practice, most parasite lines are filtered in the second step of the algorithm and the number of iterations is much smaller, between preferably 100 and 3000 on average. The RANSAC part of the algorithm is then computed in less than one second using a single thread on a 3 Ghz CPU. As said before, the rest of the computing can be made online between frame acquisitions, so the total time needed for the procedure is close to the time for performing the gesture (for instance somewhere between 0.5 to 20 seconds). In this example the position of the insertion point 1 1 in the wooden validation plate 14 is considered known by extracting the position of the fiducial markers 12 in both images and computing the 3D position in space. Fig. 6A-C shows an example of PI estimation using the first experimental setup. Comparing Fig. 6A with Fig. 6B, shows the effect of a filtering stage based on determining sets of parallel lines, which removes most of the background lines. Fig. 6C shows application of the RANSAC algorithm to find a suitable estimate 16 of the PI .

Validation of the PI estimation using the first example setup (as shown in Figs. 3A-C) illustrated that the RANSAC scheme, used in embodiments of the invention, allows one to find a good estimate of the PI. In total 90 experiments have been carried out with different instruments (two 5mm, one 8mm and one 10mm instrument) and camera positions. The error between ΡΊ and the ground truth information was found to be encouragingly low: the average error was 1 .13 pixels, with a maximum error of 2.51 pixels and a standard deviation of less than 1 pixel. Among all trials, one did not succeed (error between ΡΊ and ground truth of several centimeters), making the success rate of our statistical recognition scheme to be 98.9%.

The same set of experiments was conducted using the calibrated stereo camera. Among the 90 experiments, 5 experiments were unsuccessful due to a failure in the RANSAC algorithm. All those experiments were detected since the epipolar constraint was not fulfilled and subsequently removed. The success rate of the 3D estimation algorithm is 94.4%. Among the 85 remaining experiments, the average distance error between the estimated OPP position and the ground truth was 1.85 mm (standard deviation of 1.47 mm, 90th percentile 3.16 mm). Fig.4 presents the set of estimated PP 10 (after removing failed detections using the epipolar constraint) together with the wooden plate fiducial points 12 and the incisions 1 1 .

In addition, the exact location of the PI was performed for the second example, comprising the second experimental setup within a deformable body wall. The focus here was to derive the precision of the entire method including errors induced by the flexible wall and the user manipulation of the instrument. Six subjects were instructed to take an instrument of their choice, insert it into the 10 mm trocar and move it around for a few seconds while images were being recorded. The only instructions were to make large movements in order to describe a cone similar to the one of Fig. 1 and to grasp the instrument gently. No particular instruction about the speed of the gesture was given. The six subjects did the experiment 15 times each, making a total of 90 experiments. For each run the 3D position of the OPP was estimated. Among those experiments, three experiments failed. The failures were successfully detected using the epipolar constraint. For the 87 remaining experiments, since no ground-truth information was here available, the size of the obtained cluster of points was studied. This parameter corresponds to the precision - rather than the accuracy - of the OPP detection method. The centroid of the cluster of points was computed as the average of the X, Y and Z positions of each valid point in the cluster. The centroid was located 19 mm under the wall outer surface, which is coherent with the fact that the wall of the mockup was 40 mm thick and consisted out of homogeneous material. The distance D, to the centroid was computed for each point /^' of the cluster of points. The average distance error to the centroid was found to be 3.54 mm (standard deviation 2.81 mm, 90th percentile 8.00 mm). The discrepancy of the data around the centroid is significantly larger for this experiment than for the wooden frame experiment (unequal variances t-test, p<0.01 ). This difference most certainly comes from the flexible nature of the simulated body wall and the manipulation of the user. In fact, the user is required to apply a null torque in order to be able to estimate the true OPP position. This is never the case in practice, hence a possible error in the determination of the OPP. Nevertheless, one can see that the discrepancy in the results remains low with a 90th percentile of the error at 8.00 mm.

Methods according to the invention further may comprise a calibration step which incorporates the case where the pivot point is dynamic, e.g. due to the fact that the patient is breathing and/or the incision point is not static. In these embodiments the tolerances ε and Z2 will need to be tuned accordingly.

Embodiments of the present invention provide a statistical method which can estimate the position of the optimal pivot point in minimally invasive surgery in order to align the RCM of a surgical robot or program a redundant surgical robot. The proposed methods advantageously take only 5 to 20 seconds of time for the surgeon, most of the computations being done online. Methods according to aspects of the present invention furthermore can be carried out with only a single external stereo camera placed statically at a convenient location approximately 30 cm, preferably at least 30 cm, preferably up to 1 m or more, preferably at least 30 cm to 100 cm from the instrument, in particular the entry port or pivot point. Contrarily to other methods known in the art that determine the pivot point, the present invention uses images from outside the body (i.e. ex-vivo images) and advantageously does neither rely on special markers that must be placed on instruments nor on complex, expensive and error-prone instrument tracking techniques. However, in further embodiments, integration of the present invention with such algorithms could be done, for instance during the Hough step of the method.

Advantageously, the cameras' zoom and focus are fixed when performing methods of the invention. The intrinsic and extrinsic camera parameters are advantageously pre-calibrated in order to avoid problems of radial distortion in images, a phenomenon that frequently complicates the use of laparoscopic images. On the long run, methods according to aspects of the invention can be used for surgical robots with an RCM using a visual servoing scheme that helps automatically determine and align the RCM with the optimal pivot point. It will be convenient to note that methods according to aspects of the invention can also be used for passive devices, such as manipulators, that implement RCMs to guide the surgeon in its positioning. Methods according to aspects of the invention will allow for quicker and safer positioning of the robots or passive instrument holders at an optimal pivot point position, thus reducing interaction forces between the patient's body wall and the instrument. It is to be understood that this invention is not limited to the particular features of the means and/or the process steps of the methods described as such means and methods may vary. It is also to be understood that the terminology used herein is for purposes of describing particular embodiments only, and is not intended to be limiting. It is also to be understood that plural forms include singular and/or plural referents unless the context clearly dictates otherwise. It is moreover to be understood that, in case parameter ranges are given which are delimited by numeric values, the ranges are deemed to include these limitation values.

The above description is for the purpose of teaching the person of ordinary skill in the art how to practice the present invention, and it is not intended to detail all those obvious modifications and variations of it which will become apparent to the skilled worker upon reading the description. It is intended, however, that all such obvious modifications and variations be included within the scope of the present invention, which is defined by the following claims. The claims are intended to cover the claimed components and steps in any sequence which is effective to meet the objectives there intended, unless the context specifically indicates the contrary.

Claims

1 . A method for determining a pivot point (PP), the method comprising:

- moving an instrument (15) about a pivot point (PP);

- providing at least two ex-vivo images of the instrument's movement in time; - extracting straight lines from said ex-vivo images;

- filtering said straight lines, wherein said filtering comprises:

• obtaining at least one pair of substantially parallel straight lines from a first one of said ex-vivo images, resulting in a first set (20, 21 ) of substantially parallel straight lines; · obtaining at least one pair of substantially parallel straight lines from a second one of said ex-vivo images, resulting in a second set (20', 22) of substantially parallel straight lines;

^• comparing said first (20, 21 ) and second sets (20', 22) of substantially parallel straight lines, comprising classifying substantially parallel straight lines of said first and second sets in pairs of substantially parallel instrument lines (20, 20') and background lines (21 , 22), wherein substantially parallel straight lines forming each of said pairs of substantially parallel instrument lines are separated by a same distance (d);

- determining the pivot point (Pl^b) based on at least two of said pairs of substantially parallel instrument lines (20, 20').

2. The method according to claim 1 , further comprising the step of obtaining at least one pair (20", 20"') of substantially parallel straight lines from additional ex-vivo images, resulting in additional sets of substantially parallel straight lines, and further comparing said additional sets of substantially parallel straight lines, comprising classifying substantially parallel straight lines of the additional sets in pairs of the substantially parallel instrument lines and the background lines, wherein pairs of the substantially parallel straight lines having parallel lines in at least three subsequent images are considered background lines.

3. The method according to claim 1 or 2, wherein filtering the extracted straight lines is performed statistically.

4. The method according to any of previous claims, wherein extracting straight lines is performed using a Hough transform.

5. The method according to any of previous claims, wherein said ex-vivo images are provided by at least one camera placed statically at least 30 cm from the pivot point.

6. The method according to any one of the preceding claims, wherein at least two of the ex-vivo images are provided from each of at least two different viewing orientations relative to the instrument, wherein a two dimensional location of the pivot point is determined from the ex-vivo images of each viewing orientation independently, and wherein a three dimensional location of the pivot point is determined by triangulation of the two dimensional locations.

7. The method according to any of previous claims, wherein said distance (d) is determined by a distance between edges of said instrument (15).

8. The method according to any of previous claims, wherein determining the pivot point is performed by a RANSAC fitting.

9. The method according to any of previous claims, wherein moving the instrument about the pivot point is performed by making the instrument (15) follow a conical trajectory (C).

10. The method according to any of previous claims, further comprising monitoring a quality of movement of the instrument around a pivot point, comprising measuring and/or evaluating a parameter related to an interaction between the instrument and a material body supporting the instrument during movement for a period of time.

1 1 . The method according to claim 10, wherein the parameter is representative of a force of the interaction between the instrument and a material body supporting the instrument.

12. The method of claim 10 or 1 1 , comprising monitoring the parameter in time, and providing a signal when the parameter exceeds a predetermined threshold.

13. The method of claim 12, wherein the signal is provided visually and/or by using sounds and/or by using a display and/or by using a laser based system and/or by providing kinesthetic, tactile or vibro-tactile feedback.

14. The method according to any of previous claims, further comprising moving an apparatus comprising a remote centre of motion to position the remote centre of motion at the pivot point.

15. A computer program product for, if implemented on a control unit, performing a method according to any of claims 1 to 14.

16. A data carrier storing a computer program product according to claim 15.

17. Assembly for determining a pivot point (PP), comprising : - an instrument (15) adapted to be moved about the pivot point;

- at least one image providing means (17), positioned to take images of the instrument at a distance of at least 30 cm from the pivot point;

- a processing unit (18) configured to be connected to the image providing means and configured for processing the images of the instrument according to the method of any one of claims 1 to 14 to determine the pivot point.

18. Assembly according to claim 17, comprising monitoring means adapted to determine a quality of movement of the instrument around a pivot point.

19. Assembly according to claim 18, wherein the monitoring means comprise a force sensor adapted to sense a force of interaction of the instrument with an environment.

20. Assembly according to any one of claims 17 to 19, comprising means for providing feedback to a user coupled to the processing unit, the means for providing feedback being operable for guiding a user to the determined pivot point.

21 . Assembly according to any one of claims 17 to 20 for minimally invasive surgery, comprising an apparatus (51 ) operable to move a member (55) about a remote centre of motion (RCM) and a laser (53), wherein the processing unit (18) is configured to:

- determine a position of the remote centre of motion (RCM);

- determine a position (P) of an incision in an outer skin surface (50) of a patient; and

- compute a three dimensional vector (u) from the pivot point (PP) to the position of the incision (P); wherein the laser (53) is operable to point towards a point (54) defined by the position of the remote centre of motion (RCM) plus the three dimensional vector (u), or a point defined by adding the three dimensional vector (u) and a predetermined vector ( ) to the position of the remote centre of motion (RCM).

22. Assembly according to any one of the claims 17 to 20, comprising a surgical robot, wherein the instrument is adapted for attachment to the robot, the robot being operable to move the instrument about the pivot point for determining the pivot point.

23. Assembly of claim 22, comprising monitoring means adapted to determine a quality of movement of the instrument around a pivot point being coupled to the surgical robot for providing motion feedback to the surgical robot.