WO2019209347A1

WO2019209347A1 - Nonrotating nonuniform electric field object rotation

Info

Publication number: WO2019209347A1
Application number: PCT/US2018/030037
Authority: WO
Inventors: Viktor Shkolnikov; Daixi XIN; Yang Lei
Original assignee: Hewlett Packard Development Co LP
Current assignee: Hewlett Packard Development Co LP
Priority date: 2018-04-27
Filing date: 2018-04-27
Publication date: 2019-10-31
Anticipated expiration: 2020-10-27
Also published as: EP3756165A4; EP3756165A1; US20210033842A1; US12204085B2; CN112005278A

Abstract

A three-dimensional object modeling method may include applying a nonrotating nonuniform electric field to apply a dielectrophoretic torque to a three-dimensional object to rotate the three-dimensional object, capturing images of the object at different angles during rotation of the object and forming a three-dimensional model of the object based on the captured images.

Description

NONROTATING NONUNIFORM ELECTRIC FIELD OBJECT

ROTATION

BACKGROUND

[0001] Objects are sometimes analyzed or simulated through the use of three-dimensional image reconstructions or three-dimensional modeling of the objects such three-dimensional models are sometimes made using multiple images captured while the object is rotating.

BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 is a schematic diagram illustrating portions of an example object rotation system.

[0003] Figure 2 is a diagram illustrating the example rotation system of

Figure 1 applying a nonrotating nonuniform electric field to rotate an example object.

[0004] Figure 3 is a diagram illustrating an example rotation system applying a nonrotating nonuniform electric field to rotate an example object.

[0005] Figure 4 is a schematic diagram illustrating portions of an example three-dimensional object modeling system.

[0006] Figure 5 is a flow diagram of an example three-dimensional object modeling method.

[0007] Figure 6 a flow diagram of an example three-dimensional volume modeling method.

[0008] Figure 7 is a diagram schematically illustrating the capture of two-dimensional image frames of a rotating object at different angles. [0009] Figure 8 is a diagram depicting an example image frame including the identification of features of an object at a first angular position.

[00010] Figure 9 is a diagram depicting an example image frame including the identifications of the features of the object at a second different angular position.

[00011] Figure 10 is a diagram illustrating triangulation of the different identified features for the merging and alignment of features from the frames.

[00012] Figure 11 is a diagram illustrating an example three-dimensional volumetric parametric model produced from the example image frames of Figures 7 and 8.

[00013] Throughout the drawings, identical reference numbers designate similar, but not necessarily identical, elements. The figures are not

necessarily to scale, and the size of some parts may be exaggerated to more clearly illustrate the example shown. Moreover, the drawings provide examples and/or implementations consistent with the description; however, the description is not limited to the examples and/or implementations provided in the drawings.

DETAILED DESCRIPTION OF EXAMPLES

[00014] Disclosed herein are example systems and methods for rotating very small objects to facilitate three-dimensional imaging or modeling of such objects. Disclosed herein are example systems and methods that rotate objects for three-dimensional imaging or modeling with less complexity and cost. Disclosed herein are example systems and methods that apply a non- rotating nonuniform electric field so as to apply a die electrophoretic torque to a three-dimensional object so as to rotate the three-dimensional object. [00015] The example systems and methods facilitate rotation of the very small objects by suspending the very small objects in a fluid, such as a liquid. As a result, the system the methods are well suited for the imaging of objects where direct physical contact and direct manipulation the objects is difficult. The example systems and methods for rotating objects may employ as few as to spaced electrodes to form the nonrotating nonuniform electric field that creates a dielectrophoretic torque and that rotate the three-dimensional object.

[00016] The example systems and methods for rotating objects facilitate rotation of the object about a rotational axis that is parallel to planes of the electrodes. In some implementations, this facilitates rotation of the objects about a rotational axis that is also parallel to a plane of a microfluidic chip, slide or a platform/stage containing the fluid that suspends the object during their rotation. The rotation of the object by the nonrotating nonuniform electric field facilitates rotation the object about the rotational axis that is

perpendicular to an optical axis of the camera or imager capturing images of the object during his rotation. Because the rotation axis perpendicular to the optical axis, the overall imaging system may be more compact and less complex.

[00017] In some implementations, the three-dimensional object being rotated, imaged and modeled may comprise biological elements, such as cells. In some implementations, three-dimensional object being rotated may comprise a cellular object. For purposes of this disclosure, a cellular object comprises a 3D culture or an organoid. 3D cultures are cells grown in droplets or hydrogels that mimic a physiologically relevant environment. Organoids are miniature organs grown in a lab derived from stem cells and clusters of tissue, wherein the specific cells mimic the function of the organ they model. 3-D cultures and organaids may be used to study basic biological processes within specific organs or to understand the effects of particular drugs. 3-D cultures and organoids may provide crucial insight into mechanisms of cells and organs in a more native environment.

[00018] Disclosed herein is an example three-dimensional object modeling method. The method may include applying a nonrotating nonuniform electric field to apply a dielectrophoretic torque to a three-dimensional object to rotate the three-dimensional object. Images are captured of the object at different angles during rotation of the object. A three-dimensional model of the object is formed based on the captured images.

[00019] Disclosed is an example three-dimensional object modeling system. The system may include a first electrode, a second electrode, a power supply connected to the first electrode and the second electrode, a camera and a controller. The controller may output control signals controlling the power supply such that the first electrode and the second electrode cooperate to apply a nonrotating nonuniform electric field to an object suspended in the fluid so as to rotate the object. The controller may further output control signals controlling the camera to capture images of the object at different angles during rotation of the object, wherein the controller is to form a three-dimensional model of the object based on the captured images.

[00020] Disclosed herein is an example cellular object rotation system for use with a cellular object imaging system. The cellular object rotation system may include a first electrode, a second electrode, a power supply connected to the first electrode and the second electrode and a controller to output control signals controlling the power supply such that the first electrode and the second electrode cooperate to apply a nonrotating nonuniform electric field to a cellular object suspended in the fluid so as to rotate the object.

[00021] Figure 1 schematically illustrates portions of an example object rotation system 20 for use with a three-dimensional imaging system. In one implementation, the object rotation system 20 comprises a cellular object rotation system for use with a cellular object imaging system. Object rotation system 20 facilitate low cost and less complex rotation of objects, such as cellular objects, as such object are being imaged for three-dimensional modeling. Object rotation system 20 comprises electrodes 60, power supply 61 and controller 62.

[00022] Electrodes 60 comprise a pair of spaced electrodes that cooperate to form a nonrotating nonuniform electric field through a cellular object suspension region 50. The cellular object suspension region 50 comprises a volume of fluid 54 in which a three-dimensional object, such as a cellular object 52 is suspended. In the example illustrated, electrodes 60 comprise a pair of electrodes located on one side of the cellular object 52. In implementations where imaging is performed through the plane or planes containing electrodes 60, such electrodes 60 may be formed from a transparent electrically conductive material such as indium tin oxide. In other implementations, electrodes may be formed from other electrically conductive materials. In one implementation electrodes 62 each comprise a flat planar electrode, wherein the electrodes 60 are coplanar. As a result, object rotation system 20 may be more compact.

[00023] Power supply 61 comprise a source of power for allegedly charging at least one of electrodes 60. In one implementation, power supply 61 supplies power to electrodes 60 under the control of controller 62.

[00024] Controller 62 comprises a processing unit that follows instructions contained in a non-transitory computer-readable medium. In one implementation, controller 62 may comprise an application-specific integrated circuit. In one implementation, controller 62 serves as a signal generator controlling the frequency and voltage of the nonrotating nonuniform electric field. Figure 2 is a schematic diagram illustrating the application of the nonrotating nonuniform electric field to the example cellular object 52. As indicated by arrow 63, the electric field applies a dielectrophoretic torque to object 52 so as to rotate object 52 about a rotational axis 65. Such rotation facilitates the capturing of images of the cellular object 52 at different angles to facilitate three-dimensional reconstruction or modeling of cellular object 52 for analysis. In one implementation, controller 62 outputs control signals such that electrodes 60 apply a sinusoidal nonrotating nonuniform alternating current electric field having a frequency of at least 30 kHz and no greater than 500 kHz. In one implementation, the nonrotating nonuniform electric field has a voltage of at least 0.1 V rms and no greater than 100 V rms. Between taking consecutive images, the cellular object must have rotated a distance that at least equals to the diffraction limit dlim of the imaging optics. The relationship between minimum rotating angle 0min, radius r and diffraction limit distance dlim is 0min= dlim/r. For example, for imaging with light of A=500nm and a lens of 0.5 NA, the diffraction limit dlim = l/(2NA) = 500nm. In the meanwhile, the cellular object cannot rotate too much that there is no overlap between consecutive image frames. So the maximum rotating angle between

consecutive images 0max = 180 - 0min.

[00025] In one implementation, the nonuniform nonrotating electric field produces a dielectrophoretic torque on the cellular object so as to rotate the cellular object at a speed such that an image or may capture images every 2.4 degrees while producing output in a reasonably timely manner. In one implementation where the capture speed of the imager is 30 frames per second, the produced dielectrophoretic torque rotates the cellular object at a rotational speed of at least 12 rpm and no greater than 180 rpm. In one implementation, the produced dielectrophoretic torque rotates the cellular object at least one pixel shift between adjacent frames, but where the picture shift is not so great so as to not be captured by the imager 280. In other implementations, cellular object 52 may be rotated at other rotational speeds.

[00026] Figure 3 illustrates portions of another example three- dimensional object rotation system 120. As with object rotation system 20, object rotation system 120 is well-suited for rotating very small objects while such small objects are suspended in a fluid. As with object rotation system 20, object rotation system 120 well-suited for rotating cellular objects as such objects are being imaged to form three-dimensional reconstructions or models of the cellular objects. System 120 is similar to system 20 described above except that system 120 comprises an alternative arrangement having electrodes 160 in lieu of electrodes 60. Those remaining components of system 120 which correspond to components of system 20 are numbered similarly in Figure 3 or are shown in Figure 1.

[00027] Electrodes 160 comprise a pair of electrodes that cooperate to form a nonrotating nonuniform electric field through the cellular object suspension region 50. In the example illustrated, electrodes 160 comprise a pair of electrodes located on opposite sides of the cellular object 52 with the electrodes 160 facing one another. In implementations where imaging of the rotating cellular object is with an image or having an optical axis that passed through her intersects either of the two electrodes, such electrodes may be formed from a transparent or translucent electrically conductive material such as indium tin oxide. In other implementations, electrodes 160 may form from other electrically conductive materials.

[00028] In operation, object rotation system 120 may perform in a similar fashion as compared object rotation system 20. Controller 62 (shown in Figure 1 ) outputs control signals or generate signals such that electrodes 160 apply a nonrotating nonuniform electric field about the suspended object, shown as cellular object 52. In one implementation, controller! 62 outputs control signals such that electrodes 160 apply a sinusoidal nonrotating nonuniform alternating current electric field having a frequency of at least 30 kFIz and no greater than 500 kFIz. In one implementation, the nonrotating nonuniform electric field has a voltage of at least 0.1 V rms and no greater than 100 V rms.

[00029] The electric field is applied such that it applies a dialect referred to torque to the object 52 so as to rotate the object 52 as indicated by arrow 163 about an axis 165. The rotational axes 165 of object 52 is parallel to the slide, stage or platform containing the fluid and may be perpendicular to the optical axis of the image or camera capturing images of object 52 at different angular positions. Because the rotation axis perpendicular to the optical axis, the overall imaging system may be more compact and less complex.

[00030] Figure 4 schematically illustrates portions of an example three- dimensional object modeling system 210. Modeling system 210 facilitates the rotation of an object, such as a cellular object, as the object is being imaged to facilitate the output of a three-dimensional reconstruction or model of the object. Modeling system 210 facilitates such rotation of the object with an architecture that is less complex and more compact, potentially reducing cost. Modeling system 210 comprises object rotation system 220, controller 270 and imager 280, in the form of a camera.

[00031] Object rotation system 220 is similar to object rotation system 20 described above except that controller 270 controls power supply 61 and imager 280. Controller 270 comprises processing unit 272 and a non- transitory computer readable medium in the form of memory 274. Processing unit 272 follows instructions contained in memory 274. Memory 274 contains instructions that direct processing unit 272 to control the operation of electrodes 60, 160 and imager 280. For example, controller 270 outputs control signals controlling the rate at which so the object 52 is rotated during imaging. As with controller 62, controller 270 serves as a signal generator controlling the frequency and voltage of the nonrotating nonuniform electric field such as shown in Figures 2 or 3. The electric field applies a

dielectrophoretic torque to object 52 so as to rotate object 52 about a rotational axis 65. Such rotation facilitates the capturing of images of the cellular object 52 at different angles to facilitate three-dimensional

reconstruction or modeling of cellular object 52 for analysis. In one

implementation, controller 270 outputs control signals such that electrodes 60, 160 apply a sinusoidal nonrotating nonuniform alternating current electric field having a frequency of at least 30 kHz and no greater than 500 kHz. In one implementation, the nonrotating nonuniform electric field has a voltage of at least 0.1 V rms and no greater than 100 V rms. Between taking consecutive images, the cellular object must have rotated a distance that at least equals to the diffraction limit dn_m of the imaging optics. The relationship between minimum rotating angle 0min, radius r and diffraction limit distance dn_m is 0min= diim/r. For example, for imaging with light of A=500nm and a lens of 0.5 NA, the diffraction limit dnm = l/(2NA) = 500nm. In the meanwhile, the cellular object cannot rotate too much that there is no overlap between consecutive image frames. So the maximum rotating angle between consecutive images

0max ⁼ 180 - 0min.

[00032] In one implementation, the nonuniform nonrotating electric field produces a dielectrophoretic torque on the cellular object so as to rotate the cellular object at a speed such that an image or may capture images every 2.4 degrees while producing output in a reasonably timely manner. In one implementation where the capture speed of the imager is 30 frames per second, the produced dielectrophoretic torque rotates the cellular object at a rotational speed of at least 12 rpm and no greater than 180 rpm. In one implementation, the produced dielectrophoretic torque rotates the cellular object at least one pixel shift between adjacent frames, but where the picture shift is not so great so as to not be captured by the imager 280. In other implementations, cellular object 52 may be rotated at other rotational speeds.

[00033] Imager 280 comprise at least one camera to capture images of the rotating cellular object 52 at different angles during rotation of cellular object 52 about axis 65. Imager 280 has an optical axis 282 which is perpendicular to axis 65. In one implementation, imager 280 may comprise multiple cameras. Imager 280 captures images of the rotating cellular object 52 at different angular positions during his rotation to facilitate subsequent three-dimensional image reconstruction of the cellular object 52 as will be described hereafter. In one implementation come imager 280 may comprise a camera having an optical lens 282 facility microscopic viewing and imaging of cellular object 52.

[00034] In addition to outputting control signals to electrodes 60, 160 so as to create the nonrotating nonuniform electric field that rotates cellular object 52, controller 270 may additionally control imager 280. Controller 270 receives images or image signals from imager 280. Based upon the different images of the rotating cellular object 52 captured at different rotational angles, controller 270 triangulates identified points of the image to form a three- dimensional reconstruction or model 290 of cellular object 52 for analysis.

[00035] Figure 5 is a flow diagram of an example three-dimensional object modeling method 300. Modeling method 300 captures images of an object rotated with a nonrotating nonuniform electric field. Modeling method 300 may be implemented with less complex and lower-cost components. Although method 300 is described in the contest of being carried out by three- dimensional object modeling system 210 having object rotation system 220, it should be appreciated that method 300 may likewise be carried out with other similar modeling systems and other similar object rotation systems, such as object rotation systems 20 or 120.

[00036] As indicated by block 304, a nonrotating nonuniform electric field is applied so as to apply a dielectrophoretic torque to a three-dimensional object, such as a cellular object, to rotate the three-dimensional object. In one implementation, a sinusoidal nonrotating nonuniform alternating current electric field having a frequency of at least 30 kHz and no greater than 500 kHz is applied to the object or the object suspended in a fluid. In one implementation, the nonrotating nonuniform electric field has a voltage of at least 0.1 V rms and no greater than 100 V rms. [00037] As indicated by block 308, controller 270 outputs control signals causing camera 282 capture images of the object 52 at different angles during rotation of object 52. As indicated by block 312, upon receiving the captured images from imager 280, controller 270 formed a three-dimensional reconstruction or model of the object 52 based upon the captured images.

[00038] Figure 6 is a flow diagram of an example three-dimensional volumetric modeling method 500 that may be carried out by controller 470 using captured two-dimensional images of the rotating object 52. As indicated by block 504, a controller, such as controller 470, receives video frames or two-dimensional images captured by the imager/camera 60 during rotation of object 52. As indicated by block 508, various preprocessing actions are taken with respect to each of the received two-dimensional image video frames. Such preprocessing may include filtering, binarization, edge detection, circle fitting and the like.

[00039] As indicated by block 514, utilizing such edge detection, circle fitting and the like, controller 470 retrieves and consults a predefined three- dimensional volumetric template of the object 52, to identify various internal structures of the object are various internal points in the object. The three- dimensional volumetric template may identify the shape, size and general expected position of internal structures which may then be matched to those of the two-dimensional images taken at the different angles. For example, a single cell may have a three-dimensional volumetric template comprising a sphere having a centroid and a radius. The three-dimensional location of the centroid and radius are determined by analyzing multiple two-dimensional images taken at different angles.

[00040] Based upon a centroid and radius of the biological object or cell, controller 470 may model in three-dimensional space the size and internal depth/location of internal structures, such as the nucleus and organelles. For example, with respect to cells, controller 470 may utilize a predefined template of a cell to identify the cell wall and the nucleus As indicated by block 518, using a predefined template, controller 470 additionally identifies regions or points of interest, such as organs or organelles of the cell. As indicated by block 524, controller 470 matches the centroid of the cell membrane, nucleus and organelles amongst or between the consecutive frames so as to estimate the relative movement (R, T) between the consecutive frames per block 528.

[00041] As indicated by block 534, based upon the estimated relative movement between consecutive frames, controller 470 reconstructs the centroid coordinates in three-dimensional space. As indicated by block 538, the centroid three-dimensional coordinates reconstructed from every two frames are merged and aligned. A single copy of the same organelle is preserved. As indicated by block 542, controller 470 outputs a three- dimensional volumetric parametric model of object 52.

[00042] Figures 7-11 illustrate one example modeling process 600 that may be utilized by 3-D modeler 70 or controller 470 in the three-dimensional volumetric modeling of the biological object. Figure 7-11 illustrate an example three-dimensional volumetric modeling of an individual cell. As should be appreciated, the modeling process depicted in Figures 7-11 may likewise be carried out with other biological objects.

[00043] As shown by Figure 6, two-dimensional video/camera images or frames 604A, 604B and 604C (collectively referred to as frame 604) of the biological object 52 (schematically illustrated) are captured at different angles during rotation of object 52. In one implementation, the frame rate of the imager or camera is chosen such as the object is to rotate no more than 5° per frame by no less than 0.1 °. In one implementation, a single camera captures each of the three frames during rotation of object 52 (schematically illustrated with three instances of the same camera a different angular positions about object 52) in other implementations, multiple cameras may be utilized. [00044] As shown by Figures 7 and 8, after image preprocessing set forth in block 508 in Figure 5, edge detection, circle fitting another feature detection techniques are utilized to distinguish between distinct structures on the surface and within object 52, wherein the structures are further identified through the use of a predefined template for the object 52. For the example cell identify the cell, controller 470 identifies wall 608, its nucleus 610 and internal points of interest, such as cell organs or organelles 612 in each of the frames (two of which are shown by Figures 7 and 8).

[00045] As shown by Figure 9 and as described above with respect to blocks 524-538, controller 470 matches a centroid of a cell membrane, nucleus and organelles between consecutive frames, such as between frame 604A and 604B. Controller 470 further estimates a relative movement between the consecutive frames, reconstructs a centroid’s coordinates in three-dimensional space and then utilizes the reconstructed centroid coordinates to merge and align the centroid coordinates from all of the frames. The relationship for the relative movement parameters R and T is derived, assuming that the rotation axis is kept still, and the speed is constant all the time. Then, just the rotation speed is utilized to determine R and T

(0,0₂ ) as shown in Figure 9, where:

0,0₂ 00, Re-00,

based on the following assumptions:

Q is constant;

rotation axis doesn’t change (along y axis); and

00, is known. As shown by Figure 10, the above reconstruction by controller 470 results in the output of a parametric three-dimensional volumetric model of the object 52, shown as a cell.

[00046] Although the present disclosure has been described with reference to example implementations, workers skilled in the art will recognize that changes may be made in form and detail without departing from the spirit and scope of the claimed subject matter. For example, although different example implementations may have been described as including features providing one or more benefits, it is contemplated that the described features may be interchanged with one another or alternatively be combined with one another in the described example implementations or in other alternative implementations. Because the technology of the present disclosure is relatively complex, not all changes in the technology are foreseeable. The present disclosure described with reference to the example implementations and set forth in the following claims is manifestly intended to be as broad as possible. For example, unless specifically otherwise noted, the claims reciting a single particular element also encompass a plurality of such particular elements. The terms“first”,“second”,“third” and so on in the claims merely distinguish different elements and, unless otherwise stated, are not to be specifically associated with a particular order or particular numbering of elements in the disclosure.

Claims

WHAT IS CLAIMED IS:

1. A three-dimensional object modeling method comprising:

applying a nonrotating nonuniform electric field to apply a dielectrophoretic torque to a three-dimensional object to rotate the three-dimensional object;

capturing images of the object at different angles of the object during rotation of the object;

forming a three-dimensional model of the object based on the captured images.

2. The method of claim 1 , wherein the capturing of the images is with a camera having an optical axis and wherein the applying of the nonrotating nonuniform electric field is to apply the dielectrophoretic torque so as to rotate the object about a rotational axis perpendicular to the optical axis.

3. The method of claim 1 , wherein the object supported by a slide extending in a plane and wherein the applying of the nonrotating nonuniform electric field is to apply the die electrophoretic torque so as to rotate the object about a rotational axis parallel to the plane.

4. The method of claim 1 , wherein the nonrotating, nonuniform electric field is a sinusoidal electric field between parallel electrodes.

5. The method of claim 1 , wherein the applying of the nonrotating nonuniform electric field is with electrodes consisting of a pair of electrodes on one side of the object. i 6. The method of claim 1 , wherein the nonrotating nonuniform electric

2 field is an alternating current electric field having a frequency of at least 30 kHz

3 and no greater than 500 kHz.

1 7. The method of claim, wherein the nonrotating nonuniform electric

2 field has a voltage of at least 0.1 V rms and no greater than 100 v rms.

1 8. The method of claim 1 , wherein the objects comprise at least one

2 biological cell.

1 9. A three-dimensional object modeling system comprising:

2 a first electrode;

3 a second electrode;

a power supply connected to the first electrode and the second electrode;

a camera;

a controller to:

output control signals controlling the power supply such that the first electrode and the second electrode cooperate to apply a nonrotating nonuniform electric field to an object suspended in the fluid so as to rotate the object; and

output control signals controlling the camera to capture images of the object at different angles during rotation of the object, wherein the controller is to form a6 three-dimensional model of the object based on the captured7 images.

1 10. The system of claim 9, wherein the camera has an optical axis and

2 wherein the applying of the nonrotating nonuniform electric field is to apply the dielectrophoretic torque so as to rotate the object about a rotational axis perpendicular to the optical axis. 1 1. The system of claim 9 further comprising a stage extending in a plane for supporting a slide supporting the fluid suspended object, wherein the applying of the nonrotating nonuniform electric field is to apply the

dielectrophoretic torque so as to rotate the object about a rotational axis parallel to the stage. 12. The system of claim 9, wherein the nonrotating, nonuniform electric field is a sinusoidal electric field between the first electrode and the second electrode. 13. The system of claim 9, wherein the applying of the nonrotating nonuniform electric field is with electrodes consisting of a pair of electrodes on one side of the object. 14. A cellular object rotation system for use with a cellular object

imaging system, the rotation system comprising: a first electrode; a second electrode; a power supply connected to the first electrode and the second electrode; and

a controller to output control signals controlling the power supply such that the first electrode and the second electrode cooperate to apply a nonrotating nonuniform electric field to a cellular object suspended in the fluid so as to rotate the object. i 15. The cellular object rotation system of claim 14, wherein the

2 controller is to cause the first electrode and the second electrode to form an

3 alternating current sinusoidal electric field having a frequency of at least 30 kHz

4 and no greater than 500 having a voltage of at least 0.1 V rms and no greater

5 than 100 v rms.

1