In operating rooms (OR), physicians have to work in strict compliance to asepsis rules to not endanger the health of patients. In the case of laparoscopic mini-invasive surgery, where the field of view of the surgeon is restricted, the use of computers is necessary to provide the missing information. Physicians must therefore interact with computers either by directly manipulating mouse and keyboard, by removing their gloves or by using a protection for the devices, or by voicecommanding an assistant to do so. However, in addition to be time-consuming, it may cause hygiene issues in the first case and a lack of precision in the second. The need to have better way of interactions with computers had led to important researches in that area during the last ten years, especially in Touchless Human Machine Interaction (THMI). Indeed, THMI, including gesture recognition, voice recognition and eye-tracking, has a promising future in the medical field, allowing surgeons to interact by themselves with devices, thereby avoiding error-prone process while complying to asepsis rules. In this context, the “Intelligent Touchless Glassless Human-Machine Interface” (ITG-HMI) project aims to provide a new tool for viewing and manipulating 3D objects. In this article, we present how this interface was implemented, through the detection and recognition of hand gestures using Deep Learning, the establishment of a graphical interface to display 3D models and the adaptation of gestures recognized in actions to achieve.
Medical field has always benefited from the latest technological headways such as radiography, robotics or more recently augmented reality. Indeed, the progress in image analysis and augmented reality have led to major therapeutic progress in the surgical field as well as in the diagnosis field. Thus, one of the most important technique of medical image analysis is the registration. Image registration is the process of matching two or more images. More concretely, it consists in finding the transformation that minimizes the difference between two or more images. The transformation can be rigid (composed of rotations and translations only), affine (composed of rotations, translations and scales), or non-rigid. Even though rigid registration can seem quite easy to perform, developing and implementing solutions that realize fast, precise and robust rigid registration on complex objects is still challenging, especially when we deal with 3D objects. One of the most known and used rigid-registration algorithm is the Iterative Closest Point algorithm that has been implemented notably by the Open3D library. However, this method was unable to handle non-rigid registration. That is the reason why we have decided to use the Coherent Point Drift algorithm with non-rigid deformations. To this end, we have used the PyCPD library. In this paper, we present an efficient method for non-rigid registration applied to deformed liver models, robust to translations, rotations and cropping even though it fails to handle the most complex cases.
KEYWORDS: Microscopy, Super resolution, Image resolution, Dielectrics, Optical microscopes, Photonic nanostructures, Real time imaging, Spatial resolution, Image acquisition, Near field optics
The demands of biological imaging microscopy continue to progress, requiring high spatial resolution, real time acquisition, easy-to-use devices and with preference being label-free to avoid cell damage. While recent advances have allowed significant progress in resolution, several limitations remain such as the use of labelling, the complexity of the imaging technique or low image rate acquisition. Recently, a new optical technique, known as microsphere-assisted microscopy, seems to provide several advantages. The principle is based on the collection of a virtual image of the sample through a dielectric microsphere by a classical optical microscope. The dielectric microsphere with a diameter of tens of micrometres, can be deposited on the sample in air or in immersion. A resolution of up to 7 times smaller than the wavelength has been experimentally demonstrated in full-field imaging, i.e. without the need for point scanning nor labelling, with a classical optical microscope. Our team has also demonstrated that beyond a classical image the phase can also be measured using this technique with an interferometric configuration, so providing depth information. The presentation will be focused on the physical understanding of the phenomenon through simulations. The role of the photonic jet, light coherence and near-field effects have been numerically investigated. We will show how the photonic jet can be used to explain the imaging process but does not explain the super-resolution phenomenon. The role of coherence in the resolution limit criterion will also be illustrated, as well as a discussion on the contribution of the evanescent wave.
The loss of the information, due to the diffraction and the evanescent waves, limits the resolving power of classical optical microscopy. In air, the lateral resolution of an optical microscope can approximated at half of the wavelength using a low-coherence illumination. Recently, several methods have been developed in order to overcome this limitation and, in 2011, a new far-field and full-field imaging technique was proposed where a sub-diffraction-limit resolution has been achieved using a transparent microsphere. In this article, the phenomenon of super-resolution using microsphere-assisted microscopy is analysed through rigorous electro-magnetic simulations. The performances of the imaging technique are estimated as function of optical and geometrical parameters. Furthermore, the role of coherence is introduced through the temporal coherence of the light source and the phase response of the object.
Microscopic surface topography measurement is an important aspect of industrial inspection. Optical and near field
scanning techniques are increasingly replacing the use of the traditional mechanical stylus since they provide better lateral
resolutions and higher measurement speeds. The main far field optical techniques used are interference microscopy and
confocal microscopy, with the advantages of having larger fields of view and higher measurement speeds. Interference
microscopy is now widely used, mainly because of its nanometric axial measurement sensitivity and its ease of use but
suffers from a limitation in lateral resolution of about /2 due to diffraction. A new technique for high resolution 2D
imaging using a microsphere placed on the sample has been recently combined with interferometry by several groups to
greatly improve the lateral resolution. In this paper we present some of our own first results using glass microspheres with
a white light Linnik interferometer and demonstrate a lateral resolution of /4 and an axial measurement sensitivity of
several nm. Results are shown on calibrated square profile gratings with periods down to 400 nm, with a minimum feature
size of 200 nm and a height of 148 nm and a field of view of several μm. While these features are not visible directly with
the microscope objective, they become observable and measurable through the microsphere. An analysis using rigorous
electromagnetic simulations is also given to help better understand the imaging properties of the technique. These first
experimental and simulation results clearly indicate that this is an important new technique that opens new possibilities for
surface metrology with a lateral resolution well beyond the diffraction limit.
The development of new nanomaterials, devices and systems is very much dependent on the availability of new techniques for nanometrology. There now exists many advanced optical imaging techniques capable of subwavelength resolution and detection, recently brought to the forefront through the 2014 Nobel Prize for chemistry for fluorescent STED and single molecule microscopy. Label-free nanoscopy techniques are particularly interesting for nanometrology since they have the advantages of being less intrusive and open to a wider number of structures that can be observed compared with fluorescent techniques. In view of the existence of many nanoscopy techniques, we present a practical classification scheme to help in their understanding. An important distinction is made between superresolution techniques that provide resolutions better than the classical λ/2 limit of diffraction and nanodetection techniques that are used to detect or characterize unresolved nanostructures or as nanoprobes to image sub-diffraction nanostructures. We then highlight some of the more important label-free techniques that can be used for nanometrology. Superresolution techniques displaying sub-100 nm resolution are demonstrated with tomographic diffractive microscopy (TDM) and submerged microsphere optical nanoscopy (SMON). Nanodetection techniques are separated into three categories depending on whether they use contrast, phase or deconvolution. The use of increased contrast is illustrated with ellipsometric contrast microscopy (SEEC) for measuring nanostructures. Very high sensitivity phase measurement using interference microscopy is then shown for characterizing nanometric surface roughness or internal structures. Finally, the use of through-focus scanning optical microscopy (TSOM) demonstrates the measurement and characterization of 60 nm linewidths in microelectronic devices.
Full-field optical coherence tomography (FF-OCT) based on white-light interference microscopy, is an emerging noninvasive imaging technique for characterizing biological tissue or optical scattering media with micrometer resolution. Tomographic images can be obtained by analyzing a sequence of interferograms acquired with a camera. This is achieved by scanning an interferometric microscope objectives along the optical axis and performing appropriate signal processing for fringe envelope extraction, leading to three-dimensional imaging over depth. However, noise contained in the images can hide some important details or induce errors in the size of these details. To firstly reduce temporal and spatial noise from the camera, it is possible to apply basic image post processing methods such as image averaging, dark frame subtraction or flat field division. It has been demonstrate that this can improve the quality of microscopy images by enhancing the signal to noise ratio. In addition, the dynamic range of images can be enhanced to improve the contrast by combining images acquired with different exposure times or light intensity. This can be made possible by applying a hybrid high dynamic range (HDR) technique, which is proposed in this paper. High resolution tomographic analysis is thus performed using a combination of the above-mentioned image processing techniques. As a result, the lateral resolution of the system can be improved so as to approach the diffraction limit of the microscope as well as to increase the power of detection, thus enabling new sub-diffraction sized structures contained in a transparent layer, initially hidden by the noise, to be detected.
Transparent layers such as polymers are complex and can contain defects which are not detectable with classical optical inspection techniques. With an interference microscope, tomographic analysis can be used to obtain initial structural information over the depth of the sample by scanning the fringes along the Z axis and performing appropriate signal processing to extract the fringe envelope. By observing the resulting XZ section, low contrast, sub-μm sized defects can be lost in the noise which is present in images acquired with a CCD camera. It is possible to reduce temporal and spatial noise from the camera by applying image processing methods such as image averaging, dark frame subtraction or flat field division. In this paper, we present some first results obtained by this means with a white light scanning interferometer on a Mylar polymer, used currently as an insulator in electronics and micro-electronics. We show that sub-μm sized structures contained in the layer, initially lost in noise and barely observable, can be detected by applying a combination of image processing methods to each of the scanned XY images along the Z-axis. In addition, errors from optical imperfections such as dust particles on the lenses or components of the system can be compensated for with this method. We thus demonstrate that XZ section images of a transparent sample can be denoised by improving each of the XY acquisition images. A quantitative study of the noise reduction is presented in order to validate the performance of this technique.
KEYWORDS: 3D image processing, Fringe analysis, 3D metrology, Phase retrieval, Phase shifts, Digital Light Processing, Reconstruction algorithms, Image processing, Digital filtering, Projection systems
The processing of structured light images in real time is a challenging task for the development of three-dimensional (3-D) shape measurement methods. This paper presents a high speed and low-cost optical profiler implemented using a projection method based on the use of a digital light-processing device to illuminate the object to be measured. The image processing of the reflected structured light pattern allows potential real-time capabilities. The proposed method of absolute phase retrieval for unwrapping the relative phase uses a single additional staircase intensity pattern to determine and correct 2π discontinuities in the phase. Good results are obtained when the method is compared to another which uses three additional fringe patterns to determine the stair phase and then the absolute phase. Since the proposed technique uses only one extra pattern instead of three, it is less costly in terms of computation complexity and is thus faster. The hardware of the developed fringe projection system for a 3-D macroscopic reconstruction is presented and the performance of the method is evaluated. Simulated and experimental results are presented and compared to the other absolute phase-retrieval method. The proposed method is suitable for measuring 3-D object surfaces for a possible implementation in real time.
Based on a miniature digital light projector (pico-DLP), a prototype of a Structured Illumination Microscope (SIM) has been developed. The pico-DLP is used to project fringes onto a sample and applying the three-step phase shifting algorithm together with the absolute phase retrieval method, the 3D shape of the object surface is extracted. By using a specific optical system instead of a conventional microscope objective, the device allows 3D reconstructions of surfaces with both a 10× magnification and a high depth of field obtained thanks to a small numerical aperture of 0.06 offering an acceptable lateral resolution of 6.2 μm. An image processing algorithm has been developed to reduce the noise in the acquired images before applying the reconstruction algorithm and so optimize the reconstruction method. Compared with interference microscopy and confocal microscopy that have a shallower depth of field per XY image, the microscope developed achieves a depth of field about 700 μm and requires no vertical scanning, which greatly reduces the acquisition time. Although the system at this stage does not have the same resolution performance as interference microscopy, it is nonetheless faster and cheaper. One possible application of this SIM technique would be to first reconstruct in real-time parts of an object before performing higher resolution 3D measurements with interference microscopy. As with all classical optical instruments, the lateral resolution is limited by diffraction. Work is being carried out with the prototype SIM system to be able to exceed the lateral resolution limits and thus achieve super resolution.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.