阅读说明：本技术 用于使用结构化照射检测颗粒的方法 (Method for detecting particles using structured illumination ) 是由 柳济宽 李昊辰 吴雅琳 于 2019-01-30 设计创作，主要内容包括：一种颗粒检测方法使用来自多个结构化照射图案的经测量的信号对目标上的颗粒的存在和位置进行检测。该颗粒检测方法使用通过利用结构化照射图案对目标进行照射而获得的经测量的信号来对颗粒进行检测。具体地,计算原始图像中这些测量信号的变化程度,以确定在目标上的特定的感兴趣区域处是否存在颗粒。(A particle detection method uses measured signals from multiple structured illumination patterns to detect the presence and location of particles on a target. The particle detection method uses measured signals obtained by illuminating a target with a structured illumination pattern to detect particles. In particular, the degree of variation of these measurement signals in the raw image is calculated to determine whether particles are present at a particular region of interest on the target.)

1. A method for detecting particles on a target, the method comprising the steps of:

illuminating the target with a plurality of structured illumination patterns, each characterized by a spatial frequency and an illumination phase;

generating a plurality of raw images of the illuminated target by measuring optical signals from the target, each raw image comprising at least one raw intensity value, each raw image being obtained by measurement of the target illuminated with a respective structured illumination pattern; and

generating, for each of one or more regions of the target, a first estimate indicating whether particles are present at each of the one or more regions of the target, the generating a first estimate comprising:

for each of the one or more regions of the target, determining a modulation score by combining sets of raw intensity values from the plurality of raw images, the modulation score indicating a degree of variation of the sets of raw intensity values in each of the one or more regions of the target, and

generating the first estimate for the region by comparing the modulation for each of the one or more regions of the target to a first threshold.

2. The method of claim 1, wherein combining the set of raw intensity values comprises: calculating a variance of two or more raw intensity values in the set of raw intensity values; calculating a range of two or more raw intensity values in the set of raw intensity values; calculating a variance of the two or more raw intensity values in the set of raw intensity values normalized by an average of the two or more raw intensity values in the set of raw intensity values, or calculating a goodness-of-fit metric to a reference curve of two or more raw intensity values in the set of raw intensity values.

3. The method of claim 1, wherein the plurality of structured illumination patterns are selective excitation patterns, and the set of illumination features includes at least a spatial frequency and a phase of the selective excitation patterns.

4. The method of claim 1, wherein determining the modulation score comprises:

determining a set of sub-modulation scores, each sub-modulation score determined by combining subsets of the raw intensity values from respective subsets of the plurality of raw images; and

combining the sub-modulation scores to determine the modulation score.

5. The method of claim 4, wherein the subset of the plurality of original images are characterized by the same spatial frequency.

6. The method of claim 1, wherein the particle is a biomolecule comprising at least one of a DNA fragment, an mRNA fragment, or incrna.

7. The method of claim 1, further comprising:

generating a reconstructed image by processing at least one of the plurality of original images of the object, the reconstructed image comprising a set of reconstructed intensity values obtained by processing the at least one of the plurality of original images;

for each of the one or more regions of the object, determining a second estimate by comparing one or more reconstructed intensity values to a second threshold, the second estimate indicating whether particles are present at the region of the object; and

generating a combined set of estimates for at least one region of the target by comparing the first estimate to the second estimate for the at least one region of the target.

8. The method of claim 7, wherein generating the combined set of estimates comprises:

identifying a subset of pixel locations in the reconstructed image, wherein the second estimate for the identified subset of pixel locations indicates the presence of particles at a respective region of the object; and

for each identified pixel location, generating a combined estimate for the pixel location, wherein the combined estimate for the pixel location indicates the presence of the particle at the identified pixel location if the first estimate for the respective region of the object is a first value; and wherein the combined estimate for the pixel locations indicates that the particle is not present at the identified pixel locations if the first estimate for the respective region of the target is a second value.

9. The method of claim 7, wherein the reconstructed image has a resolution higher than a resolution of the plurality of original images.

10. The method of claim 7, wherein generating the reconstructed image comprises performing a Synthetic Aperture Optical (SAO) reconstruction process on the at least one of the plurality of raw images.

11. A system for detecting particles on a target, the system comprising:

a plurality of illumination modules configured to illuminate the target with a plurality of structured illumination patterns, each characterized by a spatial frequency and an illumination phase;

an optical imaging module configured to generate a plurality of raw images of the illuminated target by measuring optical signals from the target, each raw image comprising at least one raw intensity value, each raw image obtained by measurement of the target illuminated with a respective structured illumination pattern;

a detection module configured to generate, for each of one or more regions of the target, a first estimate indicative of whether particles are present at each of the one or more regions of the target, wherein the detection module is further configured to: determining, for each of the one or more regions of the target, a modulation score by combining sets of raw intensity values from the plurality of raw images, the modulation score indicating a degree of variation of the sets of raw intensity values in each of the one or more regions of the target; and generating the first estimate for each of the one or more regions of the target by comparing the modulation to a first threshold.

12. The system of claim 11, wherein the detection module is further configured to combine the set of raw intensity values by: calculating a variance of two or more raw intensity values in the set of raw intensity values; calculating a variance of the two or more raw intensity values in the set of raw intensity values normalized by an average of the two or more raw intensity values in the set of raw intensity values; calculating a range of two or more raw intensity values of the set of raw intensity values, or calculating a goodness-of-fit metric to a reference curve of two or more raw intensity values of the set of raw intensity values.

13. The system of claim 11, wherein the plurality of structured illumination patterns are selective excitation patterns, and the set of illumination features includes at least a spatial frequency and a phase of the selective excitation patterns.

14. The system of claim 1, wherein the detection module is further configured to:

determining a set of sub-modulation scores, each sub-modulation score determined by combining subsets of the raw intensity values from respective subsets of the plurality of raw images; and

combining the sub-modulation scores to determine the modulation score.

15. The system of claim 14, wherein a subset of the plurality of original images are characterized by the same spatial frequency.

16. The system of claim 11, wherein the particle is a biomolecule comprising at least one of a DNA fragment, an mRNA fragment, or incrna.

17. The system of claim 11, further comprising:

generating a reconstructed image by processing at least one of the plurality of original images of the object, the reconstructed image comprising a set of reconstructed intensity values obtained by processing the at least one of the plurality of original images,

wherein the detection module is further configured to: for each of the one or more regions of the object, determining a second estimate by comparing one or more reconstructed intensity values to a second threshold, the second estimate indicating whether particles are present at the region of the object; and generating a set of combined estimates for at least one region of the target by comparing the first estimate to the second estimate for the at least one region of the target.

18. The system of claim 17, wherein the detection module is further configured to:

identifying a subset of pixel locations in the reconstructed image, wherein the second estimate for the identified subset of pixel locations indicates the presence of particles at a respective region of the object; and

for each identified pixel location, generating a combined estimate for the pixel location, wherein the combined estimate for the pixel location indicates the presence of the particle at the identified pixel location if the first estimate for the respective region of the target is at a first value; and wherein the combined estimate for the pixel location indicates that the particle is not present at the identified pixel location if the first estimate for the respective region of the target is at a second value.

19. The system of claim 17, wherein the reconstructed image has a resolution higher than a resolution of the plurality of original images.

20. The system of claim 17, wherein the processing module is further configured to perform a Synthetic Aperture Optical (SAO) reconstruction process on the at least one of the plurality of raw images.

1. Field of the invention

The present invention relates generally to the field of optical microscopy imaging using structured or selective illumination or excitation, and more particularly to a method for detecting particles using measurement signals from a structured illumination pattern.

Background

Disclosure of Invention

Embodiments of the invention include a method for Synthetic Aperture Optics (SAO) that minimizes the number of selective excitation patterns used to illuminate an imaging target based on target physical features corresponding to spatial frequency content from the illuminated target and/or one or more parameters of an optical imaging system for the SAO. Embodiments of the present invention also include SAO devices that include a plurality of interference pattern generation modules arranged in a semi-annular shape.

In one embodiment, a SAO method comprises: illuminating a target comprising one or more objects with a predetermined number (N) of selective excitation patterns, wherein the number (N) of selective excitation patterns is determined based on physical characteristics of the objects corresponding to spatial frequency content from the illuminated target; optically imaging the illuminated target at a resolution insufficient to resolve objects on the target; and processing the optical image of the illuminated target using the information about the selective excitation pattern to obtain a final image of the illuminated target at a resolution sufficient to resolve objects on the target. In another embodiment, the number (N) of selective excitation patterns corresponds to the number of k-space sampling points in k-space sampling space in the frequency domain, wherein the extent of the k-space sampling space is substantially proportional to the inverse of the minimum distance (Δ x) between objects to be resolved by the SAO, and wherein the inverse of the k spatial sampling intervals between the k spatial sampling points is smaller than the width (w) of the detection region captured by the pixels of the system for optical imaging.

In another embodiment, an SAO device includes a plurality of Interference Pattern Generation Modules (IPGMs), wherein each IPGM is configured to generate a pair of beams that interfere to generate a selective excitation pattern for a target at a predetermined orientation and a predetermined spacing, and wherein the IPGMs are arranged in a semi-circular shape. The SAO device also includes an optical imaging module configured to optically image the illuminated target at a resolution insufficient to resolve objects on the target. The optical image of the illuminated target is further processed using information about the selective excitation pattern to obtain a final image of the illuminated target at a resolution sufficient to resolve the target. The number of IPGMs is equal to the number of selective excitation patterns for performing SAO on the target. The IPGM may be substantially symmetrically disposed on a monolithic structure having a semi-toroidal shape.

According to various embodiments of the present invention, an optimized minimum number of excitation patterns are used in the SAO, thereby enabling the SAO to be used for applications such as DNA sequencing requiring massively parallel SAO imaging in a small amount of time to make SAO-utilizing DNA sequencing commercially viable. Therefore, a significant increase in throughput and a reduction in cost for DNA sequencing can be achieved by using the SAO according to the present invention.

Embodiments of the present disclosure also include a method for detecting particles on a target. Embodiments of the present disclosure also include a system for detecting particles on a target.

In one embodiment, a particle detection method includes: illuminating a target with a plurality of structured illumination patterns, each characterized by a spatial frequency and an illumination phase; generating a plurality of raw images of the illuminated target by measuring optical signals from the target, each raw image comprising at least one raw intensity value, each raw image being obtained by measurement of the target illuminated with a respective structured illumination pattern; and generating, for each of the one or more regions of the target, a first estimate indicative of whether a particle is present at each of the one or more regions of the target. Generating the first estimate comprises: for each of one or more regions of the target, determining a modulation score by combining a set of raw intensity values from a plurality of raw images, the modulation score indicating a degree of variation of the set of raw intensity values in each of the one or more regions of the target, and generating a first estimate for each of the one or more regions of the target by comparing the modulation score for said region to a first threshold.

In another embodiment, a system for detecting particles on a target includes a plurality of illumination modules configured to illuminate the target with a plurality of structured illumination patterns, each characterized by a spatial frequency and an illumination phase. The system also includes an optical imaging module configured to generate a plurality of raw images of the illuminated target by measuring optical signals from the target, each raw image including at least one raw intensity value, each raw image obtained by measuring the target illuminated with a respective structured illumination pattern. The system also includes a detection module configured to: generating, for each of one or more regions of a target, a first estimate indicative of whether particles are present at each of the one or more regions of the target, wherein the detection module is further configured to: determining, for each of the one or more regions of the target, a modulation score by combining the sets of raw intensity values from the plurality of raw images, the modulation score indicating a degree of variation of the sets of raw intensity values in each of the one or more regions of the target; and generating a first estimate for each of the one or more regions of the target by comparing the modulation to a first threshold.

The features and advantages described in the specification are not all-inclusive and, in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter.

Drawings

The teachings of embodiments of the present invention can be readily understood by considering the following detailed description in conjunction with the accompanying drawings.

Fig. 1A illustrates a conventional SAO method.

Fig. 1B illustrates a conventional SAO system.

FIG. 1C shows an example of a selective excitation pattern in the spatial and frequency domains.

Fig. 2A, 2B and 2C show different types of individual sequencing microparticles that can be used for DNA sequencing.

Fig. 3A and 3B show some examples of the distribution of DNA microparticles.

Fig. 4 illustrates a SAO method according to an embodiment.

Fig. 5A shows k-space sampling points (selective excitation patterns) used in SAO according to one embodiment.

Fig. 5B illustrates the selection of k-space sampling intervals for use in SAO, according to one embodiment.

Fig. 5C illustrates the use of selective excitation patterns corresponding to k-space sampling points within a circular region, according to one embodiment.

Fig. 5D illustrates reducing the number of k-space sampling points by sparse k-space sampling, according to one embodiment.

Fig. 6A shows how aliasing may occur in SAO by using a pixel field of view (PFOV) that is smaller than the detection region, according to one embodiment.

FIG. 6B illustrates how the actual signal at a pixel of the imaging system is determined by unwrapping the measured signal at the pixel to remove aliasing, according to one embodiment.

FIG. 6C illustrates a method of unwrapping a measurement signal at a pixel to remove aliasing, according to one embodiment.

FIG. 7A illustrates a structured illumination apparatus for selectively exciting particles according to one embodiment.

FIG. 7B illustrates an illumination pattern generation module arranged in a semi-annular configuration, according to one embodiment.

Fig. 7C illustrates an internal structure of an irradiation pattern generation module according to an embodiment.

Fig. 7D shows an internal structure of an irradiation pattern generation module according to another embodiment.

FIG. 8 illustrates a particle detection method according to one embodiment.

Fig. 9 illustrates a particle detection method according to another embodiment.

Fig. 10 shows an example of a particle detection method performed on a tissue slice target region according to an embodiment.

Fig. 11 shows an example of a particle detection method performed on a single molecule mRNAFISH (fluorescent in situ hybridization) sample according to an embodiment.

Detailed Description

The drawings and the following description relate to preferred embodiments of the invention by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be employed without departing from the principles of the claimed invention.

Reference will now be made in detail to several embodiments of the invention, examples of which are illustrated in the accompanying drawings. Note that where feasible, similar or like reference numbers may be used in the figures and may indicate similar or like functionality. The figures depict embodiments of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein.

Synthetic Aperture Optical (SAO) imaging methods according to various embodiments of the present invention minimize the number of selective excitation patterns used to illuminate an imaging target based on target physical features corresponding to spatial frequency content from the illuminated target and/or one or more parameters of an optical imaging system for the SAO. Embodiments of the present invention also include SAO devices optimized to perform the SAO method according to the present invention. The SAO device includes a plurality of interference pattern generation modules arranged in a semi-annular shape, each of the plurality of interference pattern generation modules generating one selective excitation pattern for the SAO.

Turning to fig. 4, fig. 4 illustrates a SAO method according to an embodiment. As is typical for SAO imaging, selective excitation (or illumination) 104 is applied to the imaging target 102 and light scattered or fluorescently emitted from the imaging target 102 is captured by the optical imaging device 106. Here, it is assumed that the imaging target 102 is a DNA microparticle such as those shown in fig. 2A to 2C, fig. 3A, and 3B, an mRNA (messenger RNA) fragment, lncRNA (long non-coding RNA), or a protein that appears as a spot or a particle when fluorescently labeled and imaged such as those shown in fig. 10 and 11, but is not limited thereto. As will be explained in more detail below with reference to fig. 7A-7D, a selective excitation 104 may be applied to the imaging target 102 by an illumination device, the selective excitation 104 being configured to cause interference of two beams on the imaging target 102. The excitation target 102 emits signals (or photons) and the emitted signals are captured in an optical imaging system 106 that includes an objective lens and an imaging sensor (or imager). It is then determined 408 whether images corresponding to the 2D sinusoidal excitation patterns for all M phases are obtained. If images corresponding to the 2D sinusoidal excitation patterns of all phases are not obtained in step 408, the excitation phase is changed 402 and steps 104, 106, 408 are repeated for the changed excitation phase. If images corresponding to all phases of the 2D sinusoidal excitation patterns are acquired in step 408, a determination 410 is made as to whether images corresponding to all 2D sinusoidal selective excitation patterns are acquired. If images corresponding to all 2D sinusoidal excitation patterns are not obtained in step 410, the excitation patterns are changed by using a different spatial frequency (e.g., changing the pitch 143 and orientation φ of the 2D sinusoidal patterns), and steps 104, 106, 408, 402, 410, 404 are repeated for the next selective excitation pattern. Then, if images corresponding to all 2D sinusoidal excitation patterns are obtained in step 410, the captured images are sent to a computer for SAO post-processing 412 and visualization to obtain a high resolution image 114 of the imaging target 102 from the captured low resolution raw images. As described above, the raw image captured by the optical imaging device 106 has a resolution insufficient to resolve objects on the imaging target 102, while the high resolution image 114 reconstructed by the SAO post-processing 412 has a resolution sufficient to resolve objects on the imaging target 102.

The SAO method of the present invention uses an optimized number N of selective excitation patterns and an optimized number M of excitation phases per selective excitation pattern, such that the SAO can be used for imaging targets such as DNA particles in a massively parallel manner in a small amount of time. As described above, the number of selective excitation patterns used in the conventional SAO is determined only by the hardware features of the illumination system, independent of and without regard to the imaging target or the imaging system (objective and camera). Thus, in conventional SAO, the number of k-space sampling points corresponding to the selective excitation pattern is not optimized, and the k-space sampling points have many redundant and sometimes uncorrelated k-space sampling points. In contrast, SAO according to embodiments of the present invention herein uses the following selective excitation patterns: the number N of said selectively excited patterns is optimized and minimized depending on the physical characteristics of the imaged object corresponding to the spatial frequency content, e.g. the size, shape and/or spacing of the objects on the imaged object. The SAO according to embodiments herein may also use the following selective excitation patterns: the number N of the selective excitation patterns is determined according to various parameters of the imaging system (e.g., magnification (Mag) of the objective lens, Numerical Aperture (NA) of the objective lens, wavelength λ of light emitted from the imaging target_EAnd/or the effective pixel size p of the pixel sensitive area of the CCD, etc.) are alternatively or additionally optimized. In this way, the resulting number N of excitation patterns used in SAO becomes much smaller than in conventional SAO, therebyEnabling SAO for DNA sequencing requiring massively parallel SAO imaging in a small amount of time to make DNA sequencing commercially viable. Thus, a significant reduction in cost and an increase in throughput of DNA sequencing can be achieved.

Fig. 5A shows k-space sampling points (selective excitation patterns) used in SAO according to one embodiment. In fig. 5A, the CCD imaging region is assumed to have a square shape, and thus a square-shaped k-space sampling space 500 for SAO is also assumed, but the description below with respect to fig. 5A may also be applied to a non-square-shaped (e.g., rectangular) k-space sampling space. The k-space sampling space 500 has a field of view FOV²Wherein the range of the k-space sampling space in each of the horizontal direction and the vertical direction is the FOV. Here, FOV represents the k-space field of view. In the k-space frequency domain, the FOV should be equal to (1/Δ x), where Δ x is the spatial resolution of the imaging system (i.e., Δ x is the minimum distance between two point objects that can be resolved by the imaging system). Each conjugate pair 502, 506 and its DC point 504 correspond to one selective excitation pattern for the SAO as used in the present invention. Thus, the number of selective excitation patterns used in SAO corresponds to the number of conjugate pairs of k-space points in k-space sampling space 500(FOV × FOV). Δ k_xIs the k-space sampling interval and is equal to (1/PFOV), where PFOV is the pixel field of view. k-space sampling interval Δ k in k-space sampling space 500_xThe smaller the number of k-space points and the corresponding number of excitation patterns. Specifically, the following equation holds:

floor (L/2) … … (formula 1),

where L is the number of k-space points in k-space, N is the number of selectively excited patterns, and floor () rounds the number to the nearest and smallest integer;

L＝round((FOV/Δk_x)²)＝round((PFOV/Δx)²) … … (formula 2) in the following manner,

where round () rounds the number to the nearest integer, and PFOV is the range of reciprocal (or fourier) space (reciprocal space) of the sampling space (k-space) to be reconstructed from the samples.

Fig. 5B illustrates the selection of k-space sampling intervals for use in SAO, according to one embodiment. As described above, the size of the imaged object determines the required spatial resolution Δ x. The magnification (Mag) and CCD pixel size (Z) determine the effective pixel size p on the imaging target plane, p ═ Z/Mag. As shown in fig. 5B, the detection area w (x) (i.e., the area captured by the pixels) may be represented as a convolution (e.g., bell-shaped curve) of a pixel sensitivity function p (x) (e.g., a rectangular function having a width p) and a Point Spread Function (PSF) h (x) of the lens. The width w may be defined as 1/e of the detection area w (x)²Width. Since the PSF of a lens is determined by the NA of the lens, the extent (w) of the detection area and the weight on the detection area (i.e., the effective sensitivity profile on the detection area) are functions of the magnification (Mag) of the lens, the Numerical Aperture (NA) of the lens, and the CCD pixel size (Z).

As can be seen from the above, the k-space sampling space (FOV) is determined by the desired spatial resolution Δ x and is determined by the imaging subject. Particles of interest, such as biological particles like DNA particles or mRNA fragments, are usually of very small size, resulting in a large k-space sampling space. In conventional SAO, the k-space sampling interval Δ kx is set without regard to the physical characteristics of the imaging target or parameters of the imaging system, but is only randomly set according to any interval allowed by the SAO illumination system. This makes the number of k-space points and resulting selective excitation patterns for DNA sequencing applications using SAO too large, since DNA sequencing using such a large number of selective excitation patterns in SAO causes high costs and low throughput.

In contrast, SAO according to embodiments of the present invention herein uses the following selective excitation patterns: the number N of said selectively excited patterns is optimized depending on the physical characteristics of the imaged object corresponding to the spatial frequency content, e.g. the size, shape and/or spacing of the imaged object. As shown in figure 5B, in one embodiment the pixel field of view PFOV is chosen to be smaller than the extent (w) of the detection zone, i.e. PFOV<w. Using a smaller PFOV results in a larger k-space sampling interval Δ k_xThereby reducing k-space acquisitionThe number of k-space points (L) in sample space 500 and the resulting number of selective excitation patterns (N) for SAO. As will be described in more detail below with reference to fig. 6A and 6B, using a PFOV that is smaller than the range (w) of the detection region may cause aliasing (aliasing) in a high-resolution image obtained from the SAO, but such aliasing may be removed using a method as described below with reference to fig. 6C. In other embodiments, the PFOV may be set to be equal to or larger than the range (w) of the detection region, thereby preventing aliasing of the high-resolution image obtained from the SAO. It is also noted that setting the PFOV in view of the extent (w) of the detection zone effectively sets the k-space sampling interval (Δ k) based on various parameters of the imaging system_x) And the resulting number of selective excitation patterns (N), because, as explained above, the extent of the detection area (w) is a function of the magnification of the lens (Mag), the numerical aperture of the lens (NA) and the CCD pixel size (Z).

Furthermore, SAO according to embodiments herein further reduce the number of iterations of selective excitation and imaging by minimizing the number of phase changes (M in steps 402, 408 of fig. 4). Referring back to fig. 5A and as explained above, a conjugate pair of k-space points 502, 506 corresponds to an SAO interference pattern generation module that produces a specific spacing and orientation of a selective excitation pattern. The DC point 504 corresponds to a signal offset of the 2D sinusoidal selective excitation pattern. Thus, in one embodiment, three different measurements are taken at the same pitch and orientation at three different phases of the interference pattern to distinguish the two conjugate points 502, 506 in k-space and the DC point 504. This is in contrast to conventional SAO, in which more than three phases are used to illuminate and image each selective excitation pattern for the SAO. In another embodiment, since the DC spot 504 is common to all conjugate pairs 502, 506, the resulting DC spot 504 at a particular pitch and orientation in one 2D sinusoidal pattern may also be utilized to eliminate the need to illuminate and image the selective excitation pattern at a DC spot 504 at a different pitch and orientation in another selective excitation pattern, thereby reducing the number of changed phases M required for imaging in steps 402, 408 (fig. 4) to two (2) phases for the other selective excitation pattern. In other words, each interference pattern generation module only produces patterns in two different phases, except that one module produces patterns in three different phases to acquire the DC spot 504. For best tolerance to noise, a particular phase may be selected for the pattern. The optimum phase difference may be 0 degrees, 120 degrees and 240 degrees for three different phases of each particular selectively excited pattern. The optimal phase difference may be 0 degrees and 90 degrees for the two different phases of each particular selectively excited pattern.

Since the object of interest (i.e. a biological particle such as a DNA microparticle, an mRNA fragment, lncRNA or a protein) is typically circularly symmetric, the k-space spectrum of the object of interest will also be circularly symmetric, and thus only k-space samples with a diameter FOV (═ 1/Δ x) in a circular region may be needed for SAO. Thus, in one embodiment, the SAO according to the present invention uses selective excitation patterns corresponding to k-space sampling points within the circular region 512, as shown in fig. 5C.

Fig. 5D illustrates reducing the number of k-space sampling points by sparse k-space sampling, according to one embodiment. Conventional SAO methods do not utilize frequency information of objects in an image scene. Solid objects such as beads used in microparticles have much less energy in high spatial frequencies than low frequencies. Therefore, undersampling at high spatial frequencies is more tolerable than undersampling in low spatial frequency regions. Thus, in one embodiment of the present invention, the number of selective excitation patterns (N) is further reduced by non-uniform or variable density sampling in fourier space, as shown in fig. 5D. In SAO for DNA sequencing applications, the disadvantage of not meeting the nyquist sampling rate in high spatial frequencies is tolerable, and thus, SAO according to embodiments herein relaxes the nyquist sampling criterion in higher frequencies, thereby reducing the number of selective excitation patterns by almost half of the number required for uniform sampling. For example, the number of k-space samples in the embodiment of fig. 5D is only 54% of the number of k-space samples in the embodiment of fig. 5C.

Fig. 6A illustrates how aliasing may occur in an SAO by using a smaller pixel field of view than the detection region, according to one embodiment. As mentioned above with reference to fig. 5B, using a pixel field of view (PFOV) that is smaller than the extent (w) of the detection region, i.e. PFOV < w, results in aliasing of the image obtained for the SAO, since each pixel in the CCD will detect a larger region than the pixel itself. The extra regions (i.e., the left and right portions of the range (w) outside of p (x)) are regions that would also be detected by their neighboring pixels in the CCD. This is illustrated in fig. 6A, where objects 602, 604, 606, 608 detected in the extra region of neighboring pixels will enter the central pixel 610 (assuming straight-line sampling in k-space), resulting in aliasing and undesirable artifacts that degrade image quality.

Fig. 6B shows how the actual signal at a pixel of the imaging system can be determined by unwrapping the measurement signal at the pixel to remove aliasing, according to one embodiment. In order to remove aliasing of the measured image signal and to obtain the actual image signal, a CCD is arranged at a pixel of the CCD_iA linear equation in the form of y-Ax can be formulated at a particular sub-pixel k. Referring to FIG. 6B, m_k,iCCD at ith CCD pixel_iThe measured signal at a particular k-th sub-pixel location within (including aliasing). Note that all measurement signals m_k,iThe relative position of (i ═ 1, … …, infinity) in its ith CCD pixel is the same. s_k,iCCD pixel representing object at ith_iActual or ideal signal at the kth sub-pixel location within. Alpha, beta and gamma respectively indicate that the ith CCD pixel is connected with s_k,i-1、s_k,iAnd s_k,i+1The value of the weighting function w (x) at the corresponding position, and s_k,i(i is1, … …, infinity) is the ith CCD pixel CCD_iThe actual (ideal) signal at a particular k-th sub-pixel location within. As explained above, the weighting function w (x) may be expressed as a convolution (e.g. bell curve) of the pixel sensitivity function p (x) (e.g. a rectangular function with a width p) and the Point Spread Function (PSF) h (x) of the lens. Using these defined parameters and assuming the number of pixels of the CCDIs infinite, the sequence of signal equations for a particular kth sub-pixel position can be written as a linear matrix equation y ═ Ax, where y ═ m_k,1，m_k,2，……]，x＝[s_k,1，s_k,2，……]And a is a matrix with elements of zero and weighting function values (e.g., α, β, and γ). The linear matrix equation y Ax shows the "unfolding" process (i.e. recovering the actual signal s)_k,i) Can be viewed as a common inverse problem of y Ax (i.e., x a)^-1y). In other embodiments, s is if a non-linear sampling pattern (e.g., variable density, radial sampling, etc.) is used_iAnd m_iThe actual relationship therebetween will vary according to the relationship shown in fig. 6B, in which case the point spread function (i.e., impulse response) can be measured in a simulation experiment or an actual experiment to construct the inverse matrix (a)^-1)。

FIG. 6C illustrates a method of unwrapping a measurement signal at a pixel to remove aliasing, according to one embodiment. Together, steps 652, 654, 656 constitute post-processing steps for the SAO. In conventional SAO, the post-processing includes only conventional SAO reconstruction 652, which conventional SAO reconstruction 652 is used to generate a high spatial resolution image 653 from a low resolution image (M × N)650 obtained by selective excitation of the imaging target. Here, however, in the SAO according to the embodiment of the present invention, the post-processing includes a "unfolding" step 670 for removing aliasing from the high spatial resolution image 653 containing aliasing caused by using a PFOV smaller than the extent (w) of the detection region for selective excitation. The expansion process 670 includes solving the linear equation y Ax to repeat 656 at each sub-pixel location to recover the actual signal x for all CCD pixels. Thus, although PFOV smaller than the extent (w) of the detection region is used for selective excitation in SAO, a high spatial resolution image 658 without aliasing may be obtained from SAO.

Note that even when PFOV greater than or equal to the extent (w) of the detection region is used for selective excitation in SAO, the "unfolding" explained herein may be used to improve SAO image reconstruction quality. In conventional SAO, the reconstructed pixels are simply clipped (to a width of p) and stitched together. This way of "cropping and stitching" still does not allow to undo the apodization caused by the weighting function w (x). In contrast, "unfolding" may be used according to the invention in SAO even when PFOV greater than or equal to the extent (w) of the detection region are used for selective excitation, so that aliasing does not occur. Since the unfolding process fundamentally cancels (i.e., undoes) the weighting function w (x), the "unfolding" process can be used to improve image reconstruction even when PFOV > -w is used for SAO selective excitation.

FIG. 7A illustrates a structured illumination apparatus for selectively exciting particles according to one embodiment. The irradiation device shown in fig. 7A is merely exemplary, and various modifications may be made to the configuration of the irradiation device for SAO according to the present invention. The exemplary illumination apparatus in fig. 7A shows only two Interference Pattern Generation Modules (IPGM)712, 713, but for biological sequencing applications such as authentic DNA sequencing applications, mRNA sequencing applications, or lncRNA sequencing applications, there may be a greater number of IPGM. Each IPGM has a modular form and is configured to generate a selective excitation pattern at a given pitch and orientation corresponding to a conjugate pair of k-space sampling points. Thus, there is a one-to-one relationship between IPGM and a conjugate pair of 2-D sinusoidally selective excitation patterns at a given pitch and orientation and k-space sampling points. A larger number of selectively excited patterns would require a larger number of IPGMs in the SAO irradiation device.

The structured illumination apparatus 700 generates a plurality of mutually coherent laser beams whose interference produces an interference pattern. Such an interference pattern is projected onto the particle array substrate 204 and selectively excites the DNA particles 202. The use of interference of multiple laser beams to generate an interference pattern is advantageous for a number of reasons. This enables, for example, high resolution excitation patterns with very large FOV and DOF. Although the structured illumination apparatus of fig. 7A is described herein as an example of generating an excitation pattern for DNA microparticles, it should be noted that the structured illumination apparatus of fig. 7A may be used in any other type of application to generate an excitation pattern for imaging any other type of target, such as a biological particle, including DNA fragments, mRNA (messenger RNA) fragments, incrna (long non-coding RNA), or proteins that appear as spots or particles when fluorescently labeled and imaged. Examples of using selective excitation patterns to image mRNA fragments are described below in conjunction with fig. 10-11.

Referring to fig. 7A, the structured illumination apparatus 700 includes: a laser 702; a beam splitter 704; shutters 705, 707; fiber couplers 708, 709; a pair of optical fibers 710, 711; and a pair of Interference Pattern Generation Modules (IPGM)712, 713. As described above, each IPGM 712, 713 generates an interference pattern (selective excitation pattern) that corresponds to a conjugate pair of k-space sample points. The beam 703 of the laser 702 is split into two beams 740, 742 by the beam splitter 704. Pairs of high speed shutters 705, 707 are used to "turn" or "turn" each beam 740, 742 on or off, respectively, or to modulate the amplitude of each beam 740, 742, respectively. This switched laser beam is coupled via fiber couplers 709, 708 into the pair of polarization maintaining fibers 711, 710. Each optical fiber 711, 710 is connected to a corresponding interference pattern generation module 713, 712, respectively. The interference pattern generation module 713 includes a collimating lens 714', a beam splitter 716', and a translating mirror 718', and likewise, the interference pattern generation module 712 includes a collimating lens 714, a beam splitter 716, and a translating mirror 718.

The beam 744 from the optical fiber 710 is collimated by the collimating lens 714 and split into two beams 724, 726 by the beam splitter 716. Mirror 718 is translated by actuator 720 to change the optical path length of beam 726. Thus, an interference pattern 722 is generated on the substrate 204 in the region of overlap between the two laser beams 724, 726, where the phase of the pattern is changed by changing the optical path length of one of the beams 726 (i.e., by modulating the optical phase of the beam 726 by use of the translating mirror 718).

Similarly, a light beam 746 from the optical fiber 711 is collimated by the collimating lens 714 'and split into two light beams 728, 730 by the beam splitter 716'. Mirror 718 'is translated by actuator 720' to change the optical path length of beam 728. Thus, an interference pattern 722 is generated on the substrate 204 in the region of overlap between the two laser beams 728, 730, wherein the pattern is changed by changing the optical path length of one of the beams 728 (i.e., by modulating the optical phase of the beam 728 by use of the translating mirror 718).

As shown in fig. 7A, each IPGM 712, 713 is implemented in modular form according to embodiments herein, and one IPGM produces an interference pattern corresponding to one conjugate pair of k-space points. According to embodiments herein, this modular one-to-one relationship between IPGM and k-space points greatly simplifies the hardware design process for SAO. As the number of selective excitation patterns for SAO increases or decreases, SAO hardware may be modified simply by increasing or decreasing the number of IPGMs in a modular manner. In contrast, conventional SAO devices do not have discrete interference pattern generation modules, but rather have a series of split beams that produce as many multiple interferences as possible. This conventional way of designing SAO devices creates non-optimized or redundant patterns, slowing and complicating the operation of the SAO system.

Although the implementation shown in FIG. 7A is used for its simplicity, various other approaches may be used within the scope of the invention. For example, the amplitude, polarization, direction, and wavelength of one or more of the beams 724, 726, 728, 730 may be modulated in addition to or instead of optical amplitude and optical phase to change the excitation pattern 722. In addition, the structured illumination may simply be translated relative to the microparticle array to change the excitation pattern. Similarly, the microparticle array may be translated relative to the structured illumination to change the excitation pattern. In addition, various types of optical modulators may be used in addition to or in place of the translating mirrors 718, 718', such as acousto-optic modulators, electro-optic modulators, galvanometer modulated rotating windows, and micro-electro-mechanical systems (MEMS) modulators. Additionally, although the structured illumination apparatus of fig. 7A is described herein as using a laser 702 as the illumination source for coherent electromagnetic radiation, other types of coherent electromagnetic radiation sources, such as SLDs (superluminescent diodes), may be used in place of the laser 702.

In addition, although FIG. 7A illustrates the use of four beams 724, 726, 728, 730 to generate the interference pattern 722, a greater number of laser beams may be used by splitting the source laser beam into more than two beams. For example, 64 beams may be used to generate the interference pattern 722. In addition, the beam combination need not be limited to a pair-wise combination. For example, the interference pattern 722 may be generated using three beams 724, 726, 728, or three beams 724, 726, 730, or three beams 724, 728, 730, or three beams 726, 729, 730, or all four beams 724, 726, 728, 730. Typically, the minimum set of beam combinations (two beams) is selected as needed to maximize speed. In addition, the light beam may be collimated, converging, or diverging. Although different from the specific implementation of fig. 7A and for different applications, additional general background information regarding the use of multiple beam pairs to generate interference patterns may be found in the following patents: (i) U.S. patent No. 6,016,196 entitled "Multiple Beam Pair Optical Imaging" to mermelelstein at 18.1.2000; (ii) U.S. patent No. 6,140,660 entitled "Optical Synthetic Aperture Array" issued to Mermelstein at 31.10.2000; and (iii) U.S. patent No. 6,548,820 entitled "Optical synthetic Aperture Array" issued to Mermelstein at 15.4.2003, the entire contents of which are incorporated herein by reference.

FIG. 7B illustrates an illumination pattern generation module having a semi-annular structural arrangement, according to one embodiment. Referring to fig. 7B, a plurality of IPGMs (IPGM 1, IPGM 2, … …, IPGM N), such as IPGMs 712, 713 (fig. 7A), are arranged substantially symmetrically on the semi-toroidal shaped monolithic structure 762 in a semi-toroidal configuration to generate a selective excitation pattern. The semi-annular structure 762 is secured to a system table 768. In the embodiment of fig. 7B, N IPGMs generate N selective excitation patterns for the SAO for the imaging target 102, and the scattered or fluorescent light 752 passes through the objective lens 124 and is captured 756 by the camera 126, which may be a CCD camera.

These arrangements of IPGM in the embodiment of fig. 7B enable a monolithic and compact holding structure with multiple advantages for implementing SAO systems to be used for DNA sequencing applications, as compared to conventional optical platform SAO systems in which each optical component is individually mounted on its holding structure. The monolithic structure 762 enables a compact and symmetric IPGM arrangement, and this compact, symmetric and monolithic structure preserves more stable channel-to-channel and beam-to-beam geometry to prevent mechanical and thermal deformation. The compact monolithic structure 762 is also not susceptible to non-flat or twisted and curved modes of the optical bench 768, and the symmetric arrangement of IPGM around the semi-annular structure 762 results in less damage to the beam geometry from the effects of thermal contraction or expansion, i.e., the channel-to-channel angle of the laser beam or the beam-to-beam angle is changed less compared to the asymmetric structure. Furthermore, the compact design shortens the travel distance of the laser beam in air, which makes it easy to prevent air disturbances that affect the stability of the interference pattern, which may cause the effective optical path length to change, thereby generating variations in the position of the interference fringes. This stability enables a more accurate calibration of the beam geometry. Furthermore, the semi-toroidal arrangement of IPGM in fig. 7B has the following additional advantages: such that the imaging module (i.e., camera 126 and objective 124), illumination structure (i.e., semi-annular 762), and imaging target 102 can be placed on a rigid structure (e.g., optical table) 768.

Fig. 7C illustrates an internal structure of an irradiation pattern generation module according to an embodiment. The embodiment of fig. 7C has a rotating window 760 in IPGM 750 set behind mirror 762. The light beam 770 from the optical fiber 710 is collimated by the collimating lens 754 and the collimated light beam 744 is split into two light beams 773, 774 by the beam splitter 756. Beam 773 is reflected by mirror 758 and reflected beam 778 is projected onto an imaging target to generate interference pattern 780. The beam 774 is reflected by a mirror 762 and the optical path length of the reflected beam 776 is modulated by an optical window 760 rotated by a galvanometer, thereby collimatingThe optical phase of the corresponding beam 776 is modulated and a modulated beam 777 is generated. An interference pattern 780 is generated in the region of overlap between two laser beams 777, 778, where the pattern is changed by changing the optical path length of one of the beams 777. By locating the rotating window 760 behind the mirror 762, the width W of the IPGM 750 may be reduced as compared to the embodiment of FIGS. 7A and 7D shown below_IPGMAnd size. Thus, the semi-annular structure 762 holding the IPGM may be made more compact because of the width W of the IPGM_IPGMDirectly affecting the radius of the semi-circle, as shown, for example, in fig. 7B.

Fig. 7D shows an internal structure of an irradiation pattern generation module according to another embodiment. The IPGM in the embodiments of fig. 7A and 7C may produce two beams that do not have equal path lengths between the interference point at the imaging target and the split point (i.e., the beam splitter). If a relatively short coherence length laser is used and also limits the applicability of the SAO system to only specific wavelengths (e.g., 532nm green laser), the unequal path lengths may significantly reduce the sinusoidal contrast, since for good sinusoidal contrast only a small number of lasers with specific wavelengths have a sufficiently long coherence length that can be used with such an unequal path IPGM. In contrast to the embodiment of fig. 7A, the embodiment of fig. 7D uses an additional fold mirror to achieve equal paths between the two separate beams. Laser beam 744 is split into beams 781, 780 by beam splitter 756. Light beam 781 is reflected by mirror 782 and the optical path length of the light beam is modulated by rotating window 760 to generate light beam 788. On the other hand, light beam 780 is reflected twice by two mirrors 784, 787 to generate reflected light beam 789. The beam 788 and the beam 789 eventually interfere at the imaging target to generate a selective excitation pattern. By using the two mirrors 784, 786, the optical paths 744-780-785-789 are configured to have lengths substantially equal to the lengths of the optical paths 781-783-788. This equal path approach allows the use of lasers with shorter coherence lengths to generate interference patterns with high contrast. Furthermore, this equal path scheme enables the SAO system to be used with wavelengths other than 532nm, making multi-color SAO feasible.

Detecting particles on a target using a structured illumination pattern

Conventionally, detection systems estimate the presence or absence of particles on multiple regions on an object by generating a set of reconstructed estimates based on intensity values of the reconstructed image. For example, a reconstruction estimate for a pixel location of a reconstructed image may indicate whether a particle is present at a corresponding region of the target 102 by comparing an intensity value for the pixel location to a predetermined threshold. However, it is often difficult to detect particles with high accuracy in this way, since e.g. the texture of the object leads to noisy reconstructed images.

Particle detection methods according to various embodiments of the present disclosure use measurement signals from multiple structured illumination patterns to detect the presence and location of particles on a target. In particular, the particle detection methods disclosed herein use measurement signals obtained by illuminating a target with a structured illumination pattern to detect particles. Particles may respond differently to illumination across multiple structured illumination patterns, and the degree of variation of these measurement signals in the raw image may provide important insight into determining whether particles are present on the target 102. Although the reconstruction process generates a reconstructed image at a higher resolution than the original image, the intensity values of the reconstructed image generally do not retain this degree of variation, which is useful for particle detection.

Turning to the drawings, FIG. 8 illustrates a particle detection method according to one embodiment. The imaging target 102 is illuminated 806 with a plurality of structured illumination patterns, each characterized by a set of illumination features. In one embodiment, the structured illumination pattern is a selective excitation pattern characterized by at least a spatial frequency and phase as described herein. For example, the imaging target 102 may be illuminated with three structured illumination patterns having a set of illumination features { spatial frequency 1, phase 1}, { spatial frequency 1, phase 2}, and { spatial frequency 1, phase 3 }. The imaging target 102 may be assumed to be a variety of biomolecules of interest, such as, but not limited to, DNA fragments, mRNA (messenger RNA) fragments, lncRNA (long non-coding RNA), or proteins that appear as spots or particles when fluorescently labeled and imaged. The phase may be generated by changing the optical path length of one of the laser beams used to generate the pattern, for example, by modulating the optical phase of the beam using a translating mirror. A plurality of raw images of the image target are generated 808 by measuring optical signals from the illuminated target 102. Each original image may be obtained by illuminating the target 102 with a particular structured illumination pattern, and each original image includes original intensity values arranged as a set of pixels for the original image. For example, when the image target 102 is illuminated with each of the three structured illumination patterns, an original image may be generated, resulting in three original images — original image 1, original image 2, and original image 3. The raw intensity values may be captured by the optical imaging system 106 when the excited target 102 emits a signal (or photons). Each pixel in the original image may correspond to a particular region in the target 102, and the intensity values for the pixel locations are obtained by measuring the signals emitted from the particular region of the target 102 when the target 102 is illuminated with the structured illumination pattern.

For each of one or more regions of the imaging target 102, a modulation estimate using the original image is generated, the modulation estimate indicating whether particles are present at the region of the target 102. Specifically, a modulation score for each region of interest is determined 810 by combining sets of raw intensity values corresponding to the regions of interest in the target 102. The set of raw intensity values is obtained from a plurality of raw images generated by imaging the target 102 with a plurality of structured illumination patterns. A particular region of the target 102 may have a corresponding pixel location in each raw image, and a modulation score for the particular region may be generated by combining together each raw intensity value from the corresponding raw image. For example, the raw intensity values for pixel locations in raw image 1, raw image 2, and raw image 3 that correspond to a particular region of the target 102 may be combined to generate one modulation score for that region of the target 102. Alternatively, modulation scores may be determined for regions of the target 102 that encompass more than a single pixel location. In this case, the measurement from the region of the target 102 may have a corresponding set of pixel locations in each raw image, and a modulation score for the region may be generated by combining together each set of raw intensity values from the corresponding raw images.

The modulation score indicates a degree of variation in the set of raw intensity values, and predicts a likelihood of a particle being present on a region of the target 102 based on the observed raw intensity values. The modulation score is compared 812 to a first predetermined threshold to generate a modulation estimate indicative of the presence of particles on the region of the target 102. Thus, each pixel position or group of pixel positions in the original image can be marked with the following modulation estimates: the modulation estimate indicates whether a particle or a portion of a particle is present in a particular region of the target 102 corresponding to the pixel location. In one embodiment, a positive modulation estimate indicates the presence of a particle on the target 102 if the modulation score is equal to or above a predetermined threshold. In general, a high modulation score indicates a higher likelihood of the presence of particles for that pixel. In general, if particles are present at the corresponding locations, the variation in pixel intensity according to the variation in illumination characteristics, such as phase variation, will be relatively small or constant. Thus, a high degree of variation indicates the presence of particles in the pixel location.

Considering a series of raw intensity value sets for a particular region of the target 102, the modulation score indicates the degree of variation of the raw intensity value sets. In one example, the modulation score is determined as the standard deviation between a set of raw intensity values. In another example, the modulation score is determined as the standard deviation between the set of raw intensity values normalized by (divided by) the average of the set of raw intensity values. In another example, the modulation score is determined as a range in the set of raw intensity values that indicates a difference between a maximum and a minimum of the set. In yet another example, the modulation score is determined as a goodness-of-fit metric indicative of a degree of fit of the set of raw intensity values to the desired curve when a particle is present at the region of the imaging target 102. For example, the goodness-of-fit may indicate how well the set of raw intensity values fit to a sinusoid of illumination phase and intensity. Further, it should be understood that the modulation score may also be generated by any transformation of these metrics, such as scaling the metrics by a constant factor, adding or subtracting terms, and the like.

In one embodiment, the modulation score for a particular region of the target 102 may also be generated by: determining one or more sub-modulation scores from one or more subsets of the original image; and combines the sub-modulation scores to determine a modulation score for the region of the target 102, as will be described in more detail below in connection with table 1.

Table 1 shows an example of determining modulation scores for a set of structured illumination patterns comprising K spatial frequencies and M phases (hence N ═ K × M). In particular, table 1 shows the following cases: k-4 spatial frequencies (spatial frequency 1, spatial frequency 2, spatial frequency 3 and spatial frequency 4) and M-3 phases (phase 1, phase 2 and phase 3), representing a total of 12 structured illumination patterns. Although the example in table 1 uses the same number of phases for all spatial frequencies, this is merely an example, and in other examples, a different number of phases may be used for each different spatial frequency.

In table 1, the raw intensity values from these pixel locations are used to determine a modulation score for a particular region of interest in the target 102 corresponding to a single pixel location in each raw image. Thus, a particular region of the target 102 is associated with 12 raw pixel intensity values, each obtained from an original image generated by illuminating the target 102 with a respective structured illumination pattern. In this example, four modulation scores MS1, MS2, MS3, and MS4 are determined for this particular region of the target 102. Each modulation score is determined by combining raw pixel intensity values obtained from a subset of the raw images having the same spatial frequency. For example, "IS 1" represents raw pixel intensity values for a region of the imaging target 102 obtained by illuminating the imaging target 102 with a structured illumination pattern { spatial frequency 1, phase 1}, { spatial frequency 1, phase 2}, { spatial frequency 1, phase 3 }; "IS 2" represents raw pixel intensity values obtained by illuminating the imaging target 102 with the structured illumination pattern { spatial frequency 2, phase 1}, { spatial frequency 2, phase 2}, { spatial frequency 2, phase 3 }; "IS 3" represents raw pixel intensity values obtained by illuminating the imaging target 102 with the structured illumination pattern { spatial frequency 3, phase 1}, { spatial frequency 3, phase 2}, { spatial frequency 3, phase 3 }; and "IS 4" represents raw pixel intensity values obtained by illuminating the imaging target 102 with the structured illumination pattern { spatial frequency 4, phase 1}, { spatial frequency 4, phase 2}, { spatial frequency 4, phase 3 }. "MS 1" represents a modulation score for subset IS1, "MS 2" represents a modulation score for subset IS2, "MS 3" represents a modulation score for subset IS3, and "MS 4" represents a modulation score for subset IS 4. As described above, each modulation score may be determined by one or a combination of the ways in which the standard deviation is taken, normalized by the mean, the range, and the goodness-of-fit metric for the corresponding subset of raw intensity values are used to determine the degree of variation. Additionally, a modulation score for a particular region of the target 102 may also be generated by combining pixel intensity value sets from each raw image.

In one example, the modulation scores for the respective subsets may be used to generate a modulation estimate of whether particles are present at a particular region of the imaging target 102. For example, the detection system may use only the modulation score MS1 to determine the presence of a particle by comparing the score to a predetermined threshold. In this example, if the modulation score MS1 is equal to or above the threshold, the modulation estimate may indicate the presence of a particle; or if the MS1 is below the threshold, the modulation estimate may indicate that no particles are present. In another example, two or more of the score MS1, the score MS2, the score MS3, and the score MS4 may be considered sub-modulation scores, and these sub-modulation scores are combined together to generate a final modulation score for the region of the imaging target 102. For example, the final modulation score "MS" may be determined as a multiplication of all four sub-modulation scores MS1 × MS2 × MS3 × MS4, and the detection system may use the final score MS to determine the presence of particles by comparing the final score to a predetermined threshold. Similarly, if the final modulation score MS is equal to or above the threshold, the modulation estimate may indicate the presence of particles; or if the MS is below the threshold, the modulation estimate may indicate that no particles are present. Although multiplication is used as an example, other operations such as addition, multiplication, etc. may be used in other embodiments to combine the sub-modulation scores.

TABLE 1

IS: a series of pixel intensities representing measurements from a particular region of interest on a target

Turning to the drawings, FIG. 9 illustrates a particle detection method according to another embodiment. In the present embodiment, modulation estimation is used for validating the reconstructed estimates obtained from the reconstructed images to generate a set of combined estimates for particle detection over multiple regions of the target 102. The reconstructed estimate is compared to the modulation estimate and used to validate the reconstructed estimate. In particular, although the reconstructed estimate may not have the best accuracy, if the reconstructed image has a higher image resolution than the plurality of original images, the reconstructed image may provide a grain detection estimate with a higher region granularity than the original images. For example, the reconstructed image may have a resolution of 2 times the plurality of original images, and the reconstructed estimate for 1 pixel location in the reconstructed image may correspond to a smaller region in the object 102 than the modulation estimate for 1 pixel location in the plurality of original images. By validating the reconstructed estimate with a higher accuracy modulation estimate, the detection system can perform particle detection with higher accuracy and higher granularity.

Returning to FIG. 9, a modulation estimate is generated using the plurality of raw images by steps 906, 908, 910, and 912, which steps 906, 908, 910, and 912 are largely identical to steps 806, 808, 810, and 812 described in connection with FIG. 8. A reconstructed image of the target 102 is generated by reconstructing 914 the plurality of original images. In one embodiment, when multiple raw images are generated by multiple selective excitation patterns, the reconstruction process is an SAO reconstruction process step or a post-processing step that generates high spatial resolution reconstructed images as described herein in connection with fig. 1A-1C and 6A-6C. A set of reconstruction estimates for the pixel locations of the reconstructed image is generated by comparing 916 the intensity values of the reconstructed image to a predetermined threshold. For example, a reconstruction estimate may be generated for each pixel location of the reconstructed image, and if the intensity values for the pixel locations are equal to or above a threshold, the positive estimate may indicate the presence of particles at the respective region of the object 102. The reconstructed estimate for a particular region of the target 102 is compared 918 to the modulation estimate for the particular region of the target 102 to generate a combined estimate for the target 102. When the resolution of the reconstructed image is higher than the resolution of the original image and the reconstructed image captures the target 102 with a higher granularity, the pixels in the reconstructed image may correspond to a smaller region on the target 102 than the region imaged by the pixels in the original image. In this example, the reconstruction estimates for the pixels in the reconstructed image may be compared to modulation estimates for a region of the object 102 that encompasses or otherwise overlaps with the region encompassed by the reconstructed pixels.

In one embodiment, modulation estimation is used for reducing false positive errors for reconstruction estimation. A false positive error occurs when there is no particle in a particular region of the target 102 but the reconstructed estimate indicates that there is a particle in that region. These errors may occur due to a noisy background image in the target 102 that includes defects or other patterns that look like the particles of interest, but in fact they are not. To reduce false positive errors, the detection system identifies reconstructed pixels using the positive reconstructed estimates, compares these estimates to the corresponding modulation estimates 918, and generates a combined estimate as the final estimate for particle detection. The combined estimate for reconstructing a pixel using positive reconstruction estimates indicates that grain is present only if the corresponding modulation estimate is positive and indicates that grain is not present if the corresponding modulation estimate is negative. In this manner, the modulation estimate may be used to validate the positive reconstruction estimate if it erroneously detected the background pattern of the object 102 as a particle of interest.

In another embodiment, modulation estimation may also be used for reducing false negative errors for reconstruction estimation. A false negative error occurs when a particle is present in a particular region of the target 102 but the reconstructed estimate indicates that no particle is present in that region. To reduce false negative errors, the detection system utilizes the negative reconstructed estimates to identify reconstructed pixels, compares these estimates to corresponding modulation estimates, and generates a combined estimate as a final estimate for particle detection. The combined estimate for reconstructing the pixel using the negative modulation estimates indicates that there is no particle present only when the corresponding modulation estimate is negative and indicates that there is a particle present when the corresponding modulation estimate is negative.

Methods and apparatus for obtaining images of biomolecules and samples are described in more detail in U.S. patent application No. 15/059,245 filed on 3.2.2016, which is now published as U.S. patent No. 9,772,505, which is incorporated by reference.

Examples of particle detection

Fig. 10 shows an example of a particle detection method performed on a tissue slice target region according to an embodiment. Part (a) of fig. 10 shows an image of a tissue section obtained by a conventional high resolution microscope with an oil immersion lens (100 x/1.4NA objective with z-stack). As shown in part (a) of fig. 10, the particles of interest (mRNA molecules) are shown as white spots across the tissue section. Part (b) of fig. 10 shows a reconstructed image of the target region obtained by performing the SAO reconstruction process on the original image. The original image is obtained by irradiating the target area with a series of 12 different structured irradiation patterns formed by interference of laser beams. As shown in part (b) of fig. 10, the particles appear as white spots on the image. Part (c) of fig. 10 shows the following image of part (b) of fig. 10: the image is annotated (labeled "x") with regions associated with the positive reconstruction estimates as described in connection with the method of fig. 9, in particular with pixels having intensity values above a threshold. As shown in part (c) of fig. 10, more annotations appear on the target region than particles due to false positive errors. Part (d) of fig. 10 shows the following image of part (b) of fig. 10: this image annotation (marked with an "x") has regions associated with the estimates being combined, which are generated by comparing the annotated regions in part (c) of fig. 10 with the corresponding modulation estimates and retaining annotations only for those regions having positive modulation estimates, as described in connection with the method of fig. 9. The modulation estimate is generated by: as shown in table 1, 4 sub-modulation scores MS1, MS2, MS3, MS4 are each determined for raw intensity values from a subset of the raw image with the same spatial frequency, respectively; and the final modulation score is determined as MS1 × MS2 × MS3 × MS 4. Each sub-modulation score is determined as a standard deviation normalized by the mean of a subset of the raw intensity values. As shown in part (d) of fig. 10, the detection system is able to detect particles with improved accuracy and lower false positive error rate by validating the reconstructed estimate with the modulation estimate.

Fig. 11 shows an example of a particle detection method performed on a single molecule mRNA FISH (fluorescence in situ hybridization) sample according to an embodiment. Part (a) of fig. 11 shows a first annotation image of U2OS cells on a 96-well plate labeled EGFR (epidermal growth factor receptor) mRNA by performing the particle detection method of fig. 9 for reducing false positive error rate. The mRNA was labeled by all from LGC Biosearch TechnologiesmRNA FISH method, use and670 dye conjugated probe. The image shown corresponds to one field of view (0.33mm x0.33mm) of the SAO device according to embodiments described herein. In this example, a total of 146 cells and 3492 mRNA spots were detected from one field of view. Part (b) of fig. 11 shows an enlarged view of the image in part (a) of fig. 11. The detected mRNA is shown as individual points and as shown by example 1102A, the lines show the boundaries of the cell nuclei detected by the cell nucleus segmentation software. Part (c) of figure 11 shows a second annotated image of a region from mouse brain tissue labeled as mRNA using a probe conjugated to Cy5 dye. The names of gene targets and specific chemical methods for labeling mRNA are unknown. The image shown corresponds to one field of view (0.33mm x0.33mm) of the SAO device according to embodiments herein. In this example, a total of 275 cells and 36,346 mRNA spots were detected from one field of view, as shown by example 1102B. Part (d) of fig. 11 shows an enlarged view of the image in part (c) of fig. 11. Mouse brain tissue showed significantly higher cell density and mRNA spots compared to cultured cells. Furthermore, the background signal inherent in tissues makes imaging and spot detection relatively more challenging compared to cultured cells. Table 2 summarizes the results of parts (a), (b), (c) and (d) of fig. 11. The example shown in fig. 11 illustrates that the particle detection methods described in connection with fig. 8-9 are capable of detecting the presence of mRNA fragments in tissue samples or well plates.

TABLE 2 comparison of dot count results for two different types of single mRNA FISH samples

Type of sample	U2OS cells on 96-well plates	Mouse brain tissue
			Gene target	EGFR	N/A
Fluorescent dyes	Q670	Cy5
			Number of cells per FOV	146	275
Number of spots per FOV	3,492	36,341
			Average number of spots per cell	24	132
Calculating time	60 seconds	198 seconds

Those skilled in the art will also appreciate additional alternative structural and functional designs for methods and apparatus for synthetic aperture optics upon reading this disclosure. Thus, while particular embodiments and applications of the present invention have been illustrated and described, it is to be understood that the invention is not limited to the precise construction and components disclosed herein and that various modifications, changes, and variations which will be apparent to those skilled in the art may be made in the arrangement, operation, and details of the method and apparatus of the present invention disclosed herein without departing from the spirit and scope of the invention as defined in the appended claims.