阅读说明：本技术 使用栅格光图案的深度感测 (Depth sensing using grid light patterns ) 是由 纳达夫·格罗辛格 里卡多·加西亚 罗伯特·哈斯本 尼韦·坎托尔 尼他耶·罗马诺 于 2018-11-20 设计创作，主要内容包括：在一个实施例中,一种系统包括：第一投影仪,其被配置为投影具有一个或更多个第一投影照明特征的第一投影图案；第二投影仪,其被配置为投影具有一个或更多个第二投影照明特征的第二投影图案；相机,其被配置为捕获包括分别与第一投影图案和第二投影图案的反射对应的第一检测图案和第二检测图案的图像；以及一个或更多个处理器,其被配置为：通过将第一检测图案和第二检测图案的检测到的照明特征与第一投影照明特征和第二投影照明特征进行比较,来识别图像中与第一投影图案和第二投影图案中的至少一个投影图案中的投影点对应的检测到的点,并且基于投影点、检测到的点以及相机和至少一个投影仪之间的相对定位来计算与检测到的点相关联的深度。(In one embodiment, a system comprises: a first projector configured to project a first projection pattern having one or more first projection illumination features; a second projector configured to project a second projection pattern having one or more second projected illumination features; a camera configured to capture an image including first and second detection patterns corresponding to reflections of the first and second projection patterns, respectively; and one or more processors configured to: the method further includes identifying a detected point in the image corresponding to a projected point in at least one of the first and second projected patterns by comparing the detected illumination features of the first and second detected patterns to the first and second projected illumination features, and calculating a depth associated with the detected point based on the projected point, the detected point, and a relative positioning between the camera and the at least one projector.)

1. A system, comprising:

a first projector configured to project a first projection pattern having one or more first projected illumination features;

a second projector configured to project a second projected pattern having one or more second projected illumination features, wherein the first projected pattern intersects the second projected pattern;

a camera configured to capture an image including first and second detection patterns corresponding to reflections of the first and second projection patterns, respectively, wherein the camera is located at a first baseline distance from the first projector and at a second baseline distance from the second projector; and

one or more processors configured to:

identifying a detected point in the image corresponding to a projected point in at least one of the first and second projected patterns by comparing the detected illumination features of the first and second detected patterns to the first and second projected illumination features; and

calculating a depth associated with the detected point based on the projected point, the detected point, and a relative positioning between the camera and at least one of the first projector and the second projector.

2. The system of claim 1, wherein the first projection pattern comprises a plurality of first projection lines and the second projection pattern comprises a plurality of second projection lines, and

wherein the detected illumination feature comprises a plurality of first reflected lines and a plurality of second reflected lines intersecting the first reflected lines, and the first reflected lines and the second reflected lines are based on reflections of the first projected lines and the second projected lines from a surface of an object.

3. The system of claim 2, wherein to identify detected points in the image that correspond to projected points in one or more of the first and second projected patterns, the processor is further configured to:

identifying a reflection intersection at which one of the first reflection lines intersects one of the second reflection lines, wherein the reflection intersection is associated with one or more reflection intersection features;

identifying a projected intersection corresponding to the reflected intersection, wherein the projected intersection is located at an intersection of one of the first projected lines and one of the second projected lines, wherein the projected intersection is associated with one or more projected intersection features determined based on: (1) the one or more first projected illumination features associated with the first projection pattern, and (2) the one or more second projected illumination features associated with the second projection pattern, and wherein one or more of the projected junction features match one or more of the reflective junction features.

4. The system of claim 3, wherein to calculate the depth of the detected point, the processor is further configured to determine a depth associated with the reflection intersection using triangulation based on: the camera, at least one of the projectors, the reflective junction, and the projection junction corresponding to the reflective junction.

5. The system of claim 3 or 4, wherein at least one of the projected junction features of the projected junction matches at least one of the reflective junction features when:

the value of the reflecting junction feature is equal to the value of the projecting junction feature, or

The difference between the value of the reflection junction feature and the value of the projection junction feature is less than a threshold feature value;

optionally, wherein the threshold feature value comprises a predetermined value or a percentage of the value of the projected intersection feature.

6. The system of any of claims 3 to 5, wherein the projected junction feature and the reflected junction feature each comprise a light intensity;

optionally, wherein at least one of the projected junction features of the projected junction matches at least one of the reflective junction features when the following conditions are met:

(1) the intensity of at least one of the first reflected lines corresponds to the intensity of at least one of the first projected lines, or

(2) The intensity of at least one of the second reflected lines corresponds to the intensity of at least one of the second projected lines.

7. The system of any of claims 3 to 6, wherein the projected junction feature and the reflected junction feature each comprise a distance between lines;

optionally, wherein at least one of the projected junction features matches at least one of the reflective junction features when the following conditions are met:

(1) a distance between the first reflected line and another one of the first reflected lines corresponds to a distance between the first projected line and another one of the first projected lines corresponding to the another one of the first reflected lines, or

(2) A distance between the second reflected line and another one of the second reflected lines corresponds to a distance between the second projected line and another one of the second projected lines corresponding to the another one of the second reflected lines.

8. The system of any one of claims 3 to 7, wherein:

the reflective junction feature comprises one or more first reflection line features associated with one of the first reflection lines,

one or more of the first projected illumination features are associated with one of the first projection lines, and

at least one of the projected junction features matches at least one of the reflected junction features when one or more of the first projected illumination features associated with one of the first projected lines matches one or more of the first reflected line features.

9. The system of any one of claims 3 to 8, wherein:

the reflective junction feature comprises one or more second reflection line features associated with one of the second reflection lines,

one or more of the second projected illumination features are associated with the second projection line, and

at least one of the projected junction features matches at least one of the reflected junction features when one or more of the second projected illumination features associated with the second projected line matches one or more of the second reflected line features.

10. The system of any of claims 3 to 9, wherein the processor is further configured to:

detecting movement of a device or object;

in response to detecting the movement, identifying a new location of the reflective intersection, wherein the new location of the reflective intersection is on or near a epipolar line that intersects a previous location of the reflective intersection; and

determining a depth associated with the new location of the reflective intersection using triangulation based on the camera, at least one of the projectors, the location of the projection intersection, and the new location of the reflective intersection;

optionally, wherein the reflection junction is on or near the epipolar line when the reflection junction is within a threshold distance of the epipolar line.

11. The system of any of claims 3 to 10, wherein the processor is further configured to:

propagating the reflective junction feature to one or more reflective junctions adjacent to the reflective junction.

12. The system of any of claims 3 to 11, wherein a confidence value is associated with the identified correspondence between the projected intersection and the reflective intersection, and the confidence value is based on how many of the projected intersection features match one or more of the reflective intersection features;

optionally, wherein the processor is further configured to:

identifying neighboring projective junctions that are adjacent to the projective junction in a particular direction; and

propagating the identification of the neighboring projected junction to a neighboring reflective junction that is adjacent to the reflective junction in the particular direction;

optionally, wherein the identification of the adjacent projected junction and the confidence value are propagated to the adjacent reflected junction when the confidence value satisfies a threshold confidence condition.

13. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

identifying a detected point in the image by comparing one or more detected illumination features of the first and second detected patterns to one or more first and second projected illumination features, wherein the detected point corresponds to a projected point in at least one of the first and second projected patterns,

wherein a camera is configured to capture the image, the image including the first detection pattern and the second detection pattern, and the first detection pattern and the second detection pattern correspond to reflections of the first projection pattern and the second projection pattern, respectively; and

calculating a depth associated with the detected point based on the projected point, the detected point, and a relative positioning between the camera and at least one of a first projector and a second projector,

wherein the first projector is configured to project the first projected pattern having the first projected illumination feature,

wherein a second projector is configured to project the second projection pattern having the second projected illumination feature, wherein the first projection pattern intersects the second projection pattern, and

wherein the camera is located at a first baseline distance from the first projector and at a second baseline distance from the second projector.

14. A method, comprising:

identifying, by the computing device, a detected point in the image by comparing one or more detected illumination features of the first and second detected patterns to one or more first and second projected illumination features, wherein the detected point corresponds to a projected point in at least one of the first and second projected patterns,

wherein a camera is configured to capture the image, the image including the first detection pattern and the second detection pattern, and the first detection pattern and the second detection pattern correspond to reflections of the first projection pattern and the second projection pattern, respectively; and

calculating, by the computing device, a depth associated with the detected point based on the projected point, the detected point, and a relative positioning between the camera and at least one of a first projector and a second projector,

wherein a first projector is configured to project the first projected pattern, the first projected pattern having the first projected illumination feature,

wherein a second projector is configured to project the second projection pattern having the second projected illumination feature, wherein the first projection pattern intersects the second projection pattern, an

Wherein the camera is located at a first baseline distance from the first projector and at a second baseline distance from the second projector.

Technical Field

The present disclosure relates generally to structured light, and more particularly to systems and methods for determining depth using a structured light pattern.

Background

Augmented Reality (AR) systems may augment a real-world environment with sensory effects, providing an enhanced experience for a user. AR systems may use devices such as cameras and head mounted displays to integrate virtual objects with the real world environment. Components of the AR system (e.g., a head mounted display) may be connected and/or networked with a host computer system, a mobile device or computing system, or any other hardware platform capable of providing AR content to one or more users.

One challenge in AR systems is to accurately determine the positioning of physical objects in a real-world environment so that virtual objects can be displayed, and may provide haptic feedback to a user based on the location of the physical and virtual objects. Depth sensing may be implemented in an AR system to determine the location of physical objects to provide a mapping of the real-world environment. The distance from a sensor (such as a camera) to an object in the scene may be determined using structured light scanning, which includes projecting a light pattern onto the scene. The sensor may be used to capture reflections of the pattern and the distance to the object may be determined by analyzing the reflections. The reflections may be distorted by the shape of objects in the scene, and the distance to a point on the surface of an object in the scene may be calculated based on the pattern distortion detected by the sensor. The calculated distances may be used to construct a depth map that may associate a distance with each pixel in an image of the scene to represent a three-dimensional surface of an object in the scene. Depth maps may be used in AR applications and may be generated by devices with light projectors and cameras, such as mobile phones, AR glasses, AR headsets, and the like.

Summary of the specific embodiments

Certain embodiments described herein relate to depth sensing using structured light. In various embodiments, the depth sensing system may include one or more projectors and detectors. The projector may emit structured light of a known pattern into the environment, and the detector may detect reflections of the emitted light from objects in the environment. The reflection may be used to compute depth information for objects in the environment. For example, a depth map representing three-dimensional features of objects in the environment may be generated by triangulating the emitted light and the detected reflected light.

In one embodiment, a depth sensing system having a camera and two projectors may project a grid light pattern into an environment and calculate a location of an object in the environment based on reflections of the grid light pattern. The grid light pattern may have one or more projected lighting (lighting) features, such as a particular lighting intensity or line spacing, and the camera may capture an image that includes the detected pattern. The system may identify a detected point (detected point) in the image corresponding to a projected point in the at least one projected pattern by comparing the detected illumination features of the detected pattern with the projected illumination features, and calculate a depth associated with the detected point based on the projected point, the detected point, and a relative positioning between the camera and the at least one projector.

Embodiments of the invention may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to the user, and may include, for example, Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), hybrid reality (hybrid reality), or some combination and/or derivative thereof. The artificial reality content may include fully generated content or generated content combined with captured content (e.g., real-world photos). The artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to a viewer). Additionally, in some embodiments, the artificial reality may be associated with an application, product, accessory, service, or some combination thereof, for example, for creating content in the artificial reality and/or for use in the artificial reality (e.g., performing an activity in the artificial reality). An artificial reality system that provides artificial reality content may be implemented on a variety of platforms, including a Head Mounted Display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited to these embodiments. Particular embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed above. Embodiments in accordance with the present invention are specifically disclosed in the accompanying claims directed to systems, storage media, and methods, wherein any feature mentioned in one claim category (e.g., systems) may also be claimed in another claim category (e.g., storage media, methods, and computer program products). The dependencies or references back in the appended claims are chosen for formal reasons only. However, any subject matter resulting from an intentional back-reference to any previous claim (in particular, multiple references) may also be claimed, such that any combination of a claim and its features is disclosed and may be claimed, irrespective of the dependencies chosen in the appended claims. The subject matter which can be claimed comprises not only the combination of features as set forth in the appended claims, but also any other combination of features in the claims, wherein each feature mentioned in the claims can be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein may be claimed in a separate claim and/or in any combination with any of the embodiments or features described or depicted herein or in any combination with any of the features of the appended claims.

In one embodiment, a system may include:

a first projector configured to project a first projection pattern having one or more first projection illumination features;

a second projector configured to project a second projected pattern having one or more second projected illumination features, wherein the first projected pattern intersects the second projected pattern;

a camera configured to capture an image including first and second detection patterns corresponding to reflections of the first and second projection patterns, respectively, wherein the camera is located at a first baseline distance from the first projector and at a second baseline distance from the second projector; and

one or more processors configured to:

identifying a detected point in the image corresponding to a projected point of at least one of the first and second projected patterns by comparing the detected illumination features of the first and second detected patterns to the first and second projected illumination features; and

based on the projected points, the detected points, and a relative positioning between the camera and at least one of the first projector and the second projector, a depth associated with the detected points is calculated.

The first projection pattern may include a plurality of first projection lines, and the second projection pattern may include a plurality of second projection lines, and

the detected illumination feature may include a plurality of first reflected lines and a plurality of second reflected lines intersecting the first reflected lines, and the first reflected lines and the second reflected lines may be based on reflections of the first projected lines and the second projected lines from the surface of the object.

To identify detected points in the image that correspond to projected points in one or more of the first projected pattern and the second projected pattern, the processor may be configured to:

identifying a reflection intersection at which one of the first reflection lines intersects one of the second reflection lines, wherein the reflection intersection is associated with one or more reflection intersection features;

identifying a projected intersection corresponding to the reflective intersection, wherein the projected intersection is located at an intersection of one of the first projection lines and one of the second projection lines, wherein the projected intersection is associated with one or more projected intersection features determined based on (1) one or more first projected illumination features associated with the first projection pattern and (2) one or more second projected illumination features associated with the second projection pattern, and wherein one or more of the projected intersection features match one or more of the reflective intersection features.

To calculate the depth of the detected point, the processor may be configured to determine a depth associated with the reflective intersection based on the at least one of the camera, the projector, the reflective intersection, and the projected intersection corresponding to the reflective intersection using triangulation.

At least one of the projected junction features of the projected junction matches at least one of the reflected junction features when the following conditions are met: :

the value of the reflecting junction feature is equal to the value of the projecting junction feature, or

The difference between the value of the reflected junction feature and the value of the projected junction feature is less than a threshold feature value.

The threshold feature value may comprise a predetermined value or percentage of the projected intersection feature value.

The projected junction feature and the reflected junction feature may each include an intensity of light.

At least one of the projected junction features of the projected junction may match at least one of the reflected junction features if:

(1) the intensity of at least one of the first reflected lines corresponds to the intensity of at least one of the first projected lines, or

(2) The intensity of at least one of the second reflected lines corresponds to the intensity of at least one of the second projected lines.

The projected junction feature and the reflected junction feature may each comprise a distance between lines.

At least one of the projected junction features may match at least one of the reflected junction features in the following cases:

(1) the distance between the first reflected line and another one of the first reflected lines corresponds to the distance between the first projected line and another one of the first projected lines corresponding to the another one of the first reflected lines, or

(2) The distance between the second reflected line and another one of the second reflected lines corresponds to the distance between the second projected line and another one of the second projected lines corresponding to the another one of the second reflected lines.

The reflective intersection point features may include one or more first reflection line features associated with one of the first reflection lines,

one or more of the first projected illumination features can be associated with one of the first projection lines, and

at least one of the projected junction features may match at least one of the reflected junction features when one or more of the first projected illumination features associated with one of the first projected lines match one or more of the first reflected line features.

The reflective intersection point features may include one or more second reflection line features associated with one of the second reflection lines,

one or more of the second projected illumination features can be associated with a second projection line, an

At least one of the projected junction features may match at least one of the reflected junction features when one or more of the second projected illumination features associated with the second projected line match one or more of the second reflected line features.

The processor may be configured to:

detecting movement of a device or object;

in response to detecting the movement, identifying a new location of the reflective intersection, wherein the new location of the reflective intersection is on or near a epipolar line that intersects the previous location of the reflective intersection; and

triangulation is used to determine a depth associated with the new location of the reflective intersection based on at least one of the camera, the projector, the location of the projected intersection, and the new location of the reflective intersection.

When the reflection junction is within a threshold distance of the epipolar line, the reflection junction may be on or near the epipolar line.

The processor may be configured to:

the reflective junction feature is propagated to one or more reflective junctions adjacent to the reflective junction.

The confidence value may be associated with the identified correspondence between the projected intersection and the reflective intersection, and the confidence value may be based on how many projected intersection features match one or more of the reflective intersection features.

The processor may be configured to:

identifying neighboring projection intersections adjacent to the projection intersections in a particular direction; and

the identification of neighboring projection junctions is propagated to neighboring reflection junctions that are adjacent to the reflection junctions in a particular direction.

When the confidence values satisfy a threshold confidence condition, the identity and confidence values of adjacent projection junctions may be propagated to adjacent reflection junctions.

In an embodiment, one or more computer-readable non-transitory storage media may embody software that, when executed, is operable to:

identifying a detected point in the image by comparing one or more detected illumination features of the first and second detected patterns to one or more first and second projected illumination features, wherein the detected point corresponds to a projected point in at least one of the first and second projected patterns,

wherein the camera is configured to capture an image, the image including a first detection pattern and a second detection pattern, and the first detection pattern and the second detection pattern correspond to reflections of the first projection pattern and the second projection pattern, respectively; and

calculating a depth associated with the detected point based on the projected point, the detected point and a relative positioning between the camera and at least one of the first projector and the second projector,

wherein the first projector is configured to project a first projected pattern, the first projected pattern having a first projected illumination feature,

wherein the second projector is configured to project a second projection pattern having a second projection illumination characteristic, wherein the first projection pattern intersects the second projection pattern, an

Wherein the camera is located at a first baseline distance from the first projector and at a second baseline distance from the second projector.

In one embodiment, a method may comprise:

identifying, by the computing device, a detected point in the image by comparing one or more detected illumination features of the first detected pattern and the second detected pattern to one or more first projected illumination features and second projected illumination features, wherein the detected point corresponds to a projected point in at least one of the first projected pattern and the second projected pattern,

wherein the camera is configured to capture an image, the image including a first detection pattern and a second detection pattern, and the first detection pattern and the second detection pattern correspond to reflections of the first projection pattern and the second projection pattern, respectively; and

calculating, by the computing device, a depth associated with the detected point based on the projected point, the detected point, and a relative positioning between the camera and at least one of the first projector and the second projector,

wherein the first projector is configured to project a first projected pattern, the first projected pattern having a first projected illumination feature,

wherein the second projector is configured to project a second projection pattern having a second projection illumination characteristic, wherein the first projection pattern intersects the second projection pattern, an

Wherein the camera is located at a first baseline distance from the first projector and at a second baseline distance from the second projector.

In another embodiment of the invention, one or more computer-readable non-transitory storage media embody software that, when executed, is operable to execute in a system according to the invention or any of the above-mentioned embodiments.

In another embodiment of the invention, a computer-implemented method uses a system according to the invention or any of the above-mentioned embodiments.

In another embodiment of the invention, a computer program product, preferably comprising a computer readable non-transitory storage medium, is used in a system according to the invention or any of the above mentioned embodiments.

Brief Description of Drawings

FIG. 1 illustrates an example network environment associated with a computing system.

Fig. 2A shows an example of triangulation calculations.

Fig. 2B shows an example of triangulation calculations using two projectors.

Fig. 3 shows an example of a projector.

FIG. 4A illustrates an example detection event.

FIG. 4B illustrates an example detection event for a surface anomaly.

FIG. 5 illustrates an example head mounted display.

Fig. 6 shows an example grid light pattern with varying line spacing.

Fig. 7 shows an example grid light pattern with varying line intensities.

FIG. 8 illustrates an example grid light pattern with a varying line pattern.

Fig. 9 illustrates the movement of the grid light pattern and the corresponding epipolar line.

FIG. 10 illustrates an example method of determining depth using a grid light pattern.

FIG. 11 shows an example temporal illumination pattern.

FIG. 12 illustrates an example method of determining depth using temporal patterns.

FIG. 13A illustrates an example illumination (attenuation) pattern of an illuminator for determining depth from different distances.

Fig. 13B shows an example gaussian illumination pattern viewed from different distances.

Fig. 13C shows an example projection pattern viewed from different distances.

13D and 13E illustrate example illumination patterns for a two-dimensional illuminator for determining depth from different distances.

Fig. 14 illustrates an example method for activating or deactivating one or more light emitters of a luminaire to determine depth from different distances.

Fig. 15 shows an example of reducing the grid light pattern density to reduce power consumption.

FIG. 16 illustrates an example method for reducing the grid light pattern density.

Fig. 17 shows an example of dividing a grid light pattern into a plurality of sections to reduce power consumption.

FIG. 18 illustrates an example method for dividing a grid light pattern into portions to be projected at different times.

FIG. 19 shows an example computer system.

Description of example embodiments

FIG. 1 illustrates an example network environment 100 associated with a computing system 160. Network environment 100 includes a user 101, a client system 130 (e.g., a head mounted display), a computing system 160, and a third party system 170 interconnected by a network 110. Although fig. 1 illustrates a particular arrangement of user 101, client system 130, computing system 160, third-party system 170, and network 110, this disclosure contemplates any suitable arrangement of user 101, client system 130, computing system 160, third-party system 170, and network 110. By way of example and not by way of limitation, two or more of client system 130, computing system 160, and third-party system 170 may be directly connected to each other, bypassing network 110. As another example, two or more of client system 130, computing system 160, and third-party system 170 may be wholly or partially physically or logically co-located with one another. Moreover, although fig. 1 illustrates a particular number of users 101, client systems 130, computing systems 160, third party systems 170, and networks 110, this disclosure contemplates any suitable number of users 101, client systems 130, computing systems 160, third party systems 170, and networks 110. By way of example and not limitation, network environment 100 may include a plurality of users 101, client systems 130, computing systems 160, third-party systems 170, and networks 110.

In particular embodiments, user 101 may be an individual (e.g., a human user) interacting with computing system 160 or communicating through computing system 160. In particular embodiments, computing system 160 may generate, store, receive, and transmit data related to generating an artificial reality environment, including for example, but not limited to, visual data, audio data, haptic data, and the like. Computing system 160 may be accessed by other components of network environment 100 directly or via network 110. In particular embodiments, the third party system 170 may be a structured light projector and/or detector, a wall mounted speaker system, a motion sensor system, a haptic actuator, and/or the like. The third party system 170 may be accessed by other components of the network environment 100 directly or via the network 110. In particular embodiments, one or more client systems 130 (such as a head mounted display) may access, send data to, and receive data from computing system 160 and/or third party system 170. Client system 130 may access computing system 160 or third-party system 170 directly, via network 110, or via a third-party system. By way of example and not limitation, client system 130 may access third party system 170 via computing system 160. Client system 130 may be any suitable computing device, such as a head-mounted display, an augmented/virtual reality device, and so forth.

The present disclosure contemplates any suitable network 110. By way of example and not limitation, one or more portions of network 110 may include an ad hoc network, an intranet, an extranet, a Virtual Private Network (VPN), a Local Area Network (LAN), a Wireless LAN (WLAN), a Wide Area Network (WAN), a Wireless WAN (WWAN), a Metropolitan Area Network (MAN), a portion of the internet, a portion of the Public Switched Telephone Network (PSTN), a cellular telephone network, or a combination of two or more of these. The network 110 may include one or more networks 110.

Links 150 may connect client system 130, computing system 160, and third-party system 170 to communication network 110 or to each other. The present disclosure contemplates any suitable links 150. In particular embodiments, one or more links 150 include one or more wired (such as, for example, Digital Subscriber Line (DSL) or cable-based data service interface specification (DOCSIS)) links, wireless (such as, for example, Wi-Fi, bluetooth, or Worldwide Interoperability for Microwave Access (WiMAX)) links, or optical (such as, for example, Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links. In particular embodiments, one or more links 150 each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the internet, a portion of the PSTN, a cellular technology-based network, a satellite communication technology-based network, another link 150, or a combination of two or more such links 150. Links 150 need not necessarily be the same throughout network environment 100. One or more first links 150 may differ in one or more respects from one or more second links 150.

Fig. 2A shows an example of triangulation calculations. For simplicity, the figure shows triangulation as a two-dimensional arrangement, but one of ordinary skill in the art will recognize that the same concepts may be applied to three dimensions. System environment 200 includes a projector 210, and projector 210 may be represented by a projector location 215 (e.g., a dot representation of a projector lens) for triangulation purposes. Conceptually, the light emission 240 projects from the projector location 215 (shown as a single beam for simplicity, but the emission may be multiple beams or planes) and intersects the object 260 at the reflection point 230. In a particular embodiment, system environment 200 also includes two detectors 220-1 and 220-2 (collectively detectors 220) configured to have substantially overlapping fields of view. For purposes of triangulation calculations, detectors 220-1 and 220-2 may be represented by detector location 250-1 and detector location 250-2, respectively (collectively detector locations 250). In a particular embodiment, detector location 250 may be a pinhole lens conceptually defined for detector 220.

Projector 210 may include any type of device capable of emitting light. For example, the projector 210 may include a laser emitter, such as a Vertical Cavity Surface Emitting Laser (VCSEL). Functionally, the projector 210 emits the light emission 240 into the system environment 200. The light emission 240 may have various characteristics such as beam size, beam shape, beam intensity, beam wavelength, and the like. In particular embodiments, the light emission 240 may be a coherent, collimated light emission (e.g., a laser). The light emission 240 may pass through the system environment 200 in the form of a line, grid, circle, etc. Further, the light emission 240 may include multiple instances of one or more forms (e.g., two or more lines, two or more torus shapes, two or more grids, etc.). In various embodiments, the light emission 240 interacts with an object (e.g., object 260) located in the path of the light emission 240. The object may be, for example, but not limited to, a wall, a chair, a person, an animal, a tree, a plant, a surface, a mesh, and so forth. For example, the object 260 may reflect and/or absorb some or all of the light emission 240. In particular, line c₁May represent a portion of the light emission 240 reflected by the object 260 that is incident on the detector 220-1. Similarly, line c₂May represent light reflected by object 260A portion of emission 240 that is incident on detector 220-2.

Detectors 220-1 and 220-2 may be any type of device capable of detecting light. For example, one or both of detectors 220-1 and 220-2 may be an internal and external light detector or camera mounted on a moving platform, an array sensor (e.g., a linear array sensor, a planar array sensor, a circular array sensor, etc.), and/or the like. In various embodiments, detector 220-1 and detector 220-2 may be unfiltered. Thus, detector 220-1 and detector 220-2 may exhibit similar detection sensitivity for various wavelengths of light, without exhibiting preferential detection sensitivity for a wavelength band of selected light (such as the wavelength of light emission 240). In operation, detector 220-1 detects light entering a detector aperture (not shown). In some embodiments, detectors 220-1 and 220-2 may each include one or more lenses that focus the light. For example, but not by way of limitation, the lens may be along line c₁The propagating light is focused to the image sensor of detector 220-1. In various embodiments, detector 220-1 transmits the location of the detected light to client system 130 (e.g., a head mounted display) for determining the location of the object in system environment 200.

In various embodiments, a processor in the head mounted display executes an AR application (stored in any or all of client system 130, computing system 160, or third party system 170) to process data received from detector 220. For example, the AR application may analyze the input received from the detector 220-1 to identify a signal corresponding to the reflection point 230 of the light emission 240. For example, the AR application may filter out signals corresponding to light below a preset threshold intensity. Once the AR application identifies a candidate detector signal that may correspond to reflection point 230, the AR application may verify that it actually corresponds to reflection point 230 and calculate the depth of reflection point 230 on object 260 using triangulation techniques.

The geometric relationship between the projector 210, each detector 220, and the reflection point 230 on the object 260 may be used to form the basis of the triangulation calculations. Line a₁Indicating projector position 215 and detector 2A known baseline distance between detector locations 250-1 of 20-1. Similarly, line a₂Representing a known baseline distance between projector position 215 and detector position 250-2 of detector 220-2. Further, line b represents the path of the light emission 240 emitted from the projector position 215. The light emission 240 may be reflected from the object 260 at the reflection point 230. Line c₁May represent the path of reflected light, line c, propagating from reflection point 230 to detector location 250-1₂May represent the path of reflected light traveling from reflection point 230 to detector location 250-2.

Thus, a first triangle (referred to herein as "triangle 1") may be defined by line a₁Line c₁And line b, forming line a₁And line c₁Angle alpha therebetween₁Line c₁And the angle beta between the line b₁And line b and line a₁Angle theta therebetween₁. Line a, as described above₁Is known because it can be pre-calculated based on the fixed detector position 250-1 and the fixed projector position 215. In certain embodiments, the angle θ₁May also be known because of the fixed relative positioning (by baseline a) of the light emission 240 with respect to the detector position 250-1 and the projector position 215₁Representation) is known. Although the angle θ is shown in FIG. 2A₁Appears to be a right angle (i.e., a 90 degree angle), but one of ordinary skill in the art will understand that θ is₁It is not limited thereto and may be any other angle. Finally, the angle α₁May be calculated based on the position of the reflection point 230 in the field of view of the detector 250-1. The point at which reflected point 230 is captured by the sensor of detector 220-1 corresponds to a point in an image plane defined by the field of view of detector 220-1. Conceptually, the position at which the point appears in the image plane is line c₁The position of intersection with the image plane. The intersection point in the image plane and the known center of the image plane, and the known relationship between the center of the image plane and the detector location 250-1 (e.g., the distance between the center of the image plane and the detector location 250-1, and the angle between the image plane and the line connecting the detector location 250-1 and the center) may be used to calculate the angle α₁(e.g. inBy triangulation). Once angle alpha₁Side a₁And angle theta₁Determined, the remaining dimensions of triangle 1 may be calculated based on the known geometric properties of the triangle. For example, the length of line b may be calculated to represent a first depth of reflection point 230 from projector position 215.

As previously mentioned, one challenge in depth calculations using emitted light is the need to accurately identify the corresponding reflected light captured by the detector. If a point in the captured image is incorrectly identified as a reflection of the emitted light, the triangulation calculation will be erroneous because the various triangulation assumptions will not hold (e.g., if the incorrectly identified point in the image does not correspond to a reflection point 230, it cannot be used to represent line c₁At the intersection with the image plane, thus angle alpha₁Will be erroneous). One way to help detect the emitted light is to emit light at a particular wavelength (e.g., infrared) and use a detector with a filter tailored to that wavelength. However, as previously mentioned, doing so may not be practical in various applications. For example, if the depth sensing system is to be integrated with a head mounted display with limited resources (e.g., power, real estate, and weight), it may not be feasible or desirable to add a dedicated filtered detector for detecting structured light, particularly because the head mounted display may already have unfiltered cameras for object detection and other sensory purposes.

To facilitate accurate identification of the emitted light, particular embodiments verify that the point is actually a reflection of the emitted light by performing a second depth calculation on the candidate reflection point using information captured by the second unfiltered camera. Referring again to FIG. 2A, a second triangle (referred to herein as "triangle 2") may be defined by line a₂Line c₂And line b, at line a₂And line c₂Form an angle alpha therebetween₂On-line c₂And line b forms an angle beta₂On lines b and a₂Form an angle theta therebetween₂. Like line a₁And angle theta₁Line a due to the known relatively fixed positioning of the projector position 215 and the detector position 250-2 and the trajectory of the emitted light 240₂And angle theta₂May be known. Angle alpha₂May be calculated like a₁As described in further detail above. In particular embodiments, a point in the image captured by detector 220-2 may be identified as corresponding to reflection point 230 captured in the image of detector 220-1. In particular embodiments, any suitable stereo correspondence or matching algorithm may be used to determine the correspondence between points in two captured images. Based on the location of such a point in the image plane defined by the field of view of detector 220-2, along with the known center of the image plane and the known relationship between the center of the image plane and detector location 250-2 (e.g., the distance between the center of the image plane and detector location 250-2 and the angle between the image plane and a line connecting detector location 250-2 and the center), may be used to calculate angle α₂(e.g., by triangulation). Once angle alpha₂Side a₂And angle theta₂Determined, the remaining dimensions of triangle 2 may be calculated based on the known geometric properties of the triangle. For example, the length of line b may be calculated to represent the second depth of reflection point 230 from the projector position. If the second depth of the reflection point 230 calculated using the detector 220-2 is different (e.g., exceeds a predetermined threshold) from the first depth calculated using the detector 220-1, then the reflection point 230 may be deemed not to correspond to the emitted light 240 and may be rejected.

FIG. 2B shows an example of triangulation calculations using two projectors 210-1 and 210-2. For simplicity, the figure shows triangulation as a two-dimensional arrangement, but one of ordinary skill in the art will recognize that the same concepts may be applied to three dimensions. System environment 201 includes two projectors 210-1 and 210-2, which may be represented by projector locations 215-1 and 215-2 (e.g., point representations of the projector lenses) for triangulation purposes. Conceptually, the light emissions 240-1 and 240-2 are projected from respective projector locations 215-1 and 215-2 (shown as a single beam for simplicity, but the emissions may be multiple beams or planes) and intersect the object 260 at respective reflection points 230-1 and 230-2. In a particular embodiment, the system environment 201 also includes a detector 220 having a field of view. For purposes of triangulation calculations, the detector 220 may be represented by a detector position 250. In a particular embodiment, detector location 250 may be a pinhole lens conceptually defined for detector 220. Projectors 210-1 and 210-2 may be devices capable of emitting light, as described above with reference to fig. 2A. The detector 220 may be any type of device capable of detecting light, as described above with reference to fig. 2A.

The geometric relationship between the projector 210, the detector 220, and the reflection point 230 on the object 260 may be used to form the basis of the triangulation calculations. Line a₁Representing a known baseline distance between the projector position 215-1 and the detector position 250. Similarly, line a₂Representing a known baseline distance between projector position 215-2 and detector position 250. Further, line b₁Representing the path of light emission 240-1 emitted from projector location 215-1 and line b2 represents the path of light emission 240-2 emitted from projector location 215-2. The light emissions 240-1 and 240-2 may reflect off of the object 260 at reflection points 230-1 and 230-2, respectively. Line c₃Can represent the path of reflected light propagating from reflection point 230-1 to detector location 250, and line c₄May represent the path of reflected light traveling from reflection point 230-2 toward detector location 250.

Thus, a first triangle (referred to herein as "triangle 1") may be defined by line a₁Line c₃And line b₁To describe, line a₁And line c₃Form an angle alpha therebetween₁Line c₁And line b forms an angle beta₁Line b and line a₁Form an angle theta therebetween₁. Line a, as described above₁Is known because it can be pre-calculated based on the fixed detector position 250 and the fixed projector position 215-1. In certain embodiments, the angle θ₁May also be known because of the fixed relative positioning (by baseline a) of the light emission 240-1 relative to the detector position 250 and the projector position 215-1₁Representation) is known. Finally, the angle α₁May be calculated based on the position of the reflection point 230-1 in the field of view of the detector 250. Reflected point 230-1 is detected by detector 22The point captured by the sensor of 0 corresponds to a point in the image plane defined by the field of view of the detector 220. Conceptually, the position at which the point appears in the image plane is line c₃The position of intersection with the image plane. The intersection point in the image plane and the known center of the image plane, as well as the known relationship between the center of the image plane and the detector location 250-1 (e.g., the distance between the center of the image plane and the detector location 250-1 and the angle between the image plane and the line connecting the detector location 250-1 and the center) may be used to calculate the angle α₁(e.g., by triangulation). Once angle alpha₁Side a₁And angle theta₁Determined, the remaining dimensions of triangle 1 may be calculated based on the known geometric properties of the triangle. For example, line b may be calculated₁To represent a first depth of reflection point 230-1 from projector position 215-1.

Fig. 3 shows an example of the projector 210. System environment 300 includes projector 210, laser emission 340, optical element 350, emission pattern 360, and light projection 370 on object 330. Projector 210 may include any type of device capable of emitting light. For example, the projector 210 may include a laser emitter, such as a Vertical Cavity Surface Emitting Laser (VCSEL). The light emission 340 may have various characteristics such as beam size, beam shape, beam intensity, beam wavelength, and the like. In particular embodiments, the optical emission 340 may be a coherent, collimated optical emission (e.g., a laser). The light emission 340 may propagate through the system environment 300 in the form of a line, grid, circle, or the like. Additionally, the light emission 340 may include multiple instances of one or more forms (e.g., two or more lines, two or more torus shapes, two or more grids, etc.).

In various embodiments, the light emission 340 may pass through one or more optical elements 350 to produce structured light for depth sensing. The optical element 350 may include a diffractive element, a refractive element, and/or a reflective element. The optical element 350 may collimate the light emission 340, may focus the light emission 340, may divide the light emission 340 into a plurality of light beams, may preferentially diffuse the light emission along one or more axes to produce a line and/or grid pattern, may focus and/or diffuse the light emission 340. In particular, the optical element 350 may include a collimator for collimating light, a beam splitter for splitting the light emission 340 into two or more light beams. In addition, optical element 350 may include a line generator, such as a homogenizer with a non-periodic cross-section along one axis, a diffractive optical element with a grating angular spacing smaller than the angular width of the spot, a reflective or refractive surface curved along one dimension, or the like.

In some embodiments, the optical element 350 may modify the light emission 340 to produce structured light having an emission pattern 360 (e.g., one or more lines), the emission pattern 360 propagating through the environment 300. In various embodiments, the emission pattern 360 interacts with physical objects (e.g., 330) in its path. The object may be, for example, but not limited to, a wall, a chair, a person, an animal, a tree, a plant, a surface, a mesh, and so forth. For example, object 330 may reflect and/or absorb some or all of the light emission. When the emission pattern 360 encounters an object in the environment (e.g., object 330), the lines are superimposed on the object 330, producing a light projection 370. Light projection 370 may outline object 330, circumscribe curves, cusps, discontinuities, etc., thereby providing visual cues for textured or otherwise uneven surfaces and/or edges of object 330. Accordingly, the AR application may identify distortions or discontinuities in the light projection 370 compared to the expected emission pattern 360 to determine characteristics of the object 330.

FIG. 4A illustrates an example detection event. System environment 400 includes projector 210, light emission 340, optical element 350, emission pattern 360, object 330, reflected light 410, and image sensor 430 (e.g., of detector 220-1 or detector 220-2 shown in FIG. 2A). Image sensor 430 may include a grid of pixels (also referred to as photosites), one of which is identified as pixel 420-1. Some pixels may detect the reflected light 410. For ease of reference, these pixels are referred to herein as light-reflective pixels (e.g., one of which is identified as light-reflective pixel 422-1).

In various embodiments, the emission pattern 360 forms a light projection on the object 330, which can be detected by the image sensor 430. In particular, the object 330 reflects a portion of the light emission pattern 360 toward the image sensor 430. In various embodiments, object 330 may include a specular and/or diffuse surface. The mirror may preferentially deflect light at a particular angle (e.g., at an angle to the normal of the object surface that mirrors the angle of incidence of the light pattern 360 relative to the normal). The diffusing surface reflects light in multiple directions without preferential reflection at certain angles.

In various embodiments, image sensor 430 may include an array of light-sensitive pixels, also referred to as photosites, such as pixel 420-1. When light from environment 400 encounters a pixel, the pixel may generate a signal. For example, the voltage drop across pixel 420-1 may increase or decrease in proportion to the light intensity received by pixel 420-1. Similarly, when reflected light 410 from object 330 encounters pixel 422-1, the pixel may generate a signal. As shown in fig. 4A, the reflected light 410 is detected by the pixel 422-1. In some embodiments, the intensity of the reflected light 410 may exceed the average intensity of ambient light in the system environment 400. Accordingly, the AR application may apply an intensity discriminator to disambiguate between the reflected light 410 corresponding to the emission pattern 360 and other light sources (referred to herein as parasitic light or parasitic signals). For example, the process may help select candidate pixels that are more likely to correspond to the reflected light 410. In various embodiments, each candidate pixel may undergo the process described above with reference to fig. 2A to verify that the pixel actually corresponds to reflected light 410, and its depth may be calculated based on triangulation. Thus, the AR application may obtain three-dimensional measurements of objects in system environment 400. Additionally, the AR application may continuously generate depth and/or coordinate data from the reflected light 410 as the location of the light projection generated by the light pattern 360 moves along the surface of the object 330 and/or around the system environment 400. The AR application may further compile the generated depth and/or coordinate data to form a three-dimensional depth map of object 330 and/or system environment 400. In a further embodiment, the AR application may resolve the detected coordinates corresponding to the reflected light 410 to sub-pixel accuracy, thereby increasing the detection resolution.

FIG. 4B illustrates an example detection event for a surface anomaly. System environment 405 includes projector 210, light emission 340, optical element 350, emission pattern 360, object 330, reflected light 410, and image sensor 430 (e.g., of detector 220-1 or detector 220-2 shown in FIG. 2A). Image sensor 430 may include a grid of pixels (also referred to as photosites), one of which is identified as pixel 420-1.

In addition, system environment 405 includes an object 490 that partially occludes object 330 and introduces a surface discontinuity between an edge of object 490 and surface 330. The reflected light from object 490 (referred to herein as reflected light 415) strikes certain pixels of image sensor 430, such as pixel 424-1, at locations other than where reflected light 410 from object 330 strikes pixels, such as pixel 422-1, forming a discontinuous pattern. In various embodiments, the characteristics of the reflected light 415 are indicative of surface properties of the object 490. In particular, the AR application may utilize the location, orientation, quantity, intensity, and/or distribution of pixels (e.g., pixel 424-1) that receive reflected light 415 to determine surface properties of object 490 and/or a relationship between object 490 and object 330. For example, as shown, there is a discontinuity between the pixel associated with reflected light 410 (e.g., pixel 422-1) and the pixel associated with reflected light 415 (e.g., pixel 424-1). In addition to the triangulation technique of fig. 2A, the AR application may analyze some or all of the displacement, orientation, quantity, intensity, and/or distribution of pixels associated with reflected light 410 relative to pixels associated with reflected light 415 to characterize and image object 330, object 490, and surface discontinuities between object 330 and object 490. For example, in some embodiments, the pixel (e.g., 424-1) associated with reflected light 415 may be located below, above, to the left, to the right, rotated to the left, rotated to the right, and so on, relative to the pixel (e.g., 422-1) associated with reflected light 410. Additionally or alternatively, the pixels associated with the reflected light 410 may have a different intensity, a different spatial distribution, and/or a different number of pixels relative to the pixels associated with the reflected light 415.

Further, in some embodiments, projector 210 may emit multiple instances (e.g., two or more rows of laser light) of light pattern 360 to object 330 and object 490 to improve the efficiency of the depth mapping (e.g., the depths of multiple regions of the environment may be determined simultaneously). Accordingly, the image sensor 430 may receive reflected light for each instance of the light pattern 360 reflected from objects in the environment 405. The application may analyze the signals from the image sensor 430 to associate each signal with a particular instance of the emitted light pattern 360. The application may match or associate each light reflection pixel (e.g., pixel 422-1 or 424-1 shown in fig. 4B) detected by a detector (e.g., detector 220-1 and/or detector 220-2) with a particular instance of emission pattern 360 reflected from object 330 and/or object 490. In some embodiments, to facilitate the matching process, each instance of the emission pattern 360 may have a unique identification (e.g., each line may have a different intensity, wavelength, temporal or spatial coding, etc.).

In other embodiments, the instances of the emission pattern 360 may have the same illumination characteristics (e.g., the same intensity, color, spacing, etc.) in addition to their relative ordering. This ordering may be used in particular embodiments to match each instance of the emission pattern with the detected illumination pattern. For example, the shot pattern 360 may include 5 lines spaced vertically (and/or horizontally) from each other, oriented, and/or rotated at a constant or variable separation distance and/or rotation angle. An application may analyze the detector signals produced by detector 220-1 and/or detector 220-2 to identify and determine the relative ordering of the reflected light patterns. As one example, the projector may project 5 ordered lines that are parallel to each other and spaced apart to cover the intended field of view of the detector. When the detector detects reflected light from 5 ordered lines, they will remain relatively ordered. In various embodiments, the reflected illumination patterns detected by the detectors may be logically grouped such that reflected illumination patterns belonging to the same group are considered to correspond to the same projection line. In various embodiments, the segments or pixels of the reflected illumination pattern may be grouped based on their relative positioning or other characteristics (spatial separation distance, spatial separation direction, spatial orientation, etc.) with respect to other segments or pixels of the reflected illumination pattern. For example, a segment of a reflected illumination pattern may be classified as belonging to a group if it is closer to that group than any other group. By identifying the set of reflected illumination patterns and their relative ordering, the set of reflected illumination patterns can be matched to the corresponding projection line. For example, the uppermost projected line can be associated with the uppermost group of detected illumination patterns, the second from the top projected line can be associated with the second from the top group of detected illumination patterns, and so on. In various embodiments, pixels belonging to a group may or may not actually correspond to the projection line associated with the group. The embodiment described above with reference to fig. 2A may be used to verify whether the pixel actually corresponds to a projection line. If the pixel does correspond to a projection line, the associated depth value may be determined and used to generate a depth map; otherwise, it may be discarded and not contribute to the depth map.

FIG. 5 illustrates an example head mounted display 510 that can be used to implement embodiments described herein. System environment 500 includes a head mounted display 510 that includes detector 220-1, detector 220-2, projector 210, baseline a₁And a base line a₂. In various embodiments, baseline a₁Is the distance between the projector 210 and the detector 220-1, baseline a₂Is the distance between the projector 210 and the detector 220-2. In addition, projector 210 includes an emitter (e.g., 520-1) and a voltage source (e.g., 530-1).

In various embodiments, detectors 220-1 and 220-2 (referred to herein generally as detectors 220) may be front unfiltered cameras disposed on corners of head mounted display 510. These detectors 220 may be used as inside-outside cameras for position tracking. As shown, the detector 220 may be positioned and oriented on a head mounted display 510 to enable stereoscopic imaging of the system environment 500. In particular, detector 220-1 may be located on the right side of head mounted display 510 (from the perspective of a user wearing display 510) and generate a right image of system environment 500. Further, detector 220-2 may be located to the left of head mounted display 510 and generate a left image of system environment 500.

Accordingly, detector 220 may detect light reflected from objects in system environment 500. When determining the positioning of head mounted display 510 in system environment 500, the AR application may analyze detector data generated by detector 220-1 and detector 220-2. In particular, because each of detectors 220-1 and 220-2 is unfiltered, each detector exhibits similar detection sensitivity for each wavelength of light. To isolate detector signals corresponding to reflected light emitted by projector 210, a computing system associated with head mounted display 510 may perform embodiments described herein (e.g., the triangulation and verification process described with reference to fig. 2A).

Functionally, in some embodiments, each detector 220 may include a frame buffer. The frame buffer stores detection signals generated by the detector over a time interval. A frame is a set of detector signals acquired over a time interval. When the time interval ends, the frame is stored in a frame buffer and the incoming detector signal is stored in a new frame until the next time interval elapses. The frequency at which new frames are generated may be defined as the frame rate. In various embodiments, a higher frame rate enables higher resolution detection of temporal variations in the system environment 500 while producing a higher energy load. On the other hand, a lower frame rate provides lower resolution detection of temporal variations in the system environment 500, but results in a lower energy load. In various embodiments, the AR application may configure each detector 220 to operate at the same frame rate or a different frame rate. Additionally or alternatively, the AR application may dynamically select the frame rate at which each detector 220 operates. The AR application may further increase and/or decrease the frame rate based on one or more characteristics of system environment 500.

In some embodiments, the AR application may configure the detector 220 to operate at a low frame rate. For example, the AR application may configure the detector 220 to operate between 10 and 50 frames per second, such as 30 frames per second (fps). In such an embodiment, the energy load may be low. For example, the energy load may be between, for example, 10 and 50 milliwatts, such as 30 milliwatts. Thus, a smaller power supply may be sufficient to operate the detector 220. Additionally or alternatively, the useful life of a power source (e.g., on or external to the head mounted display 510) may be increased. Further, the bill of materials for each detector 220 may be inexpensive. For example, a bill for material may be between $ 2 and $ 10, such as $ 6.

In various embodiments, projector 210 may include one or more emitters 520 that emit light at one or more wavelengths. In some embodiments, each transmitter or group of transmitters may be individually connected to a power source (e.g., a voltage source). Additionally or alternatively, one or more rows and/or one or more columns in the transmitter 520 may be connected to a power source. Thus, an AR application may control the intensity of light emitted by one or more individual emitters, one or more pairs of emitters, one or more rows of emitters, one or more columns of emitters, one or more groups of emitters, and/or the like. In this way, the AR application may increase or decrease the intensity of the light generated by the projector 210 and produce different emission patterns. The AR application may further control the direction of light emission produced by projector 210 by selectively turning on and/or off one or more emitters 520. For example, when directing the beam to the left, the AR application may turn on the emitter 520 located on the left side of the projector 210 and turn off the emitter 520 located on the right side of the projector 210. Similarly, the AR application may turn on and/or off the emitter 520 to detect beams emitted upward, downward, rightward, leftward, and at oblique angles between rightward, leftward, upward, and downward.

In further embodiments, the AR application may sequentially turn on and/or off the emitters 520 in order to produce light emissions having a particular form (e.g., a line, grid, etc.). The light emission produces light projections in the region of the system environment 500. By turning on and/or off the emitter 520, the AR application may control the portion of the area of the system environment 500 that interacts with the light projection. In particular embodiments, the AR application may configure projector 210 to generate light projections of selected areas of sequential scanning system environment 500. The AR application may select the scan frequency based on various parameters such as the detector frame rate, the number of objects in the selected area, the distance to one or more objects in the selected area, the type of object, the surface characteristics of the one or more objects, the optical noise level of system environment 500, the type of light pattern emitted by projector 210, and so forth.

In various embodiments, the AR application may turn on and/or off the emitter 520 to increase and/or decrease the sparsity and/or density of the projected light pattern. For example, if the AR application determines that the imaged object is near the head mounted display 510, the AR application may decrease the intensity of the light emitted by each emitter 520 and/or turn off one or more emitters 520 to increase the sparsity of the light pattern emitted by the projector 210. Alternatively, if the AR application determines that the object in the system environment 510 is farther away and/or the system environment is a noisy environment, the AR application may be configured to increase the intensity of the light emitted by the emitters 520 (e.g., by increasing the voltage supply to one or more of the emitters 520) and/or increase the density of the light pattern emitted by the projector 510 (e.g., by increasing the number of emitters 520 that generate light).

The head mounted display 510 may be worn by a user. As the user moves within system environment 500, the positioning of head mounted display 510 may change. Accordingly, as the position of head mounted display 510 moves within system environment 500, the AR application may configure projector 210 to continuously, periodically, and/or sporadically emit light into system environment 500. In addition, the AR application may configure the detectors 220-1 and 220-2 to continuously detect ambient light in the system environment 500. The AR application may further continuously analyze data from detector 220-1 and detector 220-2 to separate detector signals corresponding to light generated by projector 210 that is reflected from objects in system environment 500. The AR application may also calculate depth and/or coordinate information based on the isolated detector signals and add the calculated information to the system environment map.

In various embodiments, the projector 210 may be positioned at an angle (e.g., 45 degrees) from the orientation of the head-mounted device 510 such that when the user moves his or her head vertically or horizontally, the light pattern emitted by the detector sweeps the environment 500 both vertically and horizontally. Thus, the AR application may obtain vertical and/or horizontal information about the surface based on vertical or horizontal movement of the user's head.

Further, the AR application may dynamically adjust various characteristics of the light pattern generated by projector 210 based on characteristics of system environment 500. For example, if the head mounted display 510 is positioned near one or more objects, the AR application may reduce the output intensity of the projector 210. However, if head mounted display 510 is moved away from the location of the objects in system environment 500, the AR application may configure projector 210 to increase the output intensity. Further, if the AR application determines how to modify the projector 210 and detector 220, the current system environment map may be viewed. If the AR application determines that the area of the system environment 500 is well mapped, the AR application may reduce the frame rate of the detector 220. However, if the AR application determines that an unmapped object is present in system environment 500 and/or that the location of a mapped object in system environment 500 has changed, the AR application may increase the frame rate of one or more of detectors 220-1 and 220-2.

In various embodiments, the AR application may dynamically adjust the intensity of emissions produced by projector 210 based on the amount of light detectable by living objects (e.g., people, animals, etc.) located in system environment 500. In particular, AR applications may reduce the emission intensity and/or increase the sparsity of the emission pattern to keep the potential light detection intensity below a threshold intensity level. For example, if the AR application implements a dense laser pattern, the AR application may reduce the output emission intensity to keep the potential light detection intensity below a threshold intensity level. Alternatively, in various embodiments, if the AR application implements a sparse light pattern, the AR application may implement a higher output intensity for imaging distant objects as long as the potential light detection intensity remains below the threshold intensity level. Although the above description refers to two detectors, this reference is non-limiting as one detector and/or three or more detectors are within the scope of the present disclosure.

In particular embodiments, the grid light pattern may be projected onto the object and the reflection of the pattern may be captured using a camera. Triangulation techniques may be used to determine the distance to the object based on captured pattern reflections, where features of the pattern are used to match detected points in the pattern to corresponding projected points in the pattern.

Fig. 6 shows example grid light patterns 604, 606 with varying line spacing. The projected pattern 604 may be projected into the scene 600 by one or more projectors 210-1, 210-2, and the projectors 210-1, 210-2 may be mounted on a structure or device such as a head-mounted apparatus or head-mounted display. Projectors 210-1, 210-2 may each be mounted at a fixed baseline distance from the camera. Projector 210-11 may project light in the form of lines 620a-624a, and projector 210-2 may project light in the form of lines 610a-616 a. Note that although a particular number of projected lines are shown in this example, each projector may project any number of lines. Projected lines 610a-616a may intersect projected lines 620a-624a and may form projected grid pattern 604. Projection lines 620a-624a are referred to herein as "horizontal" lines, and projection lines 610a-616a are referred to herein as "vertical" lines. The terms "horizontal" and "vertical" are used for explanatory purposes and do not necessarily imply that a line is horizontal or vertical. At least a portion of the projected grid pattern 604 may be incident on one or more objects 602, reflected by the objects 602, and form a reflected grid pattern 606 on the objects 602. The reflective grid pattern 606 may be a distortion variation of the projected grid pattern 604. The reflected grid pattern 606 may have a shape based on the shape of the projected grid pattern 604 and the distance between the projector and the point on the object at which the projected line is reflected. For example, there may be curves or breaks in the reflected line.

In particular embodiments, projected junctions 630a-636a may be formed at the intersections of vertical projection lines 610a-616a and horizontal projection lines 620a-624 a. The junctions 630a-636a, 640a, 650a may be reflected by the object 602 and detected by the camera 220 as respective detected junctions 630b-638b, 640b, 650 b. Each point along the detected line 620b may correspond to a point along the corresponding projected line 620a, and each detected intersection 630b between the detected intersecting lines may correspond to a projected intersection 630a between the projected lines corresponding to the detected intersecting lines. Since the detected intersection 630b may be a point on two different detection lines 610b, the corresponding point at the projected intersection 630a may be identified more reliably than a point on a single line.

In a particular embodiment, the camera 220 may capture one or more images of the reflective grid pattern 606. The camera 220 may detect a first set of detected lines 620b-624b, which may be reflections of the lines 620a-624a, and a second set of detected lines 610b-616b, which may be reflections of the lines 610a-616 a. The lines 610b-616b, 620b-624b detected in the image captured by the camera may be represented as pixels in the captured image. Each pixel may correspond to a point on the surface of one of the objects 602.

As described above with reference to fig. 2A and 2B, triangulation calculations may be used to determine distances to points on the object based on the identified spots on the surface of the object 602 (which form the detected patterns 610B-616B, 620B-624B) and known camera and projector positions. Triangulation calculations may involve identifying correspondences between one or more features of the projected grid pattern 604 (e.g., line 620a or intersection 630a) and one or more features of the reflective grid pattern 606 (e.g., line 620b or intersection 630b), and calculating distances to the object 602 or to points on the surface of the object 602 using the locations of the identified features of the projected and reflected patterns. The projected lines 610a-616a, 620a-624a can be encoded with features that can be used to identify a particular projected line from the corresponding detection lines 610b-616b, 620b-624 b. These features may be, for example, line strength, line pitch, line pattern, and the like.

One or more features of the projected grid lines 620a-624a, 610a-616a, such as the intensity of each line 620a and/or the spacing (interval) between each pair of adjacent projected lines 620a, 622a, may be unique (at least within a particular projected grid pattern 604), such that each projected line may be associated with a unique value of these features. These features may be reflected by the object 602 and detected as detected features. The detected features may be identified in the detected lines 620b-624b, 610b-616b using the camera 220. The detected features can be used to identify the projected line 620a that corresponds to a particular detected line 620b by finding the projected line 620a with projected features that match the detected features of the detected line 620 b.

In particular embodiments, as described above, the characteristics (e.g., intensity) of each projected line may be different from the characteristics of other projected lines in projected grid pattern 604, or at least different from the characteristics of other adjacent or nearby lines. The correspondence between the projected lines and the detected lines may be determined by matching particular projected lines with particular detected lines based on their associated characteristics. For example, the projected grid pattern 604 has varying line spacing characteristics, which can be seen in the increase in distance between lines toward the lower right corner of the projected grid pattern 604. That is, the distance between adjacent pairs of projected lines may be different for different adjacent pairs of lines. By way of example, the distance between projected lines 612a and 614a (which may be referred to as d1) is different than the distance between projected lines 614a and 616a (which may be referred to as d 2). These two distances can be used to identify the detection line 622b corresponding to the projection line 622 a. For example, the ratio of d1 to d2 may be calculated as d1/d2 and compared to the ratio of the distance between the detected lines in the grid pattern 606 (such as the ratio of d3 to d4), where d3 is the distance between the detected lines 612b and 614b and d4 is the distance between the detected lines 614b and 616 b. If the spacing of projected lines 620a-624a is configured such that the ratio d1/d2 in projected grid pattern 604 is unique when compared to the ratios calculated for other adjacent line pairs in grid pattern 604, then the corresponding detected line 622b may be identified by calculating the corresponding ratio in detected grid pattern 606 and finding a detected ratio d3/d4 that matches the ratio d1/d2 (e.g., is equal to or within a threshold tolerance of the ratio d1/d 2). The ratio d3/d4 may be a ratio of the distances between the detected lines 612b and 614b (d3) and 614b and 616b (d 4).

In particular embodiments, each projection line 610a may be continuous. Each projected line 610a may have endpoints, in which case the line may be continuous between its endpoints. Since the projected patterns 604 are continuous (at least to their outer boundaries for patterns having a particular size), sudden changes in depth in the real world environment may create discontinuities in the detected grid pattern 606. Depth variations in the real-world environment may be detected by identifying discontinuities in the reflection pattern. A change in the characteristic along the reflection line may also indicate that the junction is open. When the projection line 610a passes through an edge of the object 602, the corresponding detection line 610b may have a discontinuity corresponding to the position of the edge of the object 602. Such discontinuities may be used to detect object boundaries. For example, the upper left corner of the object 602 has a diagonal boundary that intersects the detected pattern 606. In detected pattern 606, the uppermost and leftmost lines of projected pattern 604 are truncated, and the intersection between the leftmost and uppermost lines of the upper left corner of projected pattern 604 has no corresponding detected intersection in detected pattern 606. This intersection is not present from detected pattern 606, and there is a nearby intersection in detected pattern 606 (such as the intersection between detected line 620b and the leftmost line of detected pattern 606), indicating that there is an edge of object 602 near the next intersection.

In a particular embodiment, the intersection 630 may be used to uniquely identify a given pair of intersecting lines 610, 620. The correspondence between the detected intersection 630b and the projected intersection 630a may be determined based on characteristics of the intersections, such as the relative strengths of the lines 610, 620 that intersect at the intersection 630 and the horizontal and vertical relative spacing between the lines 610, 620 that intersect at the intersection and other lines (e.g., lines 612, 622). The lines 610a, 620a, and optionally other lines (e.g., lines 612a, 622a) may be used to determine characteristics of the projected intersection 630 a. The projected features determined for projected intersection 630a may be compared or correlated with the detected features of detected intersection 630b using detected lines 610b, 620b and optionally other lines (e.g., detected lines 612b, 622 b).

In particular embodiments, the detected features of the detected lines (such as the detected intensity or the distance between the detected lines) are not necessarily the same as the corresponding projected features, but may be sufficiently similar such that the corresponding projected lines or intersections may be determined by comparing a plurality of projections with the detected features or using a tolerance such that features that are not equal but within a predetermined tolerance (e.g., 1%, 3%, 5%, or other suitable tolerance) of each other are determined to be equal. For example, if the intensity of each projection line differs by 10% from the neighboring projection lines, a reflected line corresponding to (e.g., matching) the particular projection line having intensity value 20 may be identified by searching for a line in the reflected line having intensity value 20 plus or minus a tolerance, such as 5%.

In particular embodiments, the line structure of the projected and detected pattern may be used to identify lines that have unmatched features but are between other line pairs where the features match. For example, if 10 horizontal lines are projected in the grid, the first line having an intensity of 1, the second line having an intensity of 10, and so on until the tenth line has an intensity of 100, then the projected line corresponding to a particular detected line having a detected intensity value of 23 can be identified by searching for projected lines in the projected lines that have an adjacent projected line less than 23 on one side and greater than 23 on the other side.

In particular embodiments, there may be an ambiguity in determining which projected intersection 630a corresponds to a particular detected intersection 630 b. For example, in some cases, the reflected intensities of two adjacent lines may be difficult to distinguish. Determinations made for surrounding junctions may also be used to resolve or reduce ambiguities at particular junctions. The detected attributes (e.g., intensity, spacing, etc.) or the determined location of the detected junction may be propagated to neighboring junctions. The detected attribute or determined location of each junction may have an associated confidence value, and the confidence value may be propagated with the detected attribute or determined location to neighboring junctions.

In particular embodiments, the depth of object 602 in the image may be determined by identifying a detected intersection point 630b at which detected vertical line 610b and horizontal line 620b intersect on object 602 in the image. The illumination characteristics of the detected intersection 630b may then be used to identify a corresponding projected intersection 630a, which projected intersection 630a may be used to triangulate the depth (e.g., distance to) a point on the object 602 corresponding to the detected intersection 630 b. The illumination features may be understood as information encoded in the projected intersection, and may include the intensity of the detected line that intersects at intersection 630b and its corresponding intensity, the distance to the neighboring line, and time-encoded information, which will be described with reference to fig. 11. The illumination features may propagate to other junctions according to lines between the junctions. The propagation may be performed in steps, for example, by propagating illumination features that have been determined at one junction or received (from another junction) to each neighboring junction (e.g., first-order neighbors) at an appropriate time (e.g., after each frame, or after every N frames). In this way, information may be propagated between the junctions in a local manner. The propagation of information in the junction plot may converge quickly. Reliability information (e.g., confidence values) regarding the determination may also be propagated. The reliability threshold may be used to decide whether to propagate information to other junction points. The reliability information may be used to form a weighted graph, for example, where nodes correspond to junctions and edges correspond to lines between junctions. Triangulation may be used to calculate the depth of the detected intersection 630b based on information available at the intersection 630b, which may include information propagated from other intersections, such as attributes and locations or depths of nearby intersections.

Fig. 7 shows example grid light patterns 704, 706 with varying line intensities. As can be seen in fig. 7, the intensity of each line in the projected pattern 704 may be greater than the intensity of the line above it (for horizontal lines) or the line to the left of it (for vertical lines). As an example, if each of the projected lines 710, 720 has a unique intensity, when the detected intersection 730b is detected by the camera 220, the intensity of the cross detection lines 710b, 720b forming the intersection 730b can be detected by the camera and used to identify the particular projected intersection 730a corresponding to the detected intersection 730b by identifying the cross projection line with the corresponding intensity. For example, if the projection line 710a has an intensity of 20 and the projection line 720a has an intensity of 30, the detection line 710b having an intensity of 20 may correspond to the projection line 710a and the detection line 720b having an intensity of 30 may correspond to the projection line 710 b. As another example, the intensity may vary in other ways, for example, the intensity of each line may be less than the intensity of the line above it (for horizontal lines) and less than the intensity of the line to the left of it (for vertical lines). The intensity of the projected line may increase across the projected grid pattern 704, and thus the intensity of the projected line 712a may be set based on a value determined by multiplying the intensity of the other line 710a by a factor of, for example, 1.25, 1.5, 2.0, 0.5, etc. As another example, the intensity of the projection line 712a may be set based on a counter value that is incremented for each projection line, or based on another value that is unique to each projection line 710, 720 in the projection grid 704 (such as a unique randomly selected value in the projection grid 704).

FIG. 8 illustrates example grid light patterns 804, 806 with varying line patterns. As can be seen in fig. 8, each projected line 810a, 812a, 814a can have a different line pattern. Each line pattern may be, for example, dots arranged to form a row of dots (e.g., small circles), a dashed line, or other shapes arranged in a row. The line patterns may be uniform, e.g., with a constant distance between shapes, and the shapes may be the same size. Alternatively, the distance between the shapes or the size of the shapes may vary. As an example, the distance between the shapes of each projected line 810a may be greater than the distance between the shapes of adjacent projected lines 810b, or the size of the shapes in each line may be greater than the size of the shapes in adjacent lines.

As an example, if each of the projected lines 810a, 820a has a unique line pattern (e.g., the size and/or shape of an element such as a dashed line or a dot), when the detected intersection 830b is detected by the camera 220, the camera 220 may detect the pattern of intersecting detected lines 810b, 820b forming the intersection 830b and use to identify the particular projected intersection 830a corresponding to the detected intersection 830b by identifying the intersecting projected lines with the corresponding pattern. For example, if projected line 810a has a dashed pattern, the length of each scribe line (dash) is twice the length of the space between each pair of scribe lines, then detected line 810b having the same dashed pattern corresponds to projected line 810 a.

Fig. 9 illustrates the movement of the grid light pattern and the corresponding epipolar line. In certain embodiments, at certain junctions, the horizontal and vertical lines coincide in depth. When there is movement in depth, the intersection of the reflected lines may move, for example due to movement of the device (which may be detected by changes in GPS coordinates, for example) or movement of objects in the scene (which may be detected by changes in the image captured by a camera, for example). The line between the initial point where these lines intersect before movement and the subsequent point where they intersect after movement is referred to herein as the epipolar line 906. The epipolar line 906 may correspond to a path, and the two junctions may be located only on the path. If there are N lines in one direction and M lines in the vertical direction, then there are NxM junctions that have a certain path through them. These paths should have as little commonality as possible with each other. Then, if a point is located on a particular path, the corresponding two intersection points may be determined.

In particular embodiments, given a baseline distance (e.g., the distance between the first projector and the second projector), the intersection point may be one of a relatively small number of possible choices, as only a relatively small number of horizontal and vertical line pairs may intersect at that point. To resolve which lines correspond to junctions, it is not necessary to examine all junctions in the grid (a grid consisting of M horizontal lines and N vertical lines may include MxN junctions). Only a subset of the junction points that may correspond to the points detected in the image need be searched. The subset may include an intersection on or near the core line that intersects the initial location of the intersection. For example, if there are 5 vertical/horizontal pairs that may hit the point, then only those pairs need to be searched to identify the pair corresponding to the point. Furthermore, based on depth variations, the line and camera pairs of the vertical projector may only move a certain distance or at most a certain distance. Also, the line and camera pair of the horizontal projector may only move or at most a certain distance based on the same depth variation. The combination of the two (e.g., the intersection of the lines) can be moved in a constrained manner, which corresponds to the kernel. The calibration (horizontal and vertical baselines) may result in two epipolar line constraints, which may determine the line through which the intersection may move, e.g., the epipolar line. The constraint may narrow the set of possible projected junctions corresponding to the reflective junctions. The intensity and spacing codes described above may also be used to narrow down possible projected junctions corresponding to the reflection junctions if there is ambiguity in identifying the projected junctions. The encoding and epipolar constraints may be combined to narrow the likelihood to a single projected vertical-horizontal line pair (e.g., an intersection) to which the reflected vertical-horizontal line pair may correspond.

As an example, when movement of a device or object in a scene captured by a camera is detected, a reflective intersection detected at an initial point (prior to the movement) may be detected at a subsequent point (e.g., pixel) in the image (after the movement). To determine whether the reflected junction detected at the subsequent point corresponds to the junction detected at the initial point (e.g., is the same as the reflected junction detected at the initial point, and thus corresponds to the same projected junction as the junction detected at the initial point), the location of the subsequent point may be compared to an epipolar line intersecting the initial point. If the subsequent point is on the epipolar line (or close enough), the reflected junction detected at the subsequent point may correspond to the same reflected junction detected at the initial point, and thus to the same projected junction as the reflected junction detected at the initial point.

Referring to fig. 9, a projected grid light pattern 904a including an intersection 942a is projected onto an object 902. The grid light pattern 904 includes horizontal lines projected by the first projector. The horizontal grid lines of pattern 904a are projected by a first projector and the vertical grid lines are projected by a second projector. The reflection of the projected pattern 904a can be detected by a camera and used to generate a detected pattern 904b, which detected pattern 904b has a detected intersection location 942b corresponding to the projected intersection 942 a. Subsequently, a motion occurs in which the object 902 moves to the right and upward relative to the camera, so the intersection position moves to the second detected intersection position 942 c. The movement of the intersection from location 942b to location 942c may be along the epipolar line 906. Further, as the depth of the object changes, the detected lines from the projector projecting horizontal lines may move up or down in the image, and the detected lines from the projector projecting vertical lines may move left or right in the image.

Fig. 10 illustrates an example method 1000 of determining depth using a grid light pattern. The method can begin at step 1010, where the method can project a first projection pattern having one or more first projection illumination features using a first projector. At step 1020, the method 1000 may project a second projection pattern having one or more second projected illumination features using a second projector, where the first projection pattern intersects the second projection pattern. At step 1030, method 1000 may capture an image including first and second detection patterns corresponding to reflections of the first and second projection patterns, respectively, using a camera, wherein the camera is located at a first baseline distance from the first projector and at a second baseline distance from the second projector.

At step 1040, the method 1000 may identify a detected point in the image corresponding to a projected point in at least one of the first and second projected patterns by comparing the detected illumination features of the first and second detected patterns to the first and second projected illumination features. At step 1050, method 1000 may calculate a depth associated with the detected point based on the projected point, the detected point, and a relative positioning between the camera and at least one of the first projector and the second projector.

In a particular embodiment, the first projection pattern may include a plurality of first projection lines and the second projection pattern may include a plurality of second projection lines. The detected illumination feature may include a plurality of first reflected lines and a plurality of second reflected lines intersecting the first reflected lines, and the first reflected lines and the second reflected lines may be based on reflections of the first projected lines and the second projected lines from the surface of the object. By identifying a reflection intersection point at which one of the first reflection lines intersects one of the second reflection lines, a detected point on the image corresponding to a projected point in one or more of the first projected pattern and the second projected pattern may be identified.

In particular embodiments, a reflective junction may be associated with one or more reflective junction features. The projected intersection may correspond to a reflective intersection such that the projected intersection is located at an intersection of one of the first projection lines and one of the second projection lines, wherein the projected intersection is associated with one or more projected intersection features determined based on (1) one or more first projected lighting features associated with the first projection pattern and (2) one or more second projected lighting features associated with the second projection pattern, and wherein one or more of the projected intersection features match one or more of the reflective intersection features. The depth of the detected point may be determined based on a depth associated with the reflective intersection, which may be determined based on at least one of the camera, the projector, the reflective intersection, and a projected intersection corresponding to the reflective intersection.

Particular embodiments may repeat one or more steps of the method of fig. 10 where appropriate. Although this disclosure describes and illustrates particular steps of the method of fig. 10 as occurring in a particular order, this disclosure contemplates any suitable steps of the method of fig. 10 occurring in any suitable order. Further, although this disclosure describes and illustrates an example method of determining depth using a grid light pattern that includes particular steps of the method of fig. 10, this disclosure contemplates any suitable method of determining depth using a grid light pattern that includes any suitable steps, which may include all, some, or none of the steps of the method of fig. 10, where appropriate. Moreover, although this disclosure describes and illustrates particular components, devices, or systems performing particular steps of the method of fig. 10, this disclosure contemplates any suitable combination of any suitable components, devices, or systems performing any suitable steps of the method of fig. 10.

In particular embodiments, depth sensing may be performed by projecting a pattern (such as a line) and varying the projected illumination characteristics (such as illumination intensity) of the pattern over a period of time according to a temporal illumination characteristic pattern associated with the projected pattern. For example, the temporal illumination characteristic pattern may be a pattern of illumination intensities. The camera may capture projected patterns, e.g., lines, in a plurality of images over a period of time to form a detected pattern, and determine a detected temporal illumination characteristic pattern, e.g., a pattern of illumination intensities, such as 3, 5, 7, and 9, based on the detected pattern. The depth sensing system may identify a particular projection pattern corresponding to a given detected pattern by comparing the detected temporal illumination signature pattern to the projection temporal illumination signature pattern for the projection pattern. The projection patterns having the same temporal illumination feature pattern (e.g., 3, 5, 7, and 9) may be the particular projection pattern corresponding to a given detection pattern.

FIG. 11 shows example temporal illumination patterns 1102-1130. An illumination pattern may be projected using a suitable type of projector, the one or more features of which vary over time, for example as a function of the temporal illumination feature pattern. The projected pattern of illumination features may be projected as one or more shapes of illumination features. The illumination characteristic may be, for example, an intensity of the shape, a geometric characteristic of the shape (such as a distance between lines), or other suitable characteristic of one or more projected shapes. The shapes may be, for example, a grid of lines, and the shapes may have illumination characteristics specified by a pattern. One or more shapes may be projected at different times using different illumination characteristics. The camera may capture the reflected light beam in a sequence of images and identify a pattern of detected temporal illumination features based on the images. Features of a previously projected shape pattern of projected illumination features may be compared with features of a pattern detected in a sequence of captured images to find image pixels corresponding to the projected light beam, and the correspondence may be used to determine the depth of each pixel by triangulation based on the location of the corresponding image pixel. The distance to the object may be determined based on the reflection detected by the camera using triangulation techniques, wherein the correspondence between the lines or other patterns detected by the camera and the projected lines may be identified by matching the pattern of temporal illumination features of the received light beam over time to known patterns of temporal illumination features of the projected light beam to identify the projected light beam corresponding to the detected lines or other patterns.

In a particular embodiment, the temporal illumination characteristic may be a light intensity value. The intensity used by the projector may be determined using a temporal illumination characteristic pattern, which may be a sequence of values such as the example sequence 1102. The example sequence 1102 has values of 3, 5, 7, 9. Each value in the sequence may be projected for a duration, such as 5 milliseconds, 100 milliseconds, or other suitable duration. The time value is associated with the sequence 1102 in fig. 11 to illustrate that each number in the sequence can be used as the intensity of the projected line at a different time. Thus, for example, a line or other shape of intensity 3 may be projected for T milliseconds, after which the intensity may change to 5 for another T milliseconds, then to 3 for another T milliseconds, to 9 for another T milliseconds. The pattern may then be repeated. The duration of time each intensity is projected may correspond to the rate at which frames are captured, in which case the intensity of each row may vary over the course of the frame sequence. The time at which the pattern is projected may be referred to as a time period of the pattern. In this example, the time period of the pattern 1102 is 4T milliseconds. A frame may be captured every T milliseconds in order to capture each intensity in a separate frame. The pattern may then be determined based on the captured frames. For example, a detected pattern having an associated intensity may be determined based on the intensity of the lines or other features detected in each frame. The temporal patterns detected in this example are 3, 5, 7, 9. The detected temporal pattern can then be used to identify each projection line. In this example, the detected pattern 3, 5, 7, 9 may be compared with a previously projected pattern 3, 5, 7, 9, and the line projected when the pattern 3, 5, 7, 9 is projected may be a projected line corresponding to the detected line. Different sequences may be used for different projection lines. The sequence may be a unique sequence of other sequences that are projected over a period of time (e.g., over a particular hour or day) relative to the depth sensing system. Since the sequence is different from other sequences that may be detected during the time period, the projected shape pattern 604 corresponding to the sequence may be identified by finding a projected pattern that has the same sequence as the detected pattern 606. After capturing a plurality of frames corresponding to the sequence length, the projected line corresponding to a particular detected line may be identified by comparing the detected sequence to the projected sequence.

Since the intensity values determined from the camera frame may not be accurate, the intensities of many consecutive lines may match a subsequence of a known sequence of projection intensities encoded by the projector. The number of consecutive rows to match may be increased to increase the confidence that the correspondence has been correctly identified. That is, the intensity value of each projection beam may correspond to a number in a predetermined sequence, where a subsequence of a particular length N (e.g., 3 consecutive numbers) is unique. For N consecutively detected lines, the corresponding projection line can be identified by finding a subsequence of the same N consecutive numbers in the predetermined sequence.

Fig. 11 shows three different projection patterns of the coded temporal illumination feature patterns 3, 5, 7, 9. The four lines 11104-1110 of different intensities encode the patterns 3, 5, 7, 9 in line intensity. Line 1104 has an intensity of 3 and is projected for a period of time starting at time T1. The line 1106 has an intensity of 5 and is projected for a period of time starting at time T2. Line 1108 has an intensity of 7 and is projected for a period of time at time T3. Line 1110 has an intensity of 9 and is projected for a period of time at time T4.

As another example, four lines 1114-1120 of different widths encode patterns 3, 5, 7, 9 in line width. Line 1114 has a width of 3 (e.g., 3 pixels), line 1116 has a width of 5, line 1118 has a width of 7, and line 1120 has a width of 9. Each line width may be projected for a period of time. Each line may project the same duration. Alternatively, lines 1114-1120 may be projected for different durations, and detection of transitions between different features in the pattern may be accomplished based on changes in temporal illumination characteristics. For example, as the intensity of the projected pattern changes, the camera may proceed to the next frame.

The example projected patterns 1124-1130 include different densities of parallel lines based on the sequence 1102. The first projection pattern 1124 has 3 lines, the second projection pattern 1126 has 5 lines, the third projection pattern 1128 has 7 lines, and the fourth projection pattern 1130 has 9 lines. The first pattern 1124 may be projected for a period of time, followed by the second pattern 1126 being projected for that period of time, the third pattern 1128 being projected for that period of time, and the fourth pattern 1130 being projected for that period of time. If the length of each time segment is a time unit T, the total time segment is 4T. Each of the projected patterns 1124-1130 may be captured by a camera and the values encoded in each projected pattern may be extracted to generate a pattern of detected temporal illumination feature values. The depth sensing system may count the number of non-intersecting lines in each projected pattern 1124-1130, and thus may determine a number of 3 from the first pattern 1124, a number of 5 from the second pattern 1126, a number of 7 from the third pattern 1128, and a number of 9 from the fourth pattern 1130.

Examples of repeatedly detected temporal illumination feature patterns 1130 include a number 0 at time 1, followed by three consecutive occurrences of the pattern 3579, and a number 1 at time 14. The first occurs at time 2-5, the second at time 6-9, and the third at time 10-13. The projected pattern of illumination features may be repeated in this manner and may be detected over a wider time frame. This pattern may be repeated continuously until a different pattern is selected. The depth sensing system may identify the pattern as a repeating sequence 3, 5, 7, 9 based on the pattern 1130.

In particular embodiments, the projector may be mounted on a structure or device, such as a head-mounted device, that is a fixed distance from the camera. To project light of different intensities, the projector may use rows of emitters, and there may be a different number of emitters on each row, such that each row projects a different intensity. Alternatively, an addressable illuminator may be used, where different currents may be applied to the emitter to produce different intensities.

Temporal intensity coding may be combined with other coding techniques, such as using different intensities for different simultaneously projected lines (projected to different locations), or changing the distance between projected lines. These encodings can be used to resolve ambiguities regarding the location in the pattern of a particular point detected in the camera image, using triangulation to determine the depth of that point.

In particular embodiments, the projector may project a pattern, referred to herein as a projected pattern, which may include one or more shapes, such as lines or other shapes. Each projection pattern may be associated with a projection time illumination signature pattern, which may specify how the illumination signature of the projection pattern varies over time. As an example, two different projection patterns, Pat1 and Pat2, may be projected. Pat1 may be a line and may have an associated illumination feature pattern 1, 3, 5, 7. Pat2 may be a circular dot (e.g., a light spot) and may have an associated temporal illumination feature pattern 2, 4, 6.

The camera may capture an image containing the projected pattern. For example, the camera may capture images of the lines of Pat1 and the dots of Pat 2. The camera may capture images at a particular frame rate (e.g., 30 frames per second or other suitable rate). The projector may project the pattern values at the same rate. Thus, the projector may project a line of intensity 1 and a dot of intensity 2 at time T1 for a duration corresponding to 30 frames per second. The camera may capture an image containing lines of intensity 1 and circles of intensity 2, followed by another frame containing lines of intensity 3 and dots of intensity 4. The depth sensing system may attempt to match the two sequences with previously projected sequences. The previously projected sequences are 1, 3, 5, 7 and 2, 4, 6, so none of the detected sequences 1, 3 and 2, 4 match. The projector may continue by projecting a line of intensity 5 and a dot of intensity 6. The depth sensing system may compare the detected sequence with previously projected sequences and may find a match for the pattern 2, 4, 6 of the projection dots. The sequences 2, 4, 5 are associated with the projected dot pattern, and therefore the projected dot pattern is identified as the projected pattern corresponding to the detected dot pattern.

When determining depth, a confidence level may be associated with each pattern. If the confidence level is low, the line may be turned off in the next frame or projected differently, e.g., at a different frequency that may be detected. This technique can be used for dark objects or in the presence of a large amount of noise to verify a determined depth and to achieve a desired level of accuracy.

Each projected line may project a unique pattern in a specific sequence that may be used to detect the line over time. Thus, if (1) each line has a unique pattern, e.g., an intensity pattern, that varies over time, (2) the projector iterates through the pattern fast enough, (3) the frame rate of the camera is high enough to match the iteration rate of the projector, and (4) the object in the scene is stationary or moving slowly enough, the line can be uniquely identified based on the unique pattern. Due to the unique pattern, and slow or non-existent movement, the depth of the line can be resolved based on local information (i.e., the detected pattern of the line). The frame rate required to identify the unique pattern may be based on the speed of movement in the scene, so there is a trade-off between movement and time. If the scene is stationary or has only slow movement, a relatively low frame rate (e.g., 50FPS) may be sufficient. For scenes that may have faster motion, a higher frame rate is required, e.g., 500 to 1000FPS, because during intervals of 1 or 2 milliseconds there is typically no motion in nature. The number of cycles of the pattern to be detected may correspond to the density of the pattern. When the pattern changes at a sufficiently high rate and the rate of pattern change is slowed down according to the frame rate of the camera, the depth of the lines can be analyzed based on the detected line pattern.

Fig. 12 illustrates an example method 1200 for determining depth using temporal patterns. The method may begin at step 1210, where method 1200 may project a plurality of projection patterns using at least one projector, where the projected illumination characteristics of each projection pattern vary over a period of time according to a predetermined time illumination characteristic pattern associated with the projection pattern. At step 1220, method 1200 may capture a plurality of images at different times within the time period using a camera, the images including a detected pattern corresponding to a reflection of at least a portion of the plurality of projected patterns projected during the time period. At step 1230, the method 1200 may determine a detected temporal illumination feature pattern for each detected pattern based on the change in the detected illumination feature of the detected pattern during the time period. At step 1240, the method 1200 may identify one or more detected patterns corresponding to the one or more projected patterns by comparing the at least one detected temporal illumination signature pattern to at least one predetermined temporal illumination signature pattern associated with a time period. In step 1250, the method 1200 may calculate one or more depths associated with the identified detected pattern based on the corresponding projected pattern, the detected pattern, and the relative positioning between the camera and the projector.

Particular embodiments may repeat one or more steps of the method of fig. 12 where appropriate. Although this disclosure describes and illustrates particular steps of the method of fig. 12 as occurring in a particular order, this disclosure contemplates any suitable steps of the method of fig. 12 occurring in any suitable order. Further, although this disclosure describes and illustrates an example method of determining depth using a temporal pattern that includes particular steps of the method of fig. 12, this disclosure contemplates any suitable method of determining depth using a temporal pattern that includes any suitable steps, which may include all, some, or none of the steps of the method of fig. 12, where appropriate. Moreover, although this disclosure describes and illustrates particular components, devices, or systems performing particular steps of the method of fig. 12, this disclosure contemplates any suitable combination of any suitable components, devices, or systems performing any suitable steps of the method of fig. 12.

In certain embodiments, a light-emitting device, such as a projector, may have multiple individually addressable emitters. Multiple emitters may be activated to produce an illumination pattern, which may appear as a set of individual light sources that may be detected by the camera and used to determine the location of the object, for example, by matching the reflection pattern to a particular projection pattern and performing triangulation. The density of placement of emitters on a light-emitting device, such as a projector, may affect the accuracy of determining location. For example, emitters that are close to each other may provide more detectable points per unit area. However, if the emitters are too close to each other, the light from adjacent emitters may merge (e.g., overlap) due to a gaussian distribution of light intensity in the beam projected by each emitter, where the light near the center of the beam is very bright, and the beams from the individual emitters may become unresolvable when viewed by cameras closer than a threshold distance. Therefore, when the camera is relatively close to the projector, adjacent emitters should be separated from each other by a separation distance large enough to prevent their beams from merging. However, when viewed from a further distance, emitters separated by the separation distance appear further apart, and thus the depth resolution accuracy may be degraded.

In certain embodiments, to address this issue, one or more working emitters may be deactivated when gaussian beams from multiple emitters merge together as the camera moves closer to the projector to provide additional space between the working emitters. To avoid reducing accuracy when the camera is far away from the emitters, the distance between the detected light sources increases when the camera is moved further away from the projector, additional emitters may be activated. The range of distances that the camera can recognize a particular spot from an emitter can thus be extended by modifying the emitted light pattern so that the distance between each pair of adjacent illumination emitters is greater when the camera is closer to the emitter than when the camera is further from the emitter. Increasing the distance between emitters in this manner may prevent light generated by different emitters from merging together when the cameras are close together. The emitted light pattern may be modified such that the distance between adjacent pairs of emitters is smaller when the camera is further from the emitters, as the distance between the emitters may be larger than otherwise required, which may result in less accurate identification of the point location.

In particular embodiments, the density of the illumination pattern projected by a projection device comprising addressable emitters may be varied by activating or deactivating individual emitters. Higher density patterns may be projected by activating emitters that are more closely spaced, while lower density patterns may be projected by activating emitters that are more widely spaced. Higher density patterns may use more emitters and therefore may use more power than lower density patterns. Furthermore, higher density patterns may not be resolvable from a close distance. Lower density patterns may be resolved from close ranges, but may provide less accurate position determination than higher density patterns. For applications involving determining the depth of relatively close objects, such as hand movements, lower density patterns may be used. For applications involving determining depth at greater distances, such as constructing a depth map of a room or larger space, higher density patterns may be used. Addressable emitters may be used for both types of applications by activating a relatively sparse emitter pattern for closer distances (e.g., by activating half of the emitters on a projector) and a relatively dense emitter pattern for farther distances (e.g., by activating all of the emitters on a projector).

In a particular embodiment, the appropriate density of emitters to activate may be determined by projecting a pattern, capturing an image of the pattern using a camera, and determining whether to increase or decrease the density of the pattern based on the image. For example, if a dense pattern is projected using 9 emitters of the device, and light from a single emitter cannot be resolved (e.g., the light is merged together and does not appear as 9 separate light sources), the camera may be too close to the emitters to resolve the dense pattern, and the density of the pattern may decrease. The density may be reduced by deactivating half of the active emitters, for example, every other active emitter, and capturing an image of the remaining active emitters. If an individual emitter still cannot be resolved, the process may be repeated, for example, by again deactivating half of the working emitters and determining whether the remaining emitters can be resolved, and so on, until the working emitters can be resolved (e.g., the number of different light sources detected in the camera image matches the number of working emitters). The light sources can be independently analyzed without being completely separated; for example, an overlap threshold may be used to resolve partially overlapping light sources. As another example, if individual light sources can be resolved, additional emitters may be activated, for example, by doubling the number of active emitters. The camera may then capture images of the increased number of emitters. If the working transmitters can be resolved they can be used for subsequent frames or, alternatively, the number of working transmitters can be increased again until the working transmitters cannot be resolved, at which point the previous (resolvable) pattern of the working transmitter can be used for subsequent frames.

In particular embodiments, dense and sparse patterns may be projected in alternating frames by activating closer transmitters for dense patterns and activating further transmitters for sparse patterns. The camera may detect alternating sparse and dense patterns and determine whether the emitter can be resolved in each pattern. If the emitter can be resolved in a dense pattern, the dense pattern can then be used (e.g., for detecting projections in subsequent frames captured by the camera) unless the pattern is denser than necessary, in which case a sparser pattern can be used. The pattern may be higher than needed if, for example, the density of the pattern provides a higher accuracy than needed, or higher than can be used due to other limitations of accuracy (e.g., limitations from other devices, such as camera frame rate).

If individual emitters of a dense pattern cannot be resolved, the Gaussian light from the emitters may overlap and a more sparse pattern may then be projected (e.g., for frames subsequently captured by a camera). For example, if individual emitters of the sparse pattern can be resolved, the sparse pattern can be subsequently used, or a new sparse pattern with a density between the sparse pattern and the irresolvable dense pattern can be generated and projected. The new sparse pattern may be analyzed as described above to determine whether it is parsable and should be projected for subsequent frames. For example, the width of a line or the intensity of light may be used to determine that gaussian beams have merged together.

FIG. 13A shows an example illumination pattern of an illuminator 1302 for determining depth from different distances. The illuminator 1302 may be, for example, an addressable illuminator, and may be included in a projector. The luminaire in this example has 9 emitters 1311-1319. In the first pattern 1304, all 9 emitters are active, e.g., emit light. For example, the transmitter may transmit a pattern for depth sensing. However, as the camera moves closer to the projector 1302, the gaussian light distributions from the working emitters may appear to merge together due to the intensity properties of the gaussian light. Thus, at sufficiently close distances, light from multiple emitters may merge together, and various features of the light pattern (e.g., dots or lines produced by emitters 1310-1318) may no longer be resolvable. Due to this merging of gaussian light patterns, the accuracy of the structured depth sensing technique may be reduced within a short distance. In certain embodiments, one or more working emitters may be deactivated, thereby increasing the distance between the working emitters so that light from the working emitters does not merge when the camera is closer than a certain distance. It may be desirable for the emitters to remain operational if possible, as a greater number of operational emitters may provide more accurate depth sensing. Thus, an iterative process may be used to identify a particular transmitter to deactivate.

As shown in fig. 13A, when the light from the emitters begins to merge together, for example, every other active emitter may be deactivated because the camera is less than a threshold distance from the emitter. In the initial pattern 1304, all 9 transmitters are active, so deactivation of every other active transmitter results in deactivation of the second, fourth, sixth, and eighth transmitters, as shown in the second pattern 1310. Transmitters 1312, 1314, 1316, and 1318 have been deactivated, and transmitters 1311, 1313, 1315, 1317, and 1319 remain operational in pattern 1310. The light from the emitters may still appear to merge when the camera is sufficiently close to the projector 1302, or may begin to merge as the camera moves closer to the projector 1302. If the light is still merged or becomes merged again, e.g. the individual emitters are not resolvable, further emitters may be deactivated by deactivating every other active emitter again. The third and seventh emitters are thus deactivated, resulting in pattern 1320 where emitters 1311, 1315, and 1319 remain operational.

If the light remains merged or becomes merged after a period of time, additional emitters may be deactivated by again deactivating every other active emitter. The fifth transmitter is thus deactivated, resulting in pattern 1330 where transmitters 1311 and 1319 are active. Then, if the light remains merged or becomes merged after a period of time, the further emitters may be deactivated by deactivating every other active emitter again. The ninth transmitter is thus deactivated, resulting in the pattern 1340 in which only the transmitter 1311 is active. Since only one emitter is active in this example, no combination of light from the two emitters will occur. It may be desirable to reactivate the emitter if the distance between the camera and the projector 1302 increases.

As an example, the emitters may be reactivated in an order opposite to the order in which they were deactivated, e.g., by activating pattern 1330, and if there is no merging of light, then pattern 1320, and if there is still no merging of light, then pattern 1310, and finally, if there is still no merging of light, then all emitters of projector 1302 are activated. Although the emitters are described as being deactivated or activated according to patterns 1310-1340, these patterns are examples only, and light may be deactivated or activated according to other patterns, for example, by deactivating odd-numbered emitters or even-numbered emitters instead of every other emitter, or by deactivating more emitters at each step, for example, by starting deactivation with pattern 1320 instead of pattern 1310, and continuing directly to pattern 1340 after pattern 1320.

FIG. 13B shows an example light pattern viewed from different distances. As described above, the gaussian beams from adjacent emitters 1312-1316 of the addressable illuminator 1302 appear to merge together, as shown by the light pattern 1342, as the intensity of the light increases as the observer (e.g., camera) moves closer to the light source, and the beams from the individual emitters 1312-1316 become indistinguishable in the camera image of the merged light pattern 1342. This combination of light may occur when the distance D1 between the illuminator 1302 and the camera is less than the threshold distance D. In the combined light pattern 1342, the beams from adjacent emitters 1312, 1313 are separated by a distance d 1. The width of the resolvable feature (i.e., the combined light pattern) is labeled w 1.

As the distance between the illuminator 1302 and the camera increases, the beam becomes larger and can be resolved separately, as shown by the light pattern 1344. In light pattern 1344, the appearance of light from illuminator 1302 at distance D2> D is shown, with beams from adjacent emitters 1312, 1313 separated by distance D2. The width of the resolvable feature (i.e., the light beam) is labeled w 2. As the distance between the illuminator 1302 and the camera is further increased, the beam becomes larger, as shown by the light pattern 1346. In the light pattern 1346, the appearance of light from the illuminator 1302 at distances D3> D2 is shown, with beams from adjacent emitters 1312, 1313 separated by a distance D3. The width of the resolvable feature (i.e., the light beam) is labeled w 3.

Fig. 13C shows an example projection pattern viewed from different distances. In this figure, the emitters of the illuminator 1302 project light. When viewed from a short distance, the gaussian beams from adjacent ones of the emitters 1312-1316 may appear to merge together as shown by the merged light pattern 1347, and the beams from the individual emitters 1312-1316 become indistinguishable in the camera image of the merged light pattern 1347. The distance D1 between the illuminator 1302 and the camera is less than the threshold distance D. The width of the resolvable feature (i.e. the combined light pattern) is marked w 4.

As the distance between the illuminator 1302 and the camera increases, the beam becomes larger and can be resolved separately, as shown by the light pattern 1348. In the light pattern 1348, the appearance of light from the illuminator 1302 at a distance D2> D is shown, with beams from adjacent emitters 1312, 1313 separated by a distance D4. The width of the resolvable feature, which is a line (and may include one or more beams) produced by the emitter, is labeled w 5. As the distance between the illuminator 1302 and the camera is further increased, the beam becomes larger, as shown by the light pattern 1349. In the light pattern 1349, the appearance of light from the illuminator 1302 at distances D3> D2 is shown, with beams from adjacent emitters 1312, 1313 separated by a distance D5. The width of the resolvable feature (the line produced by the emitter) is labeled w 6.

13D and 13E illustrate example illumination patterns for a two-dimensional illuminator for determining depth from different distances. In a particular embodiment, a group of emitters may be deactivated when light from the emitters begins to merge together, e.g., because the camera is less than a threshold distance from the emitters. The transmitters to be deactivated may be selected such that there are no horizontally or vertically adjacent pairs of active transmitters. That is, there are no active transmitters separated by a horizontal or vertical distance 1. Alternatively, half of the active transmitters may be deactivated by selecting every other active transmitter to deactivate. As shown in fig. 13D, in the initial pattern 1350, 25 emitters are active. Every other active transmitter may be identified by starting at the upper left corner of the transmitter pattern 1350 and selecting every other transmitter. When the end of a row is reached, the selection may continue to the next row. Every other active transmitter deactivation pattern 1350 results in even transmitters being deactivated as shown by the second pattern 1352. In the second pattern 1352, the emitters 1354, 1356, 1358, 1360, 1362, 1364, 1366, 1368, 1370, 1372, 1374, and 1376 have been deactivated. When the camera is close enough to the projector, the light from the emitters may still appear to merge, or when the camera is close enough to the projector, merging may begin. Additional emitters may be deactivated if the light is still combined or combined again, e.g. individual emitters are not resolvable. The transmitters to be deactivated may be selected such that there are no active transmitters separated by a horizontal or vertical distance 2. The result of deactivating these emitters is shown in fig. 13E as pattern 1380, where emitters 1381, 1382, 1383, 1384, 1385, 1386, 1387 and 1388 have been deactivated. Subsequently, if the light still merges or becomes merged after a period of time, the additional emitters may be deactivated. Transmitters 1391, 1392, 1393, and 1394 can thus be deactivated, resulting in pattern 1390 in which only the center transmitter is active. Since only one emitter is active in this example, the combination of light from the two emitters does not occur. It may be desirable to reactivate the active emitters if the distance between the camera and the projector 1302 subsequently increases.

In particular embodiments, the distance between the emitters and the camera may be determined and used to control the distance between the emitters. Alternatively, the camera image may be analyzed to determine that adjacent light has merged (e.g., formed a line) based on the width of the detected light, and the distance between each pair of emitters may be increased when the light appears to merge. As another example, the bright light detected by the camera may or may not come from the structured light pattern. The emitter density, e.g., the number of active emitters in a particular area, may be changed in the next frame and the image in that frame evaluated to determine whether light is coming from the pattern by, for example, activating or deactivating certain emitters, and then determining whether the camera detects that those emitters have been activated or deactivated. This process may be repeated by activating or deactivating a different transmitter in each iteration of the loop and determining that the transmitted pattern has been detected or has not been detected after a threshold number of iterations.

Fig. 14 shows an example method 1400 for activating or deactivating one or more light emitters of a luminaire to determine depth from different distances. The method may begin at step 1410, where method 1400 may project a projection pattern including a plurality of projection features having different locations using at least one projector. At step 1420, the method 1400 may capture an image including a detected pattern corresponding to a reflection of the projected pattern using a camera. At step 1430, the method 1400 may identify at least one detected feature of the detected pattern, wherein the detected feature corresponds to at least one reflection of the projected feature. At step 1440, the method 1400 may activate or deactivate one or more light emitters based on the detected characteristic.

Particular embodiments may repeat one or more steps of the method of fig. 14 where appropriate. Although this disclosure describes and illustrates particular steps of the method of fig. 14 as occurring in a particular order, this disclosure contemplates any suitable steps of the method of fig. 14 occurring in any suitable order. Further, while this disclosure describes and illustrates an example method for activating or deactivating one or more light emitters of a luminaire to determine depth from different distances, including certain steps of the method of fig. 14, this disclosure contemplates any suitable method for activating or deactivating one or more light emitters of a luminaire to determine depth from different distances, including any suitable steps, which may include all, some, or none of the steps of the method of fig. 14, where appropriate. Moreover, although this disclosure describes and illustrates particular components, devices, or systems performing particular steps of the method of fig. 14, this disclosure contemplates any suitable combination of any suitable components, devices, or systems performing any suitable steps of the method of fig. 14.

In particular embodiments, in systems that perform depth sensing using structured light, power consumption may be reduced by deactivating transmitters of luminaires that are not needed or used, such as by projecting sparse patterns in a low power mode to scan for motion in a scene, and upon identifying motion, projecting denser patterns to update a depth map based on the motion. The disclosed technology may use addressable luminaires, which may have individually controllable emitters. The intensity of the structured light projected by the addressable emitters may vary based on the input current. The structured light may be projected in a pattern such as a set of dots, lines, or a grid of lines.

In a particular embodiment, the technique for saving power includes projecting a dense pattern to create a dense model of the scene, and then operating in a low-power scan mode for changing portions of the sampled scene by projecting a sparse pattern. If any changes are detected (such as motion intersecting one or more lines of the sparse pattern), the dense pattern may be projected again to create or update a dense model of the scene. Further, portions of the scene may be sampled for changes by projecting sparse patterns (such as lines) onto portions of the scene and using a camera to detect reflections of the sparse patterns indicative of motion. Periodically (e.g., every second), a single frame burst of dense patterns may be projected to create or update a dense model of the scene.

Fig. 15 shows an example of reducing the grid light pattern density to reduce power consumption. The dense pattern 1502 is projected onto the scene at time T1. At time T2, a more sparse pattern is projected onto the scene because, for example, no motion is detected in the scene between times T1 and T2. Since there is no motion, the existing depth map may be used at least by time T2. Since the depth map does not need to be updated between time T1 and at least T2, a more sparse pattern 1504 may be projected at time T2 to detect motion. Using a more sparse pattern 1504 at time T2 may reduce power consumption because fewer lines are projected in the sparse pattern than in the dense pattern. At time T3, while the sparse pattern 1504 is still being projected, motion is detected in the scene, for example, by detecting distortion of one or more grid lines of the pattern 1504 or by using a motion detection algorithm on an image of the scene. As motion is detected, the scene may have changed and a new depth map should be generated using the dense pattern 1502. The dense pattern 1502 is thus projected again at time T4. At time T5, no further motion is detected, so a sparse pattern 1506 that is more sparse than the initial sparse pattern 1504 may be projected onto the scene. The density of the sparse pattern 1506 (e.g., the distance between its lines) may be determined based on the size of the smallest detected object in the scene. For example, the smallest object detected in the T5 scene is the Mad Hatter, so the distances between the lines of the sparse pattern 1506 are chosen such that the motion of the Mad Hatter will be detected by the sparse pattern. The size of the object may be determined by generating a bounding box for the object. The distance between the horizontal gridlines may then be set based on the height of the bounding box, and the distance between the vertical gridlines may be set based on the width of the bounding box. At time T5, motion is detected when another person appears in the scene. Dense patterns 1504, which are denser than the previous patterns 1506 but less dense than the dense patterns 1502, may then be projected onto the scene to detect further motion or to generate or update a depth map of the scene.

In particular embodiments, portions of a scene may be sampled for changes by projecting a sparse pattern (such as lines) onto the portions of the scene and using a camera to detect reflections of the sparse pattern indicative of motion. The portion of the scene to be sampled may be selected such that the entire scene is covered after a number of frames, or such that objects of at least a threshold minimum size are detected after a number of frames. The size of the scanned portion may correspond to the distance between the projection lines of the sparse pattern and may be based on the expected size of the object. If a relatively large object is expected to move, a relatively small number of relatively widely spaced lines may be projected. For example, if the minimum size is one quarter of the scene area and the size of the scanned portion corresponds to the minimum size, movement of the object of the minimum size may be detected within four consecutive frames.

In particular embodiments, the low power scan mode may project a pattern onto a different portion (e.g., region) of the scene in each subsequent frame until the entire scene (or a region of a particular size) is scanned, at which point the process may repeat. In other examples, the sample portion of the scene may be selected randomly, or according to a particular application (such as searching for hands known to be connected to arms). The sampling process may modify the search based on feedback generated during the search (e.g., the location of the reflected line). In particular embodiments, the area of the scene portion may be less than the area of the scene. The portions may be identified by identifying a rectangle that surrounds the first projected pattern and dividing the rectangle that surrounds the first projected pattern into equally sized rectangles, where each different portion corresponds to a rectangle. In particular embodiments, when the sampling process detects a change, the dense pattern may be projected again, for example, in the next frame. In a particular embodiment, movement of the object may be detected based on the one or more second images, and the projector is configured to project the first projection pattern in response to detecting the movement of the object. In a particular embodiment, one or more emitters may project each portion of the first projected pattern, and to project each portion, the projector is configured to activate the emitter that projected the portion.

FIG. 16 illustrates an example method 1600 for reducing the grid optical pattern density. The method may begin at step 1610, where method 1600 may project onto a scene, using at least one projector, a plurality of projection patterns including a first projection pattern, the first projection pattern including a plurality of first projection features. At step 1620, method 1600 may capture a plurality of images including a first detection pattern corresponding to a reflection of a first projection pattern using a camera. At step 1630, the method 1600 may calculate a depth map of the scene based on the first projection pattern, the first detection pattern, and the relative positioning between the camera and the projector. At step 1640, method 1600 may project a second projection pattern comprising a plurality of second projection features onto a portion of the scene using the projector, where the second projection pattern is sparser than the first projection pattern. At step 1650, the method 1600 may capture a second detection pattern corresponding to a reflection of the second projection pattern using the camera. At step 1660, the method 1600 may calculate a depth map of the portion of the scene based on the second projection pattern, the second detection pattern, and the relative positioning of the camera and the projector. At step 1670, method 1600 may update the depth map of the scene based on the depth map of the scene portion.

Particular embodiments may repeat one or more steps of the method of fig. 16 where appropriate. Although this disclosure describes and illustrates particular steps of the method of fig. 16 as occurring in a particular order, this disclosure contemplates any suitable steps of the method of fig. 16 occurring in any suitable order. Further, although this disclosure describes and illustrates an example method for reducing the grid optical pattern density including particular steps of the method of fig. 16, this disclosure contemplates any suitable method for reducing the grid optical pattern density, including any suitable steps, which may or may not include all, some, or all of the steps of the method of fig. 16, where appropriate. Moreover, although this disclosure describes and illustrates particular components, devices, or systems performing particular steps of the method of fig. 16, this disclosure contemplates any suitable combination of any suitable components, devices, or systems performing any suitable steps of the method of fig. 16.

In particular embodiments, to reduce power consumption, a pattern of points or lines projected in a frame at a particular time may be divided into two or more partial patterns, and the partial patterns may be projected in successive time intervals. The camera may capture partial patterns at time intervals in a "partial" frame. Projecting a portion of the pattern in each frame instead of the entire pattern may reduce power consumption. For example, projecting half of the pattern lines in one frame and the other half of the pattern lines in the next frame may reduce power consumption by 50%. The camera may receive successive partial frames and combine them to reconstruct the complete pattern.

In particular embodiments, as a result of dividing the projected pattern into N partial patterns, each frame may be divided into N "partial" frames, and each partial frame may have 1/N of the complete pattern. The value N may be understood as an "interlace factor" and may be selected for a particular application depending on a trade-off between the speed of receiving the pattern and the desired battery power saving. The interlacing factor may be changed between frames to adapt to the scene. For example, fast detection may be used for a scenario where a person plays a piano, in which case all lines of the pattern may be used in each frame without segmenting the pattern (N ═ 1). When the person stops playing the piano, the interlace factor may be increased to N-5. For a camera that captures images at a speed of 60FPS and constructs a depth map while moving within a room, each frame can be divided into five frames while maintaining a frame rate of 60 FPS. In another example, one "line" (unit of pattern) may be projected in each frame. The camera frame rate may be changed to correspond to the projector frame rate.

Fig. 17 shows an example of dividing a grid light pattern into a plurality of sections to reduce power consumption. The pattern 1700 may be divided into four partial patterns, as indicated by dotted lines 1702, with dotted lines 1702 dividing the pattern 1700 into four quarters. Each quarter of the pattern 1700 corresponds to a partial pattern. Each partial pattern may be projected onto a respective portion of the scene over a period of time during which the camera may capture reflections from the partial pattern of the scene and generate a depth map of the area of the scene 1704 covered by the projected partial pattern 1706. The partial patterns may be projected at different times. At time T1, the first partial pattern 1706 is projected to the upper left corner of the scene 1704. At time T2, second partial pattern 1710 is projected onto the upper right corner of scene 1704. The camera may capture a reflection of second portion pattern 1710 from scene 1704 during the projection of second portion pattern 1710. The depth sensing system may identify the detected pattern of one quarter of the screen based on the reflection of the partial pattern. At time T3, third partial pattern 1714 is projected into the lower left corner of scene 1704. The camera may capture reflections of the third portion of the pattern 1714 from the scene 1704 during the time that the third portion of the pattern 1714 is projected. At time T4, the fourth partial pattern 1718 is projected to the lower left corner of the scene 1704. The camera may capture reflections of the third portion of the pattern 1718 from the scene 1704 during the time that the third portion of the pattern 1718 is projected.

In particular embodiments, when projecting the partial patterns, the depth sensing system may identify partially detected patterns based on reflections of the four partial patterns. Alternatively or additionally, the depth sensing system may identify the detected pattern based on a combination of reflections of the four partial patterns after the four partial patterns have been projected.

FIG. 18 shows an example method 1800 for dividing a grid light pattern into portions to be projected at different times. The method may begin at step 1810, where method 1800 may determine a specified number of different portions of the first projected pattern. In step 1820, the method 1800 may use at least one projector to project different determined portions of the first projected pattern at different times within the time interval. At step 1830, method 1800 may capture, using the camera, a plurality of images including a first detection pattern corresponding to a reflection of the first projected pattern, where the first detection pattern is based on a set of images captured by the camera at times based on different times during the time interval. At step 1840, method 1800 may generate an image of the first detected pattern based on a plurality of images, each image (a) including detected portions of the first detected pattern and (b) a reflection corresponding to a respective determined portion of the first projected pattern. At step 1840, the method 1800 may calculate a depth map based on the first projected pattern, the image of the first detected pattern, and the relative positioning between the camera and the at least one projector.

Particular embodiments may repeat one or more steps of the method of fig. 18 where appropriate. Although this disclosure describes and illustrates particular steps of the method of fig. 18 as occurring in a particular order, this disclosure contemplates any suitable steps of the method of fig. 18 occurring in any suitable order. Further, although this disclosure describes and illustrates an example method for dividing a grid light pattern into portions to be projected at different times that includes particular steps of the method of fig. 18, this disclosure contemplates any suitable method for dividing a grid light pattern into portions to be projected at different times, including any suitable steps, which may include all, some, or none of the steps of the method of fig. 18, where appropriate. Moreover, although this disclosure describes and illustrates particular components, devices, or systems performing particular steps of the method of fig. 18, this disclosure contemplates any suitable combination of any suitable components, devices, or systems performing any suitable steps of the method of fig. 18.

Fig. 19 shows an example computer system 1900. In certain embodiments, one or more computer systems 1900 perform one or more steps of one or more methods described or illustrated herein. In certain embodiments, one or more computer systems 1900 provide the functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems 1900 performs one or more steps of one or more methods described or illustrated herein or provides functions described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems 1900. Herein, reference to a computer system may include a computing device, and vice versa, where appropriate. Further, references to a computer system may include one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems 1900. The present disclosure contemplates computer system 1900 taking any suitable physical form. By way of example, and not limitation, computer system 1900 may be an embedded computer system, a system on a chip (SOC), a single board computer System (SBC) (e.g., a Computer On Module (COM) or a System On Module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a computer system mesh, a mobile phone, a Personal Digital Assistant (PDA), a server, a tablet computer system, an augmented/virtual reality device, or a combination of two or more of these systems. Where appropriate, computer system 1900 may include one or more computer systems 1900; may be monolithic or distributed; spanning a plurality of locations; spanning multiple machines; spanning multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1900 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitation. By way of example, and not by way of limitation, one or more computer systems 1900 may perform in real time or in batch mode one or more steps of one or more methods described or illustrated herein. One or more computer systems 1900 may perform one or more steps of one or more methods described or illustrated herein at different times or at different locations, where appropriate.

In a particular embodiment, the computer system 1900 includes a processor 1902, a memory 1904, a storage 1906, an input/output (I/O) interface 1908, a communication interface 1910, and a bus 1912. Although this disclosure describes and illustrates a particular computer system with a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

In a particular embodiment, the processor 1902 includes hardware for executing instructions (e.g., those making up a computer program). By way of example, and not limitation, to execute instructions, processor 1902 may retrieve (or retrieve) instructions from an internal register, an internal cache, memory 1904, or storage 1906; decode the instructions and execute them; and then write the one or more results to an internal register, internal cache, memory 1904, or storage 1906. In particular embodiments, processor 1902 may include one or more internal caches for data, instructions, or addresses. The present disclosure contemplates processor 1902 including any suitable number of any suitable internal caches, where appropriate. By way of example, and not limitation, processor 1902 may include one or more instruction caches, one or more data caches, and one or more Translation Lookaside Buffers (TLBs). The instructions in the instruction cache may be copies of instructions in memory 1904 or storage 1906, and the instruction cache may accelerate retrieval of those instructions by processor 1902. The data in the data cache may be: a copy of the data in memory 1904 or storage 1906 for operation by instructions executed at processor 1902; the results of previous instructions executed at processor 1902, for access by subsequent instructions executed at processor 1902 or for writing to memory 1904 or storage 1906; or other suitable data. The data cache may speed up read or write operations by the processor 1902. The TLB may accelerate virtual address translations for the processor 1902. In a particular embodiment, the processor 1902 may include one or more internal registers for data, instructions, or addresses. The present disclosure contemplates processor 1902 including any suitable number of any suitable internal registers, where appropriate. The processor 1902 may include one or more Arithmetic Logic Units (ALUs), where appropriate; may be a multi-core processor; or include one or more processors 1902. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

In a particular embodiment, the memory 1904 includes main memory for storing instructions to be executed by the processor 1902 or data to be operated on by the processor 1902. By way of example, and not limitation, computer system 1900 may load instructions from storage 1906 or another source (such as, for example, another computer system 1900) to memory 1904. The processor 1902 may then load the instructions from the memory 1904 into an internal register or internal cache. To execute instructions, processor 1902 may retrieve instructions from an internal register or internal cache and decode them. During or after execution of the instructions, processor 1902 may write one or more results (which may be intermediate results or final results) to an internal register or internal cache. The processor 1902 may then write one or more of these results to the memory 1904. In a particular embodiment, the processor 1902 executes only instructions in the one or more internal registers or internal caches or in the memory 1904 (but not the storage 1906 or elsewhere) and operates only on data in the one or more internal registers or internal caches or in the memory 1904 (but not the storage 1906 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor 1902 to memory 1904. Bus 1912 may include one or more memory buses, as described below. In a particular embodiment, one or more Memory Management Units (MMUs) reside between processor 1902 and memory 1904 and facilitate accesses to memory 1904 requested by processor 1902. In a particular embodiment, the memory 1904 includes Random Access Memory (RAM). The RAM may be volatile memory, where appropriate. The RAM may be Dynamic RAM (DRAM) or Static RAM (SRAM), where appropriate. Further, the RAM may be single-port RAM or multi-port RAM, where appropriate. The present disclosure contemplates any suitable RAM. The memory 1904 may include one or more memories 1904, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

In a particular embodiment, the storage 1906 includes mass storage for data or instructions. By way of example, and not limitation, storage 1906 may include a Hard Disk Drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Storage 1906 may include removable or non-removable (or fixed) media, where appropriate. Storage 1906 may be internal or external to computer system 1900, where appropriate. In a particular embodiment, the storage 1906 is non-volatile solid-state memory. In a particular embodiment, the storage 1906 includes Read Only Memory (ROM). Where appropriate, the ROM may be mask-programmed ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), Electrically Alterable ROM (EAROM), or flash memory, or a combination of two or more of these. The present disclosure contemplates mass storage 1906 in any suitable physical form. The storage 1906 may include one or more storage control units that facilitate communication between the processor 1902 and the storage 1906, where appropriate. Storage 1906 may include one or more storage 1906, where appropriate. Although this disclosure describes and illustrates a particular storage device, this disclosure contemplates any suitable storage device.

In certain embodiments, I/O interface 1908 includes hardware, software, or both that provide one or more interfaces for communication between computer system 1900 and one or more I/O devices. Computer system 1900 may include one or more of these I/O devices, where appropriate. One or more of these I/O devices may enable communication between a person and computer system 1900. By way of example, and not limitation, an I/O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet computer, touch screen, trackball, video camera, another suitable I/O device, or a combination of two or more of these devices. The I/O device may include one or more sensors. This disclosure contemplates any suitable I/O devices and any suitable I/O interfaces 1908 for them. The I/O interface 1908 may include one or more device or software drivers that enable the processor 1902 to drive one or more of these I/O devices, where appropriate. The I/O interfaces 1908 may include one or more I/O interfaces 1908, where appropriate. Although this disclosure describes and illustrates particular I/O interfaces, this disclosure contemplates any suitable I/O interfaces.

In particular embodiments, communication interface 1910 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication) between computer system 1900 and one or more other computer systems 1900 or one or more networks. By way of example, and not limitation, communication interface 1910 may include a Network Interface Controller (NIC) or network adapter for communicating with an ethernet or other wire-based network, or a Wireless NIC (WNIC) or wireless adapter for communicating with a wireless network (e.g., WI-FI), for example. The present disclosure contemplates any suitable networks and any suitable communication interfaces 1910 therefor. By way of example, and not limitation, computer system 1900 may communicate with an ad hoc network, a Personal Area Network (PAN), a Local Area Network (LAN), a Wide Area Network (WAN), a Metropolitan Area Network (MAN), or one or more portions of the internet, or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. By way of example, computer system 1900 may communicate with a Wireless PAN (WPAN) (such as, for example, a bluetooth WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a global system for mobile communications (GSM) network), or other suitable wireless network, or a combination of two or more of these. Computer system 1900 may include any suitable communication interface 1910 for any of these networks, where appropriate. Communication interface 1910 may include one or more communication interfaces 1910, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

In particular embodiments, bus 1912 includes hardware, software, or both to couple the components of computer system 1900 to each other. By way of example, and not limitation, the bus 1912 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or any other suitable bus or combination of two or more of these. Bus 1912 may include one or more buses 1912, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

Herein, where appropriate, one or more computer-readable non-transitory storage media may include one or more semiconductor-based or other Integrated Circuits (ICs) such as, for example, a Field Programmable Gate Array (FPGA) or an Application Specific IC (ASIC), a Hard Disk Drive (HDD), a hybrid hard disk drive (HHD), an Optical Disc Drive (ODD), a magneto-optical disc drive, a Floppy Disk Drive (FDD), a magnetic tape, a Solid State Drive (SSD), a RAM drive, a SECURE DIGITAL (SECURE DIGITAL) card or drive, any other suitable computer-readable non-transitory storage medium, or any suitable combination of two or more of these. Computer-readable non-transitory storage media may be volatile, nonvolatile, or a combination of volatile and nonvolatile, where appropriate.

Herein, unless explicitly indicated otherwise or indicated otherwise by context, "or" is inclusive and not exclusive. Thus, herein, "a or B" means "A, B, or both," unless expressly indicated otherwise or indicated otherwise by context. Further, "and" are both conjunctive and disjunctive unless expressly indicated otherwise or indicated otherwise by context. Thus, herein, "a and B" means "a and B, either jointly or individually," unless expressly indicated otherwise or indicated otherwise by context.

The scope of the present disclosure includes all changes, substitutions, variations, alterations, and modifications to the example embodiments described or illustrated herein that a person having ordinary skill in the art would comprehend. The scope of the present disclosure is not limited to the example embodiments described or illustrated herein. Moreover, although the present disclosure describes and illustrates respective embodiments herein as including particular components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that a person having ordinary skill in the art would understand. Furthermore, reference in the appended claims to an apparatus or system or to a component of an apparatus or system that is suitable for, arranged to, capable of, configured to, implemented, operable to, or operative to perform a particular function includes the apparatus, system, component, whether or not it or that particular function is activated, turned on, or unlocked, provided that the apparatus, system, or component is so adapted, arranged, enabled, configured, implemented, operable, or operative. Moreover, although this disclosure describes or illustrates particular embodiments as providing particular advantages, particular embodiments may provide some, all, or none of these advantages.