阅读说明：本技术 基于物理尺寸呈现环境 (Presenting environments based on physical dimensions ) 是由 P·乔特瓦尼 于 2020-08-18 设计创作，主要内容包括：本文所公开的各种具体实施包括用于生成具有缩放的计算机生成现实(CGR)对象的尺寸上准确的CGR环境的设备、系统和方法。在一些具体实施中,一种方法包括获取对应于物理环境的环境数据。基于该环境数据来识别位于该物理环境内的已知物理物品。该已知物理物品与已知尺寸相关联。基于该已知物理物品的该已知尺寸来确定该物理环境的物理尺寸。生成表示该物理环境的CGR环境。该CGR环境的虚拟尺寸是该物理环境的该物理尺寸的函数。(Various implementations disclosed herein include apparatus, systems, and methods for generating a dimensionally accurate Computer Generated Reality (CGR) environment with scaled CGR objects. In some implementations, a method includes obtaining environmental data corresponding to a physical environment. Known physical objects located within the physical environment are identified based on the environment data. The known physical object is associated with a known dimension. Determining a physical dimension of the physical environment based on the known dimension of the known physical object. A CGR environment is generated that represents the physical environment. The virtual size of the CGR environment is a function of the physical size of the physical environment.)

1. A method, comprising:

at a device comprising a non-transitory memory and one or more processors coupled with the non-transitory memory:

obtaining environmental data corresponding to a physical environment;

identifying a known physical object located within the physical environment based on the environment data, wherein the known physical object is associated with a known size;

determining a physical dimension of the physical environment based on the known dimension of the known physical object; and

generating a computer-generated reality (CGR) environment representing the physical environment, wherein a virtual dimension of the CGR environment is a function of the physical dimension of the physical environment.

2. The method of claim 1, wherein the environmental data comprises an image of the physical environment.

3. The method of claim 2, further comprising receiving the image of the physical environment from an image sensor.

4. The method of claim 3, further comprising determining a pose of the image sensor.

5. The method of claim 4, further comprising determining a scaling factor as a function of the pose.

6. The method of any of claims 1-5, wherein the environmental data comprises depth data.

7. The method of claim 6, further comprising receiving the depth data from a depth sensor.

8. The method of any of claims 1-7, further comprising performing at least one of semantic segmentation or instance segmentation on the environmental data to identify the known physical object.

9. The method of any one of claims 1 to 8, further comprising identifying an optical machine-readable representation of data associated with the known physical object.

10. The method of any one of claims 1 to 9, further comprising obtaining the known dimensions of the known physical object.

11. The method of any one of claims 1 to 10, further comprising retrieving the known dimensions of the known physical object from a data store.

12. The method of any one of claims 1 to 11, further comprising retrieving the known dimensions of the known physical object via a network.

13. The method of any one of claims 1 to 12, wherein the known physical object corresponds to a portion of the environmental data.

14. The method of claim 13, further comprising sending a query for an image search based on the portion of the environmental data corresponding to the known physical object.

15. The method of claim 14, further comprising receiving size information of the known physical object in response to the query.

16. The method of claim 14, further comprising receiving, in response to the query, size information for a physical object having a degree of similarity to the known physical object.

17. The method of any one of claims 1 to 16, further comprising sending a query based on a product identifier corresponding to the known physical object.

18. The method of claim 17, further comprising receiving size information of the known physical object in response to the query.

19. The method of claim 17, further comprising receiving, in response to the query, size information for a physical object having a degree of similarity to the known physical object.

20. The method of any one of claims 1 to 19, further comprising receiving user input indicating the known dimensions of the known physical object.

21. The method of any one of claims 1 to 20, further comprising determining the physical dimensions of the physical environment based on the known dimensions of the known physical object and a ratio of the known physical object to the physical environment.

22. An apparatus, comprising:

an environmental sensor;

a display;

one or more processors;

a non-transitory memory; and

one or more programs stored in the non-transitory memory that, when executed by the one or more processors, cause the apparatus to perform any of the methods of claims 1-21.

23. A non-transitory memory storing one or more programs that, when executed by one or more processors of a device, cause the device to perform any of the methods of claims 1-21.

24. An apparatus, comprising:

one or more processors;

a non-transitory memory; and

means for causing the apparatus to perform any one of the methods of claims 1-21.

25. A method, comprising:

at a device comprising an environmental sensor, a display, a non-transitory memory, and one or more processors coupled with the environmental sensor, the display, and the non-transitory memory:

displaying an Augmented Reality (AR) environment corresponding to a physical environment on the display;

determining to display a Computer Generated Reality (CGR) object in the AR environment, wherein the CGR object represents a physical object associated with a physical dimension;

identifying a known physical object located within the physical environment based on environmental data acquired via the environmental sensor, wherein the known physical object is associated with a known size;

determining a virtual size of the CGR object based on the known size of the known physical object and the physical size of the physical object represented by the CGR object; and

displaying the CGR object in the AR environment on the display according to the virtual size.

26. The method of claim 25, wherein the environment sensor comprises an image sensor, and the method further comprises acquiring, via the image sensor, image data corresponding to the physical environment.

27. The method of any of claims 25 to 26, further comprising displaying a video pass-through of the physical environment.

28. The method of any one of claims 25 to 26, further comprising displaying an optical passthrough of the physical environment.

29. The method of any of claims 25 to 28, further comprising obtaining a request to display the CGR object in the AR environment.

30. The method of any of claims 25-29, further comprising receiving user input to display the CGR object in the AR environment.

31. The method of any one of claims 25 to 30, further comprising sending a query based on a product identifier corresponding to the known physical object.

32. The method of claim 31, further comprising receiving size information for the known physical object in response to the query.

33. The method of claim 31, further comprising receiving, in response to the query, size information for a physical object having a degree of similarity to the known physical object.

34. The method of any one of claims 25 to 33, further comprising receiving user input indicative of dimensional information of the known physical object.

35. The method of any of claims 25-34, further comprising determining the virtual dimensions of the CGR object based on a distance between the device and the known physical object.

36. The method of any of claims 25-35, further comprising determining the virtual dimension of the CGR object based on a distance between the device and a placement location of the CGR object.

37. The method of any one of claims 25 to 36, further comprising determining the virtual dimension of the CGR object based on a distance between the known physical object and a placement location of the CGR object.

38. The method of any of claims 25 to 37, further comprising determining the virtual size of the CGR object based on a placement location of the CGR object.

39. The method of any of claims 25-38, further comprising performing at least one of semantic segmentation or instance segmentation on the environmental data to identify the known physical object.

40. The method of any one of claims 25 to 39, further comprising identifying an optical machine-readable representation of data associated with the known physical object.

41. The method of any of claims 25-40, further comprising determining the virtual size of the CGR object based on a virtual size of the AR environment.

42. An apparatus, comprising:

an environmental sensor;

a display;

one or more processors;

a non-transitory memory; and

one or more programs stored in the non-transitory memory that, when executed by the one or more processors, cause the apparatus to perform any of the methods of claims 25-41.

43. A non-transitory memory for storing one or more programs that, when executed by one or more processors of a device, cause the device to perform any of the methods of claims 25-41.

44. An apparatus, comprising:

one or more processors;

a non-transitory memory; and

means for causing the apparatus to perform any one of the methods of claims 25-41.

Technical Field

The present disclosure relates generally to the rendering of Computer Generated Reality (CGR) environments and objects.

Background

Some devices are capable of generating and presenting a Computer Generated Reality (CGR) environment. Some CGR environments include virtual environments that are simulated alternatives to physical environments. Some CGR environments include enhanced environments that are modified versions of the physical environment. Some devices that present CGR environments include mobile communication devices, such as smart phones, Head Mounted Displays (HMDs), glasses, Heads Up Displays (HUDs), and optical projection systems.

Disclosure of Invention

Various implementations disclosed herein include apparatus, systems, and methods for generating a dimensionally accurate Computer Generated Reality (CGR) environment with scaled CGR objects. In some implementations, a method includes obtaining environmental data corresponding to a physical environment. Known physical objects located within the physical environment are identified based on the environment data. The known physical object is associated with a known dimension. Determining a physical dimension of the physical environment based on the known dimension of the known physical object. A CGR environment is generated that represents the physical environment. The virtual size of the CGR environment is a function of the physical size of the physical environment.

Various implementations disclosed herein include apparatus, systems, and methods for instantiating a CGR object in an Augmented Reality (AR) environment and scaling the CGR object based on size information associated with the CGR object and known dimensions of a known physical object. In some implementations, a method includes displaying an AR environment corresponding to a physical environment. It is determined to display the CGR object in the AR environment. The CGR object represents a physical object associated with a physical dimension. Known physical objects located within the physical environment are identified. The known physical object is associated with a known dimension. The virtual dimensions of the CGR object are determined based on the known dimensions of the known physical object and the physical dimensions of the physical object represented by the CGR object. The CGR object is displayed in the AR environment according to the virtual size.

According to some implementations, an apparatus includes one or more processors, non-transitory memory, and one or more programs. In some implementations, the one or more programs are stored in a non-transitory memory and executed by the one or more processors. In some implementations, the one or more programs include instructions for performing or causing performance of any of the methods described herein. According to some implementations, a non-transitory computer-readable storage medium has stored therein instructions that, when executed by one or more processors of a device, cause the device to perform or cause performance of any of the methods described herein. According to some implementations, an apparatus includes one or more processors, non-transitory memory, and means for performing or causing performance of any of the methods described herein.

Drawings

Accordingly, the present disclosure may be understood by those of ordinary skill in the art and a more particular description may be had by reference to certain illustrative embodiments, some of which are illustrated in the accompanying drawings.

FIG. 1A illustrates an exemplary operating environment in accordance with some implementations.

FIG. 1B illustrates another exemplary operating environment in accordance with some implementations.

FIG. 2 illustrates an exemplary system for generating a CGR environment in accordance with various implementations.

Fig. 3 is a block diagram of an example CGR content module, according to some implementations.

Fig. 4A-4C are a flowchart representation of a method for generating a CGR environment according to some implementations.

Fig. 5 is a block diagram of an apparatus according to some implementations.

Fig. 6 illustrates an exemplary system for displaying CGR objects in an Augmented Reality (AR) environment, in accordance with various implementations.

Fig. 7 is a block diagram of an example CGR content module, according to some implementations.

Fig. 8A-8C are flow diagram representations of a method for displaying CGR objects in an AR environment, according to some implementations.

Fig. 9 is a block diagram of an apparatus according to some implementations.

In accordance with common practice, the various features shown in the drawings may not be drawn to scale. Accordingly, the dimensions of the various features may be arbitrarily expanded or reduced for clarity. Additionally, some of the figures may not depict all of the components of a given system, method, or apparatus. Finally, throughout the specification and drawings, like reference numerals may be used to refer to like features.

Detailed Description

Numerous details are described in order to provide a thorough understanding of example implementations shown in the drawings. The drawings, however, illustrate only some example aspects of the disclosure and therefore should not be considered limiting. It will be understood by those of ordinary skill in the art that other effective aspects and/or variations do not include all of the specific details described herein. Moreover, well-known systems, methods, components, devices, and circuits have not been described in detail so as not to obscure more pertinent aspects of the example implementations described herein.

The present disclosure provides methods, systems, and/or devices for generating a dimensionally accurate Computer Generated Reality (CGR) environment with scaled CGR objects. In some implementations, a method includes obtaining environmental data corresponding to a physical environment. Known physical objects located within the physical environment are identified based on the environment data. The known physical object is associated with a known dimension. Determining a physical dimension of the physical environment based on the known dimension of the known physical object. A CGR environment is generated that represents the physical environment. The virtual size of the CGR environment is a function of the physical size of the physical environment.

In some implementations, a device generates and presents computer-generated reality (CGR) content that includes a CGR environment having a virtual size proportional to a physical size of the physical environment. In some implementations, based on the sensor information, the controller detects a physical object in the physical environment and obtains a size of the physical object, for example, by searching a database that includes information about the physical object. In some implementations, the controller generates a semantic construct of the physical environment. The semantic construct may include a CGR representation of the physical object with a virtual size proportional to the physical size of the physical object. In some implementations, if the detected physical object has a certain degree of similarity to a physical object of known size, the controller uses the known size of the physical object to determine the relative sizes of the other physical objects and the physical environment based on the sensor information.

The present disclosure provides methods, systems, and/or devices for instantiating a CGR object in an Augmented Reality (AR) environment and scaling the CGR object based on size information associated with the CGR object and known dimensions of a known physical object. In some implementations, a method includes displaying an AR environment corresponding to a physical environment. It is determined to display the CGR object in the AR environment. The CGR object represents a physical object associated with a physical dimension. Known physical objects located within the physical environment are identified. The known physical object is associated with a known dimension. The virtual dimensions of the CGR object are determined based on the known dimensions of the known physical object and the physical dimensions of the physical object represented by the CGR object. The CGR object is displayed in the AR environment according to the virtual size.

In some implementations, a CGR object in an Augmented Reality (AR) environment is scaled based on a known size of a known physical object. For example, the power outlet may be identified in an AR environment corresponding to a living room. The power outlet is governed by a standard and has a known height (e.g., 4 inches or about 10 centimeters). In some implementations, CGR objects (such as chairs) are scaled based on power outlets. More generally, the CGR object may be scaled based on one or more of a known size of a known physical object, a distance of the known physical object from the device, a size of a physical object corresponding to the CGR object, and/or a distance that the CGR object is to be placed.

A physical environment refers to a physical world in which people can sense and/or interact without the aid of an electronic system. Physical environments such as physical parks include physical objects such as physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

In contrast, a computer-generated reality (CGR) environment refers to a fully or partially simulated environment in which people perceive and/or interact via electronic systems. In CGR, a subset of the human's physical movements, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one laws of physics. For example, the CGR system may detect head rotations of a person and in response adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in the physical environment. In some cases (e.g., for accessibility reasons), adjustments to the characteristics of virtual objects in the CGR environment may be made in response to representations of physical motion (e.g., voice commands).

A person may utilize any of their senses to sense and/or interact with CGR objects, including vision, hearing, touch, taste, and smell. For example, a person may sense and/or interact with audio objects that create a 3D or spatial audio environment that provides a perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sound from a physical environment with or without computer-generated audio. In some CGR environments, a person may sense and/or interact only with audio objects.

Examples of CGR include virtual reality and mixed reality.

A Virtual Reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. The VR environment includes a plurality of virtual objects that a person can sense and/or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person may sense and/or interact with a virtual object in the VR environment through simulation of the presence of the person within the computer-generated environment and/or through simulation of a subset of the physical movements of the person within the computer-generated environment.

In contrast to VR environments that are designed to be based entirely on computer-generated sensory inputs, a Mixed Reality (MR) environment refers to a simulated environment that is designed to introduce sensory inputs from a physical environment or representations thereof in addition to computer-generated sensory inputs (e.g., virtual objects). On a virtual continuum, a mixed reality environment is anything between the full physical environment as one end and the virtual reality environment as the other end, but not both ends.

In some MR environments, computer-generated sensory inputs may be responsive to changes in sensory inputs from the physical environment. Additionally, some electronic systems for presenting MR environments may track position and/or orientation relative to a physical environment to enable virtual objects to interact with real objects (i.e., physical objects or representations thereof from the physical environment). For example, the system may cause movement such that the virtual tree appears to be stationary relative to the physical ground.

Examples of mixed reality include augmented reality and augmented virtual.

An Augmented Reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed over a physical environment or representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person may directly view the physical environment. The system may be configured to present the virtual object on a transparent or translucent display such that the human perceives the virtual object superimposed over the physical environment with the system. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the image or video with the virtual object and presents the combination on the opaque display. A person utilizes the system to indirectly view the physical environment via an image or video of the physical environment and perceive a virtual object superimposed over the physical environment. As used herein, video of the physical environment displayed on the opaque display is referred to as "pass-through video," meaning that the system captures images of the physical environment using one or more image sensors and uses those images when rendering the AR environment on the opaque display. Further alternatively, the system may have a projection system that projects the virtual object into the physical environment, for example as a hologram or on a physical surface, so that a person perceives the virtual object superimposed on the physical environment with the system.

Augmented reality environments also refer to simulated environments in which representations of a physical environment are converted by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., viewpoint) that is different from the perspective captured by the imaging sensor. As another example, a representation of a physical environment may be transformed by graphically modifying (e.g., magnifying) a portion thereof, such that the modified portion may be a representative but not real version of the original captured image. As another example, a representation of a physical environment may be transformed by graphically eliminating portions thereof or blurring portions thereof.

An Augmented Virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the face of a person is realistically reproduced from an image taken of a physical person. As another example, the virtual object may take the shape or color of the physical object imaged by the one or more imaging sensors. As another example, the virtual object may employ a shadow that conforms to the positioning of the sun in the physical environment.

There are many different types of electronic systems that enable a person to sense and/or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), display-integrated vehicle windshields, display-integrated windows, displays formed as lenses designed for placement on a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smart phones, tablets, and desktop/laptop computers. The head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, the head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). The head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and/or one or more microphones for capturing audio of the physical environment. The head mounted system may have a transparent or translucent display instead of an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eye. The display may utilize digital light projection, OLED, LED, uuled, liquid crystal on silicon, laser scanning light sources, or any combination of these technologies. The medium may be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one implementation, a transparent or translucent display may be configured to be selectively rendered opaque. Projection-based systems may employ retinal projection techniques that project a graphical image onto a person's retina. The projection system may also be configured to project the virtual object into the physical environment, for example as a hologram or on a physical surface.

FIG. 1A illustrates an exemplary operating environment 100 according to some implementations. While relevant features are shown, those of ordinary skill in the art will recognize from the present disclosure that various other features are not shown for the sake of brevity and so as not to obscure more pertinent aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, operating environment 100 includes electronic device 102 and controller 104. In some implementations, the electronic device 102 is or includes a smartphone, a tablet, a laptop, and/or a desktop computer. The electronic device 102 may be worn or carried by the user 106.

As shown in fig. 1A, the electronic device 102 and/or the controller 104 obtain (e.g., receive, retrieve, and/or detect) environmental data corresponding to the physical environment 108. For example, the environmental data may include images or video captured by an image sensor 110 (such as a camera). In some implementations, the environmental data includes depth information captured by a depth sensor.

In some implementations, the electronic device 102 and/or the controller 104 identify the known physical object 112 in the physical environment 108 based on the environment data. For example, in some implementations, the electronic device 102 and/or the controller 104 perform semantic segmentation and/or instance segmentation on the environmental data to detect the known physical object 112. In some implementations, the electronic device 102 and/or the controller 104 identifies an optical machine-readable representation (e.g., a barcode or QR code) of data associated with the physical object. The optical machine-readable representation of the data may be used to identify the known physical object 112.

The known physical object 112 is associated with a known dimension 114 (e.g., a height, length, width, volume, and/or area of the known physical object 112). In some implementations, the electronic device 102 and/or the controller 104 determine (e.g., estimate) a physical dimension 116 of the physical environment 108 (e.g., a height, a length, a width, a volume, and/or an area of the physical environment 108) based on the known dimension 114. In some implementations, the electronic device 102 and/or the controller 104 obtain the known dimensions 114, for example, from a data store or via a network. In some implementations, the electronic device 102 and/or the controller 104 performs an image search based on a portion of the environmental data corresponding to the known physical object 112. In some implementations, the electronic device 102 and/or the controller 104 determine the physical dimension 116 based on the known dimension 114 and a known ratio of the physical object 112 to the physical environment 108.

Referring now to fig. 1B, in some implementations, the electronic device 102 and/or the controller 104 may present a computer-generated reality (CGR) environment 120 representing the physical environment 108. In some implementations, the CGR environment 120 includes a virtual environment that is a simulated replacement for the physical environment 108. For example, the CGR environment 120 may be simulated by the electronic device 102 and/or the controller 104. In such implementations, the CGR environment 120 is different from the physical environment 108 in which the electronic device 102 is located.

In some implementations, the CGR environment 120 includes an enhanced environment that is a modified version of the physical environment 108. For example, in some implementations, the electronic device 102 and/or the controller 104 modify (e.g., enhance) the physical environment 108 in which the electronic device 102 is located to generate the CGR environment 120. In some implementations, the electronic device 102 and/or the controller 104 generate the CGR environment 120 by simulating a copy of the physical environment 108 in which the electronic device 102 is located. In some implementations, the electronic device 102 and/or the controller 104 generate the CGR environment 120 by removing and/or adding terms from a simulated copy of the physical environment 108 in which the electronic device 102 is located.

In some implementations, the CGR environment 120 is associated with a virtual dimension 122 (e.g., a height, length, width, volume, and/or area of the CGR environment 120). In some implementations, the virtual dimension 122 is a function of the physical dimension 116 of the physical environment 108. For example, the virtual dimension 122 is proportional to the physical dimension 116 (e.g., the ratio between the physical height and the physical width of the physical environment 108 is approximately the same as the ratio between the virtual height and the virtual width of the CGR environment 120).

In some implementations, the CGR environment 120 is an Augmented Reality (AR) environment corresponding to the physical environment 108. For example, the CGR environment 120 may be rendered as an optically transparent transmission of the physical environment 108, wherein one or more CGR objects are rendered, with the physical environment 108 as a background, e.g., overlaid over the physical environment. In some implementations, the image sensor 110 acquires image data corresponding to the physical environment 108 and the CGR environment 120 is rendered as a video pass-through of the physical environment 108. In video-through, the electronic device 102 and/or the controller 104 displays one or more CGR objects having a CGR representation of the physical environment 108.

In some implementations, the electronic device 102 and/or the controller 104 determines to display the CGR object 124 in the CGR environment 120. The CGR object 120 represents a physical object associated with a physical dimension. For example, the electronic device 102 and/or the controller 104 may determine to display a CGR chair, which represents a physical chair associated with a physical size (e.g., a height of the physical chair).

In some implementations, the electronic device 102 and/or the controller 104 identify a known physical object 126 in the physical environment 108. The known physical object 126 may be the same physical object as the known physical object 112 shown in FIG. 1A, or may be a different physical object. The known physical object 126 is associated with a known dimension 128. For example, if the physical object 126 is known to be an electrical outlet, the electronic device 102 and/or the controller 104 may determine that the electrical outlet has a known height (e.g., 4 inches or about 10 centimeters, e.g., as defined by standards promulgated by standards bodies, such as the National Electrical Code (NEC)). In some implementations, the electronic device 102 and/or the controller 104 obtain the known dimensions 128 from a data store or over a network.

In some implementations, the electronic device 102 and/or the controller 104 determine the virtual dimension 130 of the CGR object 124 based on the known dimension 128 and the physical dimension of the physical object represented by the CGR object 124. For example, if the CGR object 124 represents a chair, the virtual dimension 130 may be the height of the CGR object 124. The electronic device 102 and/or the controller 104 may determine the height of the CGR object 124 based on the height of the power outlets in the physical environment 108 and the height of the physical chair represented by the CGR object 124.

In some implementations, a Head Mounted Device (HMD) worn by the user 106 presents (e.g., displays) a computer-generated reality (CGR) environment 120 according to various implementations. In some implementations, the HMD includes an integrated display (e.g., a built-in display) that displays the CGR environment 120. In some implementations, the HMD includes a head-mounted housing. In various implementations, the head-mounted housing includes an attachment region to which another device having a display may be attached. For example, in some implementations, the electronic device 102 of fig. 1A may be attached to a head-mounted housing. In various implementations, the head-mounted housing is shaped to form a receiver for receiving another device (e.g., electronic device 102) that includes a display. For example, in some implementations, the electronic device 102 slides or snaps to or is otherwise attached to the head-mounted housing. In some implementations, a display of a device attached to the head-mounted housing presents (e.g., displays) the CGR environment 120. In various implementations, examples of the electronic device 104 include a smartphone, a tablet, a media player, a laptop, and so forth.

FIG. 2 illustrates an exemplary system 200 for generating a CGR environment in accordance with various implementations. In some implementations, the environmental sensor 202 acquires environmental data 204 corresponding to a physical environment. For example, in some implementations, the environment sensor 202 includes an image sensor 206, such as a camera, that acquires an image 208 of the environment.

In some implementations, the image 208 is a still image. In some implementations, the image 208 is an image frame that forms part of a video feed. The image 208 includes a plurality of pixels. Some of the pixels (e.g., the first set of pixels) represent an object. Other pixels (e.g., a second set of pixels) represent the background, e.g., portions of the image 208 that do not represent the object. It should be understood that pixels representing one object may represent the background of a different object.

In some implementations, the environmental sensors 202 include a depth sensor 210 that acquires depth data 212 corresponding to the physical environment. The depth data 212 may be used separately from or in conjunction with the image 208 to identify one or more objects in the physical environment.

In some implementations, the CGR content module 214 receives the environmental data 204 from the environmental sensors 202. In some implementations, the CGR content module 214 identifies known physical objects in the physical environment based on the environment data 204. For example, the CGR content module 214 may perform semantic segmentation and/or instance segmentation on the environmental data 204 to identify known physical objects. In some implementations, the environmental data 204 includes an image, and the CGR content module 214 applies one or more filters and/or masks to the image to characterize pixels in the image as being associated with respective objects (such as known physical objects).

In some implementations, the image 208 includes an optically machine-readable representation of data associated with a known physical object (e.g., a barcode or QR code). The CGR content module 214 may, for example, send a query to a product database to obtain information identifying known physical objects.

The known physical object is associated with a known dimension. In some implementations, the CGR content module 214 obtains dimensional information for known physical objects. For example, the CGR content module 214 may send a query including information identifying known physical objects to a data store 216 or to a service via a network 218, such as a Local Area Network (LAN) or the internet. In some implementations, the information identifying the known physical object includes a semantic tag, a product identifier, and/or an image. In response to the query, the CGR content module 214 may receive size information for the known physical object. In some implementations, the CGR content module 214 receives size information for a physical object having a degree of similarity to a known physical object if size information for the known physical object is not available.

In some implementations, the CGR content module 214 determines the physical dimensions of the physical environment based on the known dimensions of known physical objects. In some implementations, the CGR content module 214 determines the physical dimensions of the physical environment based on the known dimensions of the known physical object (e.g., the dimensional information received in response to the query) and the ratio of the known physical object to the physical environment. For example, if the CGR content module 214 identifies a known physical object as a desk having a known width of two meters, and the desk occupies half the length of the wall, the CGR content module 214 may determine that the wall is four meters long.

In some implementations, the CGR content module 214 generates a CGR environment that represents the physical environment. The CGR environment is associated with a virtual dimension that is a function of the physical dimensions of the physical environment. The CGR content module 214 may provide the CGR environment to a display engine 220, which prepares the CGR environment for output using a display 222.

Fig. 3 is a block diagram of an exemplary CGR content module 300 according to some implementations. In some implementations, the CGR content module 300 implements the CGR content module 214 shown in fig. 2. The data obtainer 310 may obtain environmental data 302 corresponding to a physical environment.

In some implementations, the environmental data 302 includes an image 304. In some implementations, the image 304 is a still image. In some implementations, the image 304 is an image frame that forms part of a video feed. The image 304 includes a plurality of pixels. Some of the pixels (e.g., the first set of pixels) represent an object. Other pixels (e.g., a second set of pixels) represent the background, e.g., portions of image 304 that do not represent the object. It should be understood that pixels representing one object may represent the background of a different object.

In some implementations, the environment data 302 includes depth data 306 corresponding to a physical environment. The depth data 306 may be used independently of or in conjunction with the image 304 to identify one or more objects in the physical environment.

In some implementations, the data obtainer 310 can obtain the optical machine-readable representation 308 of the data associated with the physical object. The optical machine-readable representation 308 may be implemented as a barcode or QR code, for example. In some implementations, the optical machine-readable representation 308 is part of the image 304. In some implementations, the optical machine-readable representation 308 is captured separately from the image 304.

In some implementations, the object analyzer 320 identifies known physical objects in the physical environment based on one or more of the image 304, the depth data 306, and/or the optical machine-readable representation 308. In some implementations, the object analyzer 320 performs semantic segmentation and/or instance segmentation on the environmental data 302 (e.g., the image 304) to identify known physical objects. In some implementations, the known physical object is represented by a portion of the image 304, and the object analyzer 320 performs semantic segmentation and/or instance segmentation on the portion of the image 304 to identify the known physical object.

In some implementations, the object analyzer 320 determines an object identifier 322, such as a semantic tag and/or a product identifier, that identifies a known physical item. In some implementations, the object analyzer 320 determines the object identifier 322 of the known physical object based on available information related to the physical object that corresponds to or has some degree of similarity to the known physical object. This information may be obtained from one or more sources.

For example, in some implementations, the object analyzer 320 determines the object identifier 322 based on information received from a database 324 (e.g., a local database). For example, the database 324 may store product specifications of a physical object (e.g., a chair) corresponding to a known physical object (e.g., of the same model of the known physical object). In some implementations, the database 324 stores product specifications for physical objects that have a degree of similarity (e.g., a similarity threshold) to known physical objects. For example, if a product specification is not available for the same chair model corresponding to a known physical object, the object analyzer 320 may use the product specification for a similar chair model.

In some implementations, the sizer 330 receives the object identifier 322 and determines the known size of the known physical object. In some implementations, the size determiner 330 obtains size information for known physical objects. For example, the sizer 330 may send the query to the data store 326 or a service accessible via a network 328 (e.g., a local area network or the internet). The data store 326 may store dimensional information for a plurality of known physical objects.

The query may include information identifying the known physical object, such as the object identifier 322 or an image of the known physical object. In response to the query, the size determiner 330 may receive size information for the known physical object. In some implementations, the size determiner 330 receives size information of a physical object having a degree of similarity to a known physical object if size information of the known physical object is not available. For example, if the known physical object is a chair and the data store 326 does not store dimension information for the same chair model corresponding to the known physical object, the dimension determiner 330 may instead receive dimension information for a similar chair model.

In some implementations, the sizer 330 determines the physical size of the physical environment based on the known size of the known physical object. In some implementations, the size determiner 330 determines the physical size of the physical environment based on the known size and the ratio of the known physical object to the physical environment. For example, a known physical object may be a desk positioned along a wall of an office, and a known dimension of the desk may be a width of two meters. If the width of the desk to wall ratio is 1:2 (e.g., the desk occupies half of the wall along which the desk is located), the size determiner 330 may determine that the wall along which the desk is located is four meters long. In some implementations, the size determiner 330 determines other physical sizes of the physical environment based on the determination. For example, if the height of the wall is three-quarters of the length of the wall, the sizer 330 may determine that the wall is three meters high.

In some implementations, environment generator 340 generates a CGR environment that represents (e.g., models) a physical environment. In some implementations, the CGR environment is a computer-generated model of a physical environment. The CGR environment may be output as part of a CGR content item 342, which may also include one or more CGR objects. In some implementations, the CGR environment has a virtual size, e.g., a number of pixels. The virtual size is a function of the physical size of the physical environment. For example, in some implementations, environment generator 340 determines a number of pixels of a physical size for rendering the physical environment.

Fig. 4A-4C are a flowchart representation of a method 400 for generating a CGR environment, according to some implementations. In various implementations, the method 400 is performed by a device (e.g., the electronic device 102 or the controller 104 shown in fig. 1A and 1B, or the system 200 shown in fig. 2). In some implementations, the method 400 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, the method 400 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., memory). Briefly, in various implementations, the method 400 includes obtaining environment data corresponding to a physical environment, identifying a known physical object located in the physical environment based on the environment data, determining a physical dimension of the physical environment based on the known dimension of the known physical object, and generating a CGR environment representing the physical environment in virtual dimensions based on the physical dimension of the physical environment.

Referring to FIG. 4A, as shown at block 410, in various implementations, method 400 includes obtaining environmental data corresponding to a physical environment. Referring to FIG. 4B, as shown at block 410a, the environmental data may include an image of the physical environment. In some implementations, the image is a still image. In some implementations, the image is an image frame that forms part of a video feed. The image includes a plurality of pixels.

In some implementations, as shown at block 410b, the method 400 includes receiving an image of the physical environment from an image sensor (such as a camera). The image sensor may be characterized by a pose (e.g., a transformation that may be applied to a two-dimensional image captured by the image sensor to determine a three-dimensional physical environment represented by the image). In some implementations, as shown in block 410c, the pose of the image sensor is determined. As shown in block 410d, in some implementations, the scaling factor is determined based on the pose. Determining the scaling factor may be advantageous to correct for apparent distortion of the image attributable to the pose of the image sensor.

In some implementations, as shown in block 410e, the environmental data includes depth data. The depth data may be used independently of or in conjunction with the image to identify one or more objects in the physical environment. In some implementations, as shown in block 410f, depth data is received from a depth sensor.

As indicated at block 420, in some implementations, the method 400 includes identifying known physical objects located within the physical environment based on the environment data. The known physical object is associated with a known dimension. In some implementations, for example, known physical objects are identified based on the images. In some implementations, known physical objects are identified based on the depth data.

As shown in block 420a, in some implementations, semantic segmentation and/or instance segmentation is performed on the environmental data to identify known physical objects. For example, in some implementations, a portion of the image (e.g., the first set of pixels) represents a known physical object. One or more filters and/or masks may be applied to the image to distinguish the first set of pixels from a second set of pixels representing a background (e.g., a portion of the image that does not represent a known physical object). Semantic segmentation may be performed to associate known physical objects with semantic tags identifying the type of known physical object (e.g., "chair"). Instance segmentation may be performed to associate known physical objects with semantic tags that distinguish the known physical objects from other physical objects of similar type (e.g., "chair 1"). In some implementations, the semantic segmentation and/or the instance segmentation generates semantic tags that identify a model of a known physical object, e.g., a particular model of a chair.

As shown in block 420b, in some implementations, the method 400 includes identifying an optical machine-readable representation of data associated with a known physical object. The optical machine-readable representation may be implemented as a barcode or QR code, for example. In some implementations, the optical machine-readable representation is part of the environmental data. In some implementations, the optical machine-readable representation is captured separately from the environmental data, e.g., during subsequent scans and/or using different sensors. In some implementations, the optical machine-readable representation identifies a model of a known physical object.

As indicated at block 430, in some implementations, the method 400 includes determining a physical dimension of the physical environment based on a known dimension of a known physical object. Referring to fig. 4C, as shown in block 430a, the method 400 may include obtaining a known dimension of a known physical object.

In some implementations, a known size of a known physical object is retrieved from a data store, as shown in block 430 b. For example, if the physical object is known to be a desk of a particular model, the data store may return the length, width, and/or height of the particular model of the desk. If the data store does not have dimension information for a particular model of desk, the data store may return dimension information for a similar model of desk or generalized dimension information for a common (e.g., hypothetical) desk. The data store may include size information for a plurality of known physical objects (e.g., a plurality of desk models and/or a universal desk).

In some implementations, the known dimensions of the known physical object are retrieved via a network (e.g., a service via a Local Area Network (LAN) or the internet), as shown in block 430 c. For example, if the physical object is known to be a desk of a particular model, the service may return to the length, width, and/or height of the particular model of the desk. If the service does not have dimension information for a particular model of desk, the service may return dimension information for a similar model of desk or generalized dimension information for a general (e.g., phantom) desk. The service may include size information for a plurality of known physical objects (e.g., a plurality of desk models and/or a universal desk).

In some implementations, a known size of a known physical object is returned in response to a query. For example, in some implementations, the physical object is known to correspond to a portion of the environmental data (e.g., a first set of pixels), as shown in block 430 d. As shown at block 430e, the method 400 may include sending a query for an image search based on a portion of the environmental data, e.g., a first set of pixels, to which the physical object is known to correspond. In some implementations, size information for a known physical object is received in response to a query, as shown in block 430 f. In some implementations, size information for physical objects having a degree of similarity to known physical objects is received in response to the query, as shown in block 430 g. For example, if size information is not available for a particular model of desktop indicated in the query, the size information may be returned for similar desktops.

As shown at block 430h, the method 400 may include sending a query based on a product identifier corresponding to a known physical object. For example, the product identifier may be a semantic tag. In some implementations, the product identifier identifies a particular model of a known physical object. In some implementations, size information for a known physical object is received in response to a query, as shown in block 430 i. In some implementations, size information for physical objects having a degree of similarity to known physical objects is received in response to the query, as shown in block 430 j. For example, if size information is not available for a particular model of desktop indicated in the query, the size information may be returned for similar desktops.

In some implementations, as shown in block 430k, the method 400 includes receiving user input indicating a known size of a known physical object. For example, a user may provide user input indicating the length, width, and/or height of a desk using a keyboard, mouse, and/or gesture controls on a touch screen interface.

In some implementations, the physical dimensions of the physical environment are determined based on the known dimensions of the known physical object and the ratio of the known physical object to the physical environment, as shown in block 430 l. For example, a known physical object may be a desk positioned along a wall of an office, and a known dimension of the desk may be a width of two meters. If the width of the desk to wall ratio is 1:2 (e.g., the desk occupies half of the wall along which the desk is located), the size determiner 330 may determine that the wall along which the desk is located is four meters long. In some implementations, the size determiner 330 determines other physical sizes of the physical environment based on the determination. For example, if the height of the wall is three-quarters of the length of the wall, the sizer 330 may determine that the wall is three meters high.

In some implementations, as shown at block 440, the method 400 includes generating a CGR environment representing a physical environment. In some implementations, the CGR environment has a virtual size, e.g., a number of pixels. The virtual size is a function of the physical size of the physical environment. For example, in some implementations, environment generator 340 determines a number of pixels of a physical size for rendering the physical environment.

Fig. 5 is a block diagram of a device 500 enabled with one or more components of the device (e.g., the electronic device 102 and/or the controller 104 shown in fig. 1A and 1B, or the system 200 shown in fig. 2) according to some implementations. While some specific features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more pertinent aspects of the particular implementations disclosed herein. To this end, as a non-limiting example, in some implementations, the device 500 includes one or more processing units (CPUs) 502, one or more input/output (I/O) devices 506, one or more communication interfaces 508, one or more programming interfaces 510, a memory 520, and one or more communication buses 504 for interconnecting these and various other components.

In some implementations, a communication interface 508 is provided to establish and maintain, among other uses, a metadata tunnel between the cloud-hosted network management system and at least one private network that includes one or more compatible devices. In some implementations, the one or more communication buses 504 include circuitry to interconnect and control communications between system components. The memory 520 includes high speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid state storage devices. Memory 520 optionally includes one or more storage devices remotely located from the one or more CPUs 502. Memory 520 includes non-transitory computer-readable storage media.

In some implementations, the memory 520 or a non-transitory computer-readable storage medium of the memory 520 stores programs, modules, and data structures, or a subset thereof, including the optional operating system 530, data obtainer 310, object analyzer 320, sizer 330, and environment generator 340. As described herein, the data fetcher 310 may include instructions 310a and/or heuristics and metadata 310b for retrieving environmental data corresponding to a physical environment. As described herein, the object analyzer 320 may include instructions 320a and/or heuristics and metadata 320b for identifying known physical items in a physical environment based on environmental data. As described herein, the sizer 330 may include instructions 330a and/or heuristics and metadata 330b for determining a physical size of the physical environment based on a known size of a known physical object. As described herein, environment generator 340 may include instructions 340a and/or heuristics and metadata 340b for generating a CGR environment that represents a physical environment.

In some implementations, the one or more I/O devices 506 include environmental sensors to capture environmental data. In some implementations, the environmental sensor includes an image sensor (e.g., a camera) for capturing image data representing a set of one or more images. In some implementations, the environmental sensor includes a depth sensor (e.g., a depth camera) for capturing depth data. In some implementations, the one or more I/O devices 506 include a display for displaying the CGR environment. In some implementations, the display includes an optical see-through display (e.g., an optical pass-through for displaying a physical environment). In some implementations, the display includes an opaque display (e.g., a video pass-through to display the physical environment).

It should be understood that fig. 5 serves as a functional description of various features that may be present in a particular implementation, as opposed to a structural schematic of the implementations described herein. As one of ordinary skill in the art will recognize, the items displayed separately may be combined, and some items may be separated. For example, some of the functional blocks shown separately in fig. 5 may be implemented as single blocks, and various functions of a single functional block may be implemented by one or more functional blocks in various implementations. The actual number of blocks and the division of particular functions and how features are allocated therein will vary depending on the particular implementation and, in some implementations, will depend in part on the particular combination of hardware, software, and/or firmware selected for a particular implementation.

Fig. 6 illustrates an exemplary system 600 for displaying CGR objects in an Augmented Reality (AR) environment 602, according to various implementations. In some implementations, the display 604 displays the AR environment 602. In some implementations, the display 604 is implemented as part of the electronic device 102. In some implementations, the display 604 is implemented as part of an HMD.

In some implementations, the AR environment 602 corresponds to a physical environment. For example, the AR environment 602 may be rendered as an optically transparent transmission of a physical environment, where one or more CGR objects are rendered, with the physical environment as a background, e.g., overlaid over the physical environment. In some implementations, the environment sensor 606 (e.g., including the image sensor 608 and/or the depth sensor 610) acquires image data corresponding to the physical environment, and the AR environment 602 is rendered as a video pass-through of the physical environment. In video-through, the display 604 displays one or more CGR objects having CGR representations of a physical environment.

In some implementations, CGR content module 612 generates AR environment 602. In some implementations, the CGR content module 612 obtains CGR content, such as CGR content item 614, from a CGR content source 616. CGR content item 614 may include AR environment 602. In some implementations, the CGR content module 612 provides the CGR content item 614, including the AR environment 602, to a display engine 618, which prepares the CGR content item 614 for output using the display 604.

In some implementations, CGR content module 612 determines to display CGR object 620 in AR environment 602. The CGR object 620 corresponds to a physical object having a physical size. For example, the CGR object 620 may correspond to a physical chair having a physical height, width, and/or length. In some implementations, the physical object to which the CGR object 620 corresponds exists in the physical environment. In some implementations, the physical object to which the CGR object 620 corresponds does not exist in the physical environment. In either case, it is desirable to render the CGR object 620 with appropriate scaling, e.g., so that the CGR object 620 is proportional to other features of the AR environment 602.

In some implementations, the CGR content module 612 identifies known physical objects located in the physical environment. In some implementations, the environmental sensor 606 acquires environmental data 622 corresponding to the physical environment. For example, in some implementations, the image sensor 608 acquires an image 624 of the environment.

In some implementations, image 624 is a still image. In some implementations, the image 624 is an image frame that forms part of a video feed. The image 624 includes a plurality of pixels. Some of the pixels (e.g., the first set of pixels) represent an object. Other pixels (e.g., a second set of pixels) represent the background, e.g., portions of image 624 that do not represent objects. It should be understood that pixels representing one object may represent the background of a different object.

In some implementations, the depth sensor 610 acquires depth data 626 corresponding to the environment. Depth data 626 may be used independently of or in conjunction with image 624 to identify known physical objects.

In some implementations, the CGR content module 612 receives environmental data 622 from the environmental sensors 606. In some implementations, the CGR content module 612 identifies known physical objects in the physical environment based on the environment data 606. For example, the CGR content module 612 may perform semantic segmentation and/or instance segmentation on the environmental data 606 to identify known physical objects. In some implementations, the environmental data 606 includes an image, and the CGR content module 612 applies one or more filters and/or masks to the image to characterize pixels in the image as being associated with respective objects (such as known physical objects).

In some implementations, the image sensor 608 reads (e.g., detects) an optical machine-readable representation (e.g., a barcode or QR code) of data associated with a known physical object. The CGR content module 612 may, for example, send a query to a product database to obtain information identifying known physical objects.

The known physical object is associated with a known dimension. In some implementations, the CGR content module 612 obtains dimensional information for known physical objects. For example, the CGR content module 612 may send a query including information identifying known physical objects to a data store 628 via a network 630, such as a Local Area Network (LAN) or the internet, to a service. In some implementations, the information identifying the known physical object includes a semantic tag, a product identifier, and/or an image. In response to the query, the CGR content module 612 may receive size information for the known physical object. In some implementations, the CGR content module 612 receives size information for a physical object having a degree of similarity to a known physical object if size information for the known physical object is not available.

In some implementations, the CGR content module 612 determines the virtual size of the CGR object 620 based on the known size of the known physical object and the physical size of the physical object represented by the CGR object 620. For example, if the CGR object 620 is a CGR chair and the known physical object is a power outlet, the CGR content module 612 may determine a virtual height of the CGR chair based on the physical height of the physical chair represented by the CGR chair and the known physical height of the power outlet. For example, if the power outlet is four inches high and the physical chair is 20 inches high (e.g., the power outlet is five times higher), the CGR content module 612 may determine that the virtual height of the CGR chair is five times the height of the power outlet. In some implementations, when the CGR chair is displayed alongside an electrical outlet, the CGR chair occupies a height of five times the number of pixels relative to the number of pixels occupied by the electrical outlet.

In some implementations, the CGR content module 612 scales the virtual size of the CGR object 620 based on at least one factor of a plurality of factors. For example, in some implementations, the CGR content module 612 scales the virtual size of the CGR object 620 based on the known size of the known physical object (e.g., the height of the power outlet). In some implementations, the CGR content module 612 scales the virtual size of the CGR object 620 based on the distance of a known physical object from the device in which the system 600 is implemented. In some implementations, the CGR content module 612 scales the virtual size of the CGR object 620 based on the physical size of the physical object corresponding to the CGR object 620 (e.g., the physical height of a physical chair). In some implementations, the CGR content module 612 scales the virtual size of the CGR object 620 based on the placement location of the CGR object 620 within the AR environment 602 (e.g., the placement distance of the CGR object 620 from the user).

In some implementations, the CGR content module 612 displays the CGR object 620 in the AR environment 602 according to a virtual size. For example, if the CGR content module 612 determines that the virtual height of the CGR object 620 is five times the height of the known physical object, the CGR content module 612 may display the CGR object 620 with the virtual height occupying five times the height of the known physical object.

Fig. 7 is a block diagram of an exemplary CGR content module 700, according to some implementations. In some implementations, the CGR content module 700 implements the CGR content module 612 shown in fig. 6. In some implementations, the CGR content obtainer 710 obtains CGR content, such as the CGR content item 702, for display using a display (such as the display 604 of fig. 6). The CGR content item 702 may include an AR environment corresponding to a physical environment. In some implementations, the CGR content obtainer 710 generates a CGR content item 702, including an AR environment. In some implementations, the CGR content obtainer 710 obtains the CGR content item 702 from the CGR content source 704.

In some implementations, the CGR content obtainer 710 determines to display the CGR object in the AR environment. For example, the CGR content obtainer 710 may obtain the CGR object from the CGR content source 704 or from another source. The CGR object corresponds to a physical object having a physical size. For example, a CGR object may be a representation of a physical chair having a physical height, width, and/or length. The physical object to which the CGR object corresponds may or may not be present in the physical environment. In either case, it is desirable to render the CGR object with appropriate scaling, e.g., so that the CGR object is proportional to other features of the AR environment.

In some implementations, the CGR content module 700 identifies known physical objects located in the physical environment. In some implementations, the environmental sensor 706 acquires environmental data 708 corresponding to the physical environment and provides the environmental data 708 to the data obtainer 720. In some implementations, the environment sensor 706 includes an image sensor 712 (e.g., a camera) that acquires an image 714 of the physical environment. In some implementations, the environmental sensors 706 include a depth sensor 716 that acquires depth data 718 corresponding to the physical environment. The depth data 718 may be used independently of or in conjunction with the image 714 to identify known physical objects.

In some implementations, the image 714 is a still image. In some implementations, the image 714 is an image frame that forms part of a video feed. The image 714 includes a plurality of pixels. Some of the pixels (e.g., the first set of pixels) represent an object. Other pixels (e.g., a second set of pixels) represent the background, e.g., portions of the image 714 that do not represent the object. It should be understood that pixels representing one object may represent the background of a different object.

In some implementations, object analyzer 730 identifies known physical objects in the physical environment based on environment data 708. The known physical object is associated with a known physical size according to which the CGR object may be scaled. In some implementations, object analyzer 730 performs semantic segmentation and/or instance segmentation on environment data 708 to identify known physical objects. In some implementations, the environmental data 708 includes an image 714, and the object analyzer 730 applies one or more filters and/or masks to the image 714 to characterize pixels in the image 714 as being associated with a respective object (such as a known physical object). In some implementations, the known physical object is represented by a portion of the image 714, and the object analyzer 730 performs semantic segmentation and/or instance segmentation on the portion of the image 714 to identify the known physical object.

In some implementations, the data obtainer 720 can obtain an optical machine-readable representation of data associated with a known physical object. The optical machine-readable representation may be implemented as a barcode or QR code, for example. In some implementations, the optical machine-readable representation is part of the image 714. In some implementations, the optical machine-readable representation is captured separately from the image 714, e.g., in a separate scan.

In some implementations, the object analyzer 730 determines an object identifier 732 that identifies a known physical object, such as a semantic tag and/or a product identifier. In some implementations, the object analyzer 730 determines the object identifier 732 of a known physical object based on available information related to a physical object that corresponds to or has some degree of similarity to the known physical object. This information may be obtained from one or more sources.

For example, in some implementations, the object analyzer 730 determines the object identifier 732 based on information received from a database 734 (e.g., a local database). For example, the database 734 may store product specifications of a physical object (e.g., a chair) corresponding to a known physical object (e.g., of the same model of the known physical object). In some implementations, the database 734 stores product specifications for physical objects that have a degree of similarity to known physical objects. For example, if a product specification is not available for the same chair model corresponding to a known physical object, the object analyzer 730 may use the product specification for a similar chair model.

The known physical object is associated with a known dimension. In some implementations, the size determiner 740 obtains size information for known physical objects. For example, sizer 740 may send a query including information identifying known physical objects to data store 742 or to a service via network 744 (such as a Local Area Network (LAN) or the internet). In some implementations, the information identifying the known physical object includes a semantic tag, a product identifier, and/or an image. In response to the query, the size determiner 740 may receive size information for the known physical object. For example, if the physical object is known to be a standard power outlet, the sizer 740 may receive information indicating that the standard power outlet is four inches in height. In some implementations, the size determiner 740 receives size information of a physical object having a degree of similarity to a known physical object if size information of the known physical object is not available.

In some implementations, the sizer 740 determines the virtual size of the CGR object based on the known size of the known physical object and the physical size of the physical object represented by the CGR object. For example, if the CGR object is a CGR chair and the known physical object is a power outlet, the sizer 740 may determine the virtual height of the CGR chair based on the physical height of the physical chair represented by the CGR chair and the known physical height of the power outlet.

In some implementations, the sizer 740 scales the virtual size of the CGR object based on at least one factor of a plurality of factors. In some implementations, the sizer 740 scales the virtual size of the CGR object based on a known size of a known physical object (e.g., a height of a power outlet). For example, if the power outlet is four inches high and the physical chair is 20 inches high (e.g., the power outlet is five times higher), the sizer 740 may determine that the virtual height of the CGR chair is five times the height of the power outlet. In some implementations, when the CGR chair is displayed alongside a power outlet, the CGR chair occupies a height of five times the number of pixels relative to the number of pixels occupied by the power outlet.

In some implementations, the sizer 740 scales the virtual size of the CGR object based on the distance of a known physical object from the device in which the CGR content module 700 is implemented. For example, if the CGR content module 700 is implemented in an HMD located on the opposite side of a room relative to a power outlet, the power outlet may occupy fewer pixels in the display of the AR environment. Thus, scaling the CGR object relative to the power outlet may make the CGR object appear smaller than if the power outlet is closer to the HMD. In some implementations, when determining the virtual dimensions of the CGR object, the sizer 740 considers the distance of known physical objects from the device, e.g., to compensate for this potential effect.

In some implementations, the sizer 740 scales the virtual size of the CGR object based on the physical size of the physical object corresponding to the CGR object. For example, if the CGR object is a representation of a physical chair, the sizer 740 may scale the virtual height of the CGR object based on the physical height of the physical chair. For example, the physical height of the physical object may be determined by sending a query identifying the physical object to data store 742 or to a service via network 744.

In some implementations, the sizer 740 scales the virtual size of the CGR object based on the placement location of the CGR object within the AR environment (e.g., the placement distance of the CGR object from the user). For example, if the CGR object is to be placed away from the user, the sizer 740 may scale the CGR object to appear smaller. Conversely, if the CGR object is to be placed close to the user, the sizer 740 may scale the CGR object to appear larger.

In some implementations, the CGR content module 700 displays the CGR object in the AR environment according to a virtual size. In some implementations, the object generator 750 generates a modified CGR content item 752 that includes a CGR object instantiated within an AR environment consistent with the virtual dimensions. For example, if the sizer 740 determines that the virtual height of the CGR object is five times the height of the known physical object, the object generator 750 may instantiate the CGR object with the virtual height occupying five times the number of pixels of the height of the known physical object.

Fig. 8A-8C are a flowchart representation of a method 800 for displaying a CGR object in an AR environment, according to some implementations. In various implementations, the method 800 is performed by a device (e.g., the electronic device 102 and/or the controller 104 shown in fig. 1A and 1B, or the system 600 shown in fig. 6). In some implementations, the method 800 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, the method 800 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., memory). Briefly, in various implementations, the method 800 includes displaying an AR environment corresponding to a physical environment; determining to display a CGR object in an AR environment; identifying a known physical object located within a physical environment; determining a virtual size of the CGR object based on the known size of the known physical object and the physical size of the physical object represented by the CGR object; and displaying the CGR object in the AR environment according to the virtual size.

Referring to FIG. 8A, as shown at block 810, in various implementations, method 800 includes displaying an AR environment corresponding to a physical environment. Referring to fig. 8B, as shown at block 810a, the AR environment 602 may be rendered as an optically transparent transmission of a physical environment, wherein one or more CGR objects are rendered, the physical environment being a background, e.g., overlaid over the physical environment as perceived by, e.g., a lens in the HMD.

As shown in block 810b, in some implementations, an image sensor (such as a camera) acquires image data corresponding to a physical environment. The AR environment is rendered as a video pass-through of the physical environment, as shown in block 810 c. In video-through, a device displays one or more CGR objects having a CGR representation of a physical environment.

As shown at block 820, in various implementations, method 800 includes determining to display a CGR object in an AR environment. The CGR object represents a physical object associated with a physical dimension. For example, the CGR object may represent a physical chair associated with a physical height.

In some implementations, as shown in block 820a, a request to display a CGR object in an AR environment is obtained. The request may be received from a user. For example, in some implementations, as shown in block 820b, user input is received to display a CGR object in an AR environment. For example, the user input may include input from a mouse, a keyboard, and/or gesture-based input from a touch screen interface. In some implementations, the request is generated by a process or application without any intervention by the user.

In some implementations, as shown at block 830, the method 800 includes identifying known physical objects located within a physical environment. The known physical object is associated with a known size. In some implementations, as shown at block 830a, semantic segmentation and/or instance segmentation is performed on the environmental data to identify known physical objects. In some implementations, the environmental data includes images captured by an image sensor (such as a camera). One or more filters and/or masks may be applied to the image to characterize pixels in the image as being associated with respective objects (such as known physical objects). In some implementations, the known physical object is represented by a portion of the image, and semantic segmentation and/or instance segmentation is performed on the portion of the image to identify the known physical object.

In some implementations, as shown in block 830b, an optical machine-readable representation of data associated with a known physical object is identified. The optical machine-readable representation may be implemented as a barcode or QR code, for example.

In some implementations, a query is sent, for example, to a product database to obtain information identifying known physical objects. In some implementations, the information identifying the known physical object includes a semantic tag, a product identifier, and/or an image. As shown at block 830c, a query may be sent based on the product identifier corresponding to the known physical object. For example, if the optical machine-readable representation of the data includes a model number or a Uniform Product Code (UPC) identifier corresponding to a known physical object, the query may include this information. In some implementations, as shown in block 830d, the method 800 includes receiving size information for a known physical object in response to the query. For example, if this information is available, the product database may return the height of a particular model of power outlet from a particular manufacturer. In some implementations, as shown in block 830e, the method 800 includes receiving, in response to the query, size information for a physical object having a degree of similarity to a known physical object. For example, if the product database does not have information for a particular type of power outlet, the product database may instead return a general power outlet height, such as an average or standard height between manufacturers. In some implementations, as shown in block 830f, the method 800 includes receiving user input indicating dimensional information of a known physical object. For example, a user may provide user input indicating the width and/or height of the power outlet using gesture controls on a keyboard, mouse, and/or touch screen interface.

In some implementations, as shown at block 840, the method 800 includes determining a virtual size of the CGR object based on the known size of the known physical object and the physical size of the physical object represented by the CGR object. For example, if the CGR object is a CGR chair and the known physical object is a power outlet, the virtual height of the CGR chair may be based on the physical height of the physical chair represented by the CGR chair and the known physical height of the power outlet.

In some implementations, the sizer 740 scales the virtual size of the CGR object based on at least one factor of a plurality of factors. In some implementations, the virtual dimensions of the CGR object are based on known dimensions of known physical objects (e.g., height of a power outlet). For example, if the power outlet is four inches high and the physical chair is 20 inches high (e.g., five times the power outlet is high), the virtual height of the CGR chair is five times the height of the power outlet. In some implementations, when the CGR chair is displayed alongside a power outlet, the CGR chair occupies a height of five times the number of pixels relative to the number of pixels occupied by the power outlet.

Referring now to fig. 8C, as shown in block 840a, the virtual dimensions of the CGR object may be scaled based on the distance of the known physical object from the device in which system 600 is implemented. For example, if system 600 is implemented in an HMD located on the opposite side of a room relative to a power outlet, the power outlet may occupy fewer pixels in a display of an AR environment. Thus, scaling the CGR object relative to the power outlet may make the CGR object appear smaller than if the power outlet is closer to the HMD. In some implementations, the scaling of the CGR object takes into account the distance of the known physical object from the device, e.g., to compensate for this potential effect.

In some implementations, as shown in block 840b, the virtual size of the CGR object is scaled based on the placement location of the CGR object within the AR environment (e.g., the placement distance of the CGR object from the user). It may be assumed that the user is substantially collocated with the device. In some implementations, as shown in block 840c, the virtual dimensions of the CGR object are determined based on a distance between the device and a placement location of the CGR object. For example, if the CGR object is to be placed away from the user, the CGR object may be scaled to appear smaller. Conversely, if the CGR object is to be placed close to the user, the CGR object may be scaled to appear larger.

In some implementations, as shown in block 840d, the virtual dimensions of the CGR object are determined based on the distance between the known physical object and the placement location of the CGR object. For example, if the CGR object is placed close to a known physical object, the virtual size of the CGR object may be scaled according to the known size of the known physical object. In some implementations, if the CGR object is to be placed far from a known physical object in the AR environment, an additional scaling factor may be used to compensate for the apparent size difference due to the viewing angle.

In some implementations, as shown in block 840e, the virtual size of the CGR object is determined based on the virtual size of the AR environment. For example, in an AR environment corresponding to a dual car garage, CGR objects representing cars may be scaled to have a virtual width based on approximately half the virtual width of the AR environment.

In some implementations, as shown in block 850, the method 800 includes displaying the CGR object in the AR environment according to the virtual size. In some implementations, a modified CGR content item is generated. The modified CGR content item includes a CGR object instantiated within an AR environment consistent with the virtual size. For example, if the virtual height of the CGR object is five times the height of the known physical object, the CGR object may be instantiated within the AR environment with the virtual height occupying a number of pixels of five times the height of the known physical object.

Fig. 9 is a block diagram of a device 900 enabled with one or more components of the device (e.g., the electronic device 102 and/or the controller 104 shown in fig. 1A and 1B, or the system 600 shown in fig. 6) according to some implementations. While some specific features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more pertinent aspects of the particular implementations disclosed herein. To this end, as a non-limiting example, in some implementations, service system 900 includes one or more processing units (CPUs) 902, one or more input/output (I/O) devices 906, one or more communication interfaces 908, one or more programming interfaces 910, memory 920, and one or more communication buses 904 for interconnecting these and various other components.

In some implementations, a communication interface 908 is provided to establish and maintain, among other uses, a metadata tunnel between the cloud-hosted network management system and at least one private network that includes one or more compatible devices. In some implementations, the one or more communication buses 904 include circuitry to interconnect system components and control communications between system components. The memory 920 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid state storage devices. The memory 920 optionally includes one or more storage devices remotely located from the one or more CPUs 902. The memory 920 includes a non-transitory computer-readable storage medium.

In some implementations, memory 920 or a non-transitory computer readable storage medium of memory 920 stores programs, modules, and data structures, or a subset thereof, including an optional operating system 930, CGR content retriever 710, data retriever 720, object analyzer 730, sizer 740, and object generator 750. As described herein, the CGR content obtainer 710 may include instructions 710a and/or heuristics and metadata 710b for obtaining CGR content (including AR environment) to be displayed. As described herein, the data fetcher 720 may include instructions 720a and/or heuristics and metadata 720b for retrieving environmental data corresponding to a physical environment. As described herein, object analyzer 730 may include instructions 730a and/or heuristics and metadata 730b for identifying known physical objects in a physical environment based on environmental data. As described herein, the sizer 740 may include instructions 740a and/or heuristics and metadata 740b for determining known dimensions of known physical objects and/or for determining physical dimensions of the physical environment based on the known dimensions of the known physical objects. As described herein, object generator 750 may include instructions 750a and/or heuristics and metadata 750b for generating CGR objects in an AR environment.

In some implementations, the one or more I/O devices 906 include environmental sensors to capture environmental data. In some implementations, the environmental sensor includes an image sensor (e.g., a camera) for capturing image data representing a set of one or more images. In some implementations, the environmental sensor includes a depth sensor (e.g., a depth camera) for capturing depth data. In some implementations, the one or more I/O devices 906 include a display for displaying the CGR environment. In some implementations, the display includes an optical see-through display (e.g., an optical pass-through for displaying a physical environment). In some implementations, the display includes an opaque display (e.g., a video pass-through to display the physical environment).

It should be understood that fig. 9 serves as a functional description of various features that may be present in a particular implementation, as opposed to the structural schematic of the implementations described herein. As one of ordinary skill in the art will recognize, the items displayed separately may be combined, and some items may be separated. For example, some of the functional blocks shown separately in fig. 9 may be implemented as single blocks, and various functions of a single functional block may be implemented by one or more functional blocks in various implementations. The actual number of blocks and the division of particular functions and how features are allocated therein will vary depending on the particular implementation and, in some implementations, will depend in part on the particular combination of hardware, software, and/or firmware selected for a particular implementation.

While various aspects of the implementations described above are described within the scope of the appended claims, it should be apparent that various features of the implementations described above may be embodied in a wide variety of forms and that any specific structure and/or function described above is merely illustrative. Based on the disclosure, one skilled in the art should appreciate that an aspect described herein may be implemented independently of any other aspects and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method may be practiced using any number of the aspects set forth herein. In addition, such an apparatus may be implemented and/or such a method may be practiced using other structure and/or functionality in addition to or other than one or more of the aspects set forth herein.

It will also be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the claims. As used in the description of this particular implementation and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

As used herein, the term "if" may be interpreted to mean "when the prerequisite is true" or "in response to a determination" or "according to a determination" or "in response to a detection" that the prerequisite is true, depending on the context. Similarly, the phrase "if it is determined that [ the prerequisite is true ]" or "if [ the prerequisite is true ]" or "when [ the prerequisite is true ]" is interpreted to mean "upon determining that the prerequisite is true" or "in response to determining" or "according to determining that the prerequisite is true" or "upon detecting that the prerequisite is true" or "in response to detecting" that the prerequisite is true, depending on context.