阅读说明：本技术 热电堆阵列融合跟踪 (Thermopile array fusion tracking ) 是由 D.D.卡斯特尔曼 于 2020-02-18 设计创作，主要内容包括：公开了一种同时定位与地图创建(SLAM)使能的视频游戏系统、所述视频游戏系统的用户装置以及所述用户装置的计算机可读存储介质。总体上,所述视频游戏系统包括视频游戏控制台、多个热信标以及与所述视频游戏控制台通信地耦接的用户装置。所述用户装置包括热电堆阵列、处理器和存储器。所述用户装置可从所述热电堆阵列接收热数据,所述热数据对应于从所述多个热信标中的热信标发射并且由所述热电堆阵列检测到的热信号。所述用户装置可基于所述热数据确定其在3D空间中的位置,然后将所述位置传输到所述视频游戏系统。(A simultaneous localization and map creation (SLAM) -enabled video game system, a user device of the video game system, and a computer-readable storage medium of the user device are disclosed. In general, the video-game system includes a video-game console, a plurality of thermal beacons, and a user device communicatively coupled with the video-game console. The user device includes a thermopile array, a processor, and a memory. The user device may receive thermal data from the thermopile array, the thermal data corresponding to a thermal signal transmitted from a thermal beacon of the plurality of thermal beacons and detected by the thermopile array. The user device may determine its location in 3D space based on the thermal data and then transmit the location to the video game system.)

1. A video-game system, comprising:

a video game console;

a plurality of thermal beacons; and

a user device communicatively coupled with the video game console, the user device comprising:

a thermopile array;

a processor; and

a memory storing instructions that, when executed by the processor, cause the processor to:

receive thermal data from the thermopile array, the thermal data corresponding to a thermal signal emitted from a thermal beacon of the plurality of thermal beacons and detected by the thermopile array;

determining a location of the user device in a three-dimensional (3D) space based on the thermal data; and is

Transmitting the location of the user device to the video game console.

2. The video-game system of claim 1, wherein the user device further comprises an Inertial Measurement Unit (IMU), wherein execution of the instructions further causes the processor to:

receiving IMU data from the IMU, the IMU data comprising acceleration data corresponding to an acceleration of the user device in the 3D space and orientation data corresponding to a rate of rotation of the user device in the 3D space; and is

Determining the location of the user device in the 3D space by inputting the thermal data, the IMU data, and previous location data to a sensor fusion algorithm.

3. The video-game system of claim 1, wherein execution of the instructions further causes the processor to:

receive second thermal data from the thermopile array, the second thermal data corresponding to a second thermal signal transmitted from and detected by a second thermal beacon of the plurality of thermopile arrays, wherein the thermal beacon and the second thermal beacon are simultaneously in view of the thermopile array, and wherein the location of the user device is determined based further on the second thermal data.

4. The video game system of claim 1, wherein the thermal signal transmitted by the thermal beacon comprises an identifier unique to the thermal beacon, wherein the identifier is used by the thermopile array to identify the thermal beacon from the plurality of thermal beacons.

5. The video game system of claim 1, wherein each thermal beacon of the plurality of thermal beacons is positioned in a game environment, the 3D space being mapped to a portion of the game environment, the thermal beacons being positioned such that the thermopile array is capable of detecting thermal signals from at least two thermal beacons for a particular location of the user device within the 3D space.

6. A user device, comprising:

a thermopile array;

a processor; and

a memory storing instructions that, when executed by the processor, cause the processor to:

receiving thermal data from the thermopile array, the thermal data corresponding to a thermal signal emitted from a thermal beacon of a plurality of thermal beacons and detected by the thermopile array;

determining a location of the user device in a three-dimensional (3D) space based on the thermal data; and is

Transmitting the location of the user device to a video game console.

7. The user device of claim 6, wherein the user device further comprises an Inertial Measurement Unit (IMU), wherein execution of the instructions further causes the processor to:

receiving IMU data from the IMU, the IMU data comprising acceleration data corresponding to an acceleration of the user device in the 3D space and orientation data corresponding to a rate of rotation of the user device in the 3D space; and is

Determining the location of the user device in the 3D space by inputting the thermal data, the IMU data, and previous location data to a sensor fusion algorithm.

8. The user device of claim 7, wherein the sensor fusion algorithm generates a confidence value corresponding to the thermal data, the confidence value being based on a number of thermal beacons of the plurality of thermal beacons that are in a field of view of the thermopile array, the confidence value being used by the sensor fusion algorithm to determine the location.

9. The user device of claim 6, wherein each beacon of the plurality of thermal beacons is an infrared diode.

10. The user device of claim 6, wherein each thermal beacon of the plurality of thermal beacons is positioned in a gaming environment, the 3D space being mapped to a portion of the gaming environment, the thermal beacons being positioned such that one or more thermal sensors of the thermopile array are capable of detecting thermal signals from at least two thermal beacons for a particular location of the user device within the 3D space.

11. A non-transitory computer-readable storage medium storing instructions that, when executed on a user device, configure the user device to perform operations comprising:

generating, by a thermopile array of the user device, thermal data corresponding to a thermal signal emitted from a thermal beacon of a plurality of thermal beacons and detected by one or more thermal sensors of the thermopile array;

determining a location of the user device in a three-dimensional (3D) space based on the thermal data; and

transmitting, by the user device, the location of the user device to a video game console, the user device communicatively coupled with the video game console.

12. The non-transitory computer-readable storage medium of claim 11, wherein the operations further comprise:

receiving Inertial Measurement Unit (IMU) data from an IMU of the user device, the IMU data comprising acceleration data corresponding to an acceleration of the user device in the 3D space and orientation data corresponding to a rate of rotation of the user device in the 3D space; and

determining the location of the user device in the 3D space by inputting the thermal data, the IMU data, and previous location data to a sensor fusion algorithm.

13. The non-transitory computer-readable storage medium of claim 12, wherein the sensor fusion algorithm utilizes a kalman filter.

14. The non-transitory computer-readable storage medium of claim 11, wherein the operations further comprise performing calibration by:

determining a first location of the user device based on first thermal data corresponding to first thermal signals transmitted from a first set of thermal beacons of the plurality of thermal beacons positioned in the 3D space;

determining a second location of the user device based on second thermal data corresponding to second thermal signals transmitted from a second set of thermal beacons of the plurality of thermal beacons positioned in the 3D space;

generating, by the user device, a 3D model of the 3D space based on the first location and the second location; and

storing, by the user device, the 3D model.

15. The non-transitory computer-readable storage medium of claim 14, wherein the performing the calibration further comprises:

receiving, by the user device, an instruction requesting a user to move the user device to the first location; and

receiving, by the user device, an instruction requesting the user to move the user device to the second location in response to the determining the first location.

Background

Simultaneous localization and map creation (SLAM) is a technique by which a map of a particular environment is constructed while tracking the location of a target object within the environment. SLAM is used for robotic map creation and navigation in open environments (including autonomous cars). SLAM can also be used to play video games or to conduct virtual meetings in a closed environment, such as a room.

Existing SLAM technology typically uses some type of video camera configuration that includes one or more cameras, each camera including one or more optical sensors. A variety of optical sensor types may be used, including one-dimensional (1D), 2D, 3D, and the like. For example, where two fixed-point 2D cameras are employed, each camera may detect a target object. Algorithms applied by SLAM may determine the distance between the target object and each camera and triangulate this information to determine the location (e.g., position) of the target object. Typically, an Inertial Measurement Unit (IMU) may also be employed in conjunction with the camera device to detect linear accelerations and rotation rates of the target, particularly when the target may be moving and/or rotating in space. The combination of sensor inputs from both the IMU and the camera device helps achieve a higher degree of position tracking accuracy. Other SLAM technologies also exist, including radar SLAM, WiFi-SLAM, and the like.

However, there are at least two problems with existing SLAM-based applications. First, when employing an IMU to track acceleration and rotation of a moving target in space, IMU position tracking accuracy continues to decline over time due to at least offset error (also referred to as "drift"), thereby reducing the accuracy of SLAM techniques. Second, while incorporating sensed data received from other devices (e.g., cameras) can help improve the accuracy of SLAM algorithms, such devices currently add significant cost and require complex setup procedures. Therefore, there is a need to improve on existing SLAM techniques by providing a low cost but highly accurate location tracking solution.

Disclosure of Invention

In general, techniques for determining a location of a user device are described. In one example, a video game system includes a video game console, a plurality of thermal beacons, and a user device communicatively coupled with the video game console. The user device includes a thermopile array. The user device also includes a processor and a memory storing instructions that, when executed by the processor, cause the processor to perform operations. In one operation, the processor receives thermal data from the thermopile array. The thermal data corresponds to a thermal signal transmitted from a thermal beacon of the plurality of thermal beacons and detected by the thermopile array. In another operation, the processor determines a location of the user device in a three-dimensional (3D) space based on the thermal data. In another operation, the processor transmits a location of the user device to the video game console.

In one example, the user device may also include an IMU. The processor of the user device may further perform operations. In one operation, the processor receives IMU data from the IMU. The IMU data includes acceleration data corresponding to acceleration of the user device in the 3D space and orientation data corresponding to a rate of rotation of the user device in the 3D space. In another operation, the processor determines a location of the user device in the 3D space by inputting the thermal data, the IMU data, and previous location data to a sensor fusion algorithm. In one example, the processor determines an initial location of the user device based on the thermal data and independent of the IMU data of the IMU. In another example, the previous location data is stored on the user device. In yet another example, the sensor fusion algorithm utilizes an artificial intelligence model trained to determine the location. In another example, the sensor fusion algorithm utilizes a kalman filter.

In one example, the sensor fusion algorithm generates a confidence value corresponding to the thermal data. The confidence value is based on a number of thermal beacons in the plurality of thermal beacons that are in a field of view of the thermopile array. The confidence value is used by the sensor fusion algorithm to determine the location of the user device.

In one example, the processor of the user device may further perform operations. In one operation, the processor receives second thermal data from the thermopile array. The second thermal data corresponds to a second thermal signal emitted from a second thermal beacon of the plurality of thermal beacons and detected by the thermopile array. The thermal beacon and the second thermal beacon are simultaneously in view of the thermopile array. The location of the user device is further determined based on the second thermal data.

In one example, the thermal signal transmitted by a thermal beacon of the plurality of thermal beacons includes an identifier unique to the thermal beacon. The identifier is used by the thermopile array to identify the thermal beacon from the plurality of thermal beacons.

In one example, each of the plurality of thermal beacons is positioned in a gaming environment. The 3D space is mapped to a portion of the gaming environment and the thermal beacons are positioned such that the thermopile array is capable of detecting thermal signals from at least two thermal beacons for a particular location of the user device within the 3D space.

In one example, the user device is a video game controller. In another example, each beacon of the plurality of thermal beacons is an infrared diode.

A user device is also described. The user device includes a thermopile array. The user device also includes a processor and a memory storing instructions that, when executed by the processor, cause the processor to perform the operations disclosed herein above.

A non-transitory computer-readable storage medium storing instructions is also described. The instructions, when executed on a user device, configure the user device to perform the operations disclosed herein above.

In one example, the non-transitory computer-readable storage medium is further configured to perform a calibration operation. In one operation, the user device determines a first location of the user device based on first thermal data corresponding to first thermal signals transmitted from a first set of thermal beacons of the plurality of thermal beacons positioned in the 3D space. In another operation, the user device determines a second location of the user device based on second thermal data corresponding to second thermal signals transmitted from a second set of thermal beacons of the plurality of thermal beacons positioned in the 3D space. In another operation, the user device generates a 3D model of the 3D space based on the first location and the second location. In yet another operation, the user device stores the 3D model.

In one example, the non-transitory computer-readable storage medium is configured to perform additional calibration operations. In one operation, the user device receives an instruction requesting a user to move the user device to the first location. In another instruction, in response to the determining the first location, the user device receives an instruction requesting the user to move the user device to the second location.

Some embodiments of the present disclosure provide several technical advantages over current techniques for determining a user device location. First, the present disclosure provides a method of achieving similar position tracking accuracy while significantly reducing financial and human resource costs as compared to existing SLAM techniques. For example, thermal beacons (e.g., infrared diodes) are inexpensive and easy to attach to the walls of a room. Similarly, thermal sensors are also inexpensive compared to the cost of one or more optical sensors (e.g., components of a camera). In addition, the present disclosure can be used not only as an alternative to existing SLAM techniques, but also to improve the accuracy of location tracking in existing systems and methods.

A further understanding of the nature and advantages of the embodiments disclosed and suggested herein may be realized by reference to the remaining portions of the specification and the attached drawings.

Drawings

Fig. 1 illustrates an example of a system implementing a SLAM application using a thermopile array, according to an embodiment of the present disclosure.

Fig. 2 illustrates a user device including a thermopile array receiving signals from one or more thermal beacons, according to an embodiment of the present disclosure.

Fig. 3 is a block diagram of an example architecture of a user device implementing SLAM with a thermopile array, according to an embodiment of the present disclosure.

Fig. 4 illustrates an example flow for performing calibration of a user device according to an embodiment of the present disclosure.

Fig. 5 illustrates an exemplary flow for implementing a SLAM on a user device including a thermopile array according to an embodiment of the present disclosure.

Fig. 6 illustrates an exemplary flow for implementing SLAM on a user device including a thermopile array and an IMU according to an embodiment of the present disclosure.

Fig. 7 illustrates an example of a hardware system suitable for implementing a computer system, according to an embodiment of the present disclosure.

Detailed Description

In general, systems and methods for determining a location of a user device using an array of thermal sensors are described. Typically, a user may operate a user device (e.g., a video game controller, a headset, a remote wand, etc.) within a 3D space, which is mapped to a portion of a physical space (e.g., a video game room). A user may interact with a user device by moving the user device in different directions and at different speeds within the 3D space. User devices (e.g., video game controllers, Virtual Reality (VR) headsets, etc.) will also typically interact with a computing hub (e.g., a video game console), which in turn may interact with other devices and cause them to perform functions (e.g., change a video image being displayed on a Television (TV)). One of the interactions that a user device may have with a computing hub is to determine its location within 3D space and send the location to the computing hub. The user device may determine its location by utilizing a thermal sensor array of the user device. The thermal sensor array may be configured to receive thermal signals from one or more thermal beacons (e.g., IR LEDs) positioned in a physical space (e.g., attached to a wall of a game room). Based on the thermal data received from the thermal sensor array, the user device may determine its location in 3D space and transmit the location to the computing hub. The user device may also use the thermal data along with other sensor data to determine its location.

In one example, the user device may also include an IMU and receive data from the IMU. The user device may also have stored a previous location (e.g., the last determined location). In addition to the thermal data received from the thermal sensor array, the user device may also input IMU data and previous location data into the fused sensor algorithm. The fusion sensor algorithm may use different data inputs to determine the user device location. The user device may then store the location data for future use, e.g., as previous location data.

The above examples are provided for illustrative purposes. Embodiments of the present disclosure are not limited thereto. Embodiments are similarly applicable to a number of SLAM applications that use thermal sensor arrays to determine location. These and other embodiments are further described herein below.

Fig. 1 shows an example of a system 100 according to an embodiment of the present disclosure, the system 100 comprising a user device further comprising a thermal sensor array for determining a location of the user device. In one example, the location determination implements a SLAM application. In fig. 1, a game room 126 is depicted in which a user 102 wears a gaming headset 104 (e.g., a user device) to interact with a video game console 106. The video game console, in turn, may cause, for example, the TV 108 to display on the screen the movements of the object corresponding to the movements of the gaming headset 104. It should be understood that while fig. 1 and the subsequent figures may depict a SLAM application within a gaming environment, the use of this type of scenario should not be construed as limiting the scope of the present disclosure. For example, in some embodiments, a physical space such as a conference room or theater space may be used, and other location determination applications are possible. In general, and continuing the game room example, the game room 126 may allow for multiple thermal beacons 110 and 124 (e.g., IR LEDs) to be positioned in the room 126.

The user device 104 should include one or more thermal sensors (e.g., an array of thermal sensors or a thermopile array) configured to receive thermal signals from one or more thermal beacons 110 and 124 positioned in a room 126. It should be understood that while the thermopile array is of the type depicted in fig. 1 and subsequent figures, other types of thermal sensors may also be used as suitable thermal sensors. Further, while gaming headset 104 is a type of user device depicted below, other types of mobile computing devices including thermal sensors may be used as suitable user devices. This may include, but is not limited to, video game controllers, game remote control sticks, mobile phones, notebook computers, and the like.

In some embodiments, each of the plurality of thermal beacons 110 and 124 positioned in the room 126 may emit a thermal signal (e.g., infrared light). Optionally, each thermal signal transmitted by a thermal beacon may include an identifier unique to the particular thermal beacon. For example, the identifier may be a modulation frequency of an infrared signal, which the thermopile array of the user device 104 may be able to detect as a unique signal corresponding to a particular thermal beacon in the room. As discussed in more detail below, in some implementations, the user device 104 may be able to improve its location tracking accuracy by triangulating its location with respect to at least two thermal beacons in the room 126.

In some embodiments, the plurality of thermal beacons 110 and 124 may be positioned in the room 126 such that for a particular location of the user 102 within the 3D space of the room 126, the thermopile array of the user device 104 may be capable of detecting thermal signals from at least two of the plurality of thermal beacons within the field of view of the thermopile array. For example, as depicted in FIG. 1, the thermal beacons 110 and 124 may each be attached to a wall of the game room 126 and substantially equally spaced from each other. During the calibration process (discussed in detail below), the user 102 may be instructed to move the user device 104 in an arc-like (e.g., 360 degree) motion. This calibration process may allow the user device 104 to construct and store a 3D model of a 3D space on the user device 104, the 3D space corresponding to at least a portion of the room 126, and the user 102 may operate the user device 104 within the 3D space. The calibration process may also verify that the field of view of the thermopile array may detect at least two signals from specific points within the 3D space in the room 126. In other embodiments, and depending on the type of SLAM application (e.g., the expected range of use of the user device within the room 126), two or more thermal beacons may be positioned in the physical space 126 for the user device 104 to determine its location.

Fig. 2 shows an exemplary close-up view of a user device 200 receiving thermal signals from one or more thermal beacons. User device 200 is a gaming headset and may correspond to user device 104 of fig. 1. In one example, the headset 200 is a Virtual Reality (VR) or Augmented Reality (AR) headset. In general, the headset 200 includes a housing 210, which housing 210 may integrate components such as a display, processing unit, memory, audio system, I/O ports, graphics processing unit, network communication device, and other electronic, electrical, and mechanical components. The housing 210 also integrates (e.g., houses, attaches, or retains) additional components for position tracking such that the additional components are rigidly connected with the housing 210. These components include, for example, the IMU 214 and the thermopile array 218.

In one example, IMU 214 includes one or more accelerometers, one or more gyroscopes, and one or more magnetometers. One or more accelerometers measure movement along X, Y and the Z axis. In general, one or more gyroscopes measure 360 degree rotation. One or more magnetometers determine orientation towards the magnetic field. Accordingly, inertial data (e.g., including acceleration data, orientation data, and/or rotation data) indicative of rotational motion of the headset 200 may be generated from readings of these sensors. Translational motion may also be generated by the headset 200 based on the speed and time of the head motion of the user 102. For example, a motion vector is defined. Velocity is measured from acceleration and distance is measured as a function of velocity and time. The directions result from the rotation and orientation. The motion vectors define the distance and direction of motion, allowing tracking of translational motion of the headset 200 along the X, Y and Z axes. Thus, by defining inertial data, distance, and direction with motion vectors, the motion of the user headset 200 may be tracked over time. Accordingly, based on the inertial data, and by performing integration with respect to time, the position of the headphone 200 can be determined for a specific point in time in 3D space. A processing unit (e.g., a signal processor) of the IMU 214 may generate location data from data sensed by such IMU sensors.

As mentioned above, one of the limitations of utilizing an IMU to determine location is that the IMU is typically subject to cumulative errors. Since SLAM applications can integrate acceleration continuously with respect to time to calculate velocity and position, any measurement error, however small, accumulates over time, resulting in drift. Drift is a measure of the difference between where a user device can initially determine it is located compared to its actual location. Accordingly, the headset 200 may also include a thermopile array 218, which the headset 200 may use to continuously correct drift errors and determine a more accurate position.

In one example, the thermopile array 218 is a thermal infrared sensor array configured to remotely measure the temperature of one or more thermal beacons 204, 206 by detecting infrared energy from the one or more thermal beacons 204, 206 (which may correspond to one or more of the thermal beacons 110, 124 of fig. 1). The thermopile array may include thermocouples connected on the silicon chip configured to convert thermal energy received from the thermal signals of the thermal beacons 204, 206 into electrical energy in the form of voltage outputs. Typically, the input thermal energy is proportional to the voltage output. Thus, because the closer the user device 200 is to a thermal beacon (e.g., resulting in a higher temperature difference), the more light energy (e.g., signal) can be detected, the thermopile array may be used to measure the distance between the user device 200 and a particular thermal beacon. In some implementations, a processing unit connected to the thermopile array 218 (e.g., a signal processor integrated with the thermopile array 218) may transmit thermal data corresponding to a distance from the user device 200 to the thermal beacons 204, 206 based on the voltages output by the elements of the thermopile array 218. In some implementations, a thermopile may be composed of a single element (e.g., pixel), dual elements, or the like. In other embodiments, the thermopile may be a linear (e.g., 16, 32) or planar (e.g., 32x32) pixel array. Although the embodiments discussed herein discuss thermopile arrays, the present disclosure should not be construed as so limited. In some embodiments, any one or more given pixels of the thermopile array 218 may be capable of detecting a thermal signal from one or more thermal beacons at a given time, provided that the thermal beacons are within the field of view of the pixel. As discussed above, and as discussed further below with respect to fig. 3, the thermal data generated from the thermopile array 218 may be used to triangulate the distance between the user device 200 and the at least two thermal beacons 204, 206 to determine the 3D location of the user device 200 in the physical space 126. In some implementations, thermal data from the thermopile array may also be combined with IMU data to output the 3D location of the user device 200 in the physical space 126.

Fig. 3 is a block diagram 300 of an example architecture for a user device 302 (which may correspond to user device 104 of fig. 1 and/or user device 200 of fig. 2) and implementing a SLAM with a thermopile array, according to an embodiment of the present disclosure. User device 302 may include, among other components, at least one memory 304, one or more processing units (or processors) 316, a thermopile array 318, an IMU 320, and a communication device 322. The one or more processors 316 may optionally be implemented in hardware. The thermopile array 318 of the user device 302 may be configured to detect one or more thermal signals (e.g., infrared light) from one or more thermal beacons 204, 206. The communication device 322 may be further configured to communicate with the computing hub 106 (e.g., a video game console, a virtual conference server, etc.) using any suitable communication path. This may include, for example, wire or cable, fiber optics, a phone line, a cellular link, a Radio Frequency (RF) link, a WAN or LAN network, the internet, or any other suitable medium.

Memory 304 may store program instructions that are loadable and executable on one or more processors 316, as well as data generated during the execution of these programs. Depending on the configuration and type of user device 302, memory 304 may be volatile (such as Random Access Memory (RAM)) and/or non-volatile (such as Read Only Memory (ROM), flash memory, etc.). In some implementations, the memory 304 may include a variety of different types of memory, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), or ROM. User device 302 may also include additional storage (not shown), such as removable storage or non-removable storage, including but not limited to magnetic storage, optical disk, and/or tape storage. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules and other data for the computing devices.

Turning in more detail to the contents of memory 304, memory 304 may include an operating system 306 and one or more application modules or services for implementing features disclosed herein, including a calibration module 308, a sensor fusion algorithm module 312, and a thermopile array algorithm module 314. It should be understood that any of the tasks performed by one module may be performed by one or more of the other modules, and thus, the module definitions provided herein are included for illustrative purposes.

Operating system 306 may provide executable program instructions for the general management and operation of user device 302 and will typically include a computer-readable storage medium (e.g., hard disk, random access memory, read only memory, etc.) that stores instructions that, when executed by a processor of user device 302, allow user device 302 to perform its intended functions. Suitable implementations of operating systems are known or commercially available and are readily implemented by those of ordinary skill in the art, particularly in light of the present disclosure.

The calibration module 308 may be responsible for determining and maintaining a 3D model of a 3D space in the memory 304, where the 3D space may correspond to at least a portion of a physical space (e.g., the game room 126). In some implementations, the calibration module 308 may also be responsible for verifying: for a given location within the 3D space, the thermopile array 318 of the user device 302 may detect thermal signals from at least two of the plurality of beacons 110 and 124 (e.g., the beacons are within the field of view of one or more elements of the thermopile array). This facilitates position triangulation by the thermopile array algorithm module 314 discussed below. In some embodiments, the calibration module 308 may perform operations as described in fig. 4. In one embodiment, and using the game room 126 of fig. 1 as an example, the 3D model (which may be represented as a 3D wireframe model) corresponds to a 3D space within which the user 102 expects to operate the user device 104. For example, the user may only intend to operate the user device 104 within a portion of the game room that is near the center (and radiates a certain distance out from the center that is reachable by humans). The calibration module 308 may instruct the user to begin calibration by: the device is moved to the center of the room 128 (e.g., where they are primarily intended to operate the device), and then the device 104 is moved in a 360 degree visual scan of the room (divided into smaller arcs (e.g., 90 degree turns)) while standing and turning in place. At each segment, as described above, the user device 104 may verify that at least two thermal beacons are within the field of view of the thermopile. Further, the user device 104 may determine and store in the memory 304 the distance between the user device 104 and each thermal beacon 204, 206 visible within the arc segment. Once the user device 104 completes scanning the room, the calibration module 308 may use the position data determined between the user device 104 and each of the beacons 110 and 124 (relative to the center 128 of the room) to construct a 3D model of the 3D space. In some implementations, the X, Y and the origin of the Z-axis may be located at the location (e.g., center 128) where the user 102 initially positioned the user device 104 during calibration. In some embodiments, a full 360 degree visual scan may not be required to perform the calibration. For example, if the user 104 would normally operate the user device 104 forward facing the TV 108, the calibration may recommend only a partial (e.g., 180 degree) scan of the room. In this case, a smaller number of thermal beacons may need to be pre-positioned in the room in order to construct the 3D model.

The sensor fusion algorithm module 312 may be responsible for determining the location of the user device 302. In some embodiments, the sensor fusion module 312 may be performed after the calibration 308 has been performed. The sensor fusion module 312 may combine sensed data input from one or more sources to improve accuracy in determining the location of the user device 302. In one embodiment, the sensor fusion module 312 may receive thermal data from the thermopile array 318 as a sensing input and IMU data from the IMU 320 as a sensing input.

In some implementations, the sensor fusion module 312 may first execute the thermopile array algorithm module 314 before combining the sensed input data from the different sensors. The thermopile array algorithm module 314 may be responsible for triangulating the location of the user device 302 based on two or more distance values (e.g., corresponding to the distance between the user device 302 and at least two thermal beacons 204, 206). The location (e.g., position) may be in the form of X, Y and Z coordinates within the 3D model determined during calibration. In some embodiments, the thermopile array algorithm module 314 may output a confidence value corresponding to a location in addition to outputting the location. The confidence value may be increased or decreased based on the number of thermal beacons (e.g., the number of distance values obtained) detected within the field of view of the thermopile array. For example, if only one thermal beacon is detected, the corresponding confidence value may be low, although the thermopile array algorithm module 314 may still output a position value. Conversely, if two or more thermal beacons are detected within the field of view, the module 314 may output a high confidence value. In some implementations, the user device 302 can execute the thermopile array algorithm module 314 at a particular frequency. For example, the module 314 may operate at 120Hz (approximately once every 8.33 milliseconds). Generally, generating a higher frequency of updated locations based on thermal data will improve location tracking accuracy. For example, when fusing both thermal data and IMU data (discussed further below), more frequent location information based on the thermal data will help correct drift errors within the IMU data.

In another implementation, and returning to the sensor fusion module 312 discussed above, the module 312 may receive position information (X, Y and Z coordinates) from the thermopile array algorithm module 314 (e.g., after the algorithm 314 uses the thermal data to determine 3D position values and corresponding confidence values) and position information (X, Y and Z coordinates) of IMU data from the IMU 320. The sensor fusion module 312 may combine (or "fuse") the different location data together using one or more of a variety of algorithms and output a single location in 3D space with greater accuracy.

In one embodiment, the sensor fusion module 312 may employ an artificial intelligence model trained to determine the location of the user device 302 using sensor data from different sources. As used herein, the term "artificial intelligence" refers to any suitable computer-implemented artificial intelligence technique, including machine learning (with or without supervision), natural language processing, machine perception, computer vision, emotion computation, statistical learning and classification (including using hidden markov models and bayesian network models), reinforcement learning including neural networks, search algorithms and optimization algorithms (including evolutionary computations), and automated reasoning. For example, the neural network may be trained to receive IMU data, thermal data, and one or more previous location data as inputs. This information may be used to output a corrected location of user device 302 and/or predict a next location that user device 302 will be located within a particular time interval.

In another embodiment, the sensor fusion module 312 may employ a fixed point algorithm, such as employing a kalman filter. The kalman filter may use the sensed input data from the thermopile array, IMU, and previous position data to build a predictive model. The predictive model may take into account state variables (e.g., one or more previous locations, polling frequency of thermal data, previous measurements of drift error, etc.).

Fig. 4 illustrates an example flow 400 for performing calibration of the user device 102 of the system 100 of fig. 1, according to an embodiment of the present disclosure. Although the operations are shown in a particular order, some of the operations may be reordered or omitted. Further, some or all of flow 400 (or any other flow, or variations, and/or combinations thereof described herein) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that collectively execute on one or more processors. The code may be stored on a computer-readable storage medium in the form of a computer program, for example, comprising a plurality of instructions executable by one or more processors. The computer readable storage medium may be non-transitory.

In one example, the flow includes an operation 402, the operation 402 involving: the thermal beacon 110 and 124 are positioned within a physical space 126, which physical space 126 may be a gaming environment. Using the example of the video-game system 100, the video-game console 106 may prompt (e.g., visually on a television screen, audibly through headphones, or some other mechanism) the user 102 to begin the calibration process by positioning the thermal beacon 110 and 124 in the physical space 126 (e.g., a game room). Depending on the type of game application, the video game console 106 may indicate the user as to how many hot beacons should be used and what coverage (e.g., 360 degrees, 180 degrees, etc.) is recommended. For example, if an application requires that the user 102 be able to move the video game controller 104 (e.g., headset) in a 360 degree visual scan of the perimeter of the room 126 while playing a game, the user 102 may be recommended to place a thermal beacon around the perimeter of the room, as depicted in fig. 1. For example, the thermal beacons 110 and 124 may be positioned in the centers 112, 116, 120, 124 of the four walls and near the wall corners 110, 114, 118, 122. In this way, for a given location of the headset 104 within the gaming environment, the thermopile array 218 of the headset 104 may be capable of detecting thermal signals from at least two thermal beacons 204, 206 within its field of view.

In one example, the flow includes an operation 404, which operation 404 involves: an instruction is received requesting that the user move the user device 104 to a first (or next) location. It should be understood that this instruction may come directly from the user device 104 itself (e.g., audio commands from a headset), or from a computing hub 106 communicatively connected to the user device 104 (e.g., a video game console). Where the instructions may come from the computing hub 106, the user device 104 may iterate through the next step in the calibration process (e.g., operation 406 discussed below), and then send an acknowledgement message (or information/error message) to the computing hub 106, which in turn may relay the next instruction message to the user 102.

In the event that the user 102 is instructed to move the user device 104 to the first location, and continuing with the game example described above, the headset 104 (e.g., via audio instructions, flashing lights, etc.) may instruct the user 102 to move to the center 128 of the room, the user 102 intending to center the center as the game experience. The headset 104 may use this location 128 as the origin of a 3D model that the headset 104 builds and stores on the headset 104 as a result of the calibration process 400. It should be understood that the optimal location of the origin may depend on the type of application, and in some embodiments may not be located in the center of the room. In some implementations, the user 102 may then signal the video game console 106 or the headset 104 to: the user has completed the previous steps and is ready to continue.

In one example, the flow includes operation 406, where the user device 104 may determine a first (or next) location of the user device 104 based on thermal data corresponding to thermal signals transmitted by at least two of the plurality of thermal beacons. Where the user device 104 is determining the first location, the user 102 may already be in place based on the previous operation 404. Continuing with the above game example, the headset's calibration module 308 may utilize thermal data received from the thermopile array 318 to determine at least two distance values between the thermopile array 318 and at least two beacons 204, 206 within the field of view of the thermopile array 318, respectively. Based on this distance information, the calibration module 308 may triangulate the position of the headset and store the position and distance values in the memory 304. In the event that the calibration module 308 fails to detect at least two beacons within the field of view, it may prompt the user 102 with an alert (e.g., to check the location of the thermal beacon).

In some implementations, at operation 408, once the calibration module 308 has determined the first (or next) location based on information from at least two thermal beacons within its field of view, the calibration module 308 may determine whether the calibration is complete. In some implementations, the calibration module 308 can perform this determination 410 by determining whether it has sufficient information to construct a 3D model of the 3D space. In other embodiments, the calibration module 308 may predetermine that it must capture all four faces of the room 126 and continue to prompt the user to go to the next location until it has captured data for all four faces. The calibration module 308 may use any suitable mechanism to determine whether the calibration is complete. If the calibration is complete, flow may proceed to operation 412, which is described further below.

If the calibration procedure has not been completed, flow may loop back to operation 404, instructing the user to move to the next location (e.g., position). Continuing with the game example above, the headset 104 may output audible instructions, tactile feedback, or any other suitable method to indicate to the user 102 to move to the next location. In one embodiment, the headset 104 may instruct the user to "pivot about 90 degrees in place, then stop and wait for additional instructions". The calibration module 308 may then continue to calibrate and perform operation 406, as discussed above. This loop may continue until calibration is complete.

In one example, at operation 412, the calibration module 308 may determine that it has sufficient data to construct a 3D model of the physical space 126, where the first location serves as the origin of the 3D model. Each of the distances determined (e.g., to the thermal beacon 110 and 124) relative to the first location may be used to determine a size of the 3D model. In some embodiments, the calibration module 308 may use any suitable mechanism to construct the 3D model. Once the 3D model has been constructed, the calibration module 308 may store the model on the user device 302 (e.g., in the memory 304 or other storage device).

Fig. 5 illustrates an example flow 500 for implementing a SLAM on a user device including a thermopile array, according to an embodiment of the present disclosure. In some embodiments, the process 500 may be performed after the calibration process 400 of fig. 4 has been performed and a 3D model of the 3D space has been generated. Although the flow operations below discuss utilizing a remote wand as a user device (e.g., in place of headset 104 within the video-game system of fig. 1), any suitable user device and/or system may be used.

In one example, at operation 502, a user device may receive thermal data corresponding to a thermal signal transmitted from one of a plurality of thermal beacons, the thermal signal detected by a thermal sensor array of the user device. The user 102 may operate the remote control stick to play a virtual table tennis ball, where the game involves the user moving the remote control stick into a different position while the user is playing the game. The user 102 may also rotate the remote control stick and swing the remote control stick at different acceleration rates. As described above, depending in part on the type of application, the frequency at which the remote control wand can poll the thermopile array 318 to receive thermal data may be increased to achieve higher position tracking accuracy over time. In some embodiments, this may involve: the thermopile array algorithm module 314 receives voltage readings (which correspond to the IR light signals of a particular thermal beacon) from the thermopile array at a frequency of at least 120Hz (e.g., receiving an update approximately every 8.33 milliseconds).

In one example, at operation 504, a user device (e.g., a remote wand) may determine its location in 3D space based on data received from the thermopile array. Specifically, in some embodiments, for each voltage reading, the thermopile array algorithm module 312 may calculate an associated distance to the thermal beacon. The module 314 may then triangulate its position using thermal signals from at least two thermal beacons, as described above. In some implementations, module 312 may use a 3D model of the 3D space within game room 126 (which was previously stored during the calibration process) to determine the position of the remote wand in the 3D space. In some embodiments, the remote wand may utilize a confidence value (e.g., generated by the thermopile array algorithm module 314) to determine its position. The confidence value may be lower if, for example, only one hot beacon is detected at a particular time. Thus, the remote wand may decide to ignore the position value, combine it with other position data to improve accuracy, or perform any other suitable action. It should be understood that in some embodiments, the wand may utilize only thermal data (in addition to other sensor data) to determine the position of the wand. However, in other embodiments, the telewand may utilize other sensing inputs (e.g., IMU data) and/or variables in addition to thermal data to determine the position of the telewand, which may be further input into the sensing fusion algorithm 312 for combination (as discussed below in fig. 6).

It should be appreciated that one technical advantage of the present disclosure over other SLAM applications that may require calculations external to the user device 104 to determine the location of the user device (e.g., external optical sensors, processed by a video game console, etc.) is that: it enables the user device 104 to determine its location using its own internal components (based in part on the thermal signals received from the thermal beacons). This may improve system reliability, for example, in reducing network connection issues such as latency, bandwidth limitations, and the like.

In one example, at operation 506, and continuing the above example, the remote wand may transmit the position of the remote wand to video game console 106. In some implementations, the location data may be transmitted by the communication device 322 using any suitable communication path (e.g., WiFi), as discussed above. Video game console 106 may use any suitable mechanism to process the location data. In one example, the video game console 106 may use the position data to move an object displayed on the TV 108 (which corresponds to the wand) to a new position, where the new position corresponds to a change in the position of the wand in the 3D space of the game room 126.

In one example, at operation 508, the user device 104 may store its own location on the user device 104. In some implementations, the previous location data may be in the form of X, Y and Z coordinates corresponding to a 3D model (see fig. 4) previously generated during calibration. The previous location data may be stored in the memory 304 of the user device 104. In some implementations, the memory 304 may store a history of previous locations, where the sensor fusion algorithm 312 of the user device 104 (e.g., employing a machine learning algorithm or a pointing algorithm) may use one or more of the historical location data points as input to determine the current location of the user device 104. In still other embodiments, historical location data may be used by an algorithm to predict future locations.

Fig. 6 illustrates an example flow 600 for implementing a SLAM on a user device including a thermopile array and an IMU, according to an embodiment of the present disclosure. Similar to the flow 500 of fig. 5, the flow 600 may be performed after the calibration flow 400 of fig. 4 has been performed and a 3D model of the 3D space has been generated. Further, similar to flow 500, although the flow operations below discuss utilizing a joystick as user device 104 within the video-game system of FIG. 1, any suitable user device and/or system may be used. In some embodiments, the flow 600 includes exemplary operations that may be implemented as sub-operations of the exemplary flow 500.

In one example, at operation 602, which is similar to operation 502 of fig. 5, a user device may receive thermal data corresponding to a thermal signal transmitted from one of a plurality of thermal beacons, the thermal signal detected by a thermal sensor array of the user device. In some implementations, this data may be further processed by the thermopile array algorithm module 314.

In one example, at operation 604, the user device may retrieve previous location data stored on the user device. This may be, for example, the data stored by the previous operation 608. In some implementations, the previous location data may be in the form of X, Y and Z coordinates corresponding to a 3D model (see fig. 4) previously generated during calibration. In other embodiments, the previous location data may also include other data, including but not limited to inertial data (e.g., rotation rate, linear acceleration of the user device in 3D space at a certain point in time). This data may be received by one or more sensor unit devices (e.g., IMU 320) of user device 302, as discussed further below.

In one example, at operation 606, the user device may receive IMU data from the IMU 320 of the user device. In some implementations, the IMU data may include acceleration data, orientation data, and/or rotation data of the user device in 3D space, as discussed with reference to IMU 214 of fig. 2. Thus, IMU data may also be used to determine the position of the user device in 3D space at a certain point in time.

In one example, at operation 608, the user device may determine a location of the user device in 3D space by inputting thermal data (e.g., received at operation 602), previous location data (e.g., received at operation 604), and/or IMU data (e.g., received at operation 606) into a sensor fusion algorithm (which may correspond to the sensor fusion algorithm module 312 of fig. 3). In some embodiments, other sensing inputs from other sensors, including but not limited to Global Positioning System (GPS) trackers, may also be used as inputs to the sensor fusion algorithm. In some implementations, each of the data received by the sensor fusion algorithm from operations 602 and 606 may correspond to data that measures the location of the user device at substantially the same point in time. However, in other embodiments, the data received by the sensor fusion algorithm from operations 602 and 606 may correspond to data measuring the location of the user device at different time intervals. In one example, and similar to that discussed above with respect to operation 502, the thermopile array algorithm 314 may determine a user device position at a frequency of 120Hz (e.g., updated every 8.33 milliseconds) based on thermal data from the thermopile array 318. In contrast, and for example, a 20Hz IMU may output acceleration and rotation rates representing a sampling period of total IMU motion within 50 milliseconds. The sensor fusion algorithm 312 may use the position data obtained from the thermopile array as a drift correction factor to correct drift errors within the IMU position data. The sensor fusion algorithm may also utilize the previous position data from operation 604 to improve the accuracy of the algorithm, as discussed above.

In one example, at operation 610, and similar to operation 506 from fig. 5, the user device transmits its location determined in operation 608 to the video game console.

In one example, at operation 612, and similar to operation 508 from fig. 5, the user device may store its own location on the user device.

Fig. 7 illustrates an example of a hardware system suitable for implementing a computer system 700, according to various embodiments. Computer system 700 represents, for example, components of a video game system, a mobile user device, a proximity device, a wearable gesture device, and/or a central computer. Computer system 700 includes a Central Processing Unit (CPU)705 for running software applications and an optional operating system. CPU 705 may be comprised of one or more homogeneous or heterogeneous processing cores. The memory 710 stores applications and data for use by the CPU 705. Storage 715 provides non-volatile storage and other computer-readable media for applications and data, and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray disc, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. User input devices 720 communicate user inputs from one or more users to computer system 700, examples of user input devices 720 may include a keyboard, mouse, joystick, touchpad, touch screen, still or video camera, and/or microphone. Network interface 725 allows computer system 700 to communicate with other computer systems over an electronic communications network, and may include wired or wireless communications over local and wide area networks, such as the internet. The audio processor 755 is adapted to generate analog or digital audio output from instructions and/or data provided by the CPU 705, memory 710, and/or storage 715. The components of computer system 700, including CPU 705, memory 710, data storage 715, user input devices 720, network interface 725, and audio processor 755, are connected by one or more data buses 760.

Graphics subsystem 730 may be further connected to data bus 760 and the components of computer system 700. Graphics subsystem 730 includes a Graphics Processing Unit (GPU)735 and a graphics memory 740. Graphics memory 740 includes a display memory (e.g., a frame buffer) for storing pixel data for each pixel of an output image. Graphics memory 740 may be integrated in the same device as GPU 735, connected as a separate device with GPU 735, and/or implemented within memory 710. Pixel data may be provided directly from CPU 705 to graphics memory 740. Alternatively, the CPU 705 may provide the GPU 735 with data and/or instructions defining the desired output images, from which the GPU 735 generates pixel data for one or more output images. Data and/or instructions defining the desired output image may be stored in memory 710 and/or graphics memory 740. In one embodiment, the GPU 735 includes 3D rendering capabilities for generating pixel data for output images from instructions and data defining the geometry, lighting, shading, texturing, motion, and/or camera parameters of a scene. GPU 735 may also include one or more programmable execution units capable of executing shader programs.

Graphics subsystem 730 periodically outputs pixel data for an image from graphics memory 740 for display on display device 750. Display device 750 may be any device capable of displaying visual information in response to a signal from computer system 700, including CRT, LCD, plasma, and OLED displays. The computer system 700 may provide analog or digital signals to the display device 750.

According to various embodiments, CPU 705 is a general purpose microprocessor or microprocessors with one or more processing cores. Further embodiments may be implemented using one or more CPUs 705 having a microprocessor architecture that is particularly suited for highly parallel and computationally intensive applications such as media and interactive entertainment applications.

The components of the system may be connected by a network, which in various embodiments may be any combination of the following: the internet, IP network, intranet, wide area network ("WAN"), local area network ("LAN"), virtual private network ("VPN"), public switched telephone network ("PSTN"), or any other type of network that supports data communications between the devices described herein. The network may include both wired and wireless connections, including optical links. Many other examples are possible and will be apparent to those skilled in the art in light of this disclosure. In the discussion herein, a network may or may not be specifically noted.

Examples of embodiments of the present disclosure may be described according to the following clauses:

clause 1. a video-game system, comprising: a video game console; a plurality of thermal beacons; and a user device communicatively coupled with the video game console, the user device comprising: a thermopile array; a processor; and a memory storing instructions that, when executed by the processor, cause the processor to: receiving thermal data from the thermopile array, the thermal data corresponding to a thermal signal emitted from a thermal beacon of the plurality of thermal beacons and detected by the thermopile; determining a location of the user device in a three-dimensional (3D) space based on the thermal data; and transmitting the location of the user device to the video game console.

The video-game system of clause 1, wherein the user device further comprises an Inertial Measurement Unit (IMU), wherein execution of the instructions further causes the processor to: receiving IMU data from the IMU, the IMU data comprising acceleration data corresponding to an acceleration of the user device in the 3D space and orientation data corresponding to a rate of rotation of the user device in the 3D space; and determining the location of the user device in the 3D space by inputting the thermal data, the IMU data, and previous location data to a sensor fusion algorithm.

The video-game system of any preceding clause 1-2, wherein the user device further comprises an Inertial Measurement Unit (IMU), wherein execution of the instructions further causes the processor to: determining an initial position based on the thermal data and independent of the IMU data of the IMU.

Clause 4. the video-game system of any preceding clause 1-3, wherein execution of the instructions further causes the processor to: receive second thermal data from the thermopile array, the second thermal data corresponding to a second thermal signal transmitted from and detected by a second thermal beacon of the plurality of thermopile arrays, wherein the thermal beacon and the second thermal beacon are simultaneously in view of the thermopile array, and wherein the location of the user device is determined based further on the second thermal data.

Clause 5. the video-game system of any preceding clause 1-4, wherein each beacon of the plurality of thermal beacons is an infrared diode.

Clause 6. the video game system of any preceding clause 1-5, wherein the thermal signal transmitted by the thermal beacon comprises an identifier unique to the thermal beacon, wherein the identifier is used by the thermopile array to identify the thermal beacon from the plurality of thermal beacons.

Clause 7. the video game system of any preceding clause 1-6, wherein each thermal beacon of the plurality of thermal beacons is positioned in a game environment, the 3D space being mapped to a portion of the game environment, the thermal beacons being positioned such that the thermopile array is capable of detecting thermal signals from at least two thermal beacons for a particular location of the user device within the 3D space.

Clause 8. a user device, comprising: a thermopile array; a processor; and a memory storing instructions that, when executed by the processor, cause the processor to: receiving thermal data from the thermopile array, the thermal data corresponding to a thermal signal emitted from a thermal beacon of a plurality of thermal beacons and detected by the thermopile; determining a location of the user device in a three-dimensional (3D) space based on the thermal data; and transmitting the location of the user device to a video game console.

The user device of clause 9. the user device of clause 8, wherein the user device further comprises an Inertial Measurement Unit (IMU), wherein execution of the instructions further causes the processor to: receiving IMU data from the IMU, the IMU data comprising acceleration data corresponding to an acceleration of the user device in the 3D space and orientation data corresponding to a rate of rotation of the user device in the 3D space; and determining the location of the user device in the 3D space by inputting the thermal data, the IMU data, and previous location data to a sensor fusion algorithm.

Clause 10. the user device of clause 9, wherein the previous location data is stored on the user device.

Clause 11. the user device of any preceding clause 9-10, wherein the sensor fusion algorithm utilizes an artificial intelligence model trained to determine the location.

Clause 12. the user device of any preceding clause 9-11, wherein the sensor fusion algorithm generates a confidence value corresponding to the thermal data, the confidence value being based on a number of thermal beacons in the field of view of the thermopile array of the plurality of thermal beacons, the confidence value being used by the sensor fusion algorithm to determine the location.

Clause 13. the user device of any preceding clause 8-12, wherein the user device is a video game controller.

Clause 14. the user device of any preceding clause 8-13, wherein each beacon of the plurality of thermal beacons is an infrared diode.

Clause 15. the user device of any preceding clause 8-14, wherein each thermal beacon of the plurality of thermal beacons is positioned in a gaming environment, the 3D space being mapped to a portion of the gaming environment, the thermal beacons being positioned such that one or more thermal sensors of the thermopile array are capable of detecting thermal signals from at least two thermal beacons for a particular location of the user device within the 3D space.

A non-transitory computer-readable storage medium storing instructions that, when executed on a user device, configure the user device to perform operations comprising: generating, by a thermopile array of the user device, thermal data corresponding to thermal signals emitted from a thermal beacon of a plurality of thermal beacons and detected by one or more thermal sensors of the thermocouple array; determining a location of the user device in a three-dimensional (3D) space based on the thermal data; and transmitting, by the user device, the location of the user device to a video game console, the user device communicatively coupled with the video game console.

The non-transitory computer-readable storage medium of clause 16, wherein the operations further comprise: receiving Inertial Measurement Unit (IMU) data from an IMU of the user device, the IMU data comprising acceleration data corresponding to an acceleration of the user device in the 3D space and orientation data corresponding to a rate of rotation of the user device in the 3D space; and determining the location of the user device in the 3D space by inputting the thermal data, the IMU data, and previous location data to a sensor fusion algorithm.

Clause 18. the non-transitory computer-readable storage medium of clause 17, wherein the sensor fusion algorithm utilizes a kalman filter.

The non-transitory computer-readable storage medium of any preceding clause 16-18, wherein the operations further comprise performing calibration by: determining a first location of the user device based on first thermal data corresponding to first thermal signals transmitted from a first set of thermal beacons of the plurality of thermal beacons positioned in the 3D space; determining a second location of the user device based on second thermal data corresponding to second thermal signals transmitted from a second set of thermal beacons of the plurality of thermal beacons positioned in the 3D space; generating, by the user device, a 3D model of the 3D space based on the first location and the second location; and storing, by the user device, the 3D model.

Clause 20. the non-transitory computer-readable storage medium of clause 19, receiving, by the user device, an instruction requesting a user to move the user device to the first location; and receiving, by the user device, an instruction requesting the user to move the user device to the second location in response to the determining the first location.

In the foregoing specification, the invention has been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the invention is not limited thereto. Various features and aspects of the above-described invention may be used alone or in combination. Moreover, the present invention may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

It should be noted that the methods, systems, and apparatus discussed above are intended to be examples only. It must be emphasized that various embodiments may omit, substitute, or add various processes or components as appropriate. For example, it should be understood that in alternative embodiments, the methods may be performed in a different order than described, and that various steps may be added, omitted, or combined. Furthermore, features described with respect to particular embodiments may be combined in various other embodiments. Different aspects and elements of the embodiments may be combined in a similar manner. Further, it should be emphasized that technology is evolving, and thus many elements are examples and should not be interpreted as limiting the scope of the invention.

Specific details are given in the description to provide a thorough understanding of the embodiments. However, it will be understood by those of ordinary skill in the art that the embodiments may be practiced without the specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary detail in order to avoid obscuring the embodiments.

Further, it should be noted that embodiments may be described as a process, which is depicted as a flowchart or a block diagram. Although each embodiment may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process may have additional steps not included in the figure.

Furthermore, as disclosed herein, the term "memory" or "memory unit" may represent one or more devices for storing data, including Read Only Memory (ROM), Random Access Memory (RAM), magnetic RAM, core memory, magnetic disk storage media, optical storage media, flash memory devices, or other computer readable media for storing information. The term "computer-readable medium" includes, but is not limited to portable or fixed storage devices, optical storage devices, wireless channels, sim cards, other smart cards, and various other media capable of storing, containing, or carrying instruction(s) or data.

Furthermore, embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a computer readable medium such as a storage medium. The processor may perform the necessary tasks.

Unless otherwise indicated, all measurements, values, ratings, positions, quantities, sizes, and other indications set forth in this specification (including the following claims) are approximate and not precise. They are intended to have a reasonable range consistent with their associated functions and practices in the art to which they pertain. "about" includes within a tolerance of ± 0.01%, ± 0.1%, ± 1%, ± 2%, ± 3%, ± 4%, ± 5%, ± 8%, ± 10%, ± 15%, ± 20%, ± 25%, or other ranges as are known in the art. "substantially" means greater than 66%, 75%, 80%, 90%, 95%, 99%, 99.9%, or, depending on the context in which the term substantially occurs, other values as known in the art.

Having described several embodiments, it will be recognized by those of skill in the art that various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the invention. For example, the above-described elements may simply be components of a larger system, where other rules may override or otherwise modify the application of the present invention and further, multiple steps may be taken before, during, or after the above-described elements are considered. Accordingly, the above description should not be taken as limiting the scope of the invention.