阅读说明：本技术 电磁跟踪的三维空中鼠标 (Three-dimensional air mouse of electromagnetic tracking ) 是由 舍克·钟 伊恩·阿特金森 墨菲·斯泰因 阿德瓦伊特·雅因 萨基特·帕特卡 卢奇内·奥加涅相 于 2019-04-12 设计创作，主要内容包括：描述了一种手持式电子设备,其用于控制在计算设备的用户界面中显示的三维内容。手持式电子设备可以包括电磁感测系统,其用于检测手持式电子设备在三维空间中的姿态；以及至少一个通信模块,其用于基于检测到手持式电子设备的姿态的改变来触发命令的传输以操纵在计算设备中显示的三维内容。(A handheld electronic device is described for controlling three-dimensional content displayed in a user interface of a computing device. The handheld electronic device may include an electromagnetic sensing system for detecting a gesture of the handheld electronic device in three-dimensional space; and at least one communication module for triggering transmission of a command to manipulate three-dimensional content displayed in the computing device based on detecting a change in the pose of the handheld electronic device.)

1. A handheld electronic device for controlling three-dimensional content displayed in a user interface of a computing device, the handheld electronic device comprising:

an electromagnetic sensing system for detecting a gesture of the handheld electronic device in three-dimensional space for the handheld electronic device;

an inertial measurement unit sensor to detect, for the handheld electronic device, an orientation of the handheld electronic device in a three-dimensional space;

at least one processor coupled to a memory, the at least one sensor configured to generate commands to manipulate the three-dimensional content in the computing device, the commands generated based on a determined proximity of the handheld electronic device relative to a receiver module associated with the computing device, the determined proximity triggering selection of data for use in generating the commands, the data comprising:

an attitude of the electromagnetic sensing system when the determined proximity indicates that the handheld electronic device is within range of the receiver module, and

an orientation of the inertial measurement unit sensor when the determined proximity indicates that the handheld electronic device is outside of range of the receiver module; and

at least one communication module that triggers transmission of the command to manipulate three-dimensional content displayed in the computing device based on a detected change in a pose of the handheld electronic device.

2. The handheld electronic device of claim 1, wherein the handheld electronic device is an air mouse device configured to manipulate a three-dimensional computer-aided design object displayed in a user interface of the computing device, manipulation of the three-dimensional computer-aided design object being based on tracking the pose of the handheld electronic device in the three-dimensional space.

3. The handheld electronic device of claim 1, wherein the handheld electronic device is configured to:

prioritizing use of the electromagnetic sensing system utilizing six degrees of freedom when within range of the receiver module; and is

Switching to performing three degrees of freedom sensing using the inertial measurement unit upon detecting that the handheld electronic device is out of range of the receiver module associated with the computing device.

4. The handheld electronic device of claim 1, wherein the computing device is removably attached to an adapter, the adapter comprising a second processor, a first communication interface with the computing device, and a second communication interface with the handheld electronic device, the adapter operable to:

collecting, from at least one processor of the handheld electronic device and using the first communication interface, data associated with a gesture of the handheld electronic device;

converting, using the second processor, the data from the electromagnetic sensing system or from the inertial measurement unit sensor to the command; and is

Transmitting the command to the computing device using the second communication interface.

5. The handheld electronic device of claim 4, wherein the adaptor further comprises an electromagnetic receiver module to interface with a transmitter module associated with the handheld electronic device.

6. The handheld electronic device of claim 4, wherein:

the handheld electronic device is configured to communicate pose information to the adaptor via a wireless protocol; and is

The adapter is configured to be communicated to the computing device via a wired protocol.

7. The handheld electronic device defined in claim 6 wherein the wireless protocol is Radio Frequency (RF) and the wired protocol is Universal Serial Bus (USB).

8. The handheld electronic device defined in claim 1 wherein the handheld electronic device further comprises:

a microphone configured to receive a voice-based query for communication from the handheld electronic device to the computing device; and

a speaker configured to generate audio playback from the handheld electronic device, the audio playback including information responsive to the voice-based query.

9. The handheld electronic device of claim 1, wherein the handheld electronic device is detected to be out of range of the computing device if a metric related to signal noise associated with the electromagnetic sensing system is above a noise threshold.

10. The handheld electronic device of claim 1, wherein the handheld electronic device further comprises a removable trackball comprising the at least one communication module, wherein the communication module is configured to communicate commands to control three-dimensional objects in the content in accordance with movement of the trackball in the three-dimensional space.

11. The handheld electronic device of claim 10, wherein a change in the detected pose associated with the trackball causes a corresponding change in the pose of the three-dimensional object.

12. An attitude tracking system for an air mouse device, comprising:

an electromagnetic receiver device associated with the air mouse device and configured to determine a 6-DoF gesture between a remote electromagnetic transmitter and the electromagnetic receiver device, the remote electromagnetic transmitter associated with a computing device;

an inertial measurement sensor configured to determine a 3-DoF pose of the air mouse device;

at least one processor coupled to a memory and configured to:

in response to detecting an air-mouse gesture while the air-mouse device is communicatively coupled to the electromagnetic transmitter, generating commands to manipulate three-dimensional content displayed on the computing device using 6-DoF gestures; and is

In response to detecting an air-mouse gesture when the air-mouse device is beyond a predetermined range from the electromagnetic receiver device, generating a command to manipulate three-dimensional content displayed on the computing device using the 3-DoF gesture.

13. The attitude tracking system according to claim 12, wherein the predetermined range is based at least in part on at least one metric corresponding to a received signal strength detected at the electromagnetic receiver device.

14. The pose tracking system of claim 12, further comprising at least one communication module to communicate the generated commands to the computing device to control the three-dimensional content displayed on the computing device.

15. The attitude tracking system of claim 12, wherein the air mouse device further includes a removable trackball comprising:

an electromagnetic sensing system that detects a plurality of gestures associated with an air mouse device,

the at least one processor generating the commands and transmitting commands to control the three-dimensional content according to movement of the trackball in three-dimensional space.

16. The pose tracking system of claim 12, wherein the processor is configured to:

generating a command using the 3-DoF gesture to trigger manipulation of three-dimensional content displayed on the computing device until the air-mouse device is within range of the electromagnetic receiver device; and is

Automatically switching to use the 6-DoF gesture generation command to trigger manipulation of three-dimensional content displayed on the computing device when the air mouse device is within range of the electromagnetic receiver device.

17. An air mouse device for controlling content displayed in a user interface of a computing device, the air mouse device comprising:

an electromagnetic sensing system that detects a three-dimensional position and a three-dimensional orientation of the air mouse device in response to a detected movement of the air mouse device in a three-dimensional space; and

an inertial measurement unit sensor that detects a three-dimensional orientation of the air mouse device in a three-dimensional space;

at least one processor coupled to memory, the at least one processor configured to generate commands to manipulate the content in the computing device, the commands generated based on a determined proximity of the air mouse device relative to the computing device, the determined proximity triggering selection of data used in generating the commands, wherein the data comprises:

the three-dimensional position and the three-dimensional orientation associated with the electromagnetic sensing system when the determined proximity indicates that the air mouse device is within range of the computing device, an

The three-dimensional orientation of the inertial measurement unit sensor when the determined proximity indicates that the air mouse device is outside of range of the computing device; and

at least one communication module that triggers transmission of the command to manipulate the content displayed in the computing device.

18. The air mouse device of claim 17, wherein the air mouse device is configured to manipulate a three-dimensional computer-aided design object displayed in a user interface of the computing device, manipulation of the three-dimensional computer-aided design object being based on tracking the position and orientation of the air mouse device in three-dimensional space.

19. The air mouse device of claim 17, wherein the air mouse device is configured to:

when within range of the computing device, preferentially using the electromagnetic sensing system with six degrees of freedom, and

switching to performing three degrees of freedom sensing using the inertial measurement unit when the air mouse device is detected to be out of range of the computing device.

20. The air mouse device of claim 17, wherein the air mouse device is detected as being out of range of the computing device if a metric related to signal noise associated with the electromagnetic sensing system is above a predefined noise threshold.

21. A computer program product comprising instructions recorded on a non-transitory computer readable storage medium and configured to, when executed by at least one semiconductor processor, cause the at least one semiconductor processor to perform the steps of claim 1.

Technical Field

The present disclosure relates to a computer accessory incorporating non line of sight (NLOS)6-DoF (degrees of freedom) Electromagnetic (EM) tracking technology.

Background

An Electromagnetic (EM) position tracking system may use near-field EM fields, transmitters and receivers to determine position information for a particular device. In general, EM position tracking systems may use transmitters that generate EM signals (e.g., fields) that are detected at remote receivers. In one example, a transmitter generates an EM field using a transmitter coil to induce a current on a receiver coil associated with a remote receiver. The receiver generates a value corresponding to the EM field magnitude, which is then processed to calculate the position and/or orientation (i.e., pose) of the receiver relative to the transmitter.

Disclosure of Invention

A system of one or more computers may be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination thereof installed on the system that, in operation, causes the system to perform the actions. One or more computer programs may be configured to perform particular operations or actions by virtue of comprising instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

In one general aspect, a handheld electronic device for controlling three-dimensional content displayed in a user interface of a computing device. The handheld electronic device includes an electromagnetic sensing system for detecting a gesture of the handheld electronic device in three-dimensional space for the handheld electronic device. The handheld electronic device also includes an inertial measurement unit sensor for detecting, for the handheld electronic device, an orientation of the handheld electronic device in a three-dimensional space. The handheld electronic device also includes at least one processor coupled to the memory and configured to generate commands to manipulate the three-dimensional content in the computing device. The command may be generated based on a determined proximity of the handheld electronic device relative to a receiver module associated with the computing device. The determined proximity may trigger selection of data for use in generating the command. The data may include a pose of the electromagnetic sensing system when the determined proximity indicates that the handheld electronic device is within range of the receiver module, and the data may include an orientation of the inertial measurement unit sensor when the determined proximity indicates that the handheld electronic device is outside of range of the receiver module. The handheld electronic device also includes at least one communication module to trigger transmission of a command to manipulate three-dimensional content displayed in the computing device based on a detected change in the pose of the handheld electronic device. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

Implementations may include one or more of the following features. A handheld electronic device, wherein the handheld electronic device is an air mouse device configured to manipulate three-dimensional computer-aided design objects displayed in a user interface of a computing device. Manipulation of the three-dimensional computer-aided design object may be based on tracking a pose of the handheld electronic device in three-dimensional space. A handheld electronic device, wherein the handheld electronic device is configured to preferentially use an electromagnetic sensing system utilizing six degrees of freedom when within range of a receiver module, and to switch to performing sensing of three degrees of freedom using an inertial measurement unit when the handheld electronic device is detected to be out of range of the receiver module associated with the computing device. A handheld electronic device, wherein the computing device is removably attached to an adapter (dongle), wherein the adapter comprises a second processor, a first communication interface with the computing device, and a second communication interface with the handheld electronic device. The adapter is operable to collect data associated with the pose of the handheld electronic device from at least one processor of the handheld electronic device using the first communication interface, convert data from the electromagnetic sensing system or from the inertial measurement unit sensor into commands using the second processor, and transmit the commands to the computing device using the second communication interface. In some embodiments, the adapter further comprises an electromagnetic receiver module to interface with a transmitter module associated with the handheld electronic device. In some implementations, the handheld electronic device is configured to communicate pose information to the adapter via a wireless protocol, and the adapter is configured to communicate to the computing device via a wired protocol. In some embodiments, the wireless protocol is Radio Frequency (RF) and the wired protocol is Universal Serial Bus (USB).

In some embodiments, the handheld electronic device further comprises: a microphone configured to receive a voice-based query for communication from a handheld electronic device to a computing device; and a speaker configured to generate audio playback from the handheld electronic device. The audio playback may include information responsive to a voice-based query.

In some implementations, the handheld electronic device is detected to be out of range of the computing device if a metric related to signal noise associated with the electromagnetic sensing system is above a noise threshold. In some implementations, the handheld electronic device further includes a removable trackball. The trackball may comprise at least one communication module. The communication module may be configured to transmit commands to control the three-dimensional objects in the content in accordance with movement of the trackball in the three-dimensional space. In some implementations, a change in the detected pose associated with the trackball causes a corresponding change in the pose of the three-dimensional object. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium.

In another general aspect, an attitude tracking system for an air mouse device is described. The attitude tracking system includes an electromagnetic receiver device associated with the air mouse device and configured to determine a 6-DoF attitude between the remote electromagnetic transmitter and the electromagnetic receiver device. The remote electromagnetic transmitter may be associated with a computing device. The attitude tracking system may also include an inertial measurement sensor configured to determine a 3-DoF attitude of the air mouse device. The gesture tracking system may also include at least one processor coupled to the memory and configured to generate commands to manipulate three-dimensional content displayed on the computing device using the 6-DoF gesture in response to detecting the air-mouse gesture while the air-mouse device is communicatively coupled to the electromagnetic transmitter. The at least one processor may be further configured to generate commands to manipulate three-dimensional content displayed on the computing device using the 3-DoF gesture in response to detecting an air-mouse gesture when the air-mouse device is beyond a predetermined range from the electromagnetic receiver device. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

Implementations may include one or more of the following features. An attitude tracking system, wherein the predetermined range is based at least in part on at least one metric corresponding to a received signal strength detected at the electromagnetic receiver device. The pose tracking system further includes at least one communication module to communicate the generated commands to the computing device to control three-dimensional content displayed on the computing device. A gesture tracking system, wherein the air mouse device further comprises a removable trackball comprising an electromagnetic sensing system to detect a plurality of gestures associated with the air mouse device, wherein the at least one processor is configured to generate and transmit commands to control the three-dimensional content according to movement of the trackball in the three-dimensional space. A gesture tracking system, wherein the processor is configured to use the 3-DoF gesture to generate a command to trigger manipulation of three-dimensional content displayed on the computing device until the air mouse device is within range of the electromagnetic receiver device, and to automatically switch to use the 6-DoF gesture to generate a command to trigger manipulation of three-dimensional content displayed on the computing device when the air mouse device is within range of the electromagnetic receiver device. An air mouse device, wherein the air mouse device is configured to manipulate a three-dimensional computer-aided design object displayed in a user interface of a computing device, the manipulation of the three-dimensional computer-aided design object being based on tracking a position and orientation of the air mouse device in a three-dimensional space. An air mouse device, wherein the air mouse device is configured to preferentially use an electromagnetic sensing system utilizing six degrees of freedom when within range of a computing device, and to switch to performing sensing of three degrees of freedom using an inertial measurement unit when the air mouse device is detected to be outside range of the computing device. An air mouse device, wherein the air mouse device is detected to be out of range of the computing device if a metric related to signal noise associated with the electromagnetic sensing system is above a predefined noise threshold. Implementations of the described techniques may include hardware, methods and processes, or computer software on a computer-accessible medium.

In another general aspect, an air mouse device for controlling content displayed in a user interface of a computing device is described. The air mouse device may include: an electromagnetic sensing system for detecting a three-dimensional position and a three-dimensional orientation of the air mouse device in response to a detected movement of the air mouse device in a three-dimensional space; and an inertial measurement unit sensor for detecting a three-dimensional orientation of the air mouse device in a three-dimensional space. The air mouse device may also include at least one processor coupled to the memory, the at least one processor configured to generate commands to manipulate content in the computing device. The command may be generated based on the determined proximity of the air mouse device relative to the computing device. The determined proximity may trigger selection of data for use in generating the command, wherein the data includes a three-dimensional position and a three-dimensional orientation associated with the electromagnetic sensing system when the determined proximity indicates that the air mouse device is within range of the computing device, and the data includes a three-dimensional orientation of the inertial measurement unit sensor when the determined proximity indicates that the air mouse device is outside of range of the computing device. The air mouse device may also include at least one communication module to trigger transmission of commands to manipulate content displayed in the computing device. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

Implementations may include one or more of the following features. An air mouse device, wherein the air mouse device is configured to manipulate a three-dimensional computer-aided design object displayed in a user interface of a computing device, the manipulation of the three-dimensional computer-aided design object being based on tracking a position and orientation of the air mouse device in a three-dimensional space. An air mouse device, wherein the air mouse device is configured to: when in range of the computing device, the electromagnetic sensing system using six degrees of freedom is preferentially used, and when the air mouse device is detected to be out of range of the computing device, the sensing of three degrees of freedom is switched to be performed using the inertial measurement unit. An air mouse device, wherein the air mouse device is detected to be out of range of the computing device if a metric related to signal noise associated with the electromagnetic sensing system is above a predefined noise threshold. Implementations of the described techniques may include hardware, methods and processes, or computer software on a computer-accessible medium.

The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

Drawings

FIG. 1 is a block diagram of an example pose tracking system according to embodiments described herein.

FIG. 2 is a block diagram of an example pose tracking system for an air mouse device according to embodiments described herein.

FIG. 3 is a block diagram of another example tracking system for an air mouse device according to embodiments described herein.

FIG. 4 is a block diagram depicting example tracking ranges for use with one or more air mouse devices described herein.

5A-5C are block diagrams depicting example gesture recognition for use with one or more air mouse devices described herein.

6A-6B are block diagrams depicting an example air mouse device, according to embodiments described herein.

FIG. 7 is a block diagram of an example pose tracking system according to embodiments described herein.

FIG. 8 is a flow diagram of an embodiment of a process for tracking an input device according to embodiments described herein.

FIG. 9 illustrates an example of a computer device and a mobile computer device that may be used with implementations described herein.

The use of like or identical reference numbers in the various figures is intended to indicate the presence of like or identical elements or features.

Detailed Description

This document describes example input devices and tracking of such input devices for use by a computing platform. In particular, the systems and techniques described herein may be used to track input devices for manipulating three-dimensional (3D) models and/or objects provided for display in a computing device. In some implementations, the input devices described herein can be tracked in a mode using six degrees of freedom (6-DoF). In some implementations, the input devices described herein can be tracked in a mode that uses three degrees of freedom (3-DoF). The systems and techniques described herein may determine which mode to use based on the proximity of the input device to one or more other devices.

In some embodiments, the systems and techniques described in this document can be used to determine and/or convert an operation and/or tracking mode of an input device. According to example embodiments described throughout this disclosure, an input device may be capable of determining complete 6-DoF pose data for tracking the input device to allow for accurate manipulation of 3D virtual content in an intuitive manner.

For example, the systems and techniques described herein may include an input device that utilizes Electromagnetic (EM) fields to track the 3D position and/or 3D orientation of a stationary or moving input device. The input device may utilize an EM tracking system that is divided into a transmitter portion that generates the EM field and a receiver portion that senses the EM field. In some implementations, the transmitter portion resides on a computing device accessed through the input device, and the receiver portion resides on the input device. In some implementations, the transmitter portion resides on an input device and the receiver portion resides on a computing device accessed through the input device. In some embodiments, the transmitter or receiver portion is instead on a third device in the EM tracking system acting as a base station and may or may not be mechanically (or communicatively) coupled to the input device or computing device.

Embodiments described throughout this disclosure may utilize an input device, such as an air mouse device that incorporates non-line-of-sight (NLOS) sensing technology, 6-DoF electromagnetic tracking (EM) technology, and 3-DoF tracking technology, to track a position and orientation (e.g., pose) of at least one element embedded in the air mouse device. The EM-based air mouse device described herein may provide advantages over non-EM systems because the EM-based air mouse device is an NLOS and may maintain device tracking despite shielding of onboard or external inductive elements in the tracking system. Thus, the air mouse devices described herein may continue to track when their corresponding sensing elements are occluded while other technologies (e.g., light-based sensors, ultrasonic sensors, and/or other media) may not be able to maintain tracking. Employing EM tracking technology in the air mouse device described herein allows the 6-DoF device to strongly resist occlusion caused by the palm, fingers, or other body parts blocking the sensors of the air mouse device.

The systems and techniques described herein address the technical problem of accurately tracking an input device (e.g., an air mouse device) to accurately communicate manipulations of 3D content in 3D space within a user interface of a computing device to a user of the input device. An example technical solution may include an air mouse computer input device incorporating NLOS, 6-DoF EM tracking technology to track gestures associated with tracking elements embedded in the input device. In some embodiments, a secondary sensor may be installed in the input device to provide fail-safe backup tracking for the input device.

The techniques described herein may provide the technical effect of ensuring a one-to-one motion correlation in six degrees of freedom between the movement performed by the input device and any resulting movement of an object in virtual 3D space on the computing device. The systems and techniques described herein may provide improved input device movement stability, improved input device tracking accuracy, and efficient tracking in 6-DoF space.

Embodiments of the devices described herein may provide advantages over conventional devices. For example, the devices described herein may use NLOS 6-DoF EM tracking when within a particular EM base range, and may automatically switch to non-EM based sensors for tracking when outside the EM base range. Such a switching mechanism may ensure that movement of the input device (and changes in the pose) may be tracked and communicated regardless of the range of the input device to a particular computing device. The user can move content in the computing device without losing range errors and without losing input device representations on the computing device. The handheld electronic device may be detected as being out of range of the computing device if the metric related to signal noise is above a noise threshold associated with the electromagnetic sensing system. For example, in one embodiment, the EM tracking system is configured to provide acceptable tracking performance within a one meter range between the EM transmitter device and the EM receiver device. For example, the system determines that the receiver is out of range of the transmitter each time the system detects that the range has exceeded one meter. In another embodiment, the receiver is determined to be out of range of the transmitter when the signal-to-noise ratio of the received EM field falls below a threshold.

In some embodiments of the air mouse/input device described herein, the EM tracking element may be used with (or instead of) an Inertial Measurement Unit (IMU) sensor to provide absolute or relative pose tracking. In some embodiments, IMU sensors may be used in conjunction with EM tracking elements to provide additional stability and tracking accuracy. Other input devices, adapters, buttons, and touch pads may optionally be added for additional input events. Additionally, the systems described herein may execute a variety of software algorithms to resolve gesture trajectories for use with specific input gestures that may trigger commands on a computing device. For example, the system described herein may convert detected gesture trajectories into resolved gestures. The gesture may be interpreted as an input command by a computing device associated with the system.

In the embodiments described herein, the input device includes a number of example air mouse devices. In general, each air mouse device may house one or more sub-devices, one or more modules, and/or any number of mechanical and electrical components to provide an electrically powered and functional portable input device. Other external housings may be rigid, flexible, and/or communicatively attached to the air mouse device.

FIG. 1 is a block diagram of an example pose tracking system 100 according to embodiments described herein. System 100 may be used to perform position tracking for one or more air mouse devices used with one or more computing devices. Attitude tracking system 100 may provide NLOS, 6-DoF tracking for any number of air mouse devices within the range of the near-field electromagnetic field generated by system 100. The attitude tracking system 100 may provide 3-DoF tracking for any number of air mouse devices outside of the near-field electromagnetic field range, but within the communicable range of the computing device, for example.

In the example system 100, the air mouse device 102 may be moved, for example, by a user accessing content on the computing device 106 or the mobile device 108. Accessing content with the air mouse device 102 may include generating, modifying, moving, and/or selecting content shown within the computing device 106.

The base device system 110 may detect the movement and/or change in pose of the air-mouse device 102 and may perform a number of calculations, determinations, and/or processes to determine the pose and any change in pose as the air-mouse device 102 moves through the 3D space. The gestures may be used with any number of algorithms described herein to track the air mouse device 102 for moving and rendering content as appropriate, for example, for display on the computing device 106 and/or mobile device 108 over the network 104. In some implementations, gestures and other content can be used and/or transmitted directly from the air mouse device 102 without using the network 104. Similarly, gestures or other data may be communicated from base device 110 to device 106 and/or device 108 without using network 104. In some embodiments, devices of system 100 may communicate using a point-to-point communication mechanism (e.g., BLE, USB, etc.).

Once the pose of the air mouse device is calculated relative to the base device system 110, the pose can be utilized in relative space with the system 100. For example, if a user accesses the air mouse device 102 to manipulate 3D content 109 in a CAD program, the relative pose may be sufficient to allow tracking of the device 102 as the user moves the content, as the base device system 110 may be stationary and the system may not benefit from the additional computation of translating the pose to world space. In some implementations, the gestures can be converted to world space using known gestures of the base device system 110 in world space.

As shown in fig. 1, the air mouse device 102 includes a transmit coil and/or a receive coil (e.g., a transmitter/receiver coil 112), at least one processor (e.g., a Central Processing Unit (CPU)114), at least one IMU sensor 118, one or more analog-to-digital converters (ADCs) or one or more digital-to-analog converters (DACs) 120, and one or more amplifiers 122. In some embodiments, the DAC 120 may be combined with one or more amplifiers 122, which may function as an amplifier when converting a signal to an analog signal. Other hardware and software may be present in system 100, for example, to communicate via additional wired or wireless protocols.

The base unit system 110 includes a transmitter/receiver coil 130, at least one CPU132, a memory 134, an ADC/DAC 136, and one or more amplifiers 138. Other hardware and software may be present in system 100, for example, to communicate via other wired or wireless protocols.

Transmitter/receiver coil 112 and transmitter/receiver coil 130 may represent transmitter coils or receiver coils that may emit and/or sense EM fields (not shown), respectively. In general, the base equipment system 110 may use transmitter coils to generate the EM fields. The air mouse device 102 may read the EM data generated by the system 110 using the receiver coil. Other configurations are possible. For example, the EM fields and data may be generated by the air mouse device 102, while the base device system 110 reads the EM data (as shown in fig. 2).

In general, the gesture tracking system 100 uses near-field electromagnetic fields to determine gesture data associated with an input device, such as the air mouse device 102. The pose data may be used to track the air mouse device 102 as the user moves the device (or while the device is stationary). In operation, the pose tracking system 100 may utilize at least one transmitter that generates an EM field on a three-axis coil (e.g., transmitter coil 112) to induce a current on a second three-axis coil (e.g., receiver coil 130) at a receiver (e.g., base equipment system 110). The receiver generates a plurality of readings, which are then processed by the system 110, for example, to calculate the position and orientation of the transmitter (e.g., on the air mouse device 102) relative to the receiver (e.g., on the base device system 110).

In some implementations, the system 110 may generate EM fields using the transmitter coil 130 and the amplifier 138. For example, the transmitter coil 130 may be a tri-axial coil (e.g., three coils) that may generate three EM fields (one for each different axis). The EM field may be provided at a strength (i.e., transmit power) based on the electrical power provided to the transmitter coil 130 by the amplifier 138. The amplifier 138 may be a programmable amplifier generally configured to generate a certain amount of electrical power based on received control signaling. In general, amplifiers 122 and 138 may function to amplify the received signals.

Continuing with the above example, the air mouse device 102 may include a component for reading EM data. For example, the device 102 includes a receiver coil 112 and an ADC 120. In some embodiments, the receiver coil 112 is a tri-axial coil configured to generate analog electrical signals having a magnitude and/or phase indicative of the detected EM field. The ADC 120 is generally configured to receive the generated analog signal and convert the analog signal to a digital value indicative of the analog signal of the detected EM field. In this example, the digital values generated by ADC 120 represent EM data that may be used for gesture recognition, as described further below.

The air mouse device 102 may also include a processor 114, and the processor 114 may be in communication with a processor 132 in the system 110. The communication may include relative pose data of the system 110 with respect to the air mouse device 102 (or alternatively, pose data of the device 102 with respect to the system 110). Such processors 114 and 132 are configured to execute instructions (e.g., computer programs) in order to perform specific tasks. In some embodiments, at least one of processors 114 and 132 executes instructions to identify a relative pose between system 110 and device 102 based on the EM data provided by ADC 136. Memory 116 and memory 134 may be utilized throughout communications and interactions between system 110 and device 102.

The system 100 may be used to track 3D position and 3D orientation (i.e., 6-DoF tracking). The input device may utilize an EM tracking system that is divided into a transmitter portion that generates the EM field and a receiver portion that senses the EM field. In one non-limiting example, the transmitter portion is housed on the system 110 and includes a transmitter coil 130, an ADC 136 and an amplifier 138. The transmitter portion may generate three fields on a three-axis coil (not shown) while the receiver portion employs the three-axis coil (not shown) as an antenna to sense the fields generated by the transmitter portion. The receiver section may include a receiver coil 112, an ADC/DAC 120, and an amplifier 122. Each coil on the receiver portion may sense all three fields generated by the coil associated with the transmitter portion. This interaction results in at least nine EM measurements. Using these measurements, the pose of the receiver coil 112 relative to the transmitter coil 130 may be calculated from the nine measurements by employing a dipole model (not shown) to equate the sensed EM size (in the EM data) to the pose. A gesture, as used herein, may refer to a position, an orientation, or both.

In another non-limiting example, the transmitter portion is housed on the device 102 and includes the transmitter coil 112 and the DAC 120 and the amplifier 122. The transmitter portion may generate three fields on a three-axis coil, while the receiver portion employs the three-axis coil (e.g., coil 112) as an antenna to sense the fields generated by the transmitter portion. The receiver portion may be housed on the system 110 and may include a receiver coil 130, an ADC/DAC 136, and an amplifier 138. Each coil on the receiver portion may sense all three fields generated by the coil associated with the transmitter portion. This interaction results in at least nine EM measurements. Using these measurements, the pose of the receiver coil 130 relative to the transmitter coil 112 may be calculated from the nine measurements by employing a dipole model (not shown) to equate the sensed EM magnitude (in the EM data) to the pose.

As shown in FIG. 1, the air mouse device 102 may also include an IMU sensor 118. The IMU sensor 118 may function to detect a 3D orientation of the air mouse device in 3D space based on IMU measurements. The IMU sensors 118 may include one or more accelerometers, gyroscopes, magnetometers, and other such sensors. In general, the IMU sensors 118 may detect, for example, motion, movement, velocity, and/or acceleration of the air mouse device 102.

In some implementations, for example, the attitude of the air mouse device 102 can be detected based on data provided by the IMU sensors 118. Based on the detected gesture, the system 100 may update content on the computing device 106 (or the base device 110) to reflect the changed gesture of the air mouse device 102.

The term transmitter/receiver as used herein may refer to a single transmitter, a single receiver, a transmitter and a receiver (e.g., a transceiver), a plurality of transmitters, or a plurality of receivers. In some implementations, the Transmitters (TX) and Receivers (RX) are distributed on different devices, including stationary base devices (e.g., associated with or within the computing device 106) that track the position and/or orientation of the movable input device. In some implementations, the base device system 110 can include a transmitter coil, and the air mouse device 102 can include a receiver coil. In some implementations, the base device system 110 can include a receiver coil, while the air mouse device 102 can include a transmitter coil. In some implementations, one or both of the devices 102, 110 may include a transceiver that may energize the coils of the respective transceivers to transmit and receive depending on which EM field.

In some implementations, the base device system 110 is connected to the computing device 106 and/or the mobile device 108. For example, the base device system 110 may be rigidly connected to the computing device 106 and/or the mobile device 108. In some examples, base device system 110 may be communicatively coupled (e.g., wired or wirelessly) to computing device 106 and/or mobile device 108. In some examples, the base device system 110 may be an adapter coupled to the computing device 106 and/or the mobile device 108 and may communicate with the air mouse device 102 via the adapter. In some implementations, the base device system 110 is housed within the computing device 106 or the mobile device 108.

In example system 100, devices 106, 108, and 110 may be (or may be part of) a laptop computer, a desktop computer, a mobile computing device, a tablet computing device, or a gaming console. Devices 106, 108, and 110 may include hardware and/or software for executing applications and application content. Additionally, the devices 106, 108, and 110 may include (or have access to) hardware and/or software that may recognize, monitor, and track the pose of the air mouse device 102 when the devices 106, 108, and 110 are placed in front of the air mouse device 102 or remain within a range of positions relative to the air mouse device 102. In some implementations, the devices 106, 108, and 110 can provide additional content to the air-mouse device 102 over the network 104. In some implementations, the devices 102, 106, 108, and 110 can be one or more paired with each other or connected/interfaced via the network 104. The network 104 may be a public communication network or a private communication network.

System 100 may include or have access to electronic storage devices (e.g., memory 116 and memory 134). Electronic storage devices may include non-transitory storage media that electronically store information. The electronic storage device may be configured to store captured pose data, raw sensor data, application data, and/or other computational data.

In some implementations, the base device system 110 can be integrated with the computing device 106, and the computing device 106 can include (or be coupled to) the adapter 140, as detailed with respect to fig. 7 below. In some embodiments, the air mouse apparatus 102 may include a trackball 142, as detailed with respect to fig. 6A-6B below.

In some implementations, the air mouse device 102 can include a communication module 144. The communication module 144 may be used to transfer data to the CPU 114 or other devices in the system 100. In some implementations, the communication module 144 can transmit a command to change the content 109 in the manipulation device 106 based on the detected movement and/or gesture from the air mouse device 102. Although communication modules are not explicitly depicted for the computing devices 108, 106 and the base device system 110, each of these devices may have the capability to send and receive messages, data, commands, etc. between any of the devices of the system 100.

In some implementations, the air mouse device 102 includes a gesture module 146. In some implementations, the base device 110 includes a gesture module 146 instead. In yet other implementations, the computing device 106 includes a gesture module 146. Gesture module 146 represents algorithms and/or software that can convert gesture trajectories into resolved gestures. The gesture trajectory may be generated by a user accessing the air mouse device 102. For example, the gesture may be interpreted by the associated computing device 106 of the containment system 100 as an input command. In some implementations, the pose of the air mouse device can control a two-dimensional (2D) mouse cursor on the associated computing device 106. The mouse cursor may be moved by translating the air mouse device 102 or changing the pointing angle of the air mouse device 102.

In some embodiments, the air mouse device 102 includes a microphone (not shown) and/or a speaker (not shown), as detailed with respect to fig. 7.

FIG. 2 is a block diagram of an example attitude tracking system 200 for an air mouse device 202 according to embodiments described herein. The tracking system 200 may be an EM tracking system that utilizes a transmitter coil and a receiver coil to perform tracking of the airborne mouse device 202. In some implementations, the gesture tracking system 200 can include non-electromagnetic sensors and devices to assist in tracking the aerial mouse device 202 as the user moves the device 202 to interact with content in the computing device (associated with or integrated with the base device 204).

As shown in this example, the system 200 includes an input device (e.g., an air mouse device 202) and a base device 204. The air mouse device 202 may be a handheld electronic device for controlling 3D content displayed in a user interface of a computing device. For example, the device 202 may be an air mouse device 102 for use with content 109 on a computing device 106, where the base device system 110 is installed on (or accessible by) the computing device 106.

The tracking system 200 is generally configured to identify a relative pose between the air-mouse device 202 and the base device 204 by generating an EM field 201, measuring the magnitude and/or phase of the generated EM field 201 (referred to herein as "EM data"), and calculating the relative pose based on the corresponding EM data. Thus, the tracking system 200 may be incorporated into a variety of devices and systems that employ gesture recognition algorithms. In some implementations, a portion of the tracking system 200 is incorporated into an air mouse device, while other portions of the tracking system 200 are incorporated into a computing device. Thus, in some configurations, the base device 204 is a computing device (e.g., the base device system 110, the computing device 106, or the mobile device 108), and the input device is a handheld air mouse device 102/202. In some configurations, the base device 204 is an input device, while the air-mouse device 202 is a stationary computing device (e.g., the base device system 110, the computing device 106, or the mobile device 108). Other configurations are possible, including EM fields generated by device 202 to ensure that EM data can be read at base device 204.

In operation of system 200, EM field 201 is generated by base unit 204. Base device 204 includes an EM transmitter module 206 to generate an EM field. The EM transmitter module 206 includes a transmitter coil 208, an amplifier 210, and a DAC 212. The transmitter coil 208 may represent, for example, a tri-axial coil configured to generate the EM field 201 at a particular strength (e.g., transmit power). The transmit power may be based at least in part on the electrical power provided by the amplifier 210. The amplifier 210 may be configured to generate electrical power at a magnitude based on received control signaling of the device 204.

In response to detecting the generated EM field 201, the air mouse device 202 reads EM data from the EM field 201 using the onboard EM receiver module 220. In general, EM emitter module 206 may be used as part of an electromagnetic sensing system, for example, to detect gestures associated with air mouse device 202. The air mouse device 202 may include the rest of the electromagnetic sensing system in the EM receiver module, as described with respect to EM receiver module 220. In some embodiments, the EM receiver module includes one or more processors (not shown).

The base device 204 includes a CPU (i.e., processor 214) that can bi-directionally communicate with the air mouse device 202 via a communication link 218. For example, EM data, recognized gestures, recognized gesture trajectories, and/or other information may be exchanged between the air mouse device 202 and the base device 204. For example, in some implementations, processor 216 recognizes the gesture based on the EM data and communicates the recognized gesture to processor 214. In some implementations, processor 216 communicates the EM data to processor 214, which processor 214 identifies gestures based on the EM data.

The communication link 218 may be a wired communication link (e.g., USB, serial, ethernet, etc.), a wireless communication link (e.g., bluetooth, WiFi, ZigBee, RF, etc.), etc., or a combination thereof. In other embodiments, the EM data may be sent to another device and/or processor (not shown), and the other device and/or processor may calculate a pose from the EM data. In some embodiments, the EM data may be stored locally within the device 202 or 204, locally within the system 200, and/or remotely from the system 200.

As shown in fig. 2, air mouse device 202 is an input device that includes an EM receiver module 220 to generate EM data from a detected EM field. EM receiver module 220 includes a receiver coil 224, an amplifier 226, and an ADC 228. In some implementations, the receiver coil 224 may be a tri-axial coil configured to detect analog electrical signals having a magnitude and/or phase indicative of a particular detected EM field. The ADC 228 may be configured to receive the generated analog signal and convert the analog signal to a digital value indicative of the analog signal represented in the EM field 201. The digital values generated by the ADC 228 are EM data that may be used for gesture recognition of the air mouse device 202.

The air mouse device 202 further includes an IMU sensor 230. The IMU sensor 230 may provide additional pose information about the device 202. One or more of processors 214 and 216 (or another device processor) may use the additional pose information to supplement or augment the pose identified based on the EM data. For example, in some embodiments, processor 216 may use the additional pose information to identify potential errors in the pose determined based on the EM data and resolve the identified errors.

In some implementations, for example, when device 202 is outside the range of the EM field generated by transmitter module 206, processor 216 may track air mouse device 202 using additional pose information determined with IMU 230. Thus, the IMU 230 may be used as a backup pose determiner to continue tracking the device 202. For example, if the device 202 is an air mouse device 102 and the user-operated device 102 is moved out of range of the base device 204 (e.g., installed within the computing device 106), the system 200 may still determine pose information sufficiently well to track the mobile device 102 so that content (e.g., 3D objects) within the computing device 106 may be moved based on the tracked movement of the air mouse device 102. In this example, the pose information may be 3-DoF pose data based on the detected 3D orientation of device 102/202.

For example, if the IMU 230 is used as a backup pose determiner, the systems described herein may include one or more processors (e.g., processor 216 on device 202) configured to generate commands to manipulate 3D content in the computing apparatus 106. In some implementations, the IMU may generate commands to manipulate 3D content displayed on the computing device using 6-DoF gestures in response to detecting an air mouse gesture while the air mouse device is communicatively coupled to the electromagnetic transmitter. To this end, the system described herein may begin using pose information collected from IMU 230 (rather than from an EM sensing system that includes both transmitter module 206 and receiver module 220). The pose information may be used to track movement of the air mouse device 202. For example, the movement may be translated into a command that may be executed on content 109 shown in a user interface of computing device 106.

In some implementations, the command can be generated based on a determined proximity of an air mouse device (e.g., air mouse device 102) relative to a base device (e.g., base device 204). For example, the base device 204 can be associated with the computing device 106 (or in communication with the computing device 106). The determined proximity may be an indication that the air mouse device 102 is within or outside of the range of the base device 204 (and/or the computing device 106 if a transition between the base device 204 and the computing device 106 can be estimated). The determined proximity may trigger the selection of which particular data to use when generating the command. For example, when the determined proximity indicates that device 102 is within range of base device 204, system 200 may utilize data (e.g., pose data) from an electromagnetic sensing system. Instead, the system 200 may utilize pose data (e.g., three-dimensional orientation) from the IMU 230 when the determined proximity indicates that the device 102 is out of range of the base device 204 and/or the computing device 106.

Upon generating the one or more commands, communication module 144 may access CPU 216 to trigger transmission of the one or more commands to manipulate three-dimensional content displayed in the computing device.

FIG. 3 is a block diagram of another example attitude tracking system for an air mouse device 302 according to embodiments described herein. Similar to the pose tracking system 200, the system 300 may be an EM tracking system that utilizes a transmitter coil and a receiver coil to perform tracking of the airborne mouse device 302. In some implementations, the gesture tracking system 300 can include non-electromagnetic sensors and devices to assist in tracking the air mouse device 302 as a user moves the air mouse device 302 to interact with content in a computing device (e.g., computing device 106) associated with the base device 304.

Pose tracking system 300 is similar to pose tracking system 200 of FIG. 2, but pose tracking system 300 places EM receiver module 306 at base device 304 and EM transmitter module 308 at air mouse device 302. For example, as shown in fig. 3, the base device 304 includes an EM receiver module 306 having a receiver coil 310, an amplifier 312, and an ADC 314. The base device also includes a processor 316.

Input device 302 includes an EM transmitter module 308 to generate an EM field sensed at base device 304. The EM transmitter module 308 includes a transmitter coil 318, an amplifier 320, and a DAC 322. In some embodiments, the transmitter coil 318 is a tri-axial coil configured to work with the DAC 322 and the amplifier 320 to generate an analog electromagnetic field.

The air mouse device 302 also includes an IMU sensor 326. IMU sensors 326 may provide additional pose information about device 302. One or more of processors 316 and 324 (or another device processor) may use the additional pose information to supplement or augment the pose identified based on the EM data. For example, in some embodiments, processor 326 may use the additional pose information to identify potential errors in the pose determined based on the EM data and resolve the identified errors.

In some implementations, for example, when device 302 is outside the EM field range described herein, processor 316 (or processor 324 or another device processor) may track air mouse device 302 using additional pose information determined with IMU 326. Thus, if the device 302 is an air mouse device 102 and the user operating the device 102 moves out of range of the base device 304 (e.g., installed within the computing device 106), the system 300 may still well determine pose information to track the moving air mouse device 102 so that content (e.g., 3D objects) in the computing device 106 may be moved based on the tracked movement of the air mouse device 102.

In operation of system 300, EM transmit module 308 is configured to generate EM field 328. The EM receiver module 304 is configured to generate EM data from the sensed EM field 328. In some embodiments, IMU data from the air mouse may be transferred from processor 324 to processor 316 via communication link 330. In some implementations, the EM data may be used to identify the relative pose between the base device 304 and the device 302. Similarly, IMU data may be used to identify the pose of the air mouse device 302.

FIG. 4 is a block diagram depicting example tracking ranges for use with one or more air mouse devices described herein. In the depicted example, the air mouse device 402 is shown at a distance from the computing device 404. The air mouse device 402 may be used to modify 3D content in a display of the computing device 404. Air mouse device 402 may represent air mouse device 102 (FIG. 1), and computing device 404 may represent computing device 106 housing system 110.

In operation, the air mouse 402 may be moved away from the computing device 404, and the computing device 404 houses receiver circuitry and/or transmitter circuitry for determining pose information, which may be used to track the device 402. For example, a user may operate device 402 to move the device (from 402 to 402A) to manipulate 3D content in device 404. Here, the user has moved device 402 away from computing device 404, as indicated by arrows 406 and 408. The device is shown as moving in a straight line, but any path of movement is possible. The line from location 410 to location 418 is intended to illustrate a particular distance that the functionality of device 402 may change based on distance from device 404. For example, because the air mouse device 402 includes EM-based circuitry, if the device 402 travels from the computing device 404 beyond a system-defined operating distance, movement of the device 402 from a particular antenna (i.e., coil) within the device 404 may cause signal attenuation and signal failure. In particular, noise may increase as the distance between a receiver module (e.g., receiver module 220) and a transmitter module (e.g., transmitter module 206) increases. The increased noise may cause signal failure or the inability to utilize data from the devices 402, 404.

The system described herein can detect when a particular receiver module is out of range of a particular transmitter module. That is, when noise (or a noise-related metric such as distance) between the receiver and the transmitter is greater than a threshold, the system described herein may fall back to 3-DoF pose estimation and subsequent device tracking. In some embodiments, the threshold may be a predefined threshold based on the hardware used in the system, and in other embodiments, the threshold is dynamically calculated. In some examples, the threshold may be set at a distance between the transmitter and receiver that allows for attitude determination, but may be a large enough distance to induce noise effects that cause signal degradation and/or data transmission failures. In either case, if a particular receiver is detected to be out of range of the transmitter, the system may rely on a secondary sensor (e.g., IMU sensor 118) to provide 3-DoF tracking until the transmitter and receiver are detected to be in range again. In one example, the EM tracking system 100 may return to performing 6-DoF pose tracking upon detecting that the air mouse device 102 is within range of the computing device 404 (housing the base device system 110).

As shown in FIG. 4, EM-based pose determination and device tracking may be performed by system 100 if a user moves air-mouse device 402 from position 410 to a distance (within a 6-DoF range) represented by position 412 or position 414. The maximum distance may be expressed in terms of the device electronics and power protocol used. In an example operation of device 402, it may be indicated that there is approximately one meter from location 410 to location 414. However, such a distance may be programmed to be less than or greater than about one meter.

If the device 402 moves beyond the location 414, for example, to a location 416 or 418 (e.g., within a 3-DoF range), the system 100 may revert to performing 3-DoF pose determination and device tracking using the IMU sensors 118. For example, if air-mouse device 402 is returned to location 414 or otherwise placed within range of device 404 (housing an EM transmitter/receiver device), system 100 may use pose determination and device tracking of 6-DoF (based on EM).

In some implementations, an example air mouse tracking system may include a handheld air mouse device and a host computing device. The handheld air mouse device may include an IMU, a first processor, and an EM receiver device or an EM transmitter device. If the handheld air mouse device includes an EM transmitter device, the host computing device may include an EM receiver. If the handheld air mouse device includes an EM receiver device, the host computing device may include an EM transmitter device. The host computing device may also include a display and a second processor in communication with the first processor. The first processor may be configured to collect pose data derived from EM data and/or IMU data. The pose data may be used (by the first processor or the second processor) to calculate an estimated pose between the EM receiver module and the EM transmitter module. The gestures may be used to generate commands to manipulate three-dimensional content in the computing device.

In some embodiments, the EM receiver device or the EM transmitter device may be in a base device that contains the third processor and is external to the host computing device. The estimated pose may be calculated by a single processor or any combination of the first processor, the second processor, and/or the third processor. In some embodiments, the estimated pose is a 6-DoF pose calculated from the EM data when a metric related to noise in the EM data is below a threshold. When the metric is above the threshold, the estimated pose is a 3-DoF pose calculated from the IMU data.

In another example, a handheld electronic device is described to control three-dimensional content displayed in a user interface of a computing device. The handheld electronic device may include an EM receiver module, an IMU, a first communication link, and at least one processor. The at least one processor may be configured to collect EM data from the EM receiver module, collect IMU data from the IMU, and communicate pose data over the first communication link, wherein the pose data is derived from the EM data and/or IMU data, and wherein the pose data is used to estimate a pose of the handheld electronic device. Additionally, the gestures may be used to generate commands to manipulate three-dimensional content in the computing device.

In some implementations, the transmitter is housed within the computing device. In some implementations, the transmitter is embedded in an adapter that plugs into the computing device. In some implementations, the adaptor includes a second communication module that maintains a communication link with the first communication module, receives gesture data from the handheld electronic device over the communication link, and forwards data derived from the gesture data to the computing device over the USB. In some embodiments, instead, the transmitter module is housed in the air mouse device and the receiver is housed in the computing device.

5A-5C are block diagrams depicting example gesture resolution for use with one or more air mouse devices described herein. The air mouse device 502 may be described with respect to similar devices 102, 202, or 302. Gesture resolution may be performed by a gesture module within the air mouse device (e.g., gesture module 146 in air mouse device 102), base device 110, or computing device 106. The gesture module includes algorithms and/or software that can convert gesture trajectories determined for the air mouse device into resolved gestures. The gesture trajectory may be generated by a user accessing the air mouse device 102. For example, the gesture may be interpreted by the relevant computing device 106 housing the system 100 as an input command. In some implementations, the pose of the air mouse device may control a two-dimensional (2D) mouse pointer on the associated computing device 106. In general, the mouse pointer may be moved by translating the air mouse device 102 or changing the pointing angle of the air mouse device 102.

As shown in FIG. 5A, a user may be using an air mouse device 502 to access software and content in a user interface 503 in a computing device 504. The air mouse device 502 may be moved at an angle to cause a proportional change in the position of the 2D mouse cursor.

For example, the user may use device 502 to perform an arc movement from location [1]506 to location [2]508, as shown by device 502A at arrow 510. In response, system 100 may accurately move mouse pointer 512 from left position [1]514 to right position [2]516 in user interface 503 at a proportional distance using pose information obtained from the EM-based tracking system (split between system 110 and device 102), causing the mouse pointer to be shown as mouse pointer 518. That is, if the air mouse device 502 moves at an angle change as indicated by arrow 510, the mouse pointer 512 also moves from left to right a distance proportional to the angle change.

As shown in FIG. 5B, a user may be using an air mouse device 502 to access software and content in a user interface 503 in a computing device 504. The air mouse device 502 may move from left to right to cause a proportional change in the position of the 2D mouse cursor.

For example, the user may use device 502 to perform a left-to-right movement from position [1]520 to position [2]522, as shown by device 502B at arrow 524. In response, system 100 may accurately move mouse pointer 526 from left position [1]528 in user interface 503 to right position [2]530 at a proportional distance using pose information obtained from the EM-based tracking system (split between system 110 and device 102), causing the mouse pointer to be shown as mouse pointer 532. That is, if the air mouse device 502 moves as indicated by arrow 524, the mouse pointer 526 also moves from left to right a distance proportional to the change in translation. Thus, the mouse pointer position changes in proportion to the position of the air mouse device in 3D space.

As shown in fig. 5C, a user may be using the air mouse device 502 to access software and 3D content 534 in the user interface 503 in the computing device 504. The air mouse device 502 may move from left to right and forward in space to cause a proportional change in the position of a 2D or 3D mouse cursor configured to move in virtual space displayed on a computing device.

For example, the user may use device 502 to perform a left-to-right movement from position [1]536 to position [2]538, as shown by device 502C at arrow 540. In response, system 100 may use the pose information obtained from the EM-based tracking system (split between system 110 and device 102) to accurately move the mouse pointer 542 from a left position [1]544 in user interface 503 to a right position [2]546 at a proportional distance, causing the mouse pointer to be shown as mouse pointer 548. Thus, the mouse pointer position changes in proportion to the position of the air mouse device in 3D space.

Similarly, the user can use device 502 (to 502D) to perform a back-and-forth (i.e., forward) movement in 3D space from location [2]538 to location [3]550, as shown by device 502D at arrow 552. In response, the system 100 may accurately move the mouse pointer 548 from the first location [2]546 to the second location [2]544 in the user interface 503 at a proportional distance using pose information obtained from the EM-based tracking system (partitioned between the system 110 and the device 102), causing the mouse pointer to be shown as mouse pointer 556. Thus, by moving the air mouse device in 3D space, the mouse pointer position changes in proportion to the position of the air mouse device in virtual 3D space.

In some implementations, the system 100 can capture and interpret complex gestures. For example, such gestures may be captured and interpreted as commands to the associated computing device 106 of the housing system 110, for example. One example gesture may include a user shaking the air mouse device 502 back and forth from side to side. The system 100 may interpret the movement as a command to close the current operating system/application window. In another example, the gesture may include pressing a button and pointing the air mouse device 502 down to trigger the system 100 to scroll down a window focused in the interface 503. In yet another example, the gesture may include entering a letter (e.g., "x") in the air (e.g., 3D space) as input using device 502. Such a gesture may cause the system 100 to interpret the gesture as a command to close a window. Each of these example gestures may be interpreted and transmitted as commands (e.g., via communication module 144) to a software application and/or operating system on computing device 106. The gestures may be interpreted by gesture module 146 using any EM-based sensor and/or component, as well as non-EM-based sensors and/or components described in fig. 1 and/or systems throughout this disclosure.

In some implementations, the systems described herein can capture gestures in a 3D context and interpret them as commands to a computing system. In particular, the system 100 may capture gestures in a 3D context and interpret them as commands. For example, 3D object 534 may be displayed on computing device 504. The system 100 can determine that the user wishes to interact with the content 534 based on the contextual cues. If system 100 determines that the user has some background with 3D object 534, system 100 may lock the motion of air mouse device 502 to 3D object 534 such that performing the movement through device 502 causes a one-to-one movement of 3D object 534. For example, pulling (not shown) air mouse device 502 back may cause 3D object 534 to appear larger or closer in user interface 503, effectively enlarging 3D object 534. Similarly, a gesture that includes rotating air-mouse device 502 in the air may cause a corresponding rotation of 3D object 534 in user interface 503. In another example, using two air mouse devices (e.g., one device 502 in each of the left and right hands) may allow a user to intuitively grasp (e.g., naturally move) 3D object 534 and pull or stretch the object. System 100 may simulate handling a first end using left air mouse device 502 and a second end using right air mouse device 502, effectively simulating grasping the first end and the second end, respectively.

FIG. 6A is a block diagram depicting an example air mouse device 600A according to embodiments described herein. The air mouse device 600A may control a 2D pointer in a computing device by rolling a trackball 602 on a trackpad 604. For example, the air mouse device 600A may be accessed by a user to move the content 109 in two dimensions on the computing device 106 (FIG. 1) as the user scrolls the trackball 602 around the trackpad 604. If the user scrolls trackball 602 to the right, as indicated by arrow 606, content 109 may move from left to right. In this example, trackball 602 may provide gesture information, and may generate a command in response to the gesture information.

FIG. 6B is a block diagram depicting an example 3D air mouse device 600B according to embodiments described herein. The air mouse device 600B may be removed from the trackpad 610. The air mouse device 600B may control a 3D virtual object in a computing device by lifting a trackball 608 from a trackpad 610 and moving the trackball 608 in 3D space (e.g., the x-y-z axes in fig. 6B). For example, the user may lift the trackball 608 and turn the trackball from vertical y to the right (or left), as indicated by arrow 612. Such movement may tilt the content 109 (fig. 1) at any angle so that the user may choose when twisting and/or rotating the trackball 608. Similarly, the user may manipulate the content 109 to twist the content backwards (or forwards) from the horizontal z, as shown by arrow 614. Additionally, the user may rotate the trackball about the y-axis in 3D space, e.g., as shown by arrow 616. Using the tracked EM based system described herein, rotation of the trackball 608 triggers rotation of the content 109 (e.g., three-dimensional object) displayed in the computing device. Similarly, translating the trackball 608 in space causes a proportional translation of the content 109 in virtual space shown in the UI of the computing device.

The user may move the trackball 608, including tilting, rotating, and circling, left, right, up, down, forward, and backward in 3D space, for example, to effect movement of the content 109. The movement may be a one-to-one movement in which the user moves the trackball in space and the content 109 moves the same distance (or a predetermined proportional distance based on the physical movement of the trackball 608) in the computing device.

In general, the trackball 608 may provide pose information and may generate commands in response to the pose information. Such commands may manipulate, control, or otherwise modify content 109 in the computing device. To function in system 100, the trackball can include at least one emitter module (e.g., emitter module 308) to emit commands to control a three-dimensional object (e.g., content 109) in accordance with movement of trackball 608 in 3D space. In some embodiments, the trackball may instead include at least one receiver module, while the base computing device includes a transmitter module.

In some embodiments, the input devices described herein include one or more buttons that provide additional input events. Additionally, the input device may contain any or all of a slider, a one-dimensional (1D) touchpad, a scroll bar, or similar components that allow additional 1D input. In some implementations, the input devices described herein can include a 2D touchpad, a joystick, and/or similar components that allow 2D simulation input into the respective input device.

FIG. 7 is a block diagram of an example pose tracking system 700 according to embodiments described herein. The gesture tracking system 700 may include an input device 702 and a computing device 704 removably attached to an adapter 706. The adapter 706 may include components similar to those in the base device 204 (FIG. 2) or 304 (FIG. 3). The input device 702 may be a handheld electronic device for controlling 3D content displayed in a user interface (not shown) of the computing device 704. For example, the device 702 may be an input device 102 for use with content 109 on a computing device 106, where the base device system 110 is installed on or connected to the computing device 106, or otherwise accessible to the computing device 106.

In the depicted example, the adapter 706 can include a USB connector 708, and the USB connector 708 can mate with the computing device 704 through a connector 710. In general, the input device 702 may be a handheld device such as a controller, air mouse, mobile device, or tablet device. The input device 702 may be configured to communicate the gesture information to the adapter 706 via a first wireless protocol (e.g., via radio frequency, Wi-Fi, or other wireless signals in the electromagnetic spectrum). The adapter 706 may communicate with the computing device 704 using a wired protocol, such as Universal Serial Bus (USB), ZigBee, serial, firewire, thunderbolt, lightning, analog audio, digital audio, and so forth. In some implementations, the adapter 706 includes an EM transmitter module 712 to interface (and/or communicate) with an EM receiver module 714 associated with the input device 702. Although a USB interface for the adapter 706 is depicted and may provide both a reliable mechanical interface and a small form factor digital communication protocol, any other interface (such as ZigBee, serial, firewire, thunderbolt, lightning, analog audio, digital audio, etc.) may be used in any of the devices described herein.

The tracking system 700 may be an EM tracking system that utilizes a transmitter coil and a receiver coil to perform tracking of the input device 702. In some implementations, the gesture tracking system 700 can include non-electromagnetic sensors and devices that assist in tracking the input device 702 as the user moves the device 702 to interact with content in the computing device (associated with or integrated with the computing device 704 connected to the adapter 706).

In some embodiments, EM transmitter module 712 is instead in input device 702, and EM receiver module 714 is switched to adapter 706. In this embodiment of the system, EM data may be collected at the adapter and may be translated into a pose by CPU722 in adapter 706 or by CPU 724 in computing device 704.

The tracking system 700 is generally configured to identify a relative pose between the input device 702 and a base device (e.g., the adapter 706) by generating an EM field 701, measuring the magnitude and/or phase of the generated EM field 701 (collectively referred to herein as "EM data"), and calculating the relative pose based on the corresponding EM data. Other configurations are possible, including EM fields generated by device 702 to ensure that EM data can be read at base computing device 704 (e.g., via dongle 706).

In operation of system 700, an aptamer 706 generates an EM field 701. The adaptor 706 includes an EM transmitter module 712 to generate an EM field. The EM transmitter module 712 includes a transmitter coil 716, an amplifier 718, and a DAC 720. The transmitter coil 716 may represent, for example, a tri-axial coil configured to generate the EM field 701 at a particular strength (e.g., transmit power). The transmit power may be based at least in part on the electrical power provided by the amplifier 718. The amplifier 718 is configured to generate electrical power at a magnitude based on received control signaling of the device 706.

In response to detecting generated EM field 701, input device 702 reads EM data from EM field 701 using on-board EM receiver module 714. In general, EM receiver module 714 may be used as part of an electromagnetic sensing system, for example, to detect a 3D position and/or 3D orientation (i.e., gesture) associated with input device 702. In this example, the adapter 706 may comprise the remainder of the electromagnetic sensing system in the EM transmitter module 712. In some implementations, the EM receiver module 714 may be used as part of an electromagnetic sensing system to detect 3D positions and/or 3D orientations (i.e., gestures) associated with detected movements of the input device 702.

Adapter 706 also includes a CPU (i.e., processor 722) that can be in bi-directional communication with input device 702 via the CPU (i.e., processor 724) and/or the CPU (i.e., processor 726), as illustrated by communication links 729 and/or 731, respectively. EM data, recognized gestures, and/or other information are exchanged between device 202 and computing device 704, for example, via adapter 706 using any of processors 722, 724, and/or 726. For example, in some implementations, processor 726 identifies gestures based on EM data identified via module 714 and communicates the identified gestures to processor 724 directly or via processor 722 on adapter 706. The aptamer may then transmit the gesture (e.g., via USB) to the computing device 704. In some implementations, processor 726 communicates the EM data to processor 722 or processor 724, and processor 722 or processor 724 identifies gestures based on the EM data.

The communication links 729 and 731 can be wired communication links, wireless communication links (e.g., bluetooth, ZigBee, RF, etc.), and the like or combinations thereof. In other embodiments, the EM data may be sent to another device and/or processor (not shown), and the other device and/or processor may calculate the pose from the EM data. In some embodiments, the EM data may be stored locally within devices 702, 704, or 706, locally within system 700, and/or remotely from system 700.

The input device 702 also includes an EM receiver module 714 to generate EM data from the detected EM field. The EM receiver module 714 includes a receiver coil 728, an amplifier 730, and an ADC 732. In some embodiments, the receiver coil 728 is a tri-axial coil configured to detect analog electrical signals having a magnitude and/or phase indicative of a particular detected EM field. ADC 732 is generally configured to receive the generated analog signal and convert the analog signal into digital values indicative of the analog signal represented in EM field 701. The digital values generated by ADC 732 are EM data that may be used for gesture recognition of input device 702.

The input device 702 also includes an IMU sensor 734. IMU sensors 734 may provide additional pose information about device 702. One or more of processors 722, 724, or 726 (or another device processor) may use the additional pose information to supplement or augment the pose identified based on the EM data. For example, in some embodiments, processor 726 may use the additional pose information to identify potential errors in the pose determined based on the EM data and resolve the identified errors.

In some implementations, for example, when the device 702 is outside the range of the EM field generated by the transmitter module 712, the processor 726 may track the input device 702 using additional pose information determined with the IMU 734. Thus, the IMU 734 may act as a backup pose determiner to continue determining pose information and tracking data for the device 702. For example, if the device 702 is an input device 102 and the user operating the device 102 moves out of range of the computing device 704 (or the adapter 706), the system 700 may still determine pose information sufficiently well to track the mobile device 102 so that content (e.g., a cursor or 3D object) within the computing device 106 may be moved appropriately based on the tracked movement of the input device 102. In this example, the pose information may be 3-DoF pose data based on the detected 3D orientation of device 102/702.

In some implementations, the adapter 706 includes a processor 722 and a wireless interface communicatively coupled to the computing device 704 and communicatively coupled to the input device 702. The adapter 706 may be operable to collect gesture data associated with the input device 702 from the processor 726. In some implementations, the adapter 706 may be operable to determine pose information from the input device 702 when the device is not moving. The aptamer 706 can then use the processor 722 or 724 to convert the position and orientation data (i.e., pose data). In some implementations, the adapter 706 may instead convert, using the processor 722 or 724, position and/or orientation data retrieved from the IMU 734 of the device 702. In either case, the position and/or orientation data may be used to generate commands for execution by the computing device 704. For example, commands may be communicated from the adapter 706 to the computing device 704 through a USB interface generated by the connector 708/710.

Input devices 702 may also include a microphone 736 and a speaker 738. The input device 702 may include a microphone 736 to capture audio spoken by a user of the access device 702. For example, a microphone may be configured to receive a voice-based query to generate a communication from the input device 702 to the computing device 704. In some implementations, the captured audio is stored on a mouse for future playback or upload to a computer. In other embodiments, the audio is streamed to the computing device via a wireless connection for storage or processing.

In some implementations, the speaker 738 is configured to generate audio playback from the input device 702. The audio playback may include information responsive to a voice-based query received at the microphone 736. For example, if a user speaks a question or command into the microphone 736, the system 700 can stream the question (directly or via the adapter 706) to the computing device 704, which computing device 704 can respond with an answer that is played back to the user through the speaker 738.

Although the microphone 736 and speaker 738 are not shown in the systems 100, 200, 300, 600A, and 600B, both devices 736 and 738 may be included in a similar manner, as described above.

In some implementations, the adapter 706 can be attached to the input device 702 for storage when the computing device 704 is not plugged in. In some implementations, the input device 702 can include a female connector on the back surface (e.g., for charging) and the adapter 706 can include a male connector that can mate with the female connector of the input device for storage. In another embodiment, the input device 702 includes a compartment in which the adapter 706 may be stored.

FIG. 8 is a flow diagram programming an embodiment of a process 800 of tracking an input device to determine which data to use to generate a command for an associated computing device according to embodiments described herein. The method 800 is described with respect to an example implementation at the EM tracking system 200 of fig. 2, but it should be understood that the method may be implemented at EM tracking systems having other configurations.

At block 802, the process 800 may include detecting (at the air-mouse device 202) first data associated with the air-mouse device 202. The first data may include movement data, position data, orientation data, or any combination thereof. The first data may be detected using at least one EM-based sensor onboard the system 200. For example, the device 202 may use the EM receiver module 220 to detect position data and/or orientation data associated with the air mouse device 202. The EM receiver module may be associated with and/or capable of communicating with the base device 204, and the base device 204 may be installed on (or otherwise accessible to) the computing device 106 displaying content that the air-mouse device 202 may control. In some implementations, EM receiver module 220 may detect the first data to determine pose information of device 202. The first data and the second data may relate to position data, orientation data, or both (i.e., pose data).

At block 804, the process 800 may include detecting additional data (i.e., second data) associated with the air mouse device 202. The second data may be detected using at least one on-board non-EM based sensor. For example, the non-EM based sensor may include the IMU 230 and the second data may include accelerometer and gyroscope information about the device 202. As the user moves the device 202 in 3D space, the second data may be detected with respect to the device 202. In some implementations, the second data may be detected while the device 202 is not moving.

At some point, the user may move device 202 and may cause device 202 to become within or outside of an EM field associated with an infrastructure device (e.g., EM transmitter module 206 installed on device 204 in computing device 106). The range may be a predefined proximity range associated with the computing device. For example, a range may be defined based on a particular strength (e.g., transmit power) of EM transmitter module 206 used to generate an EM field used to transmit data between device 204 and device 202. Example ranges may include about one meter to about three meters. Other ranges are also possible.

At block 806, it may be determined whether device 202 is within range of computing device 106 (connected to transmitter module 206). In response to determining that the air-mouse device 202 is within the predefined proximity of the computing device 106 and/or the emitter module 206, the device 202 may generate gesture-related data using the first data (block 808), and the host computing device may translate the gesture-related data into commands to interact with content displayed on the computing device. For example, since device 202 is in range to use and transmit such data, device 202 may generate commands based at least in part on EM-based data.

In response to determining that the air-mouse device 202 is outside of the predefined proximity range of the computing device, the device 202 may generate a command using the second data (block 810) to interact with content displayed on the computing device. For example, the command may be generated using IMU data based on detecting that the EM-based element is unavailable.

Regardless of which data (first data or second data) is used, the air-mouse device 202 may trigger transmission of the gesture-related data to the host computing device at block 812, which may be used to manipulate content displayed in the computing device. The system 200 may continue to execute and determine the first data and the second data. Each time a command is generated, the system again begins determining whether the input device is within range of the computing device, as indicated by arrow 814. For example, device 202 may continue to detect proximity to device 106 to determine whether EM data can be collected to recognize gestures. Thus, process flow returns to block 806.

FIG. 9 illustrates an example computer device 900 and an example mobile computer device 950, which can be used with the techniques described herein. Features described with respect to the computer device 900 and/or the mobile computer device 950 may be included in the portable computing device 100 described above. Computing device 900 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit embodiments of the inventions described and/or claimed in this document.

Computing device 900 includes a processor 902, memory 904, a storage device 906, a high-speed interface 908 connecting to memory 904 and high-speed expansion ports 910, and a low speed interface 912 connecting to low speed bus 914 and storage device 906. Each of the components 902, 904, 906, 908, 910, and 912, are interconnected using various buses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 902 can process instructions for execution within the computing device 900, including instructions stored in the memory 904 or the storage device 906 to display graphical information for a GUI on an external input/output device, such as display 916 coupled to high speed interface 908. In other embodiments, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices 900 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a blade server bank, or a multi-processor system).

Memory 904 stores information within computing device 900. In one implementation, the memory 904 is a volatile memory unit or units. In another implementation, the memory 904 is a non-volatile memory unit or units. The memory 904 may also be another form of computer-readable medium, such as a magnetic or optical disk.

The storage device 906 can provide mass storage for the computing device 900. In one implementation, the storage device 906 may be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, a tape device, a flash memory or other similar solid state storage device, or an array of devices, including devices in a storage area network or other configurations. The computer program product may be tangibly embodied in an information carrier. The computer program product may also contain instructions that, when executed, perform one or more methods, such as the methods described above. The information carrier is a computer-or machine-readable medium, such as the memory 904, the storage device 906, or memory on processor 902.

The high speed controller 908 manages bandwidth-intensive operations for the computing device 900, while the low speed controller 912 manages lower bandwidth-intensive operations. Such allocation of functions is exemplary only. In one implementation, the high-speed controller 908 is coupled to memory 904, display 916 (e.g., through a graphics processor or accelerator), and to high-speed expansion ports 910, which may accept various expansion cards (not shown). In this embodiment, low-speed controller 912 is coupled to storage device 906 and low-speed expansion port 914. The low-speed expansion port, which may include various communication ports (e.g., USB, bluetooth, ethernet, wireless ethernet), may be coupled to one or more input/output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router, for example, through a network adapter.

Computing device 900 may be implemented in a number of different forms, as shown. For example, it may be implemented as a standard server 920, or multiple times in a group of such servers. It may also be implemented as part of a rack server system 924. It may also be implemented in a personal computer such as laptop computer 922. Alternatively, components from computing device 900 may be combined with other components in a mobile device (not shown), such as device 950. Each such device may contain one or more computing devices 900, 950, and an entire system may be made up of multiple computing devices 900, 950 communicating with each other.

Computing device 950 includes a processor 952, memory 964, an input/output device such as a display 954, a communication interface 966, and a transceiver 968, among other components. The device 950 may also be equipped with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 950, 952, 964, 954, 966, and 968, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

Processor 952 may execute instructions within computing device 950, including instructions stored in memory 964. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor may provide, for example, for coordination of the other components of the device 950 (such as control of user interfaces), applications run by device 950, and wireless communication by device 950.

The processor 952 may communicate with a user through the control interface 958 and a display interface 956 coupled to a display 954. The display 954 may be, for example, a TFT LCD (thin film transistor liquid Crystal display) or OLED (organic light emitting diode) display or other suitable display technology. The display interface 956 may comprise appropriate circuitry for driving the display 954 to present graphical and other information to a user. The control interface 958 may receive commands from a user and convert them for submission to the processor 952. In addition, an external interface 962 may be provided in communication with processor 952, so as to enable near area communication of device 950 with other devices. External interface 962 may provide, for example, for wired communication in some embodiments, or for wireless communication in other embodiments, and multiple interfaces may also be used.

The memory 964 stores information within the computing device 950. The memory 964 may be implemented as one or more of a computer-readable medium, a volatile memory unit, or one or more non-volatile memory units. Expansion memory 974 may also be provided and connected to device 950 through expansion interface 972, which may include, for example, a SIMM (Single in line memory Module) card interface. Such expansion memory 974 may provide additional storage space for device 950, or may also store applications or other information for device 950. Specifically, expansion memory 974 may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, expansion memory 974 may be provided as a security module for device 950, and may be programmed with instructions that permit secure use of device 950. In addition, secure applications may be provided via the SIMM card and additional information, such as placing identification information on the SIMM card in a non-intrusive manner.

The memory may include, for example, flash memory and/or NVRAM memory, as described below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as the methods described above. The information carrier is a computer-or machine-readable medium, such as the memory 964, expansion memory 974, or memory on processor 952, that may be received, for example, over transceiver 968 or external interface 962.

Device 950 may communicate wirelessly through communication interface 966, which may include digital signal processing circuitry if necessary. Communication interface 966 may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, through radio-frequency transceiver 968. Further, short-range communication may occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown). In addition, GPS (Global positioning System) receiver module 970 may provide other navigation-and location-related wireless data to device 950, which may be used as appropriate by applications running on device 950.

Device 950 may also communicate audibly using audio codec 960, which may receive voice information from a user and convert it to usable digital information. Audio codec 960 may similarly generate audible sound for a user, such as through a speaker, in a handset of device 950. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications running on device 950.

The computing device 950 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 980. It may also be implemented as part of a smart phone 982, personal digital assistant, or other similar mobile device.

Implementations of the various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Embodiments may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device or in a propagated signal, for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. A computer program, such as the one described above, can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

Method steps can be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method steps also may be performed by, and an apparatus may be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. Elements of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

Other kinds of devices may also be used to provide for interaction with a user, for example, feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input.

Embodiments may be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation), or any combination of such back-end, middleware, or front-end components. The components may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a Local Area Network (LAN) and a Wide Area Network (WAN), e.g., the internet.

A computing device according to example embodiments described herein may be implemented using any suitable combination of hardware and/or software configured for user interface with a user device, including a User Interface (UI) device, user terminal, client device, or customization device. The computing device may be implemented as a portable computing device, such as, for example, a laptop computer. The computing device may be implemented as some other type of portable computing device suitable for interfacing with a user, such as a PDA, notebook computer, or tablet computer. The computing device may be implemented as some other type of computing device, such as a PC, adapted to interface with a user. The computing device may be implemented as a portable communication device (e.g., a mobile phone, smart phone, wireless cellular phone, etc.) adapted to interface with a user and to communicate wirelessly over a network including a mobile communication network.

The computer system (e.g., computing device) may be configured to wirelessly communicate with the network server over the network via the communication link established with the network server using any known wireless communication techniques and protocols, including Radio Frequency (RF), microwave frequency (MWF), and/or infrared frequency (IRF) wireless communication techniques and protocols suitable for communicating over the network.

According to various aspects of the disclosure, implementations of the various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Embodiments may be implemented as a computer program product (e.g., a computer program tangibly embodied in an information carrier, a machine-readable storage device, a computer-readable medium, a tangible computer-readable medium), for processing by, or to control the operation of, data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). In some implementations, a tangible computer-readable storage medium may be configured to store instructions that, when executed, cause a processor to perform a process. A computer program, such as the computer program(s) described above, can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be processed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments. Example embodiments may, however, be embodied in many alternate forms and should not be construed as limited to only the embodiments set forth herein.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of embodiments. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises", "comprising", "includes" and/or "including", when used in this specification, specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof.

It will be understood that when an element is referred to as being "coupled," "connected," or "responsive" to or on another element, it can be directly coupled, connected, or responsive, or directly on the other element, or intervening elements may also be present. In contrast, when an element is referred to as being "directly coupled," "directly connected," or "directly responsive" to or "directly on" another element, there are no intervening elements present. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

Spatially relative terms, such as "below," "under," "above," "upper," and the like, may be used herein for ease of description to describe one element or feature's relationship to another element or feature as illustrated. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "below" or "beneath" other elements or features would then be oriented "above" the other elements or features. Thus, the term "below … …" can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.

Exemplary embodiments of the inventive concept are described herein with reference to cross-sectional illustrations that are schematic illustrations of idealized embodiments (and intermediate structures) of the exemplary embodiments. As such, variations from the shapes of the illustrations as a result, for example, of manufacturing techniques and/or tolerances, are to be expected. Accordingly, example embodiments of the inventive concept should not be construed as limited to the particular shapes of regions illustrated herein but are to include deviations in shapes that result, for example, from manufacturing. Thus, the regions illustrated in the figures are schematic in nature and their shapes are not intended to illustrate the actual shape of a region of a device and are not intended to limit the scope of example embodiments.

It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Thus, a "first" element may be termed a "second" element without departing from the teachings of the present embodiments.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this inventive concept belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and/or the present specification and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

While certain features of the described embodiments have been illustrated as described herein, many modifications, substitutions, changes, and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the scope of the embodiments. It is to be understood that they have been presented by way of example only, and not limitation, and various changes in form and details may be made. Any portions of the devices and/or methods described herein may be combined in any combination, except mutually exclusive combinations. The embodiments described herein may include various combinations and/or subcombinations of the functions, features and/or properties of the different embodiments described.