阅读说明：本技术 生成双耳音频的头戴式装置 (Head mounted device for generating binaural audio ) 是由 M·阿斯富 D·胡 P·T·M·西蒙斯 V·L·W·利梅特凯 于 2020-03-12 设计创作，主要内容包括：用于生成双耳音频内容的头戴式装置包括耦接到第一麦克风外壳的第一杆,该第一麦克风外壳分别包围生成声学信号的第一前麦克风和第一后麦克风。第一麦克风外壳包括面向下的第一前端口和面向后的第一后端口。装置包括耦接到第二麦克风外壳的第二杆,该第二麦克风外壳分别包围生成声学信号的第二前麦克风和第二后麦克风。第二麦克风外壳包括面向下的第二前端口和面向后的第二后端口。装置包括双耳音频处理器,该双耳音频处理器包括波束成形器和存储设备。波束成形器基于来自第一前麦克风和第一后麦克风的声学信号生成第一波束成形器信号,并基于来自第二前麦克风和第二后麦克风的声学信号生成第二波束成形器信号。存储设备将第一和第二波束成形器信号存储为双通道文件。(A headset for generating binaural audio content includes a first stem coupled to a first microphone housing enclosing a first front microphone and a first rear microphone, respectively, that generate acoustic signals. The first microphone housing includes a first front port facing downward and a first rear port facing rearward. The apparatus includes a second pole coupled to a second microphone housing that respectively encloses a second front microphone and a second rear microphone that generate acoustic signals. The second microphone housing includes a second front port facing downward and a second rear port facing rearward. An apparatus includes a binaural audio processor including a beamformer and a memory device. The beamformer generates a first beamformer signal based on acoustic signals from the first front microphone and the first rear microphone and generates a second beamformer signal based on acoustic signals from the second front microphone and the second rear microphone. The storage device stores the first and second beamformer signals as a two-channel file.)

1. A head-mounted device, comprising:

a frame;

a first pole coupled to a first side of the frame and to a first microphone housing enclosing a first front microphone and a first rear microphone, respectively, that generate acoustic signals,

the first microphone housing includes a first front port facing downward and a first rear port facing rearward,

a second pole coupled to a second side of the frame and to a second microphone housing enclosing a second front microphone and a second rear microphone, respectively, that generate acoustic signals,

the second microphone housing comprises a second front port facing downward and a second rear port facing rearward; and

a binaural audio processor comprising

A beam shaper of

Receiving acoustic signals from the first front microphone, the first rear microphone, the second front microphone, and the second rear microphone, and

generating

A first beamformer signal based on acoustic signals from the first front microphone and the first rear microphone, an

A second beamformer signal based on acoustic signals from the second front microphone and the second rear microphone, an

A storage device to store the first beamformer signal and the second beamformer signal as a two-channel file.

2. The headset of claim 1, wherein the wearable device is a pair of eyeglasses, wherein the first side of the frame is opposite the second side of the frame.

3. The headset of claim 2, wherein the first front port and the second front port face downward toward the user's foot and the first rear port and the second rear port face toward the back of the user's head when the user is wearing the pair of eyeglasses.

4. The headset of claim 3, wherein the frame carries a pair of lenses and is coupled to a camera, wherein a camera lens of the camera faces forward.

5. The headset of claim 1, wherein the beamformer is a fixed beamformer.

6. The headset of claim 4, wherein the fixed beamformer comprises a sub-cardioid or cardioid fixed beam pattern.

7. The headset of claim 1, wherein the first front microphone and the first rear microphone form a first order differential microphone array and the second front microphone and the second rear microphone form a first order differential microphone array.

8. The headset of claim 1, wherein the storage device is a flash memory device.

9. The headset of claim 1, wherein the binaural audio processor is a silicon on chip (SoC).

10. The headset of claim 1, further comprising:

a first audio codec that decodes acoustic signals from the first front microphone and the first rear microphone to generate a first decoded acoustic signal;

a second audio codec that decodes acoustic signals from the second front microphone and the second rear microphone to generate a second decoded acoustic signal.

11. The headset of claim 10, further comprising:

a Time Division Multiplexer (TDM) that processes the first decoded acoustic signal and the second decoded acoustic signal and generates a TDM processed signal.

12. The headset of claim 11, wherein the acoustic signals generated by the first and second front microphones and the first and second rear microphones are Pulse Density Modulated (PDM) signals.

13. The headset of claim 12, wherein the first decoded acoustic signal and the second decoded acoustic signal are Pulse Code Modulation (PCM) signals.

14. The headset of claim 13, wherein the binaural audio processor further comprises:

a noise suppressor that suppresses noise from the first and second beamformer signals and generates first and second noise suppressed signals.

15. The headset of claim 14, wherein the binaural audio processor further comprises:

a speech enhancer that enhances speech from the first noise-suppressed signal and the second noise-suppressed signal to generate a first clean signal and a second clean signal, wherein the storage device stores the first clean signal and the second clean signal.

16. A pair of eyeglasses, comprising:

a frame carrying a pair of lenses;

a camera coupled to the frame, wherein a camera lens of the camera faces in a forward direction;

a first pole coupled to a first side of the frame and to a first microphone housing enclosing a first front microphone and a first rear microphone, respectively, that generate acoustic signals,

the first microphone housing comprises a first front port facing in a downward direction and a first rear port facing in a rearward direction,

a second pole coupled to a second side of the frame and to a second microphone housing enclosing a second front microphone and a second rear microphone, respectively, that generate acoustic signals,

the second microphone housing includes a second front port facing the downward direction and a second rear port facing the rearward direction; and

a binaural audio processor comprising

Beam shaper of generation

A first beamformer signal based on acoustic signals from the first front microphone and the first rear microphone, an

A second beamformer signal based on acoustic signals from the second front microphone and the second rear microphone, an

A storage for storing the first beamformer signal and the second beamformer signal as a two-channel file.

17. A pair of spectacles according to claim 16, wherein the beamformer is a fixed beamformer, wherein the fixed beamformer comprises a sub-cardioid or cardioid fixed beam pattern.

18. A pair of spectacles according to claim 16, further comprising:

a first audio codec that decodes acoustic signals from the first front microphone and the first rear microphone to generate a first decoded acoustic signal;

a second audio codec that decodes acoustic signals from the second front microphone and the second rear microphone to generate a second decoded acoustic signal; and

a Time Division Multiplexer (TDM) that processes the first decoded acoustic signal and the second decoded acoustic signal and generates a TDM processed signal.

19. A pair of glasses according to claim 16, wherein the binaural audio processor further comprises:

a noise suppressor that suppresses noise from the first and second beamformer signals and generates first and second noise-suppressed signals; and

a speech enhancer that enhances speech from the first noise-suppressed signal and the second noise-suppressed signal to generate a first clean signal and a second clean signal, wherein the storage device stores the first clean signal and the second clean signal.

20. A method of generating binaural audio content using a head-mounted device, comprising:

generating acoustic signals by a first front microphone and a first rear microphone, respectively, wherein the first front microphone and the first rear microphone are enclosed in a first microphone housing that is coupled to a first stem of the headset;

generating acoustic signals by a second front microphone and a second rear microphone, respectively, wherein the second front microphone and the second rear microphone are enclosed in a second microphone housing coupled to a second pole of the headset, wherein the first pole and the second pole are coupled to opposite sides of a frame of the headset;

decoding, by a first audio codec, acoustic signals from the first front microphone and the first rear microphone to generate a first decoded acoustic signal, wherein the first decoded acoustic signal is a Pulse Code Modulation (PCM) signal based on the acoustic signals from the first front microphone and the first rear microphone;

decoding, by a second audio codec, the acoustic signals from the second front microphone and the second rear microphone to generate a second decoded acoustic signal, wherein the second decoded acoustic signal is a PCM signal based on the acoustic signals from the second front microphone and the second rear microphone;

processing the first decoded acoustic signal and the second decoded acoustic signal by a Time Division Multiplexer (TDM) to generate a TDM processed signal, wherein the TDM processed signal includes the first decoded acoustic signal and the second decoded acoustic signal;

beamforming, by a beamformer, the TDM processed signals, wherein beamforming the TDM processed signals comprises

Beamforming the first decoded signal to generate a first beamformer signal, an

Beamforming the second decoded signal to generate a second beamformer signal; and

storing, by a storage device, the first beamformer signal and the second beamformer signal as a two-channel file.

Background

Currently, many consumer electronic devices are adapted to capture audio and/or visual content. For example, a user may use a built-in camera on a mobile device to quickly capture events or moments that occur in the user's life.

However, the time required to retrieve the mobile device may still be too long to capture some events or moments that are more or less elapsed. Users may also feel as if they have to remove themselves from the presence and experience event or moment to retrieve their mobile device. Furthermore, when played back, the audio portion of content recorded using the mobile device may not adequately reproduce the sound perceived by the user.

Drawings

In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views. Like numerals having different letter suffixes may represent different instances of similar components. Some embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which:

fig. 1 shows a perspective view of a head mounted device for generating binaural audio according to an example embodiment.

Fig. 2 shows a bottom view of the headset from fig. 1 according to an example embodiment.

Fig. 3A-3B show details of a portion of one microphone housing from the headset of fig. 1, according to an example embodiment.

Fig. 4 is an exemplary flow diagram of a process for generating binaural audio using the headset from fig. 1 in accordance with various aspects of the disclosure.

Fig. 5 shows a block diagram of a system for generating binaural audio for inclusion in the headset from fig. 1, according to an example embodiment.

Fig. 6 shows a block diagram of a binaural audio processor comprised in the system in fig. 5 according to an example embodiment.

FIG. 7 is a block diagram illustrating a representative software architecture that may be used in connection with the various hardware architectures described herein.

Fig. 8 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments.

Fig. 9 is a high-level functional block diagram of an example headset communicatively coupling a mobile device and a server system via various networks.

Detailed Description

The following description includes systems, methods, techniques, instruction sequences, and computer machine program products that embody illustrative embodiments of the present disclosure. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments of the inventive subject matter. It will be apparent, however, to one skilled in the art that embodiments of the present subject matter may be practiced without these specific details. In general, well-known instruction instances, protocols, structures, and techniques need not be shown in detail.

To improve audio recordings captured by current electronic mobile devices, some embodiments of the present disclosure relate to a headset 100 that can capture audio content that, when played back, is an imitation of the sound perceived by a user of the headset 100. In particular, the head mounted device 100 may record audio using a microphone arranged to create a 3-dimensional (3D) sound sensation for a listener as it would exist when recording audio. This is called binaural audio. Playback of the captured audio content will have the effect of binaural audio with stereo separation and spectral content mimicking the human ear and head response. The design of the head mounted device 100 uses, among other things, the diffraction pattern of a human head, the placement of microphones on both sides of the head, and beam forming techniques.

Fig. 1 shows a perspective view of a head mounted device 100 for generating binaural audio according to an example embodiment. Fig. 2 shows a bottom view of the headset 100 from fig. 1 according to an example embodiment. In fig. 1 and 2, the head-mounted device 100 is a pair of eyeglasses. In some embodiments, the head-mounted device 100 may be sunglasses or goggles. Some embodiments may include one or more wearable devices, such as a pendant with an integrated camera integrated with, in communication with, or coupled to the head mounted apparatus 100 or client device. Any desired wearable device may be used in conjunction with embodiments of the present disclosure, such as a watch, an earphone, a wristband, an earbud, a garment (such as a hat or jacket with integrated electronics), a clip-on electronic device, or any other wearable device. It should be understood that, although not shown, one or more portions of the system included in the headset may be included in a client device (e.g., machine 800 in fig. 8) that may be used in conjunction with the headset 100. For example, one or more elements as shown in fig. 5 and 6 may be included in the headset 100 and/or the client device.

As used herein, the term "client device" may refer to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. The client device may be, but is not limited to, a mobile phone, desktop computer, laptop computer, Portable Digital Assistant (PDA), smartphone, tablet, ultrabook, netbook, laptop computer, multiprocessor system, microprocessor-based or programmable consumer electronics, game console, set-top box, or any other communication device available to a user for accessing a network.

In fig. 1 and 2, the head-mounted device 100 is a pair of glasses comprising a frame 103, the frame 103 comprising an eye-line (or a frame) coupled to two rods (or temples) via a hinge and an endpiece, respectively. The eye line of the frame 103 carries or holds a pair of lenses 104_1, 104_ 2. The frame 103 includes a first (e.g., right) side coupled to the first pole and a second (e.g., left) side coupled to the second pole. The first side is opposite to the second side of the frame 103.

The apparatus 100 further comprises a camera module comprising camera lenses 102_1, 102_2 and at least one image sensor. The camera lens may be a see-through camera lens or a non-see-through camera lens. The non-see-through camera lens may be, for example, a fisheye lens, a wide-angle lens, an omnidirectional lens, or the like. The image sensor captures digital video through the camera lens. The image may also be a still image frame or a video comprising a plurality of still image frames. The camera module may be coupled to the frame 103. As shown in fig. 1 and 2, the frame 103 is coupled to the camera lenses 102_1, 102_2 such that the camera lenses face forward. The camera lenses 102_1, 102_2 may be perpendicular to the lenses 104_1, 104_ 2. The camera module may include dual front cameras separated by the width of the frame 103 or the width of the head of the user of the device 100.

In fig. 1 and 2, two rods (or temples) are coupled to the microphone housings 101_1, 101_2, respectively. The first and second rods are coupled to opposite sides of the frame 103 of the head mounted device 100. The first rod is coupled to the first microphone housing 101_1 and the second rod is coupled to the second microphone housing 101_ 2. The microphone housings 101_1, 101_2 may be coupled to a pole between the location of the frame 103 and the temple. The microphone housings 101_1, 101_2 may be located on either side of the user's temple when the user is wearing the device 100.

As shown in fig. 2, the microphone housings 101_1, 101_2 enclose a plurality of microphones 110_1 to 110_ N (N > 1). The microphones 110_1 to 110_ N are air interface sound pickup devices that convert sound into electrical signals. More specifically, the microphones 110_1 to 110_ N are transducers that convert sound pressure into an electrical signal (e.g., an acoustic signal). The microphones 110_1 to 110_ N may be digital or analog micro-electro-mechanical systems (MEMS) microphones. The acoustic signals generated by the microphones 110_1 to 110_ N may be Pulse Density Modulation (PDM) signals.

In fig. 2, the first microphone case 101_1 surrounds the microphones 110_3 and 110_4, and the second microphone case 101_2 surrounds the microphones 110_1 and 110_ 2. In the first microphone case 101_1, the first front microphone 110_3 and the first rear microphone 110_4 are spaced apart by a predetermined distance d₁And forms a first order differential microphone array. In the second microphone case 101_2, the second front microphone 110_1 and the second rear microphone 110_2 are also spaced apart by a predetermined distance d₂And forms a first order differential microphone array. Predetermined distance d₁And d₂May be the same distance or different distances. The predetermined distance d can be set based on the nyquist frequency₁And d₂. Content above the nyquist frequency of the beamformer, particularly speech, is not recoverable. The nyquist frequency is determined by the following equation:

in this equation, c is the speed of sound, and d is the spacing between the microphones. Using this equation, in one embodiment, the distance d is predetermined₁And d₂Any value of d that results in a frequency above 6kHz, which is the cutoff frequency for wideband speech, can be set.

Although in fig. 1, the system 100 includes four microphones 110_1 to 110_4, the number of microphones may vary. In some embodiments, the microphone housings 101_1, 101_2 may include at least two microphones and may form a microphone array. Each of the microphone housings 101_1, 101_2 may further include a battery.

The user perceives the audio naturally through two ears separated by the head, so that the user can distinguish the direction from which the sound originates. Thus, by placing the microphone housings 101_1, 101_2 on the stem of the headset 100, the headset 100 may enable capturing sounds perceived by a user wearing the headset 100.

Referring to fig. 2, each of the microphone housings 101_1, 101_2 includes a front port and a rear port. The front port of the first microphone housing 101_1 is coupled to a microphone 110_3 (e.g., a first front microphone) and the rear port of the first microphone housing 101_1 is coupled to a microphone 110_4 (e.g., a first rear microphone). The front port of the second microphone housing 101_2 is coupled to the microphone 110_1 (e.g., a second front microphone) and the rear port of the second microphone housing 101_2 is coupled to the microphone 110_2 (e.g., a second rear microphone). In one embodiment, the microphones 101_ 1-101 _4 may be further moved toward a temple cuff on a shaft of the device 100 (e.g., the back of the device 100) to emphasize the binaural effect captured by the microphones.

Fig. 3A-3B show details of a portion of one microphone housing from the headset of fig. 1, according to an example embodiment. In particular, fig. 3A shows details of microphone 110_1 (e.g., the second front microphone) and its associated front port. Although fig. 3A shows details of the microphone 110_1 coupled to the front port in the second microphone housing 101_2, it should be understood that details of the microphone 110_3 (e.g., the first front microphone) coupled to the front port in the first microphone housing 101_1 are similar to those in fig. 3A.

Fig. 3A is a cross-sectional view of the front microphone 110_1 and the acoustic path 112. As shown, the acoustic path 112 passes through the space between the block 114 (e.g., the second microphone housing 101_2) and the housing 113 of the second rod. The housing 113 of the rod may be made of metal. The front microphone 110_1 and the front port point (or face) downward. For example, when the user is standing and wearing the device 100, the front port in fig. 3A opens in a direction toward the user's feet on the ground. This design allows the length and acoustic mass of the front acoustic port to be minimized while ensuring that the front microphone 110_1 is protected in a small area (pocket) that may reduce noise effects (e.g., wind noise turbulence, etc.).

Fig. 3B shows a detail of another portion of the headset from fig. 1, according to an example embodiment. In particular, fig. 3B shows details of the microphone 110_2 (e.g., the second rear microphone) and the rear port associated therewith. Although fig. 3B shows details of the microphone 110_2 coupled to the rear port in the second microphone housing 101_2, it is understood that details of the microphone 110_4 (e.g., the first rear microphone) coupled to the rear port in the first microphone housing 101_1 are similar to those in fig. 3B.

Fig. 3B is a cross-sectional view of the rear microphone 110_2 and the acoustic path 112. As shown, the acoustic path 112 passes through the space between the block 114 (e.g., the second microphone housing 101_2) and the housing 113 of the second rod. The housing 113 of the rod in fig. 3B may also be made of metal. The rear microphone 110_2 and the rear port point (or face) backward. For example, the rear port in fig. 3B opens in a direction toward the back of the user's head or the rear of the device 100 (e.g., toward the end of the temple) when the device 100 is worn by the user. Since the port is rearward in this embodiment, direct wind contact with the rear port is avoided when the user is wearing the device 100. This design further allows the length and acoustic mass of the rear acoustic port to be minimized while ensuring that the rear microphone 110_2 is protected in a small area that can reduce noise effects (e.g., wind noise turbulence, etc.). In one embodiment, a mechanical filter may be applied to the rear microphone 110_2 to further improve noise immunity.

As shown in fig. 3A-3B, microphones 110_1 through 110_4 may be part of a microphone assembly stack-up that includes a flexible circuit board and a Pressure Sensitive Adhesive (PSA) stack-up that includes a waterproof membrane sandwiched between PSA layers. The waterproof film may protect the microphones 110_1 to 110_4 from water and air leakage. The PSA stack is then coupled to the microphone assembly housing. In some embodiments, the microphone assembly housing is the block 114 or housing 113 of the second rod.

Fig. 4 is an exemplary flow diagram of a process for generating binaural audio using the headset 100 from fig. 1 in accordance with various aspects of the disclosure. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process terminates when its operations are completed. A process may correspond to a method, a program, etc. The steps of a method may be performed in whole or in part, may be performed in combination with some or all of the steps of other methods, and may be performed by any number of different systems, such as the systems described in fig. 1 and/or fig. 8. The process 400 may also be performed by a processor included in the headset 100 in fig. 1 or by a processor included in the client device 800 of fig. 8.

The process 400 begins at operation 401, where the microphones 110_1 to 110_4 generate acoustic signals. The microphones 110_1 to 110_4 may be MEMS microphones that convert sound pressure into an electrical signal (e.g., an acoustic signal). The first front microphone 110_3 and the first rear microphone 110_4 are enclosed in a first microphone 101_1 housing coupled to a first stem of the headset 100. The first front microphone 110_3 and the first rear microphone 110_4 form a first order differential microphone array. The second front microphone 110_1 and the second rear microphone 110_2 are enclosed in a second microphone housing 101_2 coupled to a second stem of the headset 100. The second front microphone 110_1 and the second rear microphone 110_2 form a first order differential microphone array. The first and second rods are coupled to opposite sides of the frame 103 of the head mounted device 100. The acoustic signal may be a Pulse Density Modulation (PDM) signal.

At operation 402, the audio codecs 501_1, 501_2 decode the acoustic signals from the microphones 110_1 to 110_ 4. The audio codec 501_2 decodes the acoustic signals from the first front microphone 110_ and the first rear microphone 110_4 to generate a first decoded acoustic signal, and the audio codec 501_1 decodes the second front microphone 110_1 and the second rear microphone 110_2 to generate a second decoded acoustic signal. The first and second decoded acoustic signals are Pulse Code Modulation (PCM) signals. The first decoded acoustic signal is a PCM signal based on the acoustic signals from the first front microphone 110_3 and the first rear microphone 110_ 4. The second decoded acoustic signal is a PCM signal based on the acoustic signals from the second front microphone 110_1 and the second rear microphone 110_ 2.

At operation 403, the TDM 502 processes the decoded acoustic signals from the audio codecs 501_1, 501_ 2. The TDM 502 processes the first and second decoded acoustic signals by time-division multiplexing the first and second decoded acoustic signals. The TDM-processed signal includes a first decoded acoustic signal and a second decoded acoustic signal.

At operation 404, the beamformer 601 beamformes the TDM processed signals. As described further below, the beamformer 601 may be a fixed beamformer including a fixed beam pattern that is sub-cardioid or cardioid. The beamformer 601 beamformes the first decoded signal to generate a first beamformer signal and beamformes the second decoded signal to generate a second beamformer signal.

At operation 405, the storage device 604 stores the beamformer signal as a two-channel file. The two-channel file may be a two-channel PCM file or a two-channel Advanced Audio Coding (AAC)/PCM file. The storage device 604 may be a flash memory device.

In one embodiment, the noise suppressor 602 suppresses noise from the first and second beamformer signals and generates first and second noise-suppressed signals. The speech enhancer 603 may enhance speech from the first noise-suppressed signal and the second noise-suppressed signal to generate a first clean signal and a second clean signal, respectively. In this embodiment, storage device 604 stores the first and second clean signals as a two-channel PCM file.

Fig. 5 shows a block diagram of a system 500 for generating binaural audio for inclusion in the headset 100 from fig. 1 according to an example embodiment. In some embodiments, one or more portions of the system 500 may be included in the headset 100 or may be included in a client device (e.g., the machine 800 in fig. 8) that may be used in conjunction with the headset 100.

The system 500 includes microphones 110_1 to 110_ N, audio codecs 501_1, 501_2, a Time Division Multiplexer (TDM)502, and a binary audio processor 503. The first front microphone 110_3 and the first rear microphone 110_4 enclosed in the first microphone case 101_1 form a first-order differential microphone array. Similarly, the second front microphone 110_1 and the second rear microphone 110_2 enclosed in the second microphone case 101_2 form another first-order differential microphone array. The microphones 110_1 to 110_4 may be analog or digital MEMS microphones. The acoustic signals generated by the microphones 110_1 to 110_4 may be Pulse Density Modulation (PDM) signals.

The audio codec 501_1 decodes the acoustic signals from the first front microphone 110_3 and the first rear microphone 110_4 to generate a first decoded acoustic signal. The audio codec 501_2 decodes the acoustic signals from the second front microphone 110_1 and the second rear microphone 110_2 to generate a second decoded acoustic signal. The first and second decoded acoustic signals may be Pulse Code Modulation (PCM) signals. In one embodiment, the audio codecs 501_1, 501_2 decode the acoustic signal as a PDM signal from a single bit PDM format to a multi-bit Pulse Code Modulation (PCM) format. The audio codecs 501_1, 501_2 may comprise a PDM input with a filter to convert the PDM signal into a PCM format. In one embodiment, the audio codecs 501_1, 501_2 use a microcontroller with a synchronous serial interface to capture the PDM data streams from the microphones 110_1 to 110_4 and use filters implemented in software to convert the PDM data streams to PCM format.

The PCM signal may be interpreted by an interface of the binaural audio processor 503. In some embodiments, the binaural audio processor 503 is a silicon on chip (SoC). The SoC may include an interface, such as an I2S interface, to receive and interpret PCM signals.

In one embodiment, the interface of the binaural audio processor 503 can only process one packet (e.g., 2 channels of audio), and a Time Division Multiplexer (TDM)502 in the system 500 receives and processes the first and second decoded acoustic signals (e.g., PCM signals) to generate TDM processed signals. The TDM 502 time-division multiplexes the first and second decoded acoustic signals to generate a TDM processed signal. In one embodiment, the system 500 further oversamples the TDM processed signals to allow for the input of four microphone signals. The system 500 may also include a switch for creating additional slots to allow microphone signals.

In one embodiment, the microphones 110_1 to 110_4 are digital MEMS microphones. The acoustic signals generated by the digital MEMS microphones are relatively immune to noise, but signal integrity remains an issue due to distortions caused by parasitic capacitances, resistances and inductances between the microphone 110_1 to 110_4 outputs and the SoC (such as the binaural audio processor 503). The impedance mismatch also produces reflections that distort the signal in applications where the distance between the digital microphones 110_1 to 110_4 and the SoC is long. In one embodiment, microphones 110_1 through 110_4 are attached to a flexible circuit designed to maximize signal integrity and also minimize trace length between elements. In this embodiment, the flexible circuit is enclosed in the microphone housing 101_1, 101_ 2.

Fig. 6 shows a block diagram of a binaural audio processor 503 comprised in the system 500 in fig. 5 according to an example embodiment. The binaural audio processor 503 comprises a beamformer 601, a noise suppressor 602, a speech enhancer 603 and a storage 604.

In one embodiment, the binaural audio processor 503 comprises an interface that receives the TDM processed signal. As described above, the TDM-processed signals are generated from the acoustic signals from the first front microphone 110_3, the first rear microphone 110_4, the second front microphone 110_1, and the second rear microphone 110_ 2. In one embodiment, the microphones 110_1 to 110_4 are inherently omni-directional digital MEMS microphones.

The beamformer with directional steering characteristics 601 is a differential beamformer which allows a flat frequency response except for the nyquist frequency. The beamformer 601 uses the transfer function of a first order differential microphone array. For two microphones, the transfer function of the first order differential microphone array is as follows:

in these equations above, theta θ is the angle and beta β is 180 degrees, which reduces to E ═ a + B cos θ for fixed frequency (or frequency independent) beams (e.g., beamformer signals). E in the simplified equation is the fixed frequency output of the beamformer. In one embodiment, the beamformer 601 is a fixed beamformer comprising a fixed beam pattern that is a sub-cardioid with a and B coefficients of 0.25 and 0.75, respectively. In one embodiment, the beamformer 601 is a fixed beamformer comprising a fixed beam pattern of cardioids with a and B coefficients of 0.5 and 0.5, respectively.

In one embodiment, the beamformer 601 receives acoustic signals from the first front microphone 110_3, the first rear microphone 110_4, the second front microphone 110_1 and the second rear microphone 110_ 2. In one embodiment, the beamformer 601 receives TDM processed signals. The beamformer 601 generates a first beamformer signal based on the acoustic signals from the first front microphone 110_3 and the first rear microphone 110_4, and generates a second beamformer signal based on the acoustic signals from the second front microphone 110_1 and the second rear microphone 110_ 2. The storage device 604 may store the first and second beamformer signals as a two-channel file.

The noise suppressor 602 suppresses noise from the first beamformer signal and the second beamformer signal. Noise suppressor 602 is a two-channel noise suppressor and generates a first noise-suppressed signal and a second noise-suppressed signal. In one embodiment, noise suppressor 602 may implement a noise suppression algorithm.

The speech enhancer 603 enhances speech from the first noise-suppressed signal and the second noise-suppressed signal to generate a first clean signal and a second clean signal. In one embodiment, speech enhancer 603 may implement model-based speech enhancement. The speech enhancer 603 may search the first and second noise-suppressed signals for a plurality of speech features. When the speech enhancer 603 identifies a portion of the first and second noise-suppressed signals that matches at least one of the speech features, the speech enhancer 603 enhances or emphasizes the identified portion. In one embodiment, the speech enhancer 603 may implement a speech enhancement algorithm.

The storage device 604 stores the first and second clean signals from the speech enhancer 603 as a two-channel file. The dual channel file may be a dual channel PCM file (or dual channel AAC/PCM file) representing the left and right channels. The storage device 604 may be a flash memory device.

FIG. 7 is a block diagram illustrating an exemplary software architecture 706 that may be used in conjunction with the various hardware architectures described herein. FIG. 7 is a non-limiting example of a software architecture, and it is understood that many other architectures can be implemented to facilitate the functionality described herein. Software architecture 706 may execute on hardware, such as machine 8 of fig. 8, including a processor 804, memory 814, and I/O components 818. A representative hardware layer 752 is shown and may represent, for example, the machine 800 of fig. 8. The representative hardware layer 752 includes a processing unit 754 having associated executable instructions 704. Executable instructions 704 represent executable instructions of software architecture 706, including implementations of the methods, components, etc. described herein. The hardware layer 752 also includes a memory or storage module memory/storage 756, which also has executable instructions 704. The hardware layer 752 may also include other hardware 758.

As used herein, the term "component" may refer to a device, physical entity, or logic having boundaries defined by function or subroutine calls, branch points, Application Program Interfaces (APIs), or other techniques for providing partitioning or modularization of particular processing or control functions. The components may be combined with other components via their interfaces to perform machine processes. A component may be a packaged-function hardware unit designed for use with other components, and typically is part of a program that performs the specified function of the associated function.

The components may constitute software components (e.g., code embodied on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit that is capable of performing certain operations and may be configured or arranged in some physical manner. In various example embodiments, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a set of processors) may be configured by software (e.g., an application or application portion) as a hardware component that operates to perform certain operations as described herein. The hardware components may also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may comprise dedicated circuitry or logic that is permanently configured to perform certain operations.

The hardware component may be a special purpose processor, such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC). The hardware components may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, the hardware components may include software executed by a general purpose processor or other programmable processor. When configured by the software, the hardware components become a specific machine (or specific components of a machine) that is uniquely customized to perform the configured functions and is no longer a general-purpose processor. It should be appreciated that the decision to mechanically implement a hardware component in a dedicated and permanently configured circuit or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations.

A processor may be or include any circuit or virtual circuit (a physical circuit emulated by logic executing on an actual processor) that manipulates data values based on control signals (e.g., "commands," "opcodes," "machine code," etc.) and generates corresponding output signals that are applied to an operating machine. For example, the processor may be a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Radio Frequency Integrated Circuit (RFIC), or any combination thereof. The processor may further be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously.

Thus, the phrase "hardware component" (or "hardware-implemented component") should be understood to encompass a tangible entity, i.e., an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. Considering embodiments in which the hardware components are temporarily configured (e.g., programmed), each of the hardware components need not be configured or instantiated at any one time. For example, where the hardware components include a general-purpose processor configured by software as a special-purpose processor, the general-purpose processor may be configured at different times as corresponding different special-purpose processors (e.g., including different hardware components). Software configures a particular processor or processors accordingly, e.g., to constitute a particular hardware component at one time and to constitute a different hardware component at a different time. A hardware component may provide information to and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled. Where multiple hardware components are present at the same time, communication may be achieved by signal transmission (e.g., over appropriate circuits and buses) between two or more of the hardware components. In embodiments in which multiple hardware components are configured or instantiated at different times, communication between such hardware components may be achieved, for example, through storage and retrieval of information in memory structures accessible to the multiple hardware components.

For example, one hardware component may perform an operation and store the output of the operation in a storage device communicatively coupled thereto. Another hardware component may then later access the storage device to retrieve and process the stored output. The hardware components may also initiate communication with input or output devices and may operate on resources (e.g., sets of information). Various operations of the example methods described herein may be performed, at least in part, by one or more processors that are temporarily configured (e.g., via software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, the processor may constitute a processor-implemented component for performing one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component that is implemented using one or more processors. Similarly, the methods described herein may be implemented at least in part by a processor, where a particular processor or processors are examples of hardware. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented components.

Further, the one or more processors may also operate to support performance of related operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some of the operations may be performed by a set of computers (as an example of machines including processors) that are accessible via a network (e.g., the internet) and via one or more appropriate interfaces (e.g., Application Program Interfaces (APIs)). The performance of some of the operations may be distributed among the processors, not only residing within a single machine, but also being deployed across multiple machines. In some example embodiments, the processors or processor-implemented components may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the processor or processor-implemented component may be distributed across multiple geographic locations.

In the exemplary architecture of FIG. 7, the software architecture 706 may be conceptualized as a stack of layers, where each layer provides specific functionality. For example, the software architecture 706 may include layers such as an operating system 702, libraries 720, applications 716, and a presentation layer 714. Operationally, an application 716 or other component within a layer may call an Application Program Interface (API) API call 708 through a software stack and receive a message 712 in response to the API call 708. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or special-purpose operating systems may not provide framework/middleware 718, while other operating systems may provide such a layer. Other software architectures may include additional layers or different layers.

Operating system 702 may manage hardware resources and provide common services. Exercise and control deviceThe operating system 702 may include, for example, a kernel 722, services 724, and drivers 726. The kernel 722 may act as an abstraction layer between hardware and other software layers. For example, the kernel 722 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, and the like. The service 724 may provide other common services for other software layers. The drivers 726 are responsible for controlling the hardware or interfacing with the underlying hardware. For example, depending on the hardware configuration, the drivers 726 include a display driver, a camera driver, Drives, flash drives, serial communication drives (e.g., Universal Serial Bus (USB) drives),Drivers, audio drivers, power management drivers, and the like.

The library 720 provides a common infrastructure used by the applications 916 or other components or layers. The library 720 provides functionality that allows other software components to perform tasks in a manner that is easier than directly interfacing with the underlying operating system 702 functionality (e.g., kernel 722, services 724, or drivers 726). The library 720 may include a system library 744 (e.g., a C-standard library) that may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. Additionally, the libraries 720 may include API libraries 746 such as media libraries (e.g., libraries to support the rendering and operation of various media formats such as MPEG4, h.264, MP3, AAC, AMR, JPG, PNG), graphics libraries (e.g., OpenGL frameworks that can be used to render two and three dimensions in graphical content on a display), database libraries (e.g., SQLite that can provide various relational database functions), web libraries (e.g., WebKit that can provide web browsing functions), and so forth. The library 720 may also include various other libraries 748 to provide many other APIs to the application 716 and other software components/modules.

Framework/middleware 718 (also sometimes referred to as middleware) provides a higher level common infrastructure that can be used by applications 716 or other software components/modules. For example, the framework/middleware 718 may provide various Graphical User Interface (GUI) functions, advanced resource management, advanced location services, and the like. The framework/middleware 718 can provide a wide range of other APIs that can be used by the applications 716 or other software components/modules, some of which can be specific to a particular operating system 702 or platform.

The applications 716 include a built-in application 738 or a third-party application 940. Examples of representative built-in applications 738 may include, but are not limited to, a contacts application, a browser application, a book reader application, a location application, a media application, a messaging application, and/or a gaming application. The third-party applications 740 may include applications developed using a Software Development Kit (SDK) by entities other than the vendor of the particular platform and may be mobile software running on a mobile operating system. The third party application 740 may invoke the API call 708 provided by a mobile operating system, such as operating system 702, to facilitate the functionality described herein.

The applications 716 may use built-in operating system functionality (e.g., kernel 722, services 724, or drivers 726), libraries 720, and framework/middleware 718 to create a user interface to interact with a user of the system. Alternatively or additionally, in some systems, interaction with a user may occur through a presentation layer (such as presentation layer 714). In these systems, the application/component "logic" may be separate from the aspects of the application/component that interact with the user.

Fig. 8 is a block diagram illustrating components (also referred to herein as "modules") of a machine 800, the machine 800 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments. In particular, fig. 8 illustrates a diagrammatic representation of machine 800 in the example form of a computer system within which instructions 810 (e.g., software, a program, an application, an applet, application software, or other executable code) may be executed for causing the machine 800 to perform any one or more of the methodologies discussed herein. As such, the instructions 810 may be used to implement the modules or components described herein. The instructions 810 transform the general purpose unprogrammed machine 800 into a specific machine 800 that is programmed to perform the functions described and illustrated in the described manner. In alternative embodiments, the machine 800 operates as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 800 may operate in the capacity of a server machine or a client machine in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 800 may include, but is not limited to, a server computer, a client computer, a Personal Computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a Personal Digital Assistant (PDA), an entertainment media system, a cellular telephone, a smartphone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing instructions 810, sequentially or otherwise, that specify actions to be taken by the machine 800. Further, while only a single machine 800 is illustrated, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute the instructions 1010 to perform any one or more of the methodologies discussed herein.

The machine 800 may include a processor 804, a memory/storage device 806, and I/O components 818 that may be configured to communicate with one another, such as via a bus 802. The memory/storage 806 may include a memory 814, such as a main memory or other memory storage device, and a storage unit 816, both of which may be accessed by the processor 804, such as via the bus 802. The storage unit 816 and the memory 814 store instructions 810 embodying any one or more of the methodologies or functions described herein. The instructions 810 may also reside, completely or partially, within the memory 814, within the storage unit 816, within at least one of the processors 804 (e.g., within a cache memory of the processor), or within any suitable combination thereof, during execution thereof by the machine 800. Thus, memory 814, storage unit 816, and the memory of processor 804 are examples of machine-readable media.

As used herein, the terms "machine-readable medium," "computer-readable medium," and the like may refer to any component, device, or other tangible medium that is capable of storing instructions and data, either temporarily or permanently. Examples of such media may include, but are not limited to, Random Access Memory (RAM), Read Only Memory (ROM), cache memory, flash memory, optical media, magnetic media, cache memory, other types of storage (e.g., erasable programmable read only memory (EEPROM)), or any suitable combination thereof. The term "machine-readable medium" shall be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that are capable of storing the instructions. The term "machine-readable medium" shall also be taken to include any medium, or combination of media, that is capable of storing instructions (e.g., code) for execution by the machine, such that the instructions, when executed by one or more processors of the machine, cause the machine to perform any one or more of the methodologies described herein. Thus, a "machine-readable medium" may refer to a single storage apparatus or device, as well as a "cloud-based" storage system or storage network that includes multiple storage apparatuses or devices. The term "machine-readable medium" does not include the signal itself.

The I/O components 818 can include a variety of components to provide a user interface for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, and the like. The specific I/O components 818 included in the user interface for a particular machine 800 will depend on the type of machine. For example, a portable machine such as a mobile phone would likely include a touch input device or other such input mechanism, while a headless server machine would likely not include such a touch input device. It should be understood that I/O components 818 may include many other components not shown in FIG. 8. The I/O components 818 are grouped by function only to simplify the following discussion, and the grouping is in no way limiting. In various exemplary embodiments, the I/O components 818 may include output components 826 and input components 828. The output components 826 may include visual components (e.g., a display such as a Plasma Display Panel (PDP), a Light Emitting Diode (LED) display, a Liquid Crystal Display (LCD), a projector, or a Cathode Ray Tube (CRT)), acoustic components (e.g., speakers), tactile components (e.g., a vibration motor, a resistive mechanism), other signal generators, and so forth. The input components 828 may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an electro-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instrument), tactile input components (e.g., physical buttons, a touch screen that provides a location or force of a touch or touch gesture, or other tactile input components), audio input components (e.g., a microphone), and so forth. The input component 828 may also include one or more image capture devices, such as a digital camera for generating digital images or videos.

In further exemplary embodiments, the I/O components 818 may include a biometric component 830, a motion component 834, an environmental component 836 or a localization component 838, as well as numerous other components. One or more such components (or portions thereof) may be collectively referred to herein as "sensor components" or "sensors" for collecting various data related to the machine 800, the environment of the machine 800, a user of the machine 800, or a combination thereof.

For example, the biometric components 830 may include components that detect expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), and so forth. The motion components 834 may include acceleration sensor components (e.g., accelerometers), gravity sensor components, velocity sensor components (e.g., speedometers), rotation sensor components (e.g., gyroscopes), and so forth. The environmental components 836 may include, for example, lighting sensor components (e.g., photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometer), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors for detecting concentrations of hazardous gases or measuring pollutants in the atmosphere for safety), or other components that may provide an indication, measurement, or signal corresponding to the surrounding physical environment. The positioning components 838 may include position sensor components (e.g., Global Positioning System (GPS) receiver components), altitude sensor components (e.g., altimeters or barometers, which may detect air pressure from which altitude may be derived), orientation sensor components (e.g., magnetometers), and so forth. For example, the location sensor component may provide location information associated with the system 800, such as GPS coordinates of the system 800 or information about where the system 1000 is currently located (e.g., the name of a restaurant or other business).

Various techniques may be used to implement communications. The I/O components 818 may include a communications component 840 operable to couple the machine 800 to a network 832 or a device 820 via a coupler 822 and a coupler 824, respectively. For example, communications component 840 may include a network interface component or other suitable device to interface with network 832. In further examples, communications component 840 may include a wired communications component, a wireless communications component, a cellular communications component, a Near Field Communications (NFC) component, a wireless communications, The components (e.g.,low power)Components and other communication components that provide communication via other modes. The device 820 may be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via a Universal Serial Bus (USB)).

Further, communications component 840 can detect the identifier or include a component operable to detect the identifier. For example, communications component 840 may include Radio Frequency Identification (RFID)RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., optical sensors for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, supercodes, UCC RSS-2D barcodes, and other optical codes), or acoustic detection components (e.g., microphones for identifying tagged audio signals). Further, various information can be derived via communications component 840, such as location via Internet Protocol (IP) geo-location, viaLocation of signal triangulation, location of NFC beacon signals that may indicate a particular location via detection, and the like.

Fig. 9 is a high-level functional block diagram of an example headset 100 communicatively coupling a mobile device 800 and a server system 998 via various networks.

The device 100 includes a camera, such as at least one of a visible light camera 950, an infrared transmitter 951, and an infrared camera 952. The camera may include a camera module having the lenses 104_1, 104_2 of fig. 1 and 2.

Client device 800 may be capable of connecting with apparatus 100 using both a low power wireless connection 925 and a high speed wireless connection 937. The client device 800 is connected to a server system 998 and a network 995. The network 995 may include any combination of wired and wireless connections.

The apparatus 100 further includes two image displays of the optical assemblies 980A-B. The two image displays 980A-980B include one display associated with the left lateral side of the device 100 and one display associated with the right lateral side. Device 100 also includes an image display driver 942, an image processor 912, low power circuitry 920, and high speed circuitry 930. The image display of the optical assemblies 980A-B is used to present images and video, including images that may include a graphical user interface, to a user of the device 100.

Image display driver 942 commands and controls the image display of optical assemblies 980A-B. Image display driver 942 may transfer image data directly to the image display of optical assembly 980A-B for presentation, or may have to convert the image data into a signal or data format suitable for transfer to an image display device. For example, the image data may be video data formatted according to a compression format, such as h.264(MPEG-4Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, etc., and the still image data may be formatted according to a compression format, such as Portable Network Group (PNG), Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), or exchangeable image file format (Exif).

As described above, the device 100 includes a frame 103 and rods (or threads) extending from lateral sides of the frame 103. The apparatus 100 further includes a user input device 991 (e.g., a touch sensor or button) that includes an input surface on the apparatus 100. A user input device 991 (e.g., a touch sensor or button) will receive input selections from a user to manipulate the graphical user interface of the presented image.

The components for the device 100 shown in fig. 9 are located on one or more circuit boards, such as PCBs or flexible PCBs, in the frame or temple. Alternatively or additionally, the depicted components may be located in a block, frame, hinge, or center sill of the device 100. Left and right visible light cameras 950 may include digital camera elements such as Complementary Metal Oxide Semiconductor (CMOS) image sensors, charge-coupled devices, lenses 104_1, 104_2, or any other corresponding visible light or light capturing elements that may be used to capture data, including images of a scene with unknown objects.

The apparatus 100 includes a memory 934, the memory 934 storing instructions to perform a subset or all of the functions described herein for generating binaural audio content. Memory 934 may also include storage device 604. The exemplary process illustrated in the flow chart of fig. 4 may be implemented in instructions stored in memory 934.

As shown in fig. 9, the high-speed circuitry 930 includes a high-speed processor 932, memory 934, and high-speed radio circuitry 936. In this example, an image display driver 942 is coupled to the high speed circuitry 930 and operated by the high speed processor 932 to drive the left and right image displays of the optical assemblies 980A-B. The high speed processor 932 may be any processor capable of managing the high speed communication and operation of any general purpose computing system required by the apparatus 100. The high-speed processor 932 includes the processing resources needed to manage high-speed data transfers over a high-speed wireless connection 937 to a Wireless Local Area Network (WLAN) using high-speed wireless circuitry 936. In some examples, the high speed processor 932 executes an operating system, such as a LINUX operating system or other such operating system of the device 100, and the operating system is stored in the memory 934 for execution. A high-speed processor 932 executing the software architecture of the device 100 is used, among other things, to manage data transfers with the high-speed radio circuit 936. In some examples, the high-speed wireless circuitry 936 is configured to implement an Institute of Electrical and Electronics Engineers (IEEE)802.11 communication standard, also referred to herein as Wi-Fi. In other examples, other high-speed communication standards may be implemented by the high-speed wireless circuitry 936.

The low-power radio circuitry 924 and high-speed radio circuitry 936 of device 100 may include a short-range transceiver (Bluetooth @)^TM) And a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or WiFi). Client device 800 (including transceivers communicating via low-power wireless connection 925 and high-speed wireless connection 937) may be implemented using details of the architecture of apparatus 100, as may other elements of network 995.

The memory 934 comprises any memory device capable of storing various data and applications, including camera data generated by the left and right visible cameras 950, the infrared camera 952, and the image processor 912, as well as images generated by the image display driver 942 for display on the image display of the optical assemblies 980A-B. Although the memory 934 is shown as being integrated with the high-speed circuitry 930, in other examples, the memory 934 may be a separate, stand-alone element of the apparatus 100. In some such examples, electrical wiring may provide a connection from the image processor 912 or the low power processor 922 to the memory 934 through a chip that includes the high speed processor 932. In other examples, the high speed processor 932 may manage addressing of the memory 934 such that the low power processor 922 will boot the high speed processor 932 whenever a read or write operation involving the memory 934 is required.

As shown in FIG. 9, the processor 932 of the apparatus 100 may be coupled to a camera (visible light camera 950; infrared emitter 951, or infrared camera 952), an image display driver 942, a user input device 991 (e.g., a touch sensor or button), and memory 934.

The device 100 is connected to a host computer. For example, the apparatus 100 is paired with the client device 800 via a high-speed wireless connection 937 or connected to a server system 998 via a network 995. The server system 998 may be one or more computing devices that are part of a service or network computing system, e.g., including a processor, memory, and a network communication interface to communicate with the client devices 800 and apparatus 100 over a network 995.

The client device 800 includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication over the network 925 or 937. The client device 800 may further store at least a portion of the instructions for generating binaural audio content in a memory of the client device 800 to implement the functionality described herein.

The output components of the apparatus 100 include visual components such as a display, such as a Liquid Crystal Display (LCD), a Plasma Display Panel (PDP), a Light Emitting Diode (LED) display, a projector, or a waveguide. The image display of the optical assembly is driven by an image display driver 942. The output components of the apparatus 100 further include acoustic components (e.g., speakers), haptic components (e.g., vibration motors), other signal generators, and the like. Input components of the apparatus 100, client device 800, and server system 998, such as the user input device 991, may include alphanumeric input components (e.g., a keyboard, a touchscreen configured to receive alphanumeric input, an electro-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing instrument), tactile input components (e.g., physical buttons, touchscreen, locations and forces that provide touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and so forth.

The apparatus 100 may optionally include additional peripheral elements. Such peripheral elements may include biometric sensors, additional sensors, or display elements integrated with the apparatus 100. For example, the peripheral elements may include any I/O components, including output components, motion components, position components, or any other such elements described herein.

For example, the biometric component includes a biometric component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measuring bio-signals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), and the like. The motion components include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), and the like. The positioning component includes a position sensor component (e.g., a Global Positioning System (GPS) receiver component) for generating position coordinates, WiFi or Bluetooth for generating positioning system coordinates^TMA transceiver, an altitude sensor component (e.g., an altimeter or barometer that detects barometric pressure from which altitude may be derived), an orientation sensor component (e.g., a magnetometer), and so forth. Such location system coordinates may also be received from client device 800 over wireless connections 925 and 937 via low-power wireless circuitry 924 or high-speed wireless circuitry 936.

Where a phrase similar to "A, B or at least one of C," "A, B and at least one of C," "one or more A, B or C," or "one or more of A, B and C" is used, it is intended that the phrase be interpreted to mean that a may exist alone in one embodiment, B may exist alone in one embodiment, C may exist alone in one embodiment, or any combination of elements A, B and C may exist in a single embodiment; for example, a and B, A and C, B and C, or a and B and C.

Variations and modifications may be made to the disclosed embodiments without departing from the scope of the disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure, as expressed in the following claims.