阅读说明：本技术 全向可听噪声源定位装置 (Omnidirectional audible noise source positioning device ) 是由 河田宏史 于 2020-02-12 设计创作，主要内容包括：本发明提供用于能够同时测量封闭或开放空间中的环绕声再现和噪声源定位的设备的系统和方法。当在封闭空间的移动对象中需要环绕声再现和噪声源定位时,特别使用该系统和方法。封闭空间中的移动对象不限于特定产品,可以考虑例如汽车、火车、电梯等。(The present invention provides systems and methods for a device capable of simultaneous measurement of surround sound reproduction and noise source localization in closed or open spaces. The system and method are particularly useful when surround sound reproduction and noise source localization are required in moving objects in an enclosed space. The moving object in the closed space is not limited to a specific product, and may be, for example, a car, a train, an elevator, or the like.)

1. A system, the system comprising:

a microphone array comprising at least four microphones arranged in a three-dimensional shape along positions relative to each other;

a 360 degree camera; and

a processor to:

calculating, for audio received by the microphone array, a three-dimensional sound intensity between at least two of the at least four microphones of the microphone array; and is

Overlaying audio onto a video feed of the 360 degree camera with a display view of the three dimensional sound intensity relative to the video feed.

2. The system of claim 1, wherein the three-dimensional shape is a regular tetrahedron.

3. The system of claim 1, wherein the processor is configured to calculate the three-dimensional sound intensity between the at least two of the at least four microphones of the microphone array by calculating the three-dimensional sound intensity based on an inverse fourier transform of a cross spectrum of sound pressures measured by the at least two of the at least four microphones.

4. The system of claim 1, wherein the processor is configured to calculate the three-dimensional sound intensity between the at least two of the at least four microphones of the microphone array by:

calculating a sound pressure of an acoustic center of the microphone array;

deriving a particle velocity between each of the at least four microphones of the microphone array and the acoustic center; and

calculating the three-dimensional sound intensity from particle velocity calculations along an x-axis, a y-axis, and a z-axis based on derived velocities between the each of the at least four microphones of the microphone array and the acoustic center.

5. The system of claim 1, wherein the microphone array is an ambient stereo microphone consisting of four microphones.

6. The system of claim 1, wherein the processor is to overlay the audio onto the video feed of a 360 degree camera with a displayed view of a three-dimensional sound intensity relative to the video feed through a heat map representation of the three-dimensional sound intensity on the video feed.

7. A method for a system comprising a microphone array and a 360 degree camera, the microphone array comprising at least four microphones arranged in a three-dimensional shape along a position relative to each other; the method comprises the following steps:

calculating, for audio received by the microphone array, a three-dimensional sound intensity between at least two of the at least four microphones of the microphone array; and

overlaying audio onto a video feed of the 360 degree camera with a display view of the three dimensional sound intensity relative to the video feed.

8. The method of claim 7, wherein the three-dimensional shape is a regular tetrahedron.

9. The method of claim 7, wherein the calculating the three-dimensional sound intensity between the at least two of the at least four microphones of the microphone array comprises: calculating the three-dimensional sound intensity based on an inverse Fourier transform of a cross-spectrum of the sound pressures measured by the at least two of the at least four microphones.

10. The method of claim 7, wherein the calculating the three-dimensional sound intensity between the at least two of the at least four microphones of the microphone array comprises:

calculating a sound pressure of an acoustic center of the microphone array;

deriving a particle velocity between each of the at least four microphones of the microphone array and the acoustic center; and

calculating the three-dimensional sound intensity from particle velocity calculations along an x-axis, a y-axis, and a z-axis based on derived velocities between the each of the at least four microphones of the microphone array and the acoustic center.

11. The method of claim 7, wherein the microphone array is an ambient stereo microphone consisting of four microphones.

12. The method of claim 7, wherein the audio is overlaid onto the video feed of the 360 degree camera with a displayed view of the three-dimensional sound intensity relative to the video feed via a heat map representation of the three-dimensional sound intensity on the video feed.

13. A non-transitory computer-readable medium storing instructions for a system comprising a microphone array and a 360 degree camera, the microphone array comprising at least four microphones arranged in a three-dimensional shape along a position relative to each other; the instructions include:

calculating, for audio received by the microphone array, a three-dimensional sound intensity between at least two of the at least four microphones of the microphone array; and

overlaying audio onto a video feed of the 360 degree camera with a display view of the three dimensional sound intensity relative to the video feed.

14. The non-transitory computer-readable medium of claim 13, wherein the three-dimensional shape is a regular tetrahedron.

15. The non-transitory computer-readable medium of claim 13, wherein the calculating the three-dimensional sound intensity between the at least two of the at least four microphones of the microphone array comprises: calculating the three-dimensional sound intensity based on an inverse Fourier transform of a cross-spectrum of the sound pressures measured by the at least two of the at least four microphones.

16. The non-transitory computer-readable medium of claim 13, wherein the calculating the three-dimensional sound intensity between the at least two of the at least four microphones of the microphone array comprises:

calculating a sound pressure of an acoustic center of the microphone array;

deriving a particle velocity between each of the at least four microphones of the microphone array and the acoustic center; and

calculating the three-dimensional sound intensity from particle velocity calculations along an x-axis, a y-axis, and a z-axis based on derived velocities between the each of the at least four microphones of the microphone array and the acoustic center.

17. The non-transitory computer-readable medium of claim 13, wherein the microphone array is an ambient stereo microphone consisting of four microphones.

18. The non-transitory computer-readable medium of claim 13, wherein the audio is overlaid onto the video feed of the 360 degree camera with a display view of the three-dimensional sound intensity relative to the video feed via a heat map representation of the three-dimensional sound intensity on the video feed.

Technical Field

The present disclosure relates generally to sound systems, and more particularly to systems and methods for providing three-dimensional sound for three-dimensional video.

Background

In the prior art, the sound (noise) problem of the product has been difficult to share between the developer and the customer. This is because sound is a sensory evaluation and generally cannot be shared unless the developer and the client are in the same situation (e.g., location, environmental conditions) to hear a particular sound.

In prior art implementations, surround sound technology has been developed for reproducing three dimensional sound directions and propagation when recording and reproducing sound. Surround sound can be played back with special microphones. Such prior art implementations are expected to facilitate information sharing with product developers and customers.

In prior art implementations, there are Virtual Realities (VRs) for generating three-dimensional audio, however, these prior art implementations do not provide any means for making an actual recording of audio, nor do they provide any implementation for noise source localization. In general, even in prior art implementations involving surround sound reproduction methods, there is no description about noise source localization or recordings for such noise source localization.

Disclosure of Invention

Drawings

Fig. 1 is a system overview of an apparatus according to an exemplary embodiment.

Fig. 2 is an exemplary front view of a microphone portion of an ambient stereo microphone according to an exemplary embodiment.

Fig. 3 shows an exemplary arrangement of microphone capsules and co-ordinate axes according to an exemplary embodiment.

Fig. 4 shows an exemplary system involving an omnidirectional image according to an exemplary embodiment.

Fig. 5 shows an exemplary flowchart of an apparatus according to an exemplary embodiment.

FIG. 6 illustrates an exemplary computing environment having an exemplary computer device suitable for use in the exemplary embodiments.

Detailed Description

The following detailed description provides further details of the figures and exemplary implementations of the present application. For clarity, reference numerals and descriptions of redundant elements between figures are omitted. The terminology used throughout the description is provided by way of example and not by way of limitation. For example, use of the term "automatic" may relate to a fully or semi-automatic implementation involving a user or administrator controlling certain aspects of the implementation, as desired by one of ordinary skill in the art practicing implementations of the present application. The selection may be made by the user through a user interface or other input means, or may be implemented through a desired algorithm. The exemplary implementations described herein may be utilized separately or in combination, and the functions of the exemplary implementations may be implemented by any means depending on the desired implementation.

The exemplary embodiments described herein relate to systems and methods for a device capable of simultaneous measurement of surround sound reproduction and noise source localization in closed or open spaces. The system and method are particularly useful when surround sound reproduction and noise source localization are required in moving objects in an enclosed space. The moving object in the closed space is not limited to a specific product, and may be, for example, a car, a train, an elevator, or the like.

For noise source localization, exemplary embodiments focus on sound intensity indicating the amount and direction of acoustic energy flow. Because the sound intensity is not easily affected by background noise, the example implementations described herein can utilize the sound intensity to locate noise sources in a room involving a moving object despite having large background noise.

In an exemplary embodiment, the sound intensity may be calculated by measuring sound pressures of a plurality of microphones and distances between the microphones.

In the exemplary embodiments described herein, special microphones for surround sound pickup are used to evaluate point microphones, where the sound intensity is calculated from the measured sound pressure and the distance between the microphones, where an omnidirectional noise source can thus be located. With such an exemplary embodiment, surround sound reproduction and noise source localization can thus be performed simultaneously.

According to the exemplary embodiments described herein, it is thus possible to be unaffected by background noise of moving objects and it is also possible to perform surround sound reproduction in all directions simultaneously without indirectly introducing a measuring device for noise source localization measurements.

Hereinafter, exemplary embodiments of an omni-directional audible noise source locating device will be described with reference to the accompanying drawings.

Fig. 1 is a system overview of an apparatus according to an exemplary embodiment. In the example of fig. 1, the measurement aspects of the apparatus relate to a special microphone (referred to herein as an ambient stereo microphone 101), a sound recording device 102 and a video recording device 103 for omni-directional shooting. The analytical aspect of the device relates to a converter 104 for converting sound picked up by an ambient stereo microphone 101 for VR ambient stereo reproduction and a calculator 105 for calculating sound intensity.

Fig. 2 is an exemplary front view of a microphone portion of an ambient stereo microphone 101 according to an exemplary embodiment. There are four microphone pods 111, 112, 113, 114 with microphones facing outward from each face of the regular tetrahedron. Surround sound reproduction or ambient stereo sound is possible by the transducer 104 generating a spherical harmonic function of the signals picked up by the four microphone pods (111 to 114).

To use the microphone arrangement described herein, exemplary embodiments are also directed to systems and methods for calculating sound intensity from the above-described ambient stereo microphone 101 signals and performing noise source localization from that calculation. In particular, the exemplary embodiment relates to two methods for calculating the operator 105 of the sound intensity, direct method and cross-spectrum method. Either method may be utilized depending on the desired implementation.

In a first embodiment, the sound intensity is calculated using the direct method, as described below.

Fig. 3 shows an exemplary arrangement of microphone pods (111 to 114) and coordinate axes according to an exemplary embodiment. Each microphone pod (111 to 114) is arranged at the vertex of a regular tetrahedron. The center of gravity G of a regular tetrahedron is the acoustic center, and the x, y, and z-axis coordinates are defined with the acoustic center as the origin as shown in FIG. 3.

First, the sound pressure p0(t) at the acoustic center is measured by each microphone and given as an average of the sound pressures p1(t) to p4 (t).

When p0(t) is used, if the sound wave is approximated as a plane wave at a distance Δ r from the acoustic center to each microphone, the particle velocity ui (t) in the direction from the acoustic center to each microphone can be obtained by the following equation.

Where ρ is the density of the propagation medium.

On the other hand, in consideration of the geometrical conditions of tetrahedron, the following relationships exist between the particle velocity ui (t) and the particle velocity components ux (t), uy (t), uz (t) in the x-axis, y-axis, and z-axis directions.

u4(t) ═ uz (t) (equation 6)

From these equations (equations 3 to 6), the particle velocity components in the x, y, z-axis directions are derived as follows.

The sound intensity can be determined from the time average of the product of the sound pressure and the particle velocity. In other words, according to equations 1, 2 and 7 to 9, the three-dimensional sound intensity can be measured by measuring the sound pressure using the four microphone pods (111 to 114). Since the described method can be utilized in real-time processing, measurement can be performed while viewing a display at a site or the like.

In a second exemplary embodiment, the three-dimensional sound intensity is measured using a cross-spectrum method, as follows. In this method, the sound intensity i (x) is approximately calculated by the following equation processed, displayed in the frequency domain (equation 2).

Wherein G is₁₂(x, ω) are cross-spectral functions of the sound pressures p1(t) and p2(t) measured by the two microphones, Im { } is the imaginary part. In other words, the imaginary part of the cross spectrum of the sound pressures measured by the two microphones is the inverse fourier transform. Since the cross spectrum is obtained from the Fourier transform of the sound pressure, the method can thereby correct the sensitivity of each microphoneDifferences in degree and phase characteristics.

From the above exemplary embodiment, the ambient stereo microphone 101 may be used to calculate the sound intensity and search for noise source localization in all directions. Thus, the exemplary embodiment makes it possible to simultaneously perform surround sound reproduction and noise source localization by using the ambient stereo microphone 101.

Fig. 4 shows an exemplary system involving an omnidirectional image according to an exemplary embodiment. As shown in fig. 4, noise source localization may be performed visually using an omnidirectional image. Also, as long as the ambient stereo microphone 101 can make measurements simultaneously, the four microphones may be separated, but the locations of the vertices of the microphones should be on a regular tetrahedron to facilitate the exemplary implementation described herein. However, other shapes than tetrahedrons may be utilized, depending on the desired implementation, as long as the sound intensity can be calculated based on measurements made between two microphones within the microphone array. In these embodiments, the equations described herein should be adjusted according to the shape used to measure the sound intensity between the two microphones.

Further, multiple microphone arrays may be used with multiple cameras, depending on the desired implementation. In an exemplary embodiment, multiple instances of the system may be provided in each room in a building and used as a surveillance system, where the audio and video feeds between the instances may be switched according to the desired implementation.

Fig. 5 shows an exemplary flowchart of an apparatus according to an exemplary embodiment. At 501, the system as shown in fig. 1 and 4 records sound through a microphone array and video through a 360 degree camera. At 502, three-dimensional sound intensities between the microphones of the microphone array are calculated according to, for example, implementations described with respect to equations provided herein. At 503, a spherical harmonic function of the microphone signals is developed and sounds for surround sound reproduction are created. At 504, surround sound is overlaid onto the video feed, where the surround sound can be played with respect to a viewpoint displayed at an appropriate three-dimensional sound intensity from the video feed. From such an exemplary implementation, a user may navigate the video feed over the interface in a 360 degree manner and then identify and locate sound sources on the interface relative to the video feed. In an exemplary embodiment, audio may be overlaid on a video feed, where a heat map indicator on the video feed indicates a location of an audio source based on the calculated sound intensity. For example, in the case where the object to be measured is in a stable state, an omnidirectional picture may be used as a substitute for omnidirectional video.

FIG. 6 illustrates an exemplary computing environment having an exemplary computer device suitable for use in the exemplary embodiments, such as the sound recording device or apparatus illustrated in the systems of FIGS. 1 and 4. The computer device 605 in the computing environment 600 may include one or more processing units, cores or processors 610, memory 615 (e.g., RAM, ROM, etc.), internal memory 620 (e.g., magnetic, optical, solid-state memory, and/or organic memory), and/or an I/O interface 625, any of which may be coupled over a communication mechanism or bus 630 for communicating information or embedded in the computer device 605.

The computer device 605 may be communicatively coupled to an input/user interface 635 and an output device/interface 640. Either or both of the input/user interface 635 and the output device/interface 640 may be wired or wireless interfaces, and may be detachable. The input/user interface 635 may include any physical or virtual device, component, sensor, or interface (e.g., buttons, touch screen interface, keyboard, pointing/cursor control, microphone, camera, braille, motion sensor, optical reader, etc.) that may be used to provide input. Output device/interface 640 may include a display, television, monitor, printer, speakers, braille, etc. In some example implementations, the input/user interface 635 and the output device/interface 640 can be embedded in or physically coupled to the computer device 605. In other exemplary implementations, other computer devices may be used as or provide the functionality of the input/user interface 635 and the output device/interface 640 of the computer device 605. In exemplary embodiments involving a touch screen display, a television display, or any other form of display, the display is configured to provide a user interface.

Examples of computer devices 605 may include, but are not limited to, highly mobile devices (e.g., smartphones, devices in vehicles and other machines, devices carried by humans and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices that are not designed for mobility (e.g., desktop computers, other computers, kiosks, televisions with one or more processors embedded and/or coupled therein, radios, etc.).

Computer device 605 may be communicatively coupled (e.g., via I/O interface 625) to external memory 645 and network 650 for communicating with any number of networked components, devices, and systems, including one or more computer devices of the same or different configurations. Computer device 605, or any connected computer device, may act as, provide services to, or be referred to as, a server, a client, a thin server, a general purpose machine, a special purpose machine, or another label.

The I/O interfaces 625 can include, but are not limited to, wired and/or wireless interfaces using any communication or I/O protocol or standard (e.g., ethernet 802.11x, a general system bus, WiMax, modem, cellular network protocol, etc.) for communicating information to and/or from at least all connected components, devices, and networks in the computing environment 600. Network 650 may be any network or combination of networks (e.g., the internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

Computer device 605 can utilize and/or communicate using computer-usable or computer-readable media, including transitory media and non-transitory media. Transitory media include transmission media (e.g., metallic cables, optical fibers), signals, carrier waves, and the like. Non-transitory media include magnetic media (e.g., disks and tapes), optical media (e.g., CD ROM, digital video disks, blu-ray disks), solid state media (e.g., RAM, ROM, flash memory, solid state memory), and other non-volatile storage or memory.

In some exemplary computing environments, computer device 605 may be used to implement techniques, methods, applications, processes, or computer-executable instructions. Computer-executable instructions may be retrieved from a transitory medium, and stored on and retrieved from a non-transitory medium. The executable instructions may be derived from one or more of any programming, scripting, and machine language (e.g., C, C + +, C #, Java, Visual Basic, Python, Perl, JavaScript, etc.).

The processor 610 may execute under any Operating System (OS) (not shown) in a local or virtual environment. One or more applications may be deployed including logic unit 660, Application Programming Interface (API) unit 665, input unit 670, output unit 675, and inter-unit communication mechanism 695 for the different units to communicate with each other, the OS, and other applications (not shown). The described units and elements may vary in design, function, configuration, or implementation and are not limited to the descriptions provided. The processor 610 may be in the form of a physical processor or Central Processing Unit (CPU) configured to execute instructions loaded from memory 615.

In some example implementations, when information or execution instructions are received by API unit 665, they may be communicated to one or more other units (e.g., logic unit 660, input unit 670, output unit 675). In some examples, in some of the example implementations described above, logic unit 660 may be used to control the flow of information between units and direct services provided by API unit 665, input unit 670, output unit 675. For example, the flow of one or more processes or implementations may be controlled by logic unit 660 alone or in conjunction with API unit 665. Input unit 670 may be used to obtain input for the computations described in the exemplary implementations, and output unit 675 may be used to provide output based on the computations described in the exemplary implementations.

The processor 610 may be configured to execute the process of fig. 5 to facilitate the functionality of the system as shown in fig. 1 and 4. Such a system may involve a microphone array involving at least four microphones arranged in a three-dimensional shape as shown in fig. 2 and 3 along a position relative to each other, and one 360 degree camera.

In an exemplary embodiment, the processor 610 may be configured to calculate a three-dimensional sound intensity between at least two of the at least four microphones of the microphone array for audio received by the microphone array; and overlays audio onto the video feed of the 360 degree camera with three dimensional sound intensity relative to the display view of the video feed as shown in fig. 5 and as described with respect to fig. 1-5.

As shown in fig. 3, the three-dimensional shape arrangement may be a regular tetrahedron.

As shown in fig. 3 and 4 and described with respect to their counterparts, the processor 610 may be operative to calculate a three-dimensional sound intensity between the at least two of the at least four microphones of the microphone array by calculating the three-dimensional sound intensity based on an inverse fourier transform of a cross spectrum of sound pressures measured by the at least two of the at least four microphones.

As shown in fig. 1 and 2 and described with respect to their counterparts, the processor 610 may be used to calculate a three-dimensional sound intensity between the at least two of the at least four microphones of the microphone array by: calculating a sound pressure of an acoustic center of the microphone array; deriving a particle velocity between each of the at least four microphones of the microphone array and the acoustic center; and calculating a three-dimensional sound intensity from particle velocity calculations along an x-axis, a y-axis, and a z-axis based on the derived velocity between the each of the at least four microphones of the microphone array and the acoustic center.

Depending on the desired implementation, the microphone array may be an ambient stereo microphone consisting of four microphones as shown in fig. 2.

The processor 610 may also be configured to overlay audio onto the video feed of the 360 degree camera with a displayed view of the three-dimensional sound intensity relative to the video feed via a heat map representation of the three-dimensional sound intensity on the video feed. Depending on the desired implementation, the heat map may be in the form of a color intensity (e.g., yellow to red) or a grayscale intensity based on the calculated sound intensity, which may set an indicator on the video feed as to the location source of the sound. Other heat map representations may be utilized in accordance with desired implementations, and the present disclosure is not limited to any particular heat map representation.

Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the substance of their innovation to others skilled in the art. An algorithm is a defined series of steps leading to a desired end state or result. In an exemplary implementation, the steps performed require physical manipulations of tangible quantities to achieve a tangible result.

Unless specifically stated otherwise as apparent from the discussion, it is appreciated that throughout the description, discussions utilizing terms such as "processing," "computing," "calculating," "determining," "displaying," or the like, may include the actions and processes of a computer system, or other information processing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other information storage, transmission or display devices.

Example embodiments may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such a computer program may be stored in a computer readable medium, such as a computer readable storage medium or a computer readable signal medium. The computer readable storage medium may include a tangible medium such as, but not limited to, an optical disk, a magnetic disk, a read-only memory, a random-access memory, a solid-state device and drive, or any other type of tangible or non-transitory medium suitable for storing electronic information. Computer readable signal media may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. A computer program may refer to a pure software implementation involving instructions to perform the operations desired to be implemented.

Various general-purpose systems may be used with programs and modules, or it may prove convenient to construct more specialized apparatus to perform the desired method steps, according to the examples herein. In addition, the exemplary implementations are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the example implementations described herein. The instructions of the programming language may be executed by one or more processing devices, such as a Central Processing Unit (CPU), processor, or controller.

The above operations may be performed by hardware, software, or some combination of software and hardware, as is known in the art. Various aspects of the exemplary implementations may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software), which if executed by a processor, would cause the processor to perform a method for performing the implementations of the present application. Furthermore, some example implementations of the present application may be performed solely in hardware, while other example implementations may be performed solely in software. Further, various functions described may be performed in a single unit or may be distributed across multiple components in any number of ways. When executed by software, the method is performed by a processor (such as a general purpose computer) based on instructions stored on a computer-readable medium. The instructions may be stored on the media in a compressed and/or encrypted format, if desired.

In addition, other implementations of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the teachings of the present application. The various aspects and/or components of the described exemplary implementations may be used alone or in any combination. It is intended that the specification and exemplary implementations be considered as examples only, with a true scope and spirit of the application being indicated by the following claims.