阅读说明：本技术 声源定位方法、装置及设备 (Sound source positioning method, device and equipment ) 是由 陈维广 黄伟隆 冯津伟 于 2021-10-09 设计创作，主要内容包括：本申请公开了会议发言展示系统,声源定位方法和装置,会议系统,拾音设备。其中,所述方法包括：通过指向性麦克风阵列采集多通道语音信号；根据阵列形状信息和麦克风指向方向信息,确定包括相位信息和振幅信息的导向矢量；根据所述导向矢量和所述语音信号,确定声源方向信息。采用这种处理方式,使得在确定导向矢量时同时考虑相位信息和振幅信息,这样可以有效提升声源定位的准确度。(The application discloses a conference speech display system, a sound source positioning method and device, a conference system and pickup equipment. Wherein the method comprises the following steps: collecting a multi-channel voice signal through a directional microphone array; determining a steering vector including phase information and amplitude information according to the array shape information and the microphone pointing direction information; and determining sound source direction information according to the guide vector and the voice signal. By adopting the processing mode, the phase information and the amplitude information are considered when the guide vector is determined, so that the accuracy of sound source positioning can be effectively improved.)

1. A conference presentation system, comprising:

the terminal equipment is used for acquiring a multi-channel voice signal of a conference space through the directional microphone array; determining a steering vector including phase information and amplitude information according to the array shape information and the microphone pointing direction information; determining the position information of a conference speaking user according to the guide vector and the voice signal; sending the voice signal and the position information to a server; and displaying conference speech texts of different conference speech users returned by the server;

the server is used for converting the voice signal into a conference speaking text through a voice recognition algorithm; and determining conference speaking texts of different conference speaking users according to the position information.

2. A sound source localization method, comprising:

collecting a multi-channel voice signal through a directional microphone array;

determining a steering vector including phase information and amplitude information according to the array shape information and the microphone pointing direction information;

and determining sound source direction information according to the guide vector and the voice signal.

3. The method according to claim 2,

the determining a steering vector including phase information and amplitude information according to the array shape information and the microphone pointing direction information includes:

determining a phase difference according to the array shape information;

determining an amplitude response according to the microphone pointing direction information;

the steering vector is determined from the phase difference and the amplitude response.

4. The method according to claim 2,

the array comprises a linear array;

the array shape information includes a distance between microphones;

the microphone pointing direction comprises a side pointing perpendicular to the array.

5. The method according to claim 2,

the array comprises a circular array;

the array shape information comprises a circular array radius;

the pointing direction of the microphone is the direction of the microphone relative to the circle center of the circular array.

6. The method according to claim 2,

determining sound source direction information according to the guide vector and the voice signal, including:

determining a spatial spectrum according to the guide vector and the voice signal;

and determining the sound source direction information according to the spatial spectrum.

7. The method according to claim 6,

the determining the sound source direction information according to the spatial spectrum includes:

the direction in which the energy response data is arranged ahead is taken as the sound source direction.

8. A sound source localization apparatus, comprising:

the sound acquisition unit is used for acquiring a multi-channel voice signal through a directional microphone array;

a steering vector determination unit for determining a steering vector including phase information and amplitude information based on the array shape information and the microphone pointing direction information;

and the sound source direction determining unit is used for determining sound source direction information according to the guide vector and the voice signal.

9. A sound pickup apparatus, comprising:

a directional microphone array;

a processor; and

a memory for storing a program for implementing the sound source localization method, the apparatus being powered on and the program for the method being run by the processor.

10. A conferencing system, comprising:

a sound source localization device and a speaker tracking device.

Technical Field

The application relates to the technical field of voice processing, in particular to a conference speech presentation system, a sound source positioning method and device, a conference system and pickup equipment.

Background

The basic functions of an audio-video device in a conference scenario include speaker tracking functionality. To implement the speaker tracking function, the speaker needs to be located in real time. Sound Localization (Sound Localization) is the determination of the spatial position of a Sound source, and the accuracy of Sound Localization directly affects the accuracy of speaker tracking.

A typical sound source localization method is a microphone-based direction of arrival estimation (DOA) method. Microphone-based DOA methods include two categories: an omni-directional microphone based DOA method and a directional microphone array based DOA method. Because the DOA method based on the omnidirectional microphone array is greatly influenced by reverberation, and the DOA method based on the directional microphone array has higher robustness, the DOA method based on the directional microphone array is widely applied. In the existing DOA method based on the directional microphone array, a circular directional microphone array is adopted, a Weighting Function (Weighting Function) is added on the basis of a controlled-Response Power (SRP) sound source positioning algorithm, and the sound source direction is estimated by using signals picked up by partial microphones facing to a sound source.

However, in implementing the present invention, the inventors found that the existing DOA scheme based on a directional microphone array has at least the following problems: since only the signals picked up by the microphones partially facing the sound source are utilized and the amplitude information is not fully utilized, the sound source localization accuracy is low.

Disclosure of Invention

The application provides a sound source positioning method to solve the problem that the sound source positioning accuracy is low in the prior art. The application additionally provides a conference speech display system, a sound source positioning device, a conference system and pickup equipment.

The application provides a conference speech presentation system, including:

the terminal equipment is used for acquiring a multi-channel voice signal of a conference space through the directional microphone array; determining a steering vector including phase information and amplitude information according to the array shape information and the microphone pointing direction information; determining the position information of a conference speaking user according to the guide vector and the voice signal; sending the voice signal and the position information to a server; and displaying conference speech texts of different conference speech users returned by the server;

the server is used for converting the voice signal into a conference speaking text through a voice recognition algorithm; and determining conference speaking texts of different conference speaking users according to the position information.

The present application also provides a sound source localization method, including:

collecting a multi-channel voice signal through a directional microphone array;

determining a steering vector including phase information and amplitude information according to the array shape information and the microphone pointing direction information;

and determining sound source direction information according to the guide vector and the voice signal.

Optionally, the determining a steering vector including phase information and amplitude information according to the array shape information and the microphone pointing direction information includes:

determining a phase difference according to the array shape information;

determining an amplitude response according to the microphone pointing direction information;

the steering vector is determined from the phase difference and the amplitude response.

Optionally, the array comprises a linear array;

the array shape information includes a distance between microphones;

the microphone pointing direction comprises a side pointing perpendicular to the array.

Optionally, the array comprises a circular array;

the array shape information comprises a circular array radius;

the pointing direction of the microphone is the direction of the microphone relative to the circle center of the circular array.

Optionally, the determining, according to the steering vector and the voice signal, sound source direction information includes:

determining a spatial spectrum according to the guide vector and the voice signal;

and determining the sound source direction information according to the spatial spectrum.

Optionally, the determining the sound source direction information according to the spatial spectrum includes:

the direction in which the energy response data is arranged ahead is taken as the sound source direction.

The present application further provides a sound source localization apparatus, including:

the sound acquisition unit is used for acquiring a multi-channel voice signal through a directional microphone array;

a steering vector determination unit for determining a steering vector including phase information and amplitude information based on the array shape information and the microphone pointing direction information;

and the sound source direction determining unit is used for determining sound source direction information according to the guide vector and the voice signal.

The present application further provides a conference system, comprising: a sound source localization device and a speaker tracking device.

The present application also provides a sound pickup apparatus, including:

a directional microphone array;

a processor and a memory; a memory for storing a program for implementing the above method, the device being powered on and the program for the method being run by the processor.

The present application also provides a computer-readable storage medium having stored therein instructions, which when run on a computer, cause the computer to perform the various methods described above.

The present application also provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the various methods described above.

Compared with the prior art, the method has the following advantages:

according to the sound source positioning method provided by the embodiment of the application, a directional microphone array is used for acquiring multi-channel voice signals; determining a steering vector including phase information and amplitude information according to the array shape information and the microphone pointing direction information; and determining sound source direction information according to the guide vector and the voice signal. By adopting the processing mode, the phase information and the amplitude information are considered when the guide vector is determined, so that the accuracy of sound source positioning can be effectively improved.

According to the conference speech display system provided by the embodiment of the application, terminal equipment acquires multi-channel speech signals of a conference space through a directional microphone array; determining a steering vector including phase information and amplitude information according to the array shape information and the microphone pointing direction information; determining the position information of a conference speaking user according to the guide vector and the voice signal; sending the voice signal and the position information to a server; the server side converts the voice signal into a conference speech text through a voice recognition algorithm; determining conference speaking texts of different conference speaking users according to the position information; and the terminal equipment displays conference speaking texts of different conference speaking users. By adopting the processing mode, the phase information and the amplitude information are considered when the guide vector is determined, so that the positioning accuracy of the conference speaking user can be effectively improved, and the conference speaking display accuracy is further improved.

Drawings

FIG. 1 is a schematic flow chart diagram of an embodiment of a sound source localization method provided by the present application;

FIG. 2 is a schematic diagram of a linear array of an embodiment of a sound source localization method provided by the present application;

FIG. 3 is a detailed flow chart of an embodiment of a sound source localization method provided by the present application;

fig. 4 is a schematic view of an application scenario of an embodiment of a conference presentation system provided in the present application.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is capable of implementation in many different ways than those herein set forth and of similar import by those skilled in the art without departing from the spirit of this application and is therefore not limited to the specific implementations disclosed below.

The application provides a conference speech presentation system, a sound source positioning method and device, a conference system and pickup equipment. Each of the schemes is described in detail in the following examples.

First embodiment

The embodiment of the application provides a sound source positioning method, which can be used for pickup equipment, audio and video conference terminals and the like, wherein the equipment comprises a directional microphone array instead of an omnidirectional microphone array.

Please refer to fig. 1, which is a schematic flow chart of an embodiment of the sound source localization method of the present application. In this embodiment, the method may include the steps of:

step S101: and acquiring a multi-channel voice signal by a directional microphone array.

The directional microphone includes, but is not limited to: heart-shaped, super-heart-shaped, gun-shaped, and double-directional.

The microphone array can be a circular array or a linear array, can also be an array with other geometric shapes, such as a square array, a triangular array and the like, and can also be an array with an irregular geometric shape.

Step S103: from the array shape information and the microphone pointing direction information, a steering vector including phase information and amplitude information is determined.

The processing flow of the method provided by the embodiment of the application adopts the same processing flow as the DOA method based on the omnidirectional microphone in the prior art, but the determination mode of the guide vector is improved, and the step S103 is the improved determination mode of the guide vector.

In specific implementation, a DOA positioning method such as a stepped-Response Power-Phase Transform (SRP-phot), MUSIC, MVDR, and the like can be adopted. Taking an SRP-PHAT positioning method as an example, the method calculates the energy response of each angle according to the steering vector and the signals received by a microphone array by scanning different angles (0-360 degrees), and further obtains a spatial spectrum; after obtaining the spatial spectrum, the angle of the higher energy response in the spatial spectrum may be selected as the sound source localization result. These DOA methods differ in the way the spatial spectrum is computed from the steering vectors and the multi-channel speech signal.

The array shape information is related to the geometry of the array. Taking a linear array as an example, the array shape information may include information such as a distance between microphones. Taking a circular array as an example, the array shape information may include information such as a radius of the circular array.

The microphone pointing direction information is also related to the geometry of the array. Taking a linear array as an example, the microphone points in a direction perpendicular to the array. Taking a circular array as an example, the pointing direction of the microphone is the direction of the microphone relative to the center of the array.

In the prior art, when an omnidirectional microphone array is used, the steering vector represents only the phase relationship of the incident signal at each array element in the microphone array. In the method provided by the present application, when the microphones in the array are directional microphones, the steering vector also takes into account the directionality of the microphones, i.e. the amplitude response in the direction to be calculated. That is, the steering vector according to the embodiment of the present application includes phase information and amplitude information. Thus, for signals in different directions, the phase information and the amplitude information can be used simultaneously for positioning.

In this embodiment, step S103 may include the following sub-steps: determining a phase difference according to the array shape information; determining an amplitude response according to the microphone pointing direction information; the steering vector is determined from the phase difference and the amplitude response.

As shown in fig. 2, in one example, where the directional microphone array is a linear array, the amplitude response can be calculated using the following equation:

in this formula, p (θ)_mθ) represents the amplitude response of the mth directional microphone, θ represents the signal incidence direction, θ) represents the amplitude response of the mth directional microphone_mIs the pointing direction of the mth directional microphone and α is the coefficient of the first-order directional microphone.

Accordingly, the following formula can be adopted for the steering vector:

as can be seen from the formula, the directional microphone array comprises m directional microphones, and the distance between adjacent microphones isd, d is the array shape information. Wherein v (w) represents the steering vector, which includes both phase difference and amplitude response components; p (theta)_iθ) represents the amplitude response of the i-th directional microphone in the direction θ, e^{-jwd/c cosθ}Indicating the phase difference of the directional microphone in the direction theta. For the first microphone, the distance difference is 0 and the phase difference is 1; for the second microphone, the distance difference is d and the phase difference is e^{-jwd/c cosθ}(ii) a By analogy, for the m-th microphone, the distance difference is (m-1) d, and the phase difference is e^{-jw(m-1)d/c cosθ}。

In the prior art, the directional microphone steering vector of the omnidirectional microphone can be calculated by the following formula:

as can be seen from the formula, the prior art does not consider amplitude information when calculating the guide vector, so the guide vector is not accurate enough.

In another example, where the directional microphone array is a circular array, the steering vector may be given by the following formula:

in this formula, θ represents the signal incidence direction, θ_mIs the pointing direction of the mth directional microphone and R is the radius of the circular array.

Step S105: and determining sound source direction information according to the guide vector and the voice signal.

After determining the steering vector comprising phase information and amplitude information, the DOA method may be used to determine the sound source direction information based on the steering vector and the speech signal.

As shown in fig. 3, the directional microphone array may be a circular array or a linear array. In a specific implementation, step S105 may include the following sub-steps: determining a spatial spectrum according to the guide vector and the voice signal; and determining the sound source direction information according to the spatial spectrum. In specific implementation, after the spatial spectrum is obtained, the angle of higher energy response in the spatial spectrum can be selected as the sound source positioning result. Since the DOA methods such as SRP-PHAT, MUSIC and MVDR belong to the mature prior art, they are not described herein again.

As can be seen from the foregoing embodiments, in the sound source localization method provided by the embodiments of the present application, a directional microphone array is used to collect a multi-channel speech signal; determining a steering vector including phase information and amplitude information according to the array shape information and the microphone pointing direction information; and determining sound source direction information according to the guide vector and the voice signal. By adopting the processing mode, the phase information and the amplitude information are considered when the guide vector is determined, so that the accuracy of sound source positioning can be effectively improved.

Second embodiment

In the foregoing embodiment, a sound source positioning method is provided, and correspondingly, the present application further provides a sound source positioning apparatus. The apparatus corresponds to an embodiment of the method described above. Since the apparatus embodiments are substantially similar to the method embodiments, they are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for relevant points. The device embodiments described below are merely illustrative.

The present application additionally provides a sound source localization apparatus, comprising:

the sound acquisition unit is used for acquiring a multi-channel voice signal through a directional microphone array;

a steering vector determination unit for determining a steering vector including phase information and amplitude information based on the array shape information and the microphone pointing direction information;

and the sound source direction determining unit is used for determining sound source direction information according to the guide vector and the voice signal.

Optionally, the steering vector determining unit includes:

a phase difference determining subunit, configured to determine a phase difference according to the array shape information;

the amplitude response determining subunit is used for determining an amplitude response according to the microphone pointing direction information;

and the determining guide vector subunit is used for determining the guide vector according to the phase difference and the amplitude response.

Optionally, the array comprises a linear array;

the array shape information includes a distance between microphones;

the microphone pointing direction comprises a side pointing perpendicular to the array.

Optionally, the array comprises a circular array;

the array shape information comprises a circular array radius;

the pointing direction of the microphone is the direction of the microphone relative to the circle center of the circular array.

Optionally, the sound source direction determining unit includes:

a determining spatial spectrum subunit, configured to determine a spatial spectrum according to the steering vector and the speech signal;

and the sound source direction determining subunit is used for determining the sound source direction information according to the spatial spectrum.

Optionally, the sound source direction determining subunit is configured to determine, as the sound source direction, a direction in which the energy response data is arranged in front.

Third embodiment

Corresponding to the sound source positioning method, the application also provides a conference system. Parts of this embodiment that are the same as the first embodiment are not described again, please refer to corresponding parts in the first embodiment. The present application provides a conference system including: a sound source localization device and a speaker tracking device.

The audio and video conference system is a system device which is used for individuals or groups in two or more different places, and can transmit sound, images and file data to each other through a transmission line, a conference terminal and other devices, so as to realize real-time and interactive communication and simultaneously carry out a conference.

The sound source positioning device corresponds to the first embodiment, and therefore, the description thereof is omitted, please refer to corresponding parts in the first embodiment. The speaker tracking device is used for determining the activity track information of the speaker according to the sound source direction information output by the sound source positioning device. Since speaker tracking is a mature prior art, it is not described here in detail.

As can be seen from the foregoing embodiments, the conference system provided in the embodiments of the present application includes a sound source localization apparatus and a speaker tracking apparatus, where the sound source localization apparatus is configured to collect a multi-channel speech signal through a directional microphone array; determining a steering vector including phase information and amplitude information according to the array shape information and the microphone pointing direction information; determining sound source direction information according to the guide vector and the voice signal; the speaker tracking device is used for determining the activity track information of the speaker according to the sound source direction information output by the sound source positioning device. The system considers the phase information and the amplitude information when determining the guide vector, so that the accuracy of sound source positioning can be effectively improved, and the accuracy of speaker tracking is further improved.

Fourth embodiment

Corresponding to the sound source positioning method, the application also provides a conference speaking display system. Parts of this embodiment that are the same as the first embodiment are not described again, please refer to corresponding parts in the first embodiment. The present application provides a conference system including: terminal equipment and server.

Please refer to fig. 4, which is a schematic view illustrating a conference presentation system according to the present application. In this embodiment, the terminal device is deployed in a conference site, the server is deployed on the cloud server, and a large screen can be deployed in the conference site to display a conference speech text and a corresponding speech user in real time for the user to view. The server and the terminal equipment can be connected through a network, for example, the terminal equipment can be networked through GPRS \4G \ WIFI and other modes. The terminal equipment is used for acquiring a multi-channel voice signal of a conference space through a directional microphone array; determining a steering vector including phase information and amplitude information according to the array shape information and the microphone pointing direction information; determining the position information of a conference speaking user according to the guide vector and the voice signal; sending the voice signal and the position information to a server; the server is used for converting the voice signal into a conference speaking text through a voice recognition algorithm; determining conference speaking texts of different conference speaking users according to the position information; and the terminal equipment displays the conference speaking text and the corresponding speaking user information on a large screen.

As can be seen from the foregoing embodiments, in the conference presentation display system provided in the embodiments of the present application, a terminal device acquires a multi-channel speech signal in a conference space through a directional microphone array; determining a steering vector including phase information and amplitude information according to the array shape information and the microphone pointing direction information; determining the position information of a conference speaking user according to the guide vector and the voice signal; sending the voice signal and the position information to a server; the server side converts the voice signal into a conference speech text through a voice recognition algorithm; determining conference speaking texts of different conference speaking users according to the position information; and the terminal equipment displays conference speaking texts of different conference speaking users. By adopting the processing mode, the phase information and the amplitude information are considered when the guide vector is determined, so that the positioning accuracy of the conference speaking user can be effectively improved, and the conference speaking display accuracy is further improved.

Although the present application has been described with reference to the preferred embodiments, it is not intended to limit the present application, and those skilled in the art can make variations and modifications without departing from the spirit and scope of the present application, therefore, the scope of the present application should be determined by the claims that follow.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

1. Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (transient media), such as modulated data signals and carrier waves.

2. As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.