阅读说明：本技术 一种声音与图像混合阵列处理系统 (Sound and image mixed array processing system ) 是由 陈劲 刘荣 关升 于 2019-08-14 设计创作，主要内容包括：本发明公开了一种声音与图像混合阵列处理系统,包括声阵列信号处理子系统、图像阵列处理子系统和显示模块；所述声阵列信号处理子系统包括声音阵列接收模块和声音数据的存储与处理模块,所述图像阵列处理子系统包括图像阵列接收模块与图像数据的存储与处理模块；显示模块用于将声阵列处理子系统与图像阵列处理子系统分别传送来的声强分布图和视频进行叠加显示,将声强分布图不同位置的声音分布与视频中该位置相对应的物体严格贴合,从而使得用户通过该模块就可得知画面中每个区域内的物体周围的声音分布情况。(The invention discloses a sound and image mixed array processing system, which comprises a sound array signal processing subsystem, an image array processing subsystem and a display module, wherein the sound array signal processing subsystem comprises a sound array signal processing module, a sound array signal processing module and a display module; the sound array signal processing subsystem comprises a sound array receiving module and a sound data storage and processing module, and the image array processing subsystem comprises an image array receiving module and an image data storage and processing module; the display module is used for overlapping and displaying the sound intensity distribution diagram and the video respectively transmitted by the sound array processing subsystem and the image array processing subsystem, and strictly attaching the sound distribution of different positions of the sound intensity distribution diagram to the object corresponding to the position in the video, so that a user can know the sound distribution situation around the object in each area in the picture through the display module.)

1. A sound and image hybrid array processing system, comprising: the system comprises an acoustic array signal processing subsystem, an image array processing subsystem and a display module;

the sound array signal processing subsystem comprises a sound array receiving module and a sound data storage and processing module, wherein the sound array receiving module is used for receiving a sound array signal data packet transmitted by the terminal and analyzing effective data in the sound array signal data packet to be used for the sound data storage and processing module to perform data processing; the storage and processing module of the sound data comprises two working modes of 'storage after processing' and 'processing before storage', and the two working modes are carried out simultaneously in a multi-thread coding mode;

wherein: under the mode of 'storage before processing', effective data is stored firstly, the data is processed after the data is stored, and the received data is displayed in a two-dimensional waveform form in a display module; under the mode of processing before storing, the storage and processing module of the sound data temporarily puts the received effective sound array data in the memory at first, after the number of data frames reaches the requirement, all data stored in the memory are converted into a two-dimensional matrix, then a direction of arrival estimation algorithm is adopted to obtain a sound intensity distribution diagram, and the sound intensity distribution diagram is finally sent to the display module;

the image array processing subsystem comprises an image array receiving module and an image data storage and processing module, wherein the image array receiving module is used for receiving an image array signal data packet transmitted by a terminal and analyzing effective data in the image array signal data packet to be used for the image data storage and processing module to perform data processing; the image data storage and processing module comprises two working modes of 'storage before processing' and 'processing before storage', and the two working modes are simultaneously carried out in a multi-thread coding mode;

wherein: under the mode of 'storage before processing', original image data is stored in a corresponding folder in a picture mode, and then target detection is carried out on the original image data according to the requirements of a user so as to obtain the detail information of the image; in the mode of processing first and storing later, received image data are directly sent to a display module, the display module can superimpose a sound intensity distribution graph obtained by a sound array processing subsystem on a video, and finally the superimposed whole video is stored.

2. The sound and image mixing array processing system of claim 1, wherein: the sound array data and the image array data received in the system are obtained by a sound and image signal mixed terminal through transmission of a kilomega Ethernet cable, the terminal adopts an image sensor and a sound sensor to collect sound and image data respectively, frames are formed on the two signals respectively, and finally the two signals are transmitted to the system through a kilomega Ethernet cable, and a virtual channel and a port in the kilomega Ethernet cable are utilized to transmit two data signals of the image and the sound which are not interfered with each other at the same time.

3. The sound and image mixing array processing system of claim 2, wherein: the sound array receiving module receives the sound signals according to the port number corresponding to the virtual channel of the terminal, firstly, the port is searched, once the system starts to operate and data arrives in the channel, the sound array receiving module can capture all transmitted data packets and analyze effective data in the data packets to be used for the storage and processing module of the sound data to process the data.

4. The sound and image mixing array processing system of claim 2, wherein: the image array receiving module receives the image signals according to the port number corresponding to the virtual channel of the terminal, firstly searches the port, once the system starts to operate and data arrives in the channel, the image array receiving module captures all transmitted data packets and analyzes effective data in the data packets to be used for data processing of the image data storage and processing module.

5. The sound and image mixing array processing system of claim 1, wherein: the storage and processing module of the sound data is under the mode of 'storage before processing': before data is stored, a space for storing the data is established in a hard disk or a magnetic disk in advance, and a memory mapping mechanism and an alternate storage method are adopted in the data storage process.

6. The sound and image mixing array processing system of claim 1, wherein: the image data storage and processing module is used for storing and processing the image data in a mode of storage before processing: before data is stored, a space for storing the data is established in a hard disk or a magnetic disk in advance, and a memory mapping mechanism and an alternate storage method are adopted in the data storage process.

7. The sound and image mixing array processing system of claim 1, wherein: the image data storage and processing module is used for storing and processing the image data in a mode of storage before processing: when the data is up, the storage and processing module of the image data stores the effective data in the format of jpg picture, and adds the time of image formation in the naming of the image, so that the user can know the time of the system obtaining the image by the name of the image conveniently, and the subsequent target detection processing is facilitated.

8. The sound and image mixing array processing system of claim 1, wherein: the image data storage and processing module is used for storing and processing the image data in a mode of storage before processing: and carrying out target identification on the image by using a neural network algorithm.

9. The sound and image mixing array processing system of claim 8, wherein: the neural network algorithm is packaged into a single packet, and the network meeting the requirements can be selected for detection according to actual requirements.

10. The sound and image mixing array processing system of claim 1, wherein: the display module is used for overlapping and displaying the sound intensity distribution diagram and the video respectively transmitted by the sound array processing subsystem and the image array processing subsystem, and strictly attaching the sound distribution of different positions of the sound intensity distribution diagram to the object corresponding to the position in the video, so that a user can know the sound distribution situation around the object in each area in the picture through the display module.

Technical Field

The invention belongs to the technical field of data processing, and particularly relates to a sound and image mixed array processing system.

Background

Sound and images are the most commonly encountered in our lives, contain a great deal of useful information, and play a critical role in many fields, such as criminal investigation, solution, teleconferencing, and the like. In practical applications, a mixed sound signal generated by a plurality of sound sources is usually extracted, and the most common processing is to perform spectrum analysis, and extract or separate signals in corresponding frequency band ranges as required. In the mixed sound signal, the sound intensity emitted by different sound sources is different, so that the signal can be processed, and the sound intensity distribution of the signal on different time and space can be visually represented.

The processing of the image is more common, for example, the image is often required to be modified in the e-commerce field, so that the image is more attractive; when an event such as theft occurs in a public place, a picture shot by a monitoring camera needs to be acquired frequently to be analyzed. In recent years, with the development of neural networks, the detection and identification of images have become a hot spot, and have attracted a large number of students to research.

In some cases, it is not enough to process only sound or image, and both need to be analyzed at the same time. Acoustic cameras are a popular application example at present, and are mainly used for positioning of noise sources. By processing the captured sound and image mixed signal, the sound distribution at different positions in the picture can be observed through the display, so as to obtain the area where the noise may exist and take corresponding measures. The system also processes the collected sound and image mixed array, so that a user can observe the distribution condition of sound signals with different intensities, and simultaneously, automatically selects the obtained image to perform target detection on the image, thereby obtaining a specific object at a certain position in a picture and the sound distribution at the position. The system can be widely applied to a plurality of fields such as remote video conferences, examination room behavior monitoring and the like.

Disclosure of Invention

The invention aims to overcome the defects of the prior art and provide a sound and image mixing array processing system.

The invention is realized by the following technical scheme:

a sound and image mixed array processing system comprises a sound array signal processing subsystem, an image array processing subsystem and a display module;

the sound array signal processing subsystem comprises a sound array receiving module and a sound data storage and processing module, wherein the sound array receiving module is used for receiving a sound array signal data packet transmitted by the terminal and analyzing effective data in the sound array signal data packet to be used for the sound data storage and processing module to perform data processing; the storage and processing module of the sound data comprises two working modes of 'storage after processing' and 'processing before storage', and the two working modes are carried out simultaneously in a multi-thread coding mode;

wherein: under the mode of 'storage before processing', effective data is stored firstly, the data is processed after the data is stored, and the received data is displayed in a two-dimensional waveform form in a display module; under the mode of processing before storing, the storage and processing module of the sound data temporarily puts the received effective sound array data in the memory at first, after the number of data frames reaches the requirement, all data stored in the memory are converted into a two-dimensional matrix, then a direction of arrival estimation algorithm is adopted to obtain a sound intensity distribution diagram, and the sound intensity distribution diagram is finally sent to the display module;

the image array processing subsystem comprises an image array receiving module and an image data storage and processing module, wherein the image array receiving module is used for receiving an image array signal data packet transmitted by a terminal and analyzing effective data in the image array signal data packet to be used for the image data storage and processing module to perform data processing; the image data storage and processing module comprises two working modes of 'storage before processing' and 'processing before storage', and the two working modes are simultaneously carried out in a multi-thread coding mode;

wherein: under the mode of 'storage before processing', original image data is stored in a corresponding folder in a picture mode, and then target detection is carried out on the original image data according to the requirements of a user so as to obtain the detail information of the image; in the mode of processing first and storing later, received image data are directly sent to a display module, the display module can superimpose a sound intensity distribution graph obtained by a sound array processing subsystem on a video, and finally the superimposed whole video is stored.

In the technical scheme, the sound array data and the image array data received in the system are obtained by transmitting a sound and image signal mixed terminal through a gigabit Ethernet line, the terminal adopts an image sensor and a sound sensor to respectively collect sound and image data, then respectively framing the two signals, and finally transmitting the two signals to the system through the gigabit Ethernet line, and a virtual channel and a port in the gigabit Ethernet line are used for simultaneously transmitting the two data signals of the image and the sound without mutual interference.

In the above technical solution, the sound array receiving module receives sound signals according to a port number corresponding to a virtual channel through which the terminal transmits the sound signals, and first searches for the port, and once the system starts to operate and data arrives in the channel, the sound array receiving module captures all transmitted data packets and analyzes valid data therein to provide for the storage and processing module of the sound data to perform data processing.

In the above technical solution, the image array receiving module receives the image signal according to the port number corresponding to the virtual channel of the terminal, and first searches the port, and once the system starts to operate and data arrives in the channel, the image array receiving module captures all transmitted data packets and analyzes valid data therein for the storage and processing module of the image data to perform data processing.

In the above technical solution, the storage and processing module of the sound data, in a "storage before processing" mode: before data is stored, a space for storing the data is established in a hard disk or a magnetic disk in advance, and a memory mapping mechanism and an alternate storage method are adopted in the data storage process.

In the above technical solution, the storage and processing module of the image data, in a "storage before processing" mode: before data is stored, a space for storing the data is established in a hard disk or a magnetic disk in advance, and a memory mapping mechanism and an alternate storage method are adopted in the data storage process.

In the above technical solution, the storage and processing module of the image data, in a "storage before processing" mode: when the data is up, the storage and processing module of the image data stores the effective data in the format of jpg picture, and adds the time of image formation in the naming of the image, so that the user can know the time of the system obtaining the image by the name of the image conveniently, and the subsequent target detection processing is facilitated.

In the above technical solution, the storage and processing module of the image data, in a "storage before processing" mode: and carrying out target identification on the image by using a neural network algorithm.

In the technical scheme, the neural network algorithm is packaged into a single packet, and the network meeting the requirement can be selected for detection according to actual requirements.

In the above technical solution, the display module is configured to superimpose and display the sound intensity distribution diagram and the video respectively transmitted by the sound array processing subsystem and the image array processing subsystem, and strictly attach the sound distribution at different positions of the sound intensity distribution diagram to an object corresponding to the position in the video, so that a user can know the sound distribution around the object in each region in the picture through the display module.

The invention has the advantages and beneficial effects that:

(1) the system adopts a mode of synchronous working of a plurality of assembly lines, each branch line runs independently, different branch lines are connected by signals, and the integrity of the system can be ensured under the condition of ensuring synchronous multitasking.

(2) Each module of the system relates to some parameters and algorithms, and the parameters and the algorithms are assembled into replaceable packets in the system building process, so that the system can adjust the corresponding parameter packets and algorithm packets according to the change of the connected terminals and application requirements, thereby greatly improving the flexibility of the system and enabling the system to be applied to more fields.

(3) The system is added with a neural network with high heat at present, and the neural network is used for carrying out target recognition on the image instead of simply relying on human eyes to carry out object recognition. The method greatly improves the practicability of the system, and particularly under the conditions that the surrounding environment is dim and the field order is disordered, the distribution of a plurality of objects in the picture cannot be accurately distinguished only by human eyes.

Drawings

Fig. 1 is a schematic diagram of a configuration of a hybrid audio and video array processing system.

Fig. 2 is a schematic diagram of reception of the audio array signal and the video array signal.

Fig. 3 is a schematic diagram of a storage and processing module for voice data.

Fig. 4 is a process diagram of data storage.

Fig. 5(a) is an interface diagram showing 2-way waveforms simultaneously.

Fig. 5(b) is an interface diagram showing 4-way waveforms simultaneously.

Figure 6 is the overall workflow of the acoustic array processing subsystem.

FIG. 7 is a schematic diagram of an image data storage and processing module.

FIG. 8 is an overall workflow of the image array processing subsystem.

For a person skilled in the art, other relevant figures can be obtained from the above figures without inventive effort.

Detailed Description

In order to make the technical solution of the present invention better understood, the technical solution of the present invention is further described below with reference to specific examples.