阅读说明：本技术 一种基于ar的三维全息实时互动系统及方法 (AR-based three-dimensional holographic real-time interaction system and method ) 是由 黄辉 徐文宇 于 2020-12-18 设计创作，主要内容包括：本发明公开了一种基于AR的三维全息实时互动系统及方法,所述系统设置于A地的3D融合处理设备,第一音视频采集设备及音视频播放设备,第一音视频采集设备采集A地场景内的音频及视频,3D融合处理设备对A地采集的视频进行3D融合形成能立体显示的融合视频；设置于B地的音视频处理设备,第二音视频采集设备、音频播放设备及全息投影显示设备,第二音视频采集设备采集B地场景内的音频及视频,音视频处理设备将B地采集的音频及视频发送至A地,音频播放设备播放A地发送来的音频,全息投影显示设备用于以全息投影的效果立体显示A地发送来的融合视频。本发明使得A地主体能以全息投影的方式在B地立体再现,从而增加了交互沟通的体验。(The invention discloses an AR-based three-dimensional holographic real-time interaction system and an AR-based three-dimensional holographic real-time interaction method, wherein the system is arranged on a 3D fusion processing device of a place A, a first audio and video acquisition device and an audio and video playing device, the first audio and video acquisition device acquires audio and video in a scene of the place A, and the 3D fusion processing device performs 3D fusion on the video acquired by the place A to form a fusion video capable of being displayed in a three-dimensional mode; the system comprises audio and video processing equipment arranged on a place B, second audio and video acquisition equipment, audio playing equipment and holographic projection display equipment, wherein the second audio and video acquisition equipment acquires audio and video in a scene of the place B, the audio and video processing equipment sends the audio and video acquired by the place B to the place A, the audio playing equipment plays the audio sent by the place A, and the holographic projection display equipment is used for three-dimensionally displaying the fused video sent by the place A with the effect of holographic projection. The invention enables the A-site main body to be stereoscopically reproduced in the B-site in a holographic projection mode, thereby increasing the experience of interactive communication.)

1. An AR-based three-dimensional holographic real-time interactive system, comprising:

the system comprises 3D fusion processing equipment arranged on a place A, first audio and video acquisition equipment and audio and video playing equipment, wherein the first audio and video acquisition equipment is connected with the 3D fusion processing equipment and is used for acquiring audio and video in a scene of the place A, the 3D fusion processing equipment is used for carrying out 3D fusion on the video acquired on the place A to form a fusion video for holographic projection three-dimensional display on the place B, the audio acquired on the place A and the fused video are sent to the place B, and the audio and video playing equipment is used for playing the audio and video sent from the place B;

the system comprises audio and video processing equipment arranged on a ground B, and second audio and video acquisition equipment, audio playing equipment and holographic projection display equipment which are connected with the audio and video processing equipment, wherein the second audio and video acquisition equipment is used for acquiring audio and video in a scene of the ground B, the audio and video processing equipment is used for sending the audio and video acquired by the ground B to the ground A, the audio playing equipment is used for playing the audio sent by the ground A, and the holographic projection display equipment is used for three-dimensionally displaying the fused video sent by the ground A by using a holographic projection effect.

2. The system of claim 1, wherein the holographic projection display device comprises a light box, a transparent holographic reflective screen positioned within the light box, and a projector positioned above the transparent holographic reflective screen, wherein the light box, the transparent holographic reflective screen, and the projector stereoscopically present images with holographic projection effects.

3. The system according to claim 1, wherein the 3D fusion processing device comprises a first main control unit and a first network communication unit, the first main control unit is configured to process audio collected by the first audio/video collection device to obtain audio data, perform 3D fusion processing on the video to obtain fused video data, send the audio/video data to the B site through the first network communication unit, and simultaneously send the audio data and video data sent from the B site to the audio/video playing device for playing;

the audio and video processing equipment comprises a second main control unit and a second network communication unit, wherein the second main control unit is used for processing audio and video collected by the second audio and video collecting equipment to obtain audio data and video data and sending the audio data and video data to the A place through the second network communication unit, and meanwhile, the second main control unit is used for sending the audio and video data sent from the A place to the audio playing equipment and the holographic projection display equipment to play.

4. The system according to claim 1, wherein the first audio/video acquisition device is provided with a first compression unit for compressing the acquired audio/video and then sending the compressed audio/video to the 3D fusion processing device;

the second audio and video acquisition equipment is provided with a second compression unit which is used for compressing the acquired audio and video and then sending the compressed audio and video to the audio and video processing equipment.

5. An AR-based three-dimensional holographic real-time interaction method applied to the system as claimed in any one of claims 1 to 4, comprising the steps of:

s10, the first audio and video acquisition equipment acquires audio and video of the A place object scene, and stores the audio and video as an audio and video file or directly sends the audio and video file to 3D fusion processing equipment;

s20, the 3D fusion processing equipment carries out image matting processing on the video image collected from the A place to remove the background and extract a main body, and then carries out 3D modeling fusion on the main body and each scene content needing interaction to obtain a fusion video;

s30, the 3D fusion processing equipment sends the audio and the fused video to the audio and video processing equipment of the B place through the network, and the 3D fusion processing equipment also receives the audio and video sent by the B place and plays the audio and video through the audio and video playing equipment;

s40, the audio and video processing device plays the audio sent from A place through the audio playing device, and the fusion video sent from A place is stereoscopically reproduced through the holographic projection display device;

and S50, the audio and video processing equipment sends the audio and video collected by the second audio and video collecting equipment from the B place to the 3D fusion processing equipment of the A place in real time through the network so as to interact with the main body of the A place.

6. The method according to claim 5, characterized in that said 3D modeling fusion comprises in particular the steps of:

s201, acquiring a main body of a video image after image matting processing;

s202, carrying out feature recognition on the scratched main body to obtain a specific action of a person or a specific mark of an object;

s203, triggering and deriving a built corresponding 3D model in a 3D model library according to a specific action or a specific mark;

and S204, superposing the derived 3D model to a video image to form a new video.

7. The method of claim 6, wherein overlaying the derived 3D model onto the video image further comprises associating the multi-frame image of the subject's change of motion with multiple states of the 3D model.

Technical Field

The invention relates to the technical field of holographic interaction equipment, in particular to an AR-based three-dimensional holographic real-time interaction system and method.

Background

The existing remote interactive system generally collects audio and video information through a network camera and a microphone, and then audio and video stream data are transmitted to a destination through a broadband, and then are displayed through a sound box display and an equal-tone video playing device.

However, the display effect of videos in the existing remote interaction system is flat and basically has no three-dimensional sense, so that the feeling of face-to-face three-dimensional interaction is lacked in the communication process, and the interaction experience is influenced.

Therefore, the prior art has yet to be improved.

Disclosure of Invention

In view of the above shortcomings in the prior art, the present invention provides an AR-based three-dimensional holographic real-time interactive system and method, so that a subject in a can be stereoscopically reproduced in a holographic projection manner in a B-site, thereby increasing the interactive communication experience.

In order to realize the purpose, the invention adopts the following technical scheme:

in a first aspect, the present invention provides an AR-based three-dimensional holographic real-time interactive system, which includes:

the system comprises 3D fusion processing equipment arranged on a place A, first audio and video acquisition equipment and audio and video playing equipment, wherein the first audio and video acquisition equipment is connected with the 3D fusion processing equipment and is used for acquiring audio and video in a scene of the place A, the 3D fusion processing equipment is used for carrying out 3D fusion on the video acquired on the place A to form a fusion video for holographic projection three-dimensional display on the place B, the audio acquired on the place A and the fused video are sent to the place B, and the audio and video playing equipment is used for playing the audio and video sent from the place B;

the system comprises audio and video processing equipment arranged on a ground B, and second audio and video acquisition equipment, audio playing equipment and holographic projection display equipment which are connected with the audio and video processing equipment, wherein the second audio and video acquisition equipment is used for acquiring audio and video in a scene of the ground B, the audio and video processing equipment is used for sending the audio and video acquired by the ground B to the ground A, the audio playing equipment is used for playing the audio sent by the ground A, and the holographic projection display equipment is used for three-dimensionally displaying the fused video sent by the ground A by using a holographic projection effect.

The holographic projection display equipment comprises a light box, a transparent holographic reflection screen and a projector, wherein the transparent holographic reflection screen is positioned in the light box, the projector is arranged above the transparent holographic reflection screen, and the light box, the transparent holographic reflection screen and the projector display images in a holographic projection effect three-dimensional mode.

The 3D fusion processing device comprises a first main control unit and a first network communication unit, wherein the first main control unit is used for processing audio collected by the first audio and video collection device to obtain audio data, performing 3D fusion processing on the video to obtain fused video data, sending the fused video data to a B place through the first network communication unit, and simultaneously sending the audio data and video data sent by the B place to an audio and video playing device for playing;

the audio and video processing equipment comprises a second main control unit and a second network communication unit, wherein the second main control unit is used for processing audio and video collected by the second audio and video collecting equipment to obtain audio data and video data and sending the audio data and video data to the A place through the second network communication unit, and meanwhile, the second main control unit is used for sending the audio and video data sent from the A place to the audio playing equipment and the holographic projection display equipment to play.

The first audio and video acquisition equipment is provided with a first compression unit for compressing the acquired audio and video and then sending the compressed audio and video to the 3D fusion processing equipment;

the second audio and video acquisition equipment is provided with a second compression unit which is used for compressing the acquired audio and video and then sending the compressed audio and video to the audio and video processing equipment.

In a second aspect, the present invention provides an AR-based three-dimensional holographic real-time interaction method, which is applied to the above system, and includes the following steps:

s10, the first audio and video acquisition equipment acquires audio and video of the A place object scene, and stores the audio and video as an audio and video file or directly sends the audio and video file to 3D fusion processing equipment;

s20, the 3D fusion processing equipment carries out image matting processing on the video image collected from the A place to remove the background and extract a main body, and then carries out 3D modeling fusion on the main body and each scene content needing interaction to obtain a fusion video;

s30, the 3D fusion processing equipment sends the audio and the fused video to the audio and video processing equipment of the B place through the network, and the 3D fusion processing equipment also receives the audio and video sent by the B place and plays the audio and video through the audio and video playing equipment;

s40, the audio and video processing device plays the audio sent from A place through the audio playing device, and the fusion video sent from A place is stereoscopically reproduced through the holographic projection display device;

and S50, the audio and video processing equipment sends the audio and video collected by the second audio and video collecting equipment from the B place to the 3D fusion processing equipment of the A place in real time through the network so as to interact with the main body of the A place.

The 3D modeling fusion specifically comprises the following steps:

s201, acquiring a main body of a video image after image matting processing;

s202, carrying out feature recognition on the scratched main body to obtain a specific action of a person or a specific mark of an object;

s203, triggering and deriving a built corresponding 3D model in a 3D model library according to a specific action or a specific mark;

and S204, superposing the derived 3D model to a video image to form a new video.

Wherein the superimposing the derived 3D model onto the video image further comprises associating and superimposing a plurality of frame images of the subject's motion change with a plurality of states of the 3D model.

According to the AR-based three-dimensional holographic real-time interaction system, the acquired videos are subjected to 3D fusion by the 3D fusion processing equipment in the place A to form a fusion video, and then the fusion video is stereoscopically and dynamically presented in the place B in a holographic projection mode through the holographic projection display equipment, so that the interaction process has the feeling of being personally on the scene and communicating face to face, and the interaction experience is improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the structures shown in the drawings without creative efforts.

FIG. 1 is a schematic diagram illustrating a first embodiment of an AR-based three-dimensional holographic real-time interaction system according to the present invention;

FIG. 2 is a schematic diagram of the holographic projection display apparatus of the present invention;

FIG. 3 is a schematic flow chart of a first embodiment of the AR-based three-dimensional holographic real-time interaction method of the present invention;

FIG. 4 is a schematic flow chart of 3D modeling fusion in accordance with the present invention;

FIG. 5 is a schematic diagram of a process of fusing a subject character with a 3D model according to the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

As shown in fig. 1, the present invention is an AR-based three-dimensional holographic real-time interactive system, which is applied to remote interaction.

The system comprises a 3D fusion processing device 11 arranged on a ground A, a first audio and video acquisition device 12 and an audio and video playing device 13 which are connected with the 3D fusion processing device 11, wherein the first audio and video acquisition device 12 is used for acquiring audio and video in a scene of the ground A, the 3D fusion processing device 11 is used for carrying out 3D fusion on the video acquired on the ground A to form a fusion video for holographic projection three-dimensional display on the ground B, the audio acquired on the ground A and the fused video are sent to the ground B, and the audio and video playing device 13 is used for playing the audio and video sent from the ground B.

The 3D fusion processing device 11 may perform 3D fusion processing using a general computer, or may perform 3D fusion processing using a dedicated computer, so as to increase the processing speed of the entire system. The first audio/video capture device 12 may be a high-definition video camera or a high-definition digital camera and a microphone to capture the video and audio. The audio/video playing device 13 may be a sound box, a liquid crystal display, or the like, for playing audio/video.

The system further comprises an audio and video processing device 21 arranged on the ground B, a second audio and video acquisition device 22, an audio playing device 24 and a holographic projection display device 23 which are connected with the audio and video processing device 21, wherein the second audio and video acquisition device 22 is used for acquiring audio and video in the scene of the ground B, the audio and video processing device 21 is used for sending the audio and video acquired by the ground B to the ground A, the audio playing device 24 is used for playing the audio sent by the ground A, and the holographic projection display device 23 is used for displaying the fused video sent by the ground A in a holographic projection mode. The audio/video processing device 21 may also use a common computer to perform audio/video processing, such as encoding and decoding, format conversion, and the like. The second audio/video capture device 22 may be a camera, a video camera or a digital camera and a microphone to capture the audio/video. The audio playing device 24 may be a sound box for playing audio. The holographic projection display device 23 is a device capable of performing stereoscopic display by holographic projection.

As an embodiment, as shown in fig. 2, the holographic projection display device 23 of the system of the present invention includes a light box 231, a transparent holographic reflection screen 232 disposed in the light box 231, and a projector 233 disposed above the transparent holographic reflection screen 232, wherein the light box 231, the transparent holographic reflection screen 232, and the projector 233 stereoscopically present an image with a holographic projection effect. The projector 233 is connected to the audio/video processing device 21 to obtain a video source for projection, the transparent holographic reflective screen 232 may be a pyramid-shaped four-sided cone transparent structure, and the video content projected by the projector 233 can reach a stereoscopic video effect suspended in the light box 231 by matching a specific position of the video content in the transparent holographic reflective screen 232 with a background, so as to obtain a holographic-like projection image.

Therefore, the interactive system sends the audio and the video collected in the place A to the place B for reproduction, and simultaneously sends the audio and the video collected in the place B to the place A for playing, so that remote interaction is realized, and simultaneously, as the video sent in the place A is the video subjected to 3D fusion processing and is stereoscopically displayed through the holographic projection display equipment 23 in the place B, the interactive system can stereoscopically reproduce main characters or contents in the place A in the place B in a holographic projection mode, so that the main characters in the place B have face-to-face communication and personally on-the-spot communication feeling, and the communication and interaction experience of users is improved.

It is understood that, in other embodiments of the present invention, the audio/video processing device 21 in the location B may also perform 3D fusion processing on the video, and then the audio/video playing device 13 in the location a may also be a holographic projection display device, so that the video of the main character in the location B can be stereoscopically presented in the location a.

Preferably, as shown in fig. 1, the 3D fusion processing device 11 of the system of the present invention includes a first main control unit 111 and a first network communication unit 112, where the first main control unit 111 is configured to process the audio collected by the first audio/video collecting device 12 to obtain audio data, perform 3D fusion processing on the video to obtain fused video data, send the audio/video data to the B site through the first network communication unit 112, and simultaneously send the audio data and video data sent from the B site to the audio/video playing device 13 for playing. The first main control unit 111 may perform 3D fusion processing on the video using a high-performance CPU. The first network communication unit 112 may be a communication module such as ethernet, WIFI, GPRS, etc. Preferably, the 3D fusion processing device 11 of the present invention further includes a database for storing 3D models required during the 3D fusion processing.

The audio and video processing device 21 includes a second main control unit 211 and a second network communication unit 212, where the second main control unit 211 is configured to process the audio and video collected by the second audio and video collecting device 22 to obtain audio data and video data, send the audio data and video data to the location a through the second network communication unit 212, and send the audio and video data sent from the location a to the audio playing device and the holographic projection display device 23 for playing. The second main control unit 211 may process the audio and video by using a high-performance CPU. The second network communication unit 212 may be a communication module such as ethernet, WIFI, GPRS, etc. for performing wired or wireless communication with the first network communication unit 112.

Further, the first audio/video capture device 12 of the system of the present invention is provided with a first compression unit 121, configured to compress the captured audio/video and send the compressed audio/video to the 3D fusion processing device 11.

The second audio and video capture device 22 is provided with a second compression unit 221 for compressing the captured audio and video and then sending the compressed audio and video to the audio and video processing device 21.

Therefore, the audio and video is the compressed audio and video in the transmission process, so that the bandwidth occupation is reduced, the video transmission speed is improved, and the blocking phenomenon in the real-time interaction process is reduced.

As shown in fig. 3, the present invention further provides an AR-based three-dimensional holographic real-time interaction method, which is applied to the above system, and includes the following steps:

and S10, the first audio and video acquisition equipment acquires the audio and video of the A place object scene, and stores the audio and video as an audio and video file or directly sends the audio and video file to the 3D fusion processing equipment.

The first audio and video acquisition device 12 is configured to convert an audio and video physical signal in the a-ground object scene into an electrical signal, convert the electrical signal into a digital signal, and send the digital signal to the 3D fusion processing device 11 for processing.

And S20, the 3D fusion processing equipment performs image matting processing on the video image collected from the A place to remove the background and extract a main body, and then performs 3D modeling fusion on the main body and each scene content needing interaction to obtain a fusion video.

The 3D fusion processing device 11 converts the audio digital signal of the a place into audio data that can be played, and performs image matting processing on the video collected from the a place to obtain a subject, such as a person or a landmark object, in the video image. And then, performing 3D modeling on scene contents required to be displayed in various interactive scenes by the main body, and performing superposition and fusion on the scene contents and the main body to obtain a fused 3D video.

If the main character in the place A speaks of rainy weather in the process of communicating with the main character in the place B, the system carries out 3D modeling on the rainy scene after triggering, and the 3D modeling is fused with the main character to obtain a 3D video of the main character in a rainy environment and the 3D video is sent to the place B for three-dimensional presentation.

And S30, the 3D fusion processing equipment sends the audio and the fused video to the audio and video processing equipment of the B place through the network, and the 3D fusion processing equipment also receives the audio and video sent by the B place and plays the audio and video through the audio and video playing equipment.

And S40, the audio and video processing device plays the audio transmitted from the A place through an audio playing device, and the fused video transmitted from the A place is stereoscopically reproduced through a holographic projection display device.

After receiving the audio and video sent from the place a, the audio and video processing device 21 separates the audio from the video, the audio is sent to the audio playing device 24 for playing, and the video is sent to the holographic projection display device 23 for three-dimensional presentation.

And S50, the audio and video processing equipment sends the audio and video collected by the second audio and video collecting equipment from the B place to the 3D fusion processing equipment of the A place in real time through the network so as to interact with the main body of the A place.

Specifically, as shown in fig. 4, the 3D modeling fusion of the method of the present invention specifically includes the following steps:

s201, acquiring the subject subjected to image matting processing in the video image.

The matting process is to remove interference in the background, and the subject can be a person or an object.

S202, carrying out feature recognition on the scratched main body to obtain a specific action of a person or a specific mark of an object.

The feature recognition adopts an image recognition technology to extract main features in the image so as to obtain a specific sign of a specific action or object of a person. Such as human gestures, expressions, two-dimensional codes on objects, and the like.

As shown in fig. 5, after the keying process, the subject performs feature recognition to obtain a specific motion of the hand of the person, and the motion is a motion of holding an article in the hand.

And S203, triggering and deriving the established corresponding 3D model in the 3D model library according to the specific action or the specific mark.

And comparing the specific action or the specific mark obtained after the characteristic identification with a preset specific action or specific mark, and triggering and deriving a built corresponding 3D model in a 3D model library if the specific action or the specific mark is matched with the preset specific action or the specific mark.

As shown in fig. 5, the specific action triggers a 3D model in the 3D model library to display a virtual cube, and virtual reality screens are further provided on two sides of the virtual cube to display cube data.

And S204, superposing the derived 3D model to a video image to form a new video.

And overlapping the triggered and derived 3D model with the main body to obtain a new overlapped video image and form a new video.

As shown in fig. 5, the superposition of the main body and the corresponding 3D model results in a video image of the main body showing the cube.

Further, the method of the present invention superimposes the derived 3D model onto the video image further comprises associating and superimposing a plurality of frame images of the subject's motion change with a plurality of states of the 3D model. Thus, a continuous animation video can be formed, for example, if each frame of image of the subject character in fig. 5, which rotates 360 degrees, is added to the shape of the cube 3D model at the corresponding angular position, an animation video of the subject character rotating 360 degrees with the cube model can be formed.

According to the AR-based three-dimensional holographic real-time interaction system and method, the acquired videos are subjected to 3D fusion by the 3D fusion processing equipment in the place A to form the fusion video, and then the fusion video is stereoscopically and dynamically presented in the place B in a holographic projection mode through the holographic projection display equipment, so that the interactive process has the feeling of being personally on the scene and communicating with each other in a face-to-face mode, and the interactive experience is greatly improved.

The above description is only for clearly illustrating the invention and is not therefore to be considered as limiting the scope of the invention, and all embodiments are not intended to be exhaustive, and all equivalent structural changes made by using the technical solutions of the present invention or other related technical fields directly/indirectly applied under the concept of the present invention are included in the scope of the present invention.