阅读说明：本技术 全景显示系统在ar眼镜上的实时传输与监控的方法 (Real-time transmission and monitoring method of panoramic display system on AR glasses ) 是由 吉涛 向艳 朱标 张骏 孙瑞 张鹏 沈玉真 于 2020-06-15 设计创作，主要内容包括：本发明公开了一种全景显示系统在AR眼镜上的实时传输与监控的方法,包括：步骤1、安装摄像头；步骤2、进行全景画面实时拼接和unity多形状动态贴图；步骤3、对场景对象进行渲染；步骤4、udp/rtsp全景画面传输；步骤5、实现头部姿态、眼动、语音和手势控制视角画面；步骤6、最终设备间同步显示全景实时画面并协同控制。该方法可以实时将设备采集的全景画面传输给移动端AR眼镜并同步显示,即便使用人员脱离全景设备也能够有效地获取安装在车载或者其他位置传出的实时全景画面,并在头部姿态控制的基础上完成全景不同方位的观察,能最大程度获取当前环境形式下的相关数据信息,结合多种智能交互模式(眼动、语音、手势)协同指挥控制。(The invention discloses a real-time transmission and monitoring method of a panoramic display system on AR glasses, which comprises the following steps: step 1, installing a camera; step 2, carrying out real-time splicing of panoramic pictures and unity multi-shape dynamic mapping; step 3, rendering the scene object; step 4, transmitting a udp/rtsp panoramic picture; step 5, controlling visual angle pictures by head gestures, eye movements, voice and gestures; and 6, synchronously displaying the panoramic real-time picture between the final devices and cooperatively controlling. The method can transmit the panoramic picture collected by the equipment to the mobile terminal AR glasses in real time and synchronously display the panoramic picture, can effectively acquire the real-time panoramic picture transmitted from a vehicle or other positions even if a user is separated from the panoramic equipment, can finish the observation of different directions of the panorama on the basis of head posture control, can acquire related data information under the current environment form to the maximum extent, and is cooperatively commanded and controlled by combining various intelligent interaction modes (eye movement, voice and gestures).)

1. A method for real-time transmission and monitoring of a panoramic display system on AR glasses is characterized by comprising the following steps:

step 1, installing a camera;

step 2, carrying out real-time splicing of panoramic pictures and unity multi-shape dynamic mapping;

step 3, rendering the scene object;

step 4, transmitting a UDP/RTSP panoramic picture;

step 5, controlling visual angle pictures by head gestures, eye movements, voice and gestures;

and 6, synchronously displaying the panoramic real-time picture between the final devices and cooperatively controlling.

2. The method for real-time transmission and monitoring of the panoramic display system on the AR glasses according to claim 1, wherein the step 1 comprises: installing a plurality of paths of cameras on the equipment, arranging the cameras at corresponding positions on the equipment, and finishing basic panoramic picture display by utilizing a real-time splicing and fusion algorithm; the installation mode of the cameras is set to be arc-shaped arrangement or hash arrangement, and the multi-path cameras are 4-path, 8-path, 16-path or 32-path.

3. The method for real-time transmission and monitoring of the panoramic display system on the AR glasses according to claim 1, wherein the step 2 comprises: accessing the real-time panoramic display picture installed in the step 1 through unity3d software, wherein when the panoramic display picture is accessed, a display object can be compatible with a plurality of models to be used as a mapping object; using the center points of the multiple model types as the center points of the visual field, and giving panoramic shaders to the objects of the multiple models; the plurality of models comprise cubes, spheres, cylinders and capsule bodies.

4. The method for real-time transmission and monitoring of the panoramic display system on the AR glasses according to claim 1, wherein the step 3 comprises: the scene object is rendered on the basis of the spherical harmonic illumination technology, and the dynamic video is pasted on the sphere and used for solving the problems of bandwidth transmission and image rendering.

5. The method for real-time transmission and monitoring of the panoramic display system on the AR glasses according to claim 1, wherein the step 4 comprises: and initializing position coordinates, transmitting a real-time panoramic picture to an AR (augmented reality) glasses end in a UDP (user datagram protocol) mode, and if the transmitted video frame is too large, compressing the transmitted video frame for improving the transmission speed.

6. The method for real-time transmission and monitoring of a panoramic display system on AR glasses according to claim 1, wherein step 5 comprises: the gyroscope at the AR glasses end is used for realizing that the omnibearing panoramic picture end moves along with the posture of the head, if the head is leftward, the picture moves leftward along with the head, and if the head is rightward, the picture moves rightward along with the head, so that the blind area of the visual field is solved.

Technical Field

The invention relates to the technical field of panoramic display, in particular to real-time transmission and monitoring of a panoramic display system on AR glasses.

Background

With the continuous development of industrial technology, computer technology, network technology and the like, products gradually change to an intelligent mode, so that great changes occur in various fields. Panorama has been used in a plurality of application fields as a new display technology. The panoramic display equipment is arranged on the automobile, so that a blind spot area of the automobile can be observed, and the occurrence of traffic accidents is effectively reduced; the panoramic equipment has an important function in monitoring, and the panoramic equipment effectively monitors a plurality of areas in real time due to the fact that the panoramic equipment has the characteristic of wide image collecting visual field. Even if the panoramic display equipment has so many advantages, the panoramic display equipment has own defects, a panoramic display system loaded in the common equipment can only display on a fixed display, does not have mobility, has poor maneuvering performance, cannot be well combined with AR glasses, and cannot ensure that the system can stably, effectively and real-timely transmit a panoramic video picture to the AR glasses and control the panoramic video picture through the head posture even if relevant attempts are made.

Therefore, it is urgently needed to provide a method for stably and effectively transmitting a panoramic real-time picture to a mobile terminal of AR glasses and observing various orientations of the panoramic picture through head pose estimation to solve the above technical problems.

Disclosure of Invention

The invention aims to provide a real-time transmission and monitoring method of a panoramic display system on AR glasses, which can transmit a panoramic picture acquired by equipment to AR glasses at a mobile terminal in real time and synchronously display the panoramic picture, can effectively acquire the real-time panoramic picture transmitted from a vehicle or other positions even if a user is separated from the panoramic equipment, can finish the observation of the panorama in different directions on the basis of head posture control, can acquire related data information in the current environment mode to the maximum extent, and is cooperatively commanded and controlled by combining various intelligent interaction modes (eye movement, voice and gestures).

In order to achieve the above object, the present invention provides a method for real-time transmission and monitoring of a panoramic display system on AR glasses, comprising:

step 1, installing a camera;

step 2, carrying out real-time splicing of panoramic pictures and unity multi-shape dynamic mapping;

step 3, rendering the scene object;

step 4, transmitting a UDP/RTSP panoramic picture;

step 5, controlling visual angle pictures by head gestures, eye movements, voice and gestures;

and 6, synchronously displaying the panoramic real-time picture between the final devices and cooperatively controlling.

Preferably, step 1 comprises: installing a plurality of paths of cameras on the equipment, arranging the cameras at corresponding positions on the equipment, and finishing basic panoramic picture display by utilizing a real-time splicing and fusion algorithm; the installation mode of the cameras is set to be arc-shaped arrangement or hash arrangement, and the multi-path cameras are 4-path, 8-path, 16-path or 32-path.

Preferably, step 2 comprises: accessing the real-time panoramic display picture installed in the step 1 through unity3d software, wherein when the panoramic display picture is accessed, a display object can be compatible with a plurality of models to be used as a mapping object; using the center points of the multiple model types as the center points of the visual field, and giving panoramic shaders to the objects of the multiple models; the plurality of models comprise cubes, spheres, cylinders and capsule bodies.

Preferably, step 3 comprises: the scene object is rendered on the basis of the spherical harmonic illumination technology, and the dynamic video is pasted on the sphere and used for solving the problems of bandwidth transmission and image rendering.

Preferably, step 4 comprises: and initializing position coordinates, transmitting a real-time panoramic picture to an AR (augmented reality) glasses end in a UDP (user datagram protocol) mode, and if the transmitted video frame is too large, compressing the transmitted video frame for improving the transmission speed.

Preferably, step 5 comprises: the gyroscope at the AR glasses end is used for realizing that the omnibearing panoramic picture end moves along with the posture of the head, if the head is leftward, the picture moves leftward along with the head, and if the head is rightward, the picture moves rightward along with the head, so that the blind area of the visual field is solved.

According to the technical scheme, the problems of illumination rendering effect and bandwidth transmission are solved by using a mapping technology of spherical harmonic illumination and panoramic dynamic video, real-time dynamic panoramic pictures collected by equipment are displayed in AR glasses in a UDP transmission mode, the scene is subjected to visual angle control by using the head posture, intelligent interactive cooperative command control is performed, and the problems of synchronous display and communication transmission of the real-time pictures in the current environment are solved. Based on the characteristics, the user can monitor and observe the situation of the whole environment in real time only by means of the AR glasses, and the working efficiency is improved.

Additional features and advantages of the invention will be set forth in the detailed description which follows.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:

FIG. 1 is a flow chart of a method for real-time transmission and monitoring of a panoramic display system on AR glasses according to the present invention;

FIG. 2 is a schematic view of an arc arrangement of the present invention;

FIG. 3 is a schematic illustration of a tandem arrangement of the present invention;

FIG. 4 is a diagram of a sphere model according to the present invention;

FIG. 5 is a diagram illustrating an acceleration process of data transmission/reception according to the present invention;

FIG. 6 is a schematic diagram of head pose control in the present invention;

fig. 7 is a block diagram of the cooperative usage flow in the present invention.

Detailed Description

The following detailed description of embodiments of the invention refers to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present invention, are given by way of illustration and explanation only, not limitation.

In the present invention, unless otherwise specified, the directional words "left, right" and the like included in the terms merely represent the orientation of the terms in the conventional use state or are colloquially known by those skilled in the art, and should not be construed as limiting the terms.

Referring to fig. 1, the present invention provides a real-time transmission and monitoring method of a panoramic display system on AR glasses, comprising:

step 1, installing a camera;

step 2, carrying out real-time splicing of panoramic pictures and unity multi-shape dynamic mapping;

step 3, rendering the scene object;

step 4, transmitting a UDP/RTSP panoramic picture;

step 5, controlling visual angle pictures by head gestures, eye movements, voice and gestures;

and 6, synchronously displaying the panoramic real-time picture between the final devices and cooperatively controlling.

In the present embodiment, step 1 includes: installing a plurality of paths of cameras on the equipment, arranging the cameras at corresponding positions on the equipment, and finishing basic panoramic picture display by utilizing a real-time splicing and fusion algorithm; the installation mode of the cameras is set to be arc-shaped arrangement or hash arrangement (as shown in fig. 2 and 3), and the multi-path cameras are 4-path, 8-path, 16-path or 32-path.

In the actual installation process, the installation of the camera is carried out according to the actual conditions, the real-time panoramic video is collected and displayed at the terminal.

The step 2 comprises the following steps: accessing the real-time panoramic display picture installed in the step 1 through unity3d software, wherein when the panoramic display picture is accessed, a display object can be compatible with a plurality of models to be used as a mapping object; using the center points of the multiple model types as the center points of the visual field, and giving panoramic shaders to the objects of the multiple models; the plurality of models comprise cubes, spheres, cylinders and capsule bodies. As shown in fig. 4, taking a sphere as an example (the total number of vertices and the total number of triangular faces rendered by the sphere are 4167 and 8192 respectively), the definition of the panoramic image can be improved by using high-precision models for all of the plurality of model categories.

In order to make the illumination effect more real, a common spherical harmonic illumination technology is adopted, the technology projects a complex illumination signal onto a basis function for storage, then data on the basis function is added up to reconstruct the illumination signal when in use, but the whole preprocessing of scene subdivision granularity consumes a large amount of time and is greatly limited to bandwidth transmission, but the time efficiency is greatly improved if a panoramic dynamic video is seamlessly pasted on a 3d model, so that the spherical harmonic illumination and the dynamic video pasting are combined on the basis of the method, the rendering effect is improved, the time efficiency is improved, and too many memory resources are not occupied. Thus, step 3 includes: the scene object is rendered on the basis of the spherical harmonic illumination technology, and the dynamic video is pasted on the sphere and used for solving the problems of bandwidth transmission and image rendering.

Step 4 comprises the following steps: and initializing position coordinates, transmitting a real-time panoramic picture to an AR (augmented reality) glasses end in a UDP (user datagram protocol) mode, and if the transmitted video frame is too large, compressing the transmitted video frame for improving the transmission speed.

Specifically, a coordinate position is initialized to ensure that a real-time panoramic picture acquired by each frame of equipment is consistent with a picture at an AR glasses end, after the initialization operation is completed, a real-time panoramic video acquired by the equipment is transmitted to the AR glasses end for display in a UDP (user datagram protocol) transmission mode, the transmission speed and the frame loss condition are influenced by the fact that each frame (about 32M) of panoramic transmission is large, picture frame compression and queue buffer are adopted for transmission, the width and the height of a transmitted picture frame are respectively set to be width and height, a scaling variable T (positive integer) is set, the width and the height of the compressed picture frame are respectively set to be width/T, height/T, the variable T is scaled according to the actual condition, and if the delay is low, the variable T can be set to be small; if the delay is large, the variable T may be set to be large, and the frame image after transmission compression may be placed in the queue buffer, as shown in fig. 5.

As shown in fig. 6, step 5 includes: the gyroscope at the AR glasses end is used for realizing that the omnibearing panoramic picture end moves along with the posture of the head, if the head is leftward, the picture moves leftward along with the head, and if the head is rightward, the picture moves rightward along with the head, so that the blind area of the visual field is solved.

The panoramic equipment and the display picture of the mobile terminal user are cooperatively commanded, when the equipment with the panoramic carrier cannot reach a certain target in the panoramic display picture, people wearing the AR glasses of the mobile terminal can be rapidly commanded to go to investigation, information is fed back in real time, intelligent interaction is carried out in the modes of eye movement, voice and gestures, and the whole actual environment condition can be effectively decided, and particularly as shown in fig. 7.

According to the technical scheme, the problems of illumination rendering effect and bandwidth transmission are solved by using a mapping technology of spherical harmonic illumination and panoramic dynamic video, real-time dynamic panoramic pictures collected by equipment are displayed in AR glasses in a UDP transmission mode, the visual angle of a scene is controlled by using the head posture, intelligent interactive cooperative command control is realized, and the problems of synchronous display and communication transmission of the real-time pictures in the current environment are solved. Based on the characteristics, the user can monitor and observe the situation of the whole environment in real time only by means of the AR glasses, and the working efficiency is improved.

The preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, however, the present invention is not limited to the specific details of the above embodiments, and various simple modifications can be made to the technical solution of the present invention within the technical idea of the present invention, and these simple modifications are within the protective scope of the present invention.

It should be noted that the various technical features described in the above embodiments can be combined in any suitable manner without contradiction, and the invention is not described in any way for the possible combinations in order to avoid unnecessary repetition.

In addition, any combination of the various embodiments of the present invention is also possible, and the same should be considered as the disclosure of the present invention as long as it does not depart from the spirit of the present invention.