阅读说明：本技术 一种视频融合方法 (Video fusion method ) 是由 邓子超 张卫 闫勇 赵常 栗健 陈�胜 程启烽 于 2020-12-15 设计创作，主要内容包括：本发明属于虚实融合视频监控相关技术领域,具体涉及一种视频融合方法,其包括：步骤1：三维建模；步骤2：效果调整；步骤3：视频流处理；步骤4：视频畸形矫正；步骤5：视频投射；本发明的视频融合技术,通过对监控视频数据流提取视频帧并将其投射到虚拟场景里,实现视频数据与虚拟场景数据的全时空立体融合,改变了地图应用只能静态展示的传统模式,将部署在不同地理位置的多路实时监控视频与监控区域的三维模型进行配准融合,生成大范围三维全景动态监控画面,实现监控区域整体安全态势的实时全局掌控。(The invention belongs to the technical field related to virtual and real fusion video monitoring, and particularly relates to a video fusion method, which comprises the following steps: step 1: three-dimensional modeling; step 2: adjusting the effect; and step 3: processing a video stream; and 4, step 4: correcting video deformity; and 5: video projection; according to the video fusion technology, the video frames are extracted from the monitoring video data stream and projected into the virtual scene, so that full-time-space three-dimensional fusion of the video data and the virtual scene data is realized, the traditional mode that map application can only be statically displayed is changed, the multi-channel real-time monitoring videos deployed at different geographic positions are registered and fused with the three-dimensional model of the monitoring area, a large-range three-dimensional panoramic dynamic monitoring picture is generated, and real-time global control of the whole safety situation of the monitoring area is realized.)

1. A video fusion method, characterized in that the video fusion method comprises the following steps:

step 1: three-dimensional modeling;

the method for modeling by aerial photography oblique photography of the unmanned aerial vehicle is adopted, the digital photo of a modeling area of a target is shot by carrying a five-mesh digital camera through the unmanned aerial vehicle, and the digital photo is converted into a three-dimensional model by using a three-dimensional model generation tool;

step 2: adjusting the effect;

loading the three-dimensional model to a geographic information system, and adjusting the three-dimensional model, wherein the three-dimensional model comprises model simplification, three-dimensional coordinate setting, position adjustment and scale adjustment, so as to build a virtual scene similar to reality;

and step 3: processing a video stream;

the method comprises the steps that RTSP video streams are pulled from a video monitoring camera and a video server device, and are converted into video streams supporting an Html5 protocol through an H5Stream service;

and 4, step 4: correcting video deformity;

adjusting a video monitoring picture of a video code stream with wide-angle parameters and barrel distortion by adopting a camera distortion correction tool in an OPENCV algorithm library and a checkerboard picture correction parameter method, cutting and intercepting an effective visual field of the video monitoring picture, and matching the video monitoring picture with a space position in a virtual scene;

and 5: video projection;

constructing a polygonal plane as a video projection carrier on a corresponding coordinate point in a virtual scene by referring to an effective visual field of a video monitoring picture, and then loading a video code stream supporting an Html5 protocol as a material of the polygonal plane; when the video is projected, multiple paths of independent videos need to be spliced to form an integral scene, and adjacent polygon blocks are subjected to position and size adjustment until edge areas of the adjacent polygon blocks are continuous without gaps.

2. The video fusion method of claim 1, wherein the three-dimensional modeling process of step 1 comprises the steps of:

step 11: planning a modeling area, sorting coordinate information, planning a horizontal flight route of the unmanned aerial vehicle, designating flight height, and covering the modeling area by the flight area;

step 12: controlling an unmanned aerial vehicle to carry a five-mesh digital camera to fly according to the flying route in the step 11, and periodically and continuously shooting ground photos by using the five-mesh digital camera;

step 13: uniformly importing the ground photos shot in the step 12 into a three-dimensional model generation tool to generate a three-dimensional model of oblique photography;

step 14: and (4) checking the three-dimensional model of the oblique photography in the step (13), and performing complementary shooting on the live-action photos in the key area to perform fine correction on the material and texture of the model.

3. The video fusion method of claim 2 wherein the three-dimensional model generation tool is a Smart3D software tool.

4. The video fusion method of claim 2 wherein the modeled area includes buildings, roads, greenery, stationary facilities.

5. The video fusion method of claim 2 wherein said taking ground photos is in JPG format.

6. The video fusion method of claim 2, wherein the effect adjustment process of step 2 comprises the steps of:

step 21: selecting a Cecum digital earth engine supporting WebGL, and building a GIS geographic information system environment;

step 22: loading the satellite front-view image data to the GIS (geographic information system) in the step 21 to serve as a basic layer to form a digital earth scene;

step 23: converting the three-dimensional model data of oblique photography output in the step 14 into an Obj file format, and loading the three-dimensional model data into the digital earth scene in the step 22; adjusting the space coordinate parameters of the three-dimensional model until the space position of the three-dimensional model is matched with the satellite front-view image in the step 22 in a comparison manner; and adjusting the scale of the three-dimensional model to be scaled in an equal proportion until the size of the three-dimensional model is close to the real scene.

7. The video fusion method of claim 6 wherein the GIS geographic information system employs an open source Cesium platform.

8. The video fusion method of claim 3, wherein the video stream processing procedure of step 3 comprises the steps of:

step 31: installing cameras, and configuring an RTSP main code stream address of each camera;

step 32: configuring the RTSP main code Stream address of each camera in the step 31 into an H5Stream service configuration file;

step 33: starting an H5Stream service to convert the RTSP video Stream into a video Stream supporting an Html5 protocol, generating an Html5 video Stream address of each camera, and simultaneously accessing a plurality of paths of video Stream preview pictures by using a browser.

9. The video fusion method of claim 8 wherein the effect adjustment process of step 4 comprises the steps of:

step 41: configuring the Html5 video stream address of the video camera of step 33 to the video stream input end of the camera distortion correction tool in the OPENCV algorithm library;

step 42: placing the printed 8X 16 black-white checkerboard A3 picture into a video picture to be corrected, and calibrating intersection points of each unit cell of the checkerboard;

step 43: according to the cell intersection points calibrated in the step 42, through uniformly stretching the video monitoring picture, the calibration points corresponding to the cell intersection points are restored to be a matrix distributed at equal intervals, and correction of barrel distortion of the video stream is realized;

step 44: and a rectangular tool clipping step 43 is used for clipping the distorted and corrected video monitoring picture to generate a video code stream output address of the effective view.

10. The video fusion method of claim 9 wherein the video projection process of step 5 comprises the steps of:

step 51: creating a polygonal plane in the digital earth scene loaded with the three-dimensional model in step 23, and adjusting the size and position of the polygonal plane with reference to the video monitoring picture of the effective field in step 44; adjusting the height of the polygon plane to match the spatial position of the three-dimensional model of the corresponding region;

step 52: setting the material parameters of the polygonal plane, and loading the video monitoring picture of the effective visual field in the step 44 as the skin material of the polygonal plane;

step 53: and adjusting the positions and the sizes of the adjacent polygonal blocks until the edge areas of the adjacent polygonal blocks are continuous without gaps, and splicing the polygons to form a continuous large-scene video monitoring picture.

Technical Field

The invention belongs to the technical field related to virtual and real fusion video monitoring, and particularly relates to a video fusion method.

Background

At present, the application of the traditional video monitoring technology mainly takes a planar matrix type video monitoring picture as a main part, and with the development and popularization and application of a Web information system technology, a GIS geographic information system technology, an H5 video streaming technology and a three-dimensional modeling technology, users put higher requirements on the implementation means of video monitoring, and hope to realize an innovative application mode combining video monitoring and a virtual space scene.

Disclosure of Invention

Technical problem to be solved

The technical problem to be solved by the invention is as follows: how to put forward a video monitoring scheme based on the combination of video data of a Web information system and virtual scenes of a GIS (geographic information system) to fuse video monitoring pictures and the virtual scenes, can view the whole situation and reduce the complexity of monitoring operation.

(II) technical scheme

In order to solve the above technical problem, the present invention provides a video fusion method, which includes the following steps:

step 1: three-dimensional modeling;

the method for modeling by aerial photography oblique photography of the unmanned aerial vehicle is adopted, the digital photo of a modeling area of a target is shot by carrying a five-mesh digital camera through the unmanned aerial vehicle, and the digital photo is converted into a three-dimensional model by using a three-dimensional model generation tool;

step 2: adjusting the effect;

loading the three-dimensional model to a geographic information system, and adjusting the three-dimensional model, wherein the three-dimensional model comprises model simplification, three-dimensional coordinate setting, position adjustment and scale adjustment, so as to build a virtual scene similar to reality;

and step 3: processing a video stream;

the method comprises the steps that RTSP video streams are pulled from a video monitoring camera and a video server device, and are converted into video streams supporting an Html5 protocol through an H5Stream service;

and 4, step 4: correcting video deformity;

adjusting a video monitoring picture of a video code stream with wide-angle parameters and barrel distortion by adopting a camera distortion correction tool in an OPENCV algorithm library and a checkerboard picture correction parameter method, cutting and intercepting an effective visual field of the video monitoring picture, and matching the video monitoring picture with a space position in a virtual scene;

and 5: video projection;

constructing a polygonal plane as a video projection carrier on a corresponding coordinate point in a virtual scene by referring to an effective visual field of a video monitoring picture, and then loading a video code stream supporting an Html5 protocol as a material of the polygonal plane; when the video is projected, multiple paths of independent videos need to be spliced to form an integral scene, and adjacent polygon blocks are subjected to position and size adjustment until edge areas of the adjacent polygon blocks are continuous without gaps.

Wherein, the three-dimensional modeling process of the step 1 comprises the following steps:

step 11: planning a modeling area, sorting coordinate information, planning a horizontal flight route of the unmanned aerial vehicle, designating flight height, and covering the modeling area by the flight area;

step 12: controlling an unmanned aerial vehicle to carry a five-mesh digital camera to fly according to the flying route in the step 11, and periodically and continuously shooting ground photos by using the five-mesh digital camera;

step 13: uniformly importing the ground photos shot in the step 12 into a three-dimensional model generation tool to generate a three-dimensional model of oblique photography;

step 14: and (4) checking the three-dimensional model of the oblique photography in the step (13), and performing complementary shooting on the live-action photos in the key area to perform fine correction on the material and texture of the model.

Wherein the three-dimensional model generation tool is a Smart3D software tool.

Wherein the modeling area comprises buildings, roads, green plants and fixed facilities.

Wherein, the ground photo is in a JPG format.

Wherein, the effect adjusting process of step 2 comprises the following steps:

step 21: selecting a Cecum digital earth engine supporting WebGL, and building a GIS geographic information system environment;

step 22: loading the satellite front-view image data to the GIS (geographic information system) in the step 21 to serve as a basic layer to form a digital earth scene;

step 23: converting the three-dimensional model data of oblique photography output in the step 14 into an Obj file format, and loading the three-dimensional model data into the digital earth scene in the step 22; adjusting the space coordinate parameters of the three-dimensional model until the space position of the three-dimensional model is matched with the satellite front-view image in the step 22 in a comparison manner; and adjusting the scale of the three-dimensional model to be scaled in an equal proportion until the size of the three-dimensional model is close to the real scene.

The GIS geographic information system adopts an open source Cesium platform.

Wherein, the video stream processing procedure of step 3 includes the following steps:

step 31: installing cameras, and configuring an RTSP main code stream address of each camera;

step 32: configuring the RTSP main code Stream address of each camera in the step 31 into an H5Stream service configuration file;

step 33: starting an H5Stream service to convert the RTSP video Stream into a video Stream supporting an Html5 protocol, generating an Html5 video Stream address of each camera, and simultaneously accessing a plurality of paths of video Stream preview pictures by using a browser.

Wherein, the effect adjusting process of the step 4 comprises the following steps:

step 41: configuring the Html5 video stream address of the video camera of step 33 to the video stream input end of the camera distortion correction tool in the OPENCV algorithm library;

step 42: placing the printed 8X 16 black-white checkerboard A3 picture into a video picture to be corrected, and calibrating intersection points of each unit cell of the checkerboard;

step 43: according to the cell intersection points calibrated in the step 42, through uniformly stretching the video monitoring picture, the calibration points corresponding to the cell intersection points are restored to be a matrix distributed at equal intervals, and correction of barrel distortion of the video stream is realized;

step 44: and a rectangular tool clipping step 43 is used for clipping the distorted and corrected video monitoring picture to generate a video code stream output address of the effective view.

Wherein, the video projection process of step 5 comprises the following steps:

step 51: creating a polygonal plane in the digital earth scene loaded with the three-dimensional model in step 23, and adjusting the size and position of the polygonal plane with reference to the video monitoring picture of the effective field in step 44; adjusting the height of the polygon plane to match the spatial position of the three-dimensional model of the corresponding region;

step 52: setting the material parameters of the polygonal plane, and loading the video monitoring picture of the effective visual field in the step 44 as the skin material of the polygonal plane;

step 53: and adjusting the positions and the sizes of the adjacent polygonal blocks until the edge areas of the adjacent polygonal blocks are continuous without gaps, and splicing the polygons to form a continuous large-scene video monitoring picture.

(III) advantageous effects

Compared with the prior art, the invention provides a video monitoring scheme combining video data based on a Web information system and a GIS (geographic information system) geographic information system virtual scene, so that a video monitoring picture and the virtual scene are fused, the overall situation can be seen, and the monitoring operation complexity is reduced.

The technical scheme of the invention is that a three-dimensional modeling is carried out on buildings, roads, green plants, fixed facilities and the like in a management area based on a geographic information system to form a three-dimensional model; putting the three-dimensional model into a geographic information system to adjust the model effect to form a virtual scene; the method comprises the steps that RTSP video streams are pulled from devices such as a video monitoring camera and a video server, and the RTSP video streams are converted into video code streams supporting an Html5 protocol; correcting the video with wide-angle parameters and barrel distortion into a corrected video matched with a spatial position in a high-precision manner; and projecting the corrected video to a virtual scene to form a virtual-real fusion scene. Meanwhile, the video fusion technology adopted by the invention extracts the video frame from the monitoring video data stream and projects the video frame into the virtual scene, so that full-time-space three-dimensional fusion of the video data and the virtual scene data is realized, the traditional mode that map application can only be statically displayed is changed, the multi-channel real-time monitoring videos deployed at different geographic positions are registered and fused with the three-dimensional model of the monitoring area, a large-range three-dimensional panoramic dynamic monitoring picture is generated, and the real-time global control of the overall security situation of the monitoring area is realized.

Drawings

Fig. 1 is a technical processing flow chart in the technical scheme of the invention.

Detailed Description

In order to make the objects, contents, and advantages of the present invention clearer, the following detailed description of the embodiments of the present invention will be made in conjunction with the accompanying drawings and examples.

In order to solve the problems in the prior art, the invention provides a video fusion method, which is applied to users with higher requirements on video monitoring safety and protection, such as military colleges, army bases, scientific research institutions and the like, and as shown in fig. 1, the video fusion method comprises the following steps:

step 1: three-dimensional modeling;

the method for modeling by aerial photography oblique photography of the unmanned aerial vehicle is adopted, the digital photo of a modeling area of a target is shot by carrying a five-mesh digital camera through the unmanned aerial vehicle, and the digital photo is converted into a three-dimensional model by using a three-dimensional model generation tool;

step 2: adjusting the effect;

loading the three-dimensional model to a geographic information system, and adjusting the three-dimensional model, wherein the three-dimensional model comprises model simplification, three-dimensional coordinate setting, position adjustment and scale adjustment, so as to build a virtual scene similar to reality;

and step 3: processing a video stream;

the method comprises the steps that RTSP video streams are pulled from a video monitoring camera and a video server device, and are converted into video streams supporting an Html5 protocol through an H5Stream service;

and 4, step 4: correcting video deformity;

adjusting a video monitoring picture of a video code stream with wide-angle parameters and barrel distortion by using a camera distortion correction tool in an OPENCV algorithm library and a checkerboard picture correction parameter method, cutting and intercepting an effective visual field of the video monitoring picture, and performing high-precision matching on the video monitoring picture and a spatial position in a virtual scene;

and 5: video projection;

constructing a polygonal plane as a video projection carrier on a corresponding coordinate point in a virtual scene by referring to an effective visual field of a video monitoring picture, and then loading a video code stream supporting an Html5 protocol as a material of the polygonal plane; when the video is projected, multiple paths of independent videos need to be spliced to form an integral scene, and adjacent polygon blocks are subjected to position and size adjustment until edge areas of the adjacent polygon blocks are continuous without gaps.

Wherein, the three-dimensional modeling process of the step 1 comprises the following steps:

step 11: planning a modeling area, sorting coordinate information, planning a horizontal flight route of the unmanned aerial vehicle, designating flight height, and covering the modeling area by the flight area;

step 12: controlling an unmanned aerial vehicle to carry a five-mesh digital camera to fly according to the flying route in the step 11, and periodically and continuously shooting ground photos by using the five-mesh digital camera;

step 13: uniformly importing the ground photos shot in the step 12 into a three-dimensional model generation tool to generate a three-dimensional model of oblique photography;

step 14: and (4) checking the three-dimensional model of the oblique photography in the step (13), and performing complementary shooting on the live-action photos in the key area to perform fine correction on the material and texture of the model.

Wherein the three-dimensional model generation tool is a Smart3D software tool.

Wherein the modeling area comprises buildings, roads, green plants and fixed facilities.

Wherein, the ground photo is in a JPG format.

The source file of the three-dimensional model of oblique photography is in osgb format, and the target file is in obj format.

Wherein, the effect adjusting process of step 2 comprises the following steps:

step 21: selecting a Cecum digital earth engine supporting WebGL, and building a GIS geographic information system environment;

step 22: loading the satellite front-view image data to the GIS (geographic information system) in the step 21 to serve as a basic layer to form a digital earth scene;

step 23: converting the three-dimensional model data of oblique photography output in the step 14 into an Obj file format, and loading the three-dimensional model data into the digital earth scene in the step 22; adjusting the space coordinate parameters of the three-dimensional model until the space position of the three-dimensional model is matched with the satellite front-view image in the step 22 in a comparison manner; and adjusting the scale of the three-dimensional model to be scaled in an equal proportion until the size of the three-dimensional model is close to the real scene.

The GIS geographic information system adopts an open source Cesium platform.

Wherein, the video stream processing procedure of step 3 includes the following steps:

step 31: installing cameras, and configuring an RTSP main code stream address of each camera;

step 32: configuring the RTSP main code Stream address of each camera in the step 31 into an H5Stream service configuration file;

step 33: starting an H5Stream service to convert the RTSP video Stream into a video Stream supporting an Html5 protocol, generating an Html5 video Stream address of each camera, and simultaneously accessing a plurality of paths of video Stream preview pictures by using a browser.

Wherein, the effect adjusting process of the step 4 comprises the following steps:

step 41: configuring the Html5 video stream address of the video camera of step 33 to the video stream input end of the camera distortion correction tool in the OPENCV algorithm library;

step 42: placing the printed 8X 16 black-white checkerboard A3 picture into a video picture to be corrected, and calibrating intersection points of each unit cell of the checkerboard;

step 43: according to the cell intersection points calibrated in the step 42, through uniformly stretching the video monitoring picture, the calibration points corresponding to the cell intersection points are restored to be a matrix distributed at equal intervals, and correction of barrel distortion of the video stream is realized;

step 44: and a rectangular tool clipping step 43 is used for clipping the distorted and corrected video monitoring picture to generate a video code stream output address of the effective view.

Wherein, the video projection process of step 5 comprises the following steps:

step 51: creating a polygonal plane in the digital earth scene loaded with the three-dimensional model in step 23, and adjusting the size and position of the polygonal plane with reference to the video monitoring picture of the effective field in step 44; adjusting the height of the polygon plane to match the spatial position of the three-dimensional model of the corresponding region;

step 52: setting the material parameters of the polygonal plane, and loading the video monitoring picture of the effective visual field in the step 44 as the skin material of the polygonal plane;

step 53: and adjusting the positions and the sizes of the adjacent polygonal blocks until the edge areas of the adjacent polygonal blocks are continuous without gaps, and splicing the polygons to form a continuous large-scene video monitoring picture.

In summary, the present invention provides a video fusion method, which is applied to users with high requirements for video monitoring security in military colleges, army bases, scientific research institutions, etc., and provides a design scheme for the requirements for video monitoring of security in large public areas, closed places, campus camps, etc. A GIS system based on a WebGL rendering engine is adopted as a bottom layer to construct a virtual-real combined video fusion monitoring system, and the system design method has high advancement and feasibility.

Example 1

The present embodiment includes:

(1) three-dimensional modeling

Planning the flight route and the flight height of the unmanned aerial vehicle, covering a modeling target area, controlling the unmanned aerial vehicle to fly along the planning route and shooting pictures by adopting an onboard five-eye camera according to the set shooting frequency. After the flight is finished, the photos are completely imported into SMART3D software for data processing and model generation. And (4) previewing the scene after the data generation is successful, performing on-site rephotography on the key area by using a digital camera, and performing material correction on the model by adopting 3 DMAX. And exporting to an obj format file.

(2) Effect adjustment

And loading the obj file of the three-dimensional model to a geographic information system, adjusting the parameter information of the three-dimensional model, including three-dimensional coordinate setting, position adjustment, scale adjustment, texture adjustment, model simplification and the like, wherein the model is attached to the position of the front view image layer of the satellite, and the height of the model is attached to the real terrain so as to build a real virtual scene.

(3) Video stream processing

The RTSP video stream is pulled from equipment such as a video monitoring camera and a video server, and video stream configuration information including an IP address, a login user password, a channel number and the like is recorded; and filling the video Stream information into a conf configuration file of the H5Stream service according to a specified format, starting the H5Stream service to access a video picture through a browser, converting the RTSP video Stream into a video Stream supporting the Html5 protocol, and recording the address of the converted H5 video Stream.

(4) Video deformity correction

Configuring an H5 video stream address into an OPENCV camera distortion correction program, adjusting a monitoring picture with wide-angle parameters and barrel distortion by using a checkerboard picture correction parameter method, cutting and intercepting a video monitoring effective visual field, and performing high-precision matching on the video picture and a space position in a virtual scene.

(5) Video projection

And constructing a polygonal plane as a video projection carrier on a corresponding coordinate point in a virtual scene by referring to the effective visual field of the video monitoring picture, and then loading a video code stream supporting an Html5 protocol as a polygonal material. When the video is projected, multiple paths of independent videos need to be spliced to form an integral scene, and splicing overlapping areas of adjacent polygonal blocks need to be flexibly configured and adjusted.

The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.