阅读说明：本技术 模糊边缘驱动的单目视频虚拟视图三维模糊表面合成方法 (Three-dimensional fuzzy surface synthesis method for monocular video virtual view driven by fuzzy edge ) 是由 韦健 王世刚 宋晨曦 赵岩 于 2020-12-09 设计创作，主要内容包括：模糊边缘驱动的单目视频虚拟视图三维模糊表面合成方法属自由立体显示技术领域,本发明包括：1.真实视图模糊边缘重建,为一组下采样视频帧重建稀疏的模糊边缘；2.通过真实视图重建结果的视点间加权求和,为已知相机位姿的虚拟视图合成模糊边缘；3.采用基于局部平滑与边缘尖锐约束的空域插值,为虚拟视图生成完整模糊表面,利用全部稀疏采样帧的模糊边缘,基于全局可见性约束,去除造成错误遮挡的模糊表面；4.利用少量稀疏采样帧的完整模糊表面,实现虚拟视图模糊表面空洞的角度域填补；本发明通过边缘驱动与模糊理论相辅相成的策略,能实现不使用给定的场景几何信息端到端的高性能、强鲁棒视图合成,为自由立体显示快速生成高品质内容。(The invention discloses a three-dimensional fuzzy surface synthesis method of a monocular video virtual view driven by fuzzy edge, belonging to the technical field of free stereo display, and the method comprises the following steps: 1. real view blurred edge reconstruction, reconstructing sparse blurred edges for a set of down-sampled video frames; 2. synthesizing a fuzzy edge for the virtual view with the known camera pose through the weighted summation among the viewpoints of the real view reconstruction result; 3. generating a complete fuzzy surface for the virtual view by adopting spatial interpolation based on local smoothness and sharp edge constraint, and removing the fuzzy surface causing error shielding by utilizing fuzzy edges of all sparse sampling frames based on global visibility constraint; 4. filling an angle domain of a virtual view fuzzy surface hole by using a complete fuzzy surface of a small number of sparse sampling frames; the invention can realize end-to-end high-performance and strong robust view synthesis without using given scene geometric information through a strategy of complementing edge driving and fuzzy theory, and quickly generate high-quality content for free three-dimensional display.)

1. A three-dimensional fuzzy surface synthesis method of a fuzzy edge-driven monocular video virtual view is characterized by comprising the following steps:

1.1 real view blurred edge reconstruction, reconstructing sparse blurred edges, i.e. blurred voxels corresponding to edge pixels, for a set of down-sampled video frames, comprising the steps of:

1.1.1 mathematical expression of blurred voxels for pixel (x, y) is:

V(x,y)＝{d_k,μ_k(d_k)|d_k∈[d_min,d_max]}

wherein: [ d_min,d_max]The depth range is obtained by adopting linear sampling of the reciprocal of the depth value in the depth interval; mu.s_k(d_k)∈[0,1]Is a candidate depth d_kA membership function whose value is related to the depth range;

1.1.2, establishing a self-adaptive depth range selection and membership degree distribution mechanism for the fuzzy edge of the real view by a double-layer frame sampling method;

1.1.3 two-layer frame sampling procedure: firstly, calculating the camera poses of all video frames by using a motion recovery structure algorithm, and simultaneously reconstructing sparse three-dimensional point cloud with significant features in a video image; then, according to the information, the video frame is subjected to dense-to-sparse sampling: firstly, densely sampling video frames to enable the difference of camera visual angles of adjacent sampling frames to be 1 degree; further down-sampling the dense sampling frame at a fixed ratio of 4:1 to obtain a sparse sampling frame;

1.1.4 depth range selection process: for each sparse sampling frame, extracting edge pixels, and then selecting a depth range from coarse to fine for each fuzzy edge, wherein the method specifically comprises the following steps:

1.1.4.1 initialize: initializing the depth range of the fuzzy edge by utilizing the depth values of the nearest and farthest three-dimensional points which are output by the motion recovery structure algorithm and can be seen in the current sampling frame;

1.1.4.2 rough selection: calculating the matching cost of all the candidate depths by using 100 adjacent dense sampling frames of the current frame, wherein the matching cost is less than tau₁Updating the depth range of the blurred edge;

1.1.4.3 Fine pick: calculating new matching cost by using 100 adjacent sparse sampling frames of the current frame, and enabling the matching cost to be less than tau₂The minimum and maximum candidate depths of the blurred edge are used as the final depth range of the blurred edge;

step 1.1.4.2 and 1.1.4.3 adopt pixel level matching cost calculation method, threshold tau₁And τ₂Setting according to the actual reconstruction effect of a specific scene, taking the depth value with the minimum matching cost obtained finally as the rough depth of the edge, and removing the background edge near the object contour and the edge with obviously unreliable rough depth through thresholding with the minimum matching cost;

1.1.5 membership assignment procedure defines the membership function for the fuzzy edge candidate depth as:

μ_k(d_k)＝(1-S_k)·(1+R_k/6)

S_k∈[0,1]for the matching costs, R, obtained in the depth range fine selection step_k∈[-6,6]The calculation formula is as follows:

wherein: d_nEdge coarse depth maps for 6 views randomly extracted from 100 adjacent sparse sampled frames of the current frame; B. a and C each represent d_kReconstructed voxels P and D_nWhether three relationships are generated between: consistent visibility, conflict in free space,If the shielding is carried out, 1 is taken, otherwise, 0 is taken;

1.2 virtual view blurred edge synthesis: synthesizing a fuzzy edge for the virtual view with the known camera pose by weighting and summing the inter-view points of the real view reconstruction result, comprising the following steps:

1.2.1 reference view selection: selecting M (more than or equal to 2) real views with the nearest camera positions and the smallest visual angle difference as interpolation reference images for the virtual views from the sparse sampling frames; the M value is set according to the actual situation so as to achieve good compromise between the operation speed and the synthesis effect; suppose a virtual view I_vHas an optical center of O_vReference to a view I_cMiddle pixel (x, y) pair I_vThe interpolation weight of (d) is defined as:

wherein: b is the average baseline of adjacent sparse sampling frames; ray_cIs I_cThe light of (2); dist is a function of the distance of the point line;

1.2.2 blurred edge synthesis: firstly synthesizing fuzzy voxels for the candidate edges, and then extracting the fuzzy edges from the fuzzy voxels, specifically comprising the following steps:

1.2.2.1 candidate edge-blurred voxel synthesis process: projecting the fuzzy edge of each reference view into a virtual view, and marking all projection points as candidate edges; for each candidate edge (x, y), projecting all reference view blurred edges projected to the pixel onto its corresponding ray, taking the union of all depth range projections as the depth range of the pixel blurred voxel; its blurred voxels are then calculated using the following formula:

wherein: v_cA blurred edge of a reference view that is a virtual view; t is_vAnd T_cRespectively representing the projective forward transformation of the virtual view and the reference view;

the summation process for two blurred voxels with different depth ranges is: the depth ranges of the two are expanded into a union of the two, the membership degree of the newly added candidate depth is set to be 0, the membership degrees of the same depth are added, and finally V is obtained_v(x, y) compacting, namely updating the depth range by adopting weighted summation operation;

1.2.2.2 blurred edge extraction procedure: taking the average value of all candidate depths with the maximum center-membership degree of each fuzzy voxel as the depth estimation of the corresponding candidate edge, and carrying out edge detection on the obtained semi-dense rough depth map by utilizing a Sobel operator;

1.3 virtual view blur surface generation: firstly, generating a complete fuzzy surface for a virtual view by adopting spatial interpolation based on local smoothness and sharp edge constraint; then, removing a fuzzy surface causing wrong occlusion by using fuzzy edges of all sparse sampling frames and simultaneously based on global visibility constraint; and finally, filling the angle domain of the virtual view fuzzy surface hole by using the complete fuzzy surface of a small number of sparse sampling frames, and specifically comprising the following steps of:

1.3.1 adopting a Jacobi iteration method to carry out viewpoint interpolation on the fuzzy edge of the virtual view, and simultaneously finishing edge retention, edge sharp constraint, region filling, smoothing and local smooth constraint on the fuzzy surface; without loss of generality, let V_v ^tFor the tth convolution result of the virtual view blurred surface, the iterative process is described as:

{(x_n,y_n) Represents a 4 neighborhood of pixel (x, y); δ (-) is an indicator function, for non-empty blurred voxels, its value is 1, otherwise it is 0; if V_v ^t+1In terms of depth range and membership distribution_v ^tAll the differences are smaller than a given threshold value, and the iteration is ended;

1.3.2 removing blurred surfaces causing false occlusion, comprising the steps of:

1.3.2.1 seed point selection: firstly, obtaining rough surface estimation from a fuzzy surface of a virtual view, namely extracting the centers of all fuzzy voxels; then projecting the rough surface to each sparse sampling frame in sequence; for the edge (x, y) of a sparsely sampled frame, assume its blurred voxel depth range is [ d ]_min,d_max]Its coarse depth estimate d_rThe reconstructed voxel is P_r(ii) a By P_wRepresenting the voxel projected from the virtual view to (x, y), with d_wAnd [ d ]_min',d_max']Respectively representing its depth value and depth range relative to the sample frame by dividing P_wThe blurred voxel is projected to the ray where (x, y) is located, if the following conditions are met:

will P_wThe label is a seed point of the region growing process, the fuzzy voxel which belongs to the seed point is not projected into the residual sampling frame, and the formula is as follows: beta is a fault tolerance performance control parameter;

1.3.2.2 region growth: gradually aggregating sparse seed points into several dense, independent fuzzy surfaces: for each seed point P_wTheir depth difference d in the corresponding sample frame_r-d_wProjecting the image to the light ray of the image in the virtual view, and recording the projection result as delta d; then P is added_wDepth range of the blurred voxel_min”,d_max”]Extend to [ d ]_min”-Δd,d_max”+Δd](ii) a Then smoothing the depth range of the virtual view fuzzy surface by a Jacobi iteration method; taking the fuzzy voxel set with the relative change of the depth range before the growth of the region exceeding a preset threshold value as a detected error fuzzy surface;

1.3.3 filling angular domains of virtual view fuzzy surface holes, comprising the following steps:

1.3.3.1, counting the number of seed points detected by each sparse sampling frame;

1.3.3.2, performing fuzzy surface spatial interpolation one by one according to the sequence of the number from more to less, and projecting the interpolation result to the cavity of the virtual view fuzzy surface; if the hole is completely filled, the operation ends, otherwise it continues.

Technical Field

The invention belongs to the technical field of free stereo display, relates to a monocular video-oriented virtual view drawing method, and particularly relates to a three-dimensional fuzzy surface synthesis method of a monocular video virtual view driven by fuzzy edges.

Background

The free stereo display technology is the leading-edge technology in the international display field and must gradually replace the existing plane display and vision-aided three-dimensional display media. Active exploration is carried out on free three-dimensional display at home and abroad, and remarkable progress is especially made in the aspects of improving resolution, visual angle, depth of field and the like. However, the bottleneck of relatively deficient high-quality content is not substantially solved, and the popularization of the method in various fields and the practicability of hardware equipment such as 3D mobile phones, 3D televisions, 3D large screens and the like are severely restricted. One of the main reasons is that the mainstream acquisition means for displaying content all have certain disadvantages, such as small parallax of the microlens array and the light field camera, high hardware cost of the camera array, complicated calibration, difficulty in surrounding shooting of a large scene along an acquired image sequence, limited detection range of a depth sensor, low precision and the like. With the development of high-speed imaging technology, the method for capturing monocular video without rail becomes an effective way for rapidly acquiring full-surface multi-scale information of a scene. Generating a virtual viewpoint Image for a video scene generally requires estimating a scene depth through Multi-view three-dimensional reconstruction (MVS), and then synthesizing each virtual view by using an Image-based rendering (IBR) technique.

The MVS process is easy to generate ambiguous matching in areas such as texture missing, shading, non-diffuse reflection, translucency and the like, so that the IBR result has the problems of edge fracture, aliasing artifacts, inconsistent virtual view angle domain content and the like, and finally the reality sense of an autostereoscopic display picture is lost. Edge-driven and fuzzy theory are important means for solving the reconstruction uncertainty by two algorithms, namely MVS and IBR. The edge driving method has high operation efficiency due to the consideration that the edge (such as object contour and color texture) reconstruction is relatively accurate, but the inherent picture jitter, jelly effect and interframe redundancy of video data increase the difficulty of edge depth estimation; the fuzzy theory uses a fault-tolerant processing mode of simulating human brain to solve the optimal solution of the 'ambiguous' problem, but the expression form of the fuzziness can cause great waste of calculation and storage resources when the fuzzy expression form is used for high frame rate or high resolution video.

Disclosure of Invention

The invention aims to provide a three-dimensional fuzzy surface synthesis method of a monocular video virtual view driven by fuzzy edges. The invention organically combines the advantages of high computational efficiency of an edge driving mechanism and good fault tolerance of a fuzzy theory, and can realize end-to-end (without using given scene geometric information) high-performance and strong robust view synthesis through a strategy of complementary edge driving and the fuzzy theory, thereby quickly generating high-quality content for free three-dimensional display and solving the problems in the background technology.

The invention discloses a three-dimensional fuzzy surface synthesis method of a fuzzy edge-driven monocular video virtual view, which comprises the following steps:

1.1 real view blurred edge reconstruction, reconstructing sparse blurred edges, i.e. blurred voxels corresponding to edge pixels, for a set of down-sampled video frames, comprising the steps of:

1.1.1 mathematical expression of blurred voxels for pixel (x, y) is:

V(x,y)＝{d_k,μ_k(d_k)|d_k∈[d_min,d_max]}

wherein: [ d_min,d_max]The depth range is obtained by adopting linear sampling of the reciprocal of the depth value in the depth interval; mu.s_k(d_k)∈[0,1]Is a candidate depth d_kA membership function whose value is related to the depth range;

1.1.2, establishing a self-adaptive depth range selection and membership degree distribution mechanism for the fuzzy edge of the real view by a double-layer frame sampling method;

1.1.3 two-layer frame sampling procedure: firstly, calculating the camera poses of all video frames by using a motion recovery structure algorithm, and simultaneously reconstructing sparse three-dimensional point cloud with significant features in a video image; then, according to the information, the video frame is subjected to dense-to-sparse sampling: firstly, densely sampling video frames to enable the difference of camera visual angles of adjacent sampling frames to be 1 degree; further down-sampling the dense sampling frame at a fixed ratio of 4:1 to obtain a sparse sampling frame;

1.1.4 depth range selection process: for each sparse sampling frame, extracting edge pixels, and then selecting a depth range from coarse to fine for each fuzzy edge, wherein the method specifically comprises the following steps:

1.1.4.1 initialize: initializing the depth range of the fuzzy edge by utilizing the depth values of the nearest and farthest three-dimensional points which are output by the motion recovery structure algorithm and can be seen in the current sampling frame;

1.1.4.2 rough selection: calculating the matching cost of all the candidate depths by using 100 adjacent dense sampling frames of the current frame, wherein the matching cost is less than tau₁Updating the depth range of the blurred edge;

1.1.4.3 Fine pick: calculating new matching cost by using 100 adjacent sparse sampling frames of the current frame, and enabling the matching cost to be less than tau₂The minimum and maximum candidate depths of the blurred edge are used as the final depth range of the blurred edge;

step 1.1.4.2 and 1.1.4.3 adopt pixel level matching cost calculation method, threshold tau₁And τ₂Setting according to the actual reconstruction effect of a specific scene, taking the depth value with the minimum matching cost obtained finally as the rough depth of the edge, and removing the background edge near the object contour and the edge with obviously unreliable rough depth through thresholding with the minimum matching cost;

1.1.5 membership assignment procedure defines the membership function for the fuzzy edge candidate depth as:

μ_k(d_k)＝(1-S_k)·(1+R_k/6)

S_k∈[0,1]for the matching costs, R, obtained in the depth range fine selection step_k∈[-6,6]The calculation formula is as follows:

wherein: d_nEdge coarse depth maps for 6 views randomly extracted from 100 adjacent sparse sampled frames of the current frame; B. a and C each represent d_kReconstructed voxels P and D_nWhether three relationships are generated between: the visibility is consistent, the free space conflicts and the shielding are carried out, if yes, 1 is taken, and if not, 0 is taken;

1.2 virtual view blurred edge synthesis: synthesizing a fuzzy edge for the virtual view with the known camera pose through the weighted summation among the viewpoints of the real view reconstruction result;

comprises the following steps:

1.2.1 reference view selection: selecting M (more than or equal to 2) real views with the nearest camera positions and the smallest visual angle difference as interpolation reference images for the virtual views from the sparse sampling frames; the M value is set according to the actual situation so as to achieve good compromise between the operation speed and the synthesis effect; suppose a virtual view I_vHas an optical center of O_vReference to a view I_cMiddle pixel (x, y) pair I_vThe interpolation weight of (d) is defined as:

b is the average baseline, Ray, of adjacent sparsely sampled frames_cIs I_cDist is a function of the distance of the point line;

1.2.2 blurred edge synthesis: firstly synthesizing fuzzy voxels for the candidate edges, and then extracting the fuzzy edges from the fuzzy voxels, specifically comprising the following steps:

1.2.2.1 candidate edge-blurred voxel synthesis process: projecting the fuzzy edge of each reference view into a virtual view, and marking all projection points as candidate edges; for each candidate edge (x, y), projecting all reference view blurred edges projected to the pixel onto its corresponding ray, taking the union of all depth range projections as the depth range of the pixel blurred voxel; its blurred voxels are then calculated using the following formula:

wherein: v_cBlurred edges of reference views, T, for virtual views_vAnd T_cRespectively representing the projective forward transformation of the virtual view and the reference view;

the summation process for two blurred voxels with different depth ranges is: the depth ranges of the two are expanded into a union of the two, the membership degree of the newly added candidate depth is set to be 0, the membership degrees of the same depth are added, and finally V is obtained_v(x, y) compacting, namely updating the depth range by adopting weighted summation operation;

1.2.2.2 blurred edge extraction procedure: taking the average value of all candidate depths with the maximum center-membership degree of each fuzzy voxel as the depth estimation of the corresponding candidate edge, and carrying out edge detection on the obtained semi-dense rough depth map by utilizing a Sobel operator;

1.3 virtual view blur surface generation: firstly, generating a complete fuzzy surface for a virtual view by adopting spatial interpolation based on local smoothness and sharp edge constraint; then, removing a fuzzy surface causing wrong occlusion by using fuzzy edges of all sparse sampling frames and simultaneously based on global visibility constraint; and finally, filling the angle domain of the virtual view fuzzy surface hole by using the complete fuzzy surface of a small number of sparse sampling frames, and specifically comprising the following steps of:

1.3.1 adopting a Jacobi iteration method to carry out viewpoint interpolation on the fuzzy edge of the virtual view, and simultaneously finishing edge retention, edge sharp constraint, region filling, smoothing and local smooth constraint on the fuzzy surface; without loss of generality, let V_v ^tFor the tth convolution result of the virtual view blurred surface, the iterative process is described as:

{(x_n,y_n) Represents a 4 neighborhood of pixel (x, y); δ (-) is an indicator function, for non-empty blurred voxels, its value is 1, otherwise it is 0; if V_v ^t+1In terms of depth range and membership distribution_v ^tAll the differences are smaller than a given threshold value, and the iteration is ended;

1.3.2 removing blurred surfaces causing false occlusion, comprising the steps of:

1.3.2.1 seed point selection: firstly, obtaining rough surface estimation from a fuzzy surface of a virtual view, namely extracting the centers of all fuzzy voxels; then projecting the rough surface to each sparse sampling frame in sequence; for the edge (x, y) of a sparsely sampled frame, assume its blurred voxel depth range is [ d ]_min,d_max]Its coarse depth estimate d_rThe reconstructed voxel is P_r(ii) a By P_wRepresenting the voxel projected from the virtual view to (x, y), with d_wAnd [ d ]_min',d_max']Respectively representing its depth value and depth range relative to the sample frame by dividing P_wThe blurred voxel is projected to the ray where (x, y) is located, if the following conditions are met:

will P_wThe label is a seed point of the region growing process, the fuzzy voxel which belongs to the seed point is not projected into the residual sampling frame, and the formula is as follows: beta is a fault tolerance performance control parameter;

1.3.2.2 region growth: gradually aggregating sparse seed points into several dense, independent fuzzy surfaces: for each seed point P_wTheir depth difference d in the corresponding sample frame_r-d_wProjecting the image to the light ray of the image in the virtual view, and recording the projection result as delta d; then P is added_wDepth range of the blurred voxel_min”,d_max”]Extend to [ d ]_min”-Δd,d_max”+Δd](ii) a Then smoothing the depth range of the virtual view fuzzy surface by a Jacobi iteration method; will compare the areaA fuzzy voxel set with the relative change of the depth range before domain growth exceeding a preset threshold value is used as a detected error fuzzy surface;

1.3.3 filling angular domains of virtual view fuzzy surface holes, comprising the following steps:

1.3.3.1, counting the number of seed points detected by each sparse sampling frame;

1.3.3.2, performing fuzzy surface spatial interpolation one by one according to the sequence of the number from more to less, and projecting the interpolation result to the cavity of the virtual view fuzzy surface; if the hole is completely filled, the operation ends, otherwise it continues.

Compared with the prior art, the invention has the beneficial effects that: the three-dimensional fuzzy surface synthesis method for the virtual view of the fuzzy edge-driven monocular video, provided by the invention, has the advantages of high computation efficiency of an edge-driven mechanism and good tolerance of a fuzzy theory in an organic combination manner in order to avoid that the integral modeling precision is extremely sensitive to errors of edge depth information and simultaneously avoid that massive fuzzy voxel data needs to be computed and accessed in the depth estimation and texture mapping processes of a video scene, and can realize end-to-end (without using given scene geometric information) high-performance and strong robust view synthesis through a strategy of complementing the edge-driven mechanism and the fuzzy theory, thereby rapidly generating high-quality content for free three-dimensional display.

Drawings

FIG. 1 is an overall framework of a three-dimensional fuzzy surface synthesis method for a virtual view of a monocular video driven by fuzzy edges

FIG. 2 is a flow chart of double-layer frame sampling and fuzzy edge reconstruction (adaptive depth range selection and membership distribution)

FIG. 3 is a diagram of global visibility constraints

FIG. 4 is a schematic diagram of virtual view blurred edge synthesis

FIG. 5 is a flow chart and schematic diagram of virtual view blur surface generation

Detailed Description

The following further describes the implementation process of the present invention, and the overall framework of the method for synthesizing a three-dimensional fuzzy surface of a virtual view of a blurred edge-driven monocular video is shown in fig. 1, and includes the following steps:

1 real view blurred edge reconstruction, reconstructing sparse blurred edges, i.e. blurred voxels corresponding to edge pixels, for a set of down-sampled video frames, comprising the steps of:

1.1 the mathematical expression of the blurred voxel for pixel (x, y) is:

V(x,y)＝{d_k,μ_k(d_k)|d_k∈[d_min,d_max]}

the components comprise: depth range [ d_min,d_max]Representing the uncertainty of depth estimation, wherein the depth interval adopts linear sampling of the reciprocal of the depth value; membership function mu_k(d_k)∈[0,1]Characterizing the candidate depth d_kA degree belonging to the correct depth value, the value of which is related to the depth range;

1.2 establishing a self-adaptive depth range selection and membership degree distribution mechanism for the fuzzy edge of the real view by a double-layer frame sampling method, wherein the specific flow is shown in FIG. 2;

1.3 two-layer frame sampling process: calculating the camera poses of all video frames by using a motion recovery structure algorithm, simultaneously reconstructing sparse three-dimensional point cloud with significant features in a video image, and then carrying out dense-to-sparse sampling on the video frames according to the information; problems that three-dimensional points are blocked, moved out of a visual field, brightness changes and the like are not prone to occurring in a narrow baseline image sequence, depth estimation is facilitated, time domain redundancy of an original video is serious, and data volume is too large, so that video frames are densely sampled first, and the camera visual angle difference of adjacent sampling frames is 1 degree; certain content difference between the wide baseline images is beneficial to improving the reliability of MVS, so that the dense sampling frame is further downsampled at a fixed ratio of 4:1 to obtain a sparse sampling frame; in order to reduce the calculation amount, only reconstructing a fuzzy edge for the sparse sampling frame;

1.4 depth range selection process: for each sparse sampling frame, extracting edge pixels, and then selecting a depth range from coarse to fine for each fuzzy edge, wherein the method specifically comprises the following steps:

1.4.1 initialization: initializing the depth range of the fuzzy edge by utilizing the depth values of the nearest and farthest three-dimensional points which are output by the motion recovery structure algorithm and can be seen in the current sampling frame;

1.4.2 coarse selection: calculating the matching cost of all the candidate depths by using 100 adjacent dense sampling frames of the current frame, wherein the matching cost is less than tau₁Updating the depth range of the blurred edge;

1.4.3 Fine selection: calculating new matching cost by using 100 adjacent sparse sampling frames of the current frame, and enabling the matching cost to be less than tau₂The minimum and maximum candidate depths of the blurred edge are used as the final depth range of the blurred edge;

considering that block matching generally reduces sharpness of geometric edges of the scene model, steps 1.4.2 and 1.4.3 adopt a pixel-level matching cost calculation method, threshold τ₁And τ₂Setting according to the actual reconstruction effect of a specific scene, taking the depth value with the minimum matching cost obtained finally as the rough depth of the edge, and removing the background edge near the object contour and the edge with obviously unreliable rough depth through thresholding with the minimum matching cost;

1.5 membership assignment procedure defines the membership function of the fuzzy edge candidate depth as:

μ_k(d_k)＝(1-S_k)·(1+R_k/6)

S_k∈[0,1]characterizing d for the matching cost found in the depth range fine selection step_kThe brightness consistency of the reconstructed voxel P in the sparse sampling frame; r_k∈[-6,6]The consistency of the visibility of P in the MVS result is measured by the following formula:

wherein: d_nEdge coarse depth maps for 6 views randomly extracted from 100 adjacent sparse sampled frames of the current frame; B. a and C represent P and D, respectively_nIf three relations are generated (if yes, 1 is taken, otherwise, 0 is taken):

the visibilities are consistent: p is in D_nReconstruct a three-dimensional point that is nearly identical to P (fig. 3 a).

Free space collision: d_nAnother three-dimensional point is reconstructed between P and the camera optical center of the viewpoint where P is located (free space) (fig. 3 b).

And c, shielding: p is in D_nReconstructed three-dimensional point and D_nBetween the camera optical centers of the viewpoints (fig. 3 c).

The above equation uses the difference between the occurrence of free space conflicts and the number of occlusions because these two cases that do not satisfy the global visibility constraint are unlikely to occur simultaneously.

2. Virtual view blurred edge synthesis: the specific flow of synthesizing the blurred edge for the virtual view with the known camera pose through the weighted summation among the viewpoints of the real view reconstruction result is shown in fig. 4a, and the method comprises the following steps:

2.1 reference view selection: selecting M (more than or equal to 2) real views with the nearest camera positions and the smallest visual angle difference as interpolation reference images for the virtual views from the sparse sampling frames; the smaller the M value is, the simpler the calculation is; the larger the M value is, the better the visual continuity between the virtual view and the real view is, and the M value is set according to the actual situation so as to achieve good compromise between the operation speed and the synthesis effect; suppose a virtual view I_vHas an optical center of O_vReference to a view I_cMiddle pixel (x, y) pair I_vThe interpolation weight of (d) is defined as:

b is the average baseline, Ray, of adjacent sparsely sampled frames_cIs I_cDist is a function of the distance of the point line;

2.2 fuzzy edge synthesis: firstly synthesizing fuzzy voxels for the candidate edges, and then extracting the fuzzy edges from the fuzzy voxels, specifically comprising the following steps:

2.2.1 candidate edge-blurred voxel synthesis procedure: as shown in fig. 4b, the blurred edge of each reference view is projected into the virtual view, and all the projected points are marked as candidate edges; for each candidate edge (x, y), projecting all reference view blurred edges projected to the pixel onto its corresponding ray, taking the union of all depth range projections as the depth range of the pixel blurred voxel; its blurred voxels are then calculated using the following formula:

wherein: v_cBlurred edges of reference views, T, for virtual views_vAnd T_cRespectively representing the projective forward transformation of the virtual view and the reference view;

the summation process for two blurred voxels with different depth ranges is: firstly, the depth ranges of the two depth ranges are expanded into a union of the two depth ranges, the membership degree of the newly added candidate depth is set to be 0, then the membership degrees of the same depth are added, and finally, in order to prevent the depth range accumulation caused by fuzzy voxel synthesis, V is used_v(x, y) compacting, namely updating the depth range by adopting weighted summation operation;

2.2.2 blurred edge extraction procedure: the non-edge information in the candidate edge-blurred voxels may affect the quality of the generation of the blurred surface, so these sets of blurred voxels must be removed; in the process, the average value of all candidate depths with the maximum center-membership degree of each fuzzy voxel is used as the depth estimation of a corresponding candidate edge, and the Sobel operator is used for carrying out edge detection on the obtained semi-dense rough depth map;

3. virtual view blur surface generation: as shown in fig. 5a, a spatial interpolation based on local smoothing and edge sharpness constraint is first adopted to generate a complete fuzzy surface for a virtual view; then, removing a fuzzy surface causing wrong occlusion by using fuzzy edges of all sparse sampling frames and simultaneously based on global visibility constraint; and finally, filling the angle domain of the virtual view fuzzy surface hole by using the complete fuzzy surface of a small number of sparse sampling frames, and specifically comprising the following steps of:

3.1 spatial interpolation adopts Jacobi iteration method to carry outPerforming viewpoint interpolation on the blurred edge of the virtual view, and simultaneously completing edge preservation, edge sharp constraint, region filling and smoothing, and local smooth constraint on the blurred surface, as shown in fig. 5 b; without loss of generality, let V_v ^tFor the tth convolution result of the virtual view blurred surface, the iterative process is described as:

{(x_n,y_n) Represents a 4 neighborhood of pixel (x, y); δ (-) is an indicator function, for non-empty blurred voxels, its value is 1, otherwise it is 0; if V_v ^t+1In terms of depth range and membership distribution_v ^tAll differences are less than a given threshold, indicating that convergence has been reached, and so the iteration is ended;

3.2 remove blurred surfaces that cause false occlusion, when some three-dimensional edges (P in FIG. 5 b)_r) When not visible in the virtual view, if only the integrity and smoothness of the blurred surface are pursued, the above-mentioned spatial interpolation process may cause the foreground blurred edge to be connected with the background blurred edge, and the resulting false blurred surface completely blocks the actual edge, thereby comprising the following steps:

3.2.1 seed point selection: as shown in fig. 5c, a rough surface estimate is obtained from the blurred surface of the virtual view, i.e. the centers of all blurred voxels are extracted; then projecting the rough surface to each sparse sampling frame in sequence; for the edge (x, y) of a sparsely sampled frame, assume its blurred voxel depth range is [ d ]_min,d_max]Its coarse depth estimate d_rThe reconstructed voxel is P_r(ii) a By P_wRepresenting the voxel projected from the virtual view to (x, y), with d_wAnd [ d ]_min',d_max']Respectively representing its depth value and depth range relative to the sample frame by dividing P_wThe blurred voxel is projected to the ray where (x, y) is located, if the following conditions are met:

description of the preferred embodiments_rIs determined to be located at P_wThen, it is clear that there is a conflict with the global visibility constraint. Due to P_rRelatively accurate, indicating P_wMust be on the wrong surface, so P will be_wMarking as a seed point of the region growing process; in order to improve the algorithm efficiency, the blurred voxels to which the seed points belong are not projected into the residual sampling frame any more, in the formula: beta is a fault-tolerant performance control parameter, the smaller the beta value is, the more sensitive the occlusion detection algorithm is to the error of depth estimation, the cleaner the occlusion region is removed, but more small-area cavities can be generated in the non-occlusion region, conversely, the larger the beta value is, the smaller the number of seed points is, but the stronger the reliability is, and the more robust the occlusion detection algorithm is;

3.2.2 region growing: gradually aggregating sparse seed points into several dense, independent fuzzy surfaces: for each seed point P_wTheir depth difference d in the corresponding sample frame_r-d_wProjecting the image to the light ray of the image in the virtual view, and recording the projection result as delta d; then P is added_wDepth range of the blurred voxel_min”,d_max”]Extend to [ d ]_min”-Δd,d_max”+Δd](ii) a Smoothing the depth range of the virtual view fuzzy surface by a Jacobi iteration method, wherein the process is similar to the fuzzy surface spatial domain interpolation, but the fuzzy voxel information corresponding to the seed points needs to be protected, and the membership degree distribution of the fuzzy surface does not need to be updated; using the fuzzy voxel set with the relative change of the depth range before the growth of the region exceeding a preset threshold as a detected error fuzzy surface, as shown in fig. 5 d;

3.3 filling the angle domain of the hole on the fuzzy surface of the virtual view, filling the hole on the fuzzy surface of the virtual view through a single-view restoration algorithm, easily causing the content difference of a plurality of generated virtual views in the angle domain, and detecting that the reconstruction result of the sparse sampling frame of the seed point in the corresponding area is reliable, so that the fuzzy surface can be used for filling the hole across viewpoints; therefore, based on the geometric consistency constraint between views, and at the same time, in order to avoid huge calculation and storage loads, the process includes the following steps:

3.3.1 counting the number of the seed points detected by each sparse sampling frame;

3.3.2, performing fuzzy surface spatial interpolation one by one according to the sequence of the number from more to less, and projecting interpolation results to cavities on the fuzzy surface of the virtual view; if the hole is completely filled, the operation ends, otherwise it continues.