阅读说明：本技术 用于图像处理的方法、装置、设备、存储介质和程序产品 (Method, apparatus, device, storage medium and program product for image processing ) 是由 李朋辉 徐静涛 范学峰 崔家华 张柳清 仲亮亮 李国洪 高菲 于 2021-09-09 设计创作，主要内容包括：根据本公开的示例实施例,提供了用于图像处理的方法、装置、设备、存储介质和程序产品。用于图像处理的方法包括获取针对二维图像的输入信息,输入信息至少包括二维图像的深度信息；利用二维图像和输入信息,获取与二维图像的各像素相对应的三维点云；以及基于目标二维像素所对应的三维点云中的点、以及与目标二维像素相邻的一组像素的所对应的三维点云中的相邻点集合,生成针对二维图像的三维图像。以此方式,能够通过二维照片实现三维模型的构建,具有良好的普遍性和通用性,由此能够大幅提升用户的沉浸式和交互式体验。(According to example embodiments of the present disclosure, methods, apparatuses, devices, storage media and program products for image processing are provided. The method for image processing includes acquiring input information for a two-dimensional image, the input information including at least depth information of the two-dimensional image; acquiring a three-dimensional point cloud corresponding to each pixel of the two-dimensional image by using the two-dimensional image and the input information; and generating a three-dimensional image for the two-dimensional image based on the points in the three-dimensional point cloud corresponding to the target two-dimensional pixel and the set of adjacent points in the three-dimensional point cloud corresponding to the set of pixels adjacent to the target two-dimensional pixel. In this way, the construction of the three-dimensional model can be realized through the two-dimensional photos, and the method has good universality and universality, so that the immersive and interactive experiences of the user can be greatly improved.)

1. An image processing method comprising:

acquiring input information aiming at a two-dimensional image, wherein the input information at least comprises depth information of the two-dimensional image;

acquiring a three-dimensional point cloud corresponding to each pixel of the two-dimensional image by using the two-dimensional image and the input information; and

a three-dimensional image for the two-dimensional image is generated based on a point in the three-dimensional point cloud to which a target two-dimensional pixel corresponds and a set of neighboring points in the three-dimensional point cloud to which a set of pixels neighboring the target two-dimensional pixel correspond.

2. The method of claim 1, wherein generating a three-dimensional image for the two-dimensional image comprises:

acquiring a planar mesh for the target pixel and the group of pixels based on the point in the three-dimensional point cloud corresponding to the target two-dimensional pixel and at least two points in the adjacent point set; and

generating the three-dimensional image for the two-dimensional image based on the acquired planar mesh.

3. The method of claim 2, wherein pixels corresponding to the at least two points in the set of neighboring points are neighboring.

4. The method of any of claims 1-3, wherein the input information further comprises at least one of:

a foreground mask of the two-dimensional image;

a background mask for the two-dimensional image; and

patched image information of the two-dimensional image.

5. The method of any of claims 1-3, wherein acquiring a three-dimensional point cloud corresponding to each pixel of the two-dimensional image comprises:

and normalizing the two-dimensional image and the input information to obtain the three-dimensional point cloud.

6. The method of any of claims 1-3, wherein acquiring a three-dimensional point cloud corresponding to each pixel of the two-dimensional image comprises:

converting a plane coordinate system where the pixels of the two-dimensional image are located into a three-dimensional coordinate system; and

generating the three-dimensional point cloud in the three-dimensional coordinate system based on pixels of the two-dimensional image.

7. The method of claim 6, wherein the planar coordinate system comprises at least one of a pixel coordinate system or an image coordinate system, and the three-dimensional coordinate system comprises at least one of a camera coordinate system or a world coordinate system.

8. The method of claim 2 or 3, wherein the planar mesh comprises a triangular mesh.

9. The method of any of claims 1-3 or 7, wherein the two-dimensional image includes at least one of texture information or color information, and wherein generating a three-dimensional image for the two-dimensional image further comprises:

drawing at least one of the texture information or the color information on the planar mesh based on the corresponding relationship between the pixels of the two-dimensional image and the midpoint of the three-dimensional point cloud;

rendering a three-dimensional image for the two-dimensional image using the rendered planar mesh.

10. A video processing method, comprising:

the method according to any of claims 1-9, generating a corresponding three-dimensional image for a two-dimensional image of each frame in a video stream; and

a three-dimensional video stream is generated using the generated three-dimensional image.

11. An apparatus for image processing, comprising:

an input information acquisition module configured to acquire input information for a two-dimensional image, the input information including at least depth information of the two-dimensional image;

a three-dimensional point cloud obtaining module configured to obtain a three-dimensional point cloud corresponding to each pixel of the two-dimensional image using the two-dimensional image and the input information; and

a three-dimensional image generation module configured to generate a three-dimensional image for the two-dimensional image based on points in the three-dimensional point cloud corresponding to a target two-dimensional pixel and a set of neighboring points in the three-dimensional point cloud corresponding to a set of pixels neighboring the target two-dimensional pixel.

12. The apparatus of claim 11, wherein the three-dimensional image generation module comprises:

a planar mesh acquisition module configured to acquire a planar mesh for the target pixel and the set of pixels based on points in the three-dimensional point cloud to which the target two-dimensional pixel corresponds and at least two points in the set of neighboring points; and

a three-dimensional image generation sub-module configured to generate the three-dimensional image for the two-dimensional image based on the acquired planar mesh.

13. The device of claim 12, wherein pixels corresponding to the at least two points in the set of neighboring points are neighboring.

14. The apparatus of any of claims 11-13, wherein the input information further comprises at least one of:

a foreground mask of the two-dimensional image;

a background mask for the two-dimensional image; and

patched image information of the two-dimensional image.

15. The apparatus of any of claims 11-13, wherein the three-dimensional point cloud acquisition module comprises:

a normalization processing module configured to perform normalization processing on the two-dimensional image and the input information to obtain the three-dimensional point cloud.

16. The apparatus of any of claims 11-13, wherein the three-dimensional point cloud acquisition module comprises:

a three-dimensional coordinate system conversion module configured to convert a planar coordinate system in which pixels of the two-dimensional image are located into a three-dimensional coordinate system; and

a three-dimensional point cloud generation module configured to generate the three-dimensional point cloud in the three-dimensional coordinate system based on pixels of the two-dimensional image.

17. The apparatus of claim 16, wherein the planar coordinate system comprises at least one of a pixel coordinate system or an image coordinate system, and the three-dimensional coordinate system comprises at least one of a camera coordinate system or a world coordinate system.

18. The apparatus of claim 12 or 13, wherein the planar mesh comprises a triangular mesh.

19. The apparatus of any of claims 11-13 or 17, wherein the two-dimensional image comprises at least one of texture information or color information, and wherein the three-dimensional image generation module further comprises:

a planar mesh rendering module configured to render at least one of the texture information or the color information on the planar mesh based on a correspondence of pixels of the two-dimensional image and the three-dimensional point cloud midpoint;

a three-dimensional image rendering module configured to render a three-dimensional image for the two-dimensional image using the rendered planar mesh.

20. A video processing apparatus comprising:

a second three-dimensional image generation module configured to generate, for the two-dimensional image of each frame in the video stream, a corresponding three-dimensional image according to the method of any one of claim 10; and

a three-dimensional video stream generation module configured to generate a three-dimensional video stream using the generated three-dimensional image.

21. An electronic device, the device comprising:

one or more processors; and

storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to carry out the method according to any one of claims 1-10.

22. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-10.

23. A computer program product comprising a computer program which, when executed by a processor, performs the method of any one of claims 1-10.

Technical Field

Embodiments of the present disclosure relate generally to the field of computers and, more particularly, relate to a method, apparatus, device, storage medium, and program product for image processing.

Background

Two-dimensional display of pictures is currently the most prominent form of picture presentation. The two-dimensional image generally refers to a planar image. The two-dimensional image only has four directions of left and right, up and down, and has no front and back. Thus, a two-dimensional image has only an area and no volume. In general, the two-dimensional image may be an RGB image or a grayscale image. When a user desires a better immersive or interactive experience, a method of converting a two-dimensional image into a three-dimensional image is required.

Disclosure of Invention

According to an embodiment of the present disclosure, there is provided a scheme of image processing.

In a first aspect of the present disclosure, there is provided an image processing method comprising: acquiring input information aiming at the two-dimensional image, wherein the input information at least comprises depth information of the two-dimensional image; acquiring a three-dimensional point cloud corresponding to each pixel of the two-dimensional image by using the two-dimensional image and the input information; and generating a three-dimensional image for the two-dimensional image based on the points in the three-dimensional point cloud corresponding to the target two-dimensional pixel and the set of adjacent points in the three-dimensional point cloud corresponding to the set of pixels adjacent to the target two-dimensional pixel.

In a second aspect of the present disclosure, there is provided a video processing method comprising generating, according to the method of the first aspect of the present disclosure, for a two-dimensional image of each frame in a video stream, a corresponding three-dimensional image; and generating a three-dimensional video stream using the generated three-dimensional image.

In a third aspect of the present disclosure, there is provided an apparatus of image processing, comprising: an input information acquisition module configured to acquire input information for the two-dimensional image, the input information including at least depth information of the two-dimensional image; a three-dimensional point cloud obtaining module configured to obtain a three-dimensional point cloud corresponding to each pixel of the two-dimensional image using the two-dimensional image and the input information; and a three-dimensional image generation module configured to generate a three-dimensional image for the two-dimensional image based on points in the three-dimensional point cloud corresponding to the target two-dimensional pixel and a set of neighboring points in the three-dimensional point cloud corresponding to a set of pixels neighboring the target two-dimensional pixel.

In a fourth aspect of the present disclosure, there is provided a video processing apparatus comprising: a second three-dimensional image generation module configured to generate a corresponding three-dimensional image for the two-dimensional image of each frame in the video stream according to the method of the second aspect of the present disclosure; and a three-dimensional video stream generation module configured to generate a three-dimensional video stream using the generated three-dimensional image.

In a fifth aspect of the present disclosure, there is provided an electronic device, the device comprising: one or more processors; and storage means for storing the one or more programs which, when executed by the one or more processors, cause the one or more processors to carry out a method according to the first or second aspect of the disclosure.

In a sixth aspect of the present disclosure, a computer readable storage medium is provided, on which a computer program is stored, which when executed by a processor, implements a method according to the first or second aspect of the present disclosure.

In a seventh aspect of the disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, performs the method according to the first or second aspect of the disclosure.

It should be understood that the statements herein reciting aspects are not intended to limit the critical or essential features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, like or similar reference characters designate like or similar elements, and wherein:

FIG. 1 illustrates a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

FIG. 2 shows a flow diagram of a process of generating a three-dimensional image according to an embodiment of the present disclosure;

FIG. 3 illustrates a schematic diagram of a process of generating a three-dimensional image based on a point cloud in generating a three-dimensional image, according to some embodiments of the present disclosure;

FIG. 4 illustrates a flow diagram of a process of generating a three-dimensional video stream, according to some embodiments of the present disclosure;

FIG. 5 shows a schematic diagram of an apparatus for generating a three-dimensional image according to some embodiments of the present disclosure;

fig. 6 shows a schematic block diagram of an apparatus for generating a three-dimensional video stream according to an embodiment of the present disclosure; and

FIG. 7 illustrates a block diagram of a computing device capable of implementing various embodiments of the present disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.

In describing embodiments of the present disclosure, the terms "include" and its derivatives should be interpreted as being inclusive, i.e., "including but not limited to. The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first," "second," and the like may refer to different or the same object. Other explicit and implicit definitions are also possible below.

In addition, herein, it is to be understood that the term "3D" may be equivalent to "three-dimensional", the term "2D" may be equivalent to "two-dimensional", and the "three-dimensional image" may be equivalent to a "three-dimensional image model".

As briefly mentioned above, there is a need for a method of converting a two-dimensional image into a three-dimensional image to satisfy a better immersive or interactive experience. Traditionally, there are two main solutions. In one approach, the design may be based on three-dimensional software, and the user may directly utilize a three-dimensional model such as Blender to generate a three-dimensional image from a two-dimensional image. However, this method needs to rely on the existing three-dimensional model, and the application scenario can only exist in the scenario of known three-dimensional information, so the application range of the scheme is relatively limited.

In another conventional approach, a hardware scanning approach, typically based on triangulation principles, converts a two-dimensional image into a three-dimensional image. However, this solution needs to rely on corresponding hardware (based on structured light, tof, etc.). For an object to be measured with a specific size, the three-dimensional model reconstruction can be realized by scanning for multiple times. In addition, for specific scanning hardware, the larger the size of an object to be reconstructed is, the more the scanning times are, and the larger the corresponding calculated amount is; moreover, since the solution is implemented based on hardware, the object reconstructed by the three-dimensional model must be scanned in situ, so the technical solution has great limitations in use.

To at least partially address the above and other potential problems, a solution for generating a three-dimensional image based on a two-dimensional image is presented herein. In the scheme, the construction of the three-dimensional model can be realized by using the two-dimensional picture, and the method has good universality and universality, and does not need to rely on extra hardware or carry out field scanning. Moreover, the color information and the texture information of the three-dimensional image can be drawn on the three-dimensional image model by utilizing the one-to-one correspondence relationship between the points in the three-dimensional point cloud and the pixels of the two-dimensional image. Therefore, the color information and the texture information of the three-dimensional image generated by the scheme cannot be lost, so that a high-quality 3D model is generated, and the immersive and interactive experience of a user is greatly improved.

Embodiments of the present disclosure will be described below in detail with reference to the accompanying drawings.

Fig. 1 illustrates a schematic diagram of an example environment 100 in which various embodiments of the present disclosure can be implemented. In this example environment 100, a two-dimensional image 110 may be input into a computing device 120, and a corresponding calculation (e.g., depth calculation, image inpainting, etc., as will be described below) may be performed at the computing device 120 to generate a three-dimensional image 130.

In some embodiments, the two-dimensional image 110 may be a planar image as previously described. It should be noted that the two-dimensional image 110 may be an image having a foreground and a background, or may be an image having no foreground or no significant foreground.

It should be noted that the foreground and background in the two-dimensional image are conventional concepts in the field of image technology. In general, the foreground of an image is the visual plane that appears closest to the viewer, while the background in the image is the compositing plane that is farther away from the viewer. For example, in a two-dimensional image of a person, the person is typically the foreground of the image, while other portions of the image are often referred to as the background. For some images, such as a blue sky white cloud image, there is no foreground or no obvious foreground, which is also a kind of two-dimensional image.

In different embodiments of the present disclosure, different processing may be performed by the computing device 120 for different kinds of two-dimensional images 110. This will be described in detail below.

Accordingly, the three-dimensional image 130 (which may also be referred to as a "three-dimensional image model") generally refers to a stereoscopic image having a height, a width, and a depth. In embodiments of the present disclosure, the three-dimensional image 130 may be an image obtained based on the two-dimensional image 110 that enables a more immersive and interactive experience for the user.

In some embodiments, the computing device 120 may include a desktop, tablet, Personal Digital Assistant (PDA), server, mainframe, etc., or any other processor-enabled device capable of wired or wireless data communication, as the present disclosure does not limit thereto.

In order to more clearly understand the scheme of generating a three-dimensional image for a two-dimensional image provided by the embodiments of the present disclosure, embodiments of the present disclosure will be further described with reference to fig. 2. Fig. 2 shows a flow diagram of a process 200 of generating a three-dimensional image according to an embodiment of the present disclosure. Process 200 may be implemented by computing device 120 of fig. 1. For ease of discussion, process 200 will be described in conjunction with FIG. 1.

At block 210, input information for the two-dimensional image 110 is obtained, the input information including at least depth information for the two-dimensional image 110. The depth information may include a depth image of the two-dimensional image 110, which may be obtained by inputting the two-dimensional image 110 into a depth model. It should be understood that the resolution of the depth image is consistent with the resolution of the two-dimensional image 110.

In some embodiments, the depth model may be deployed in computing device 120, or in other computing devices other than computing device 120, as long as the required depth information is available as input to block 220, and is not limited by this disclosure.

In some embodiments, the input information may also include a foreground mask, a background mask (the foreground mask and the background mask may also be collectively referred to as a "foreground background mask"), patched image information, and the like, for the two-dimensional image 110.

In such embodiments, the two-dimensional image 110 typically has distinct foreground and background. The foreground mask and the background mask may be obtained by a segmentation model, and the inpainted image (i.e., inpainting image information) may be obtained by an image inpainting (inpainting) model. As such, the input information may include a depth image, a patched image, a foreground mask, and a background mask.

It should be noted that the segmentation model and the image inpainting model may be deployed in the computing device 120, or may be deployed in a computing device different from the computing device 120, as long as the required depth information can be obtained as an input of the block 220, which is not limited by the present disclosure.

At block 220, a three-dimensional point cloud corresponding to each pixel of the two-dimensional image 110 is acquired using the two-dimensional image 110 and the input information. It can be seen that the two-dimensional image 110 may also be considered as an input information at block 220.

In some embodiments, the two-dimensional image 110 and the input information may be normalized to obtain a three-dimensional point cloud. Normalization is a way to simplify the calculation, and is to change a number into a decimal number between (0, 1), or transform a dimensional expression into a dimensionless expression, and then convert the dimensionless expression into a pure quantity. In this way, data processing can be facilitated, and the amount of calculation can be reduced.

In some embodiments, a planar coordinate system in which the pixels of the two-dimensional image 110 reside may be converted to a three-dimensional coordinate system, and a three-dimensional point cloud in the three-dimensional coordinate system may be generated based on the pixels of the two-dimensional image 110. The planar coordinate system may include at least one of a pixel coordinate system or an image coordinate system, and the three-dimensional coordinate system includes at least one of a camera coordinate system or a world coordinate system. In this way, the transition from the two-dimensional image 110 to the three-dimensional image can be achieved in a fast manner.

In the above embodiment of the two-dimensional image 110 with no foreground or no significant foreground, the normalized three-dimensional point cloud may be generated by performing a three-dimensional transformation (e.g., a mathematical projective transformation) directly based on the depth image. It should be noted that the above three-dimensional transformation is only exemplary, and any way capable of implementing two-dimensional to three-dimensional transformation is feasible, and the disclosure does not limit this.

In embodiments in a two-dimensional image 110 having a foreground and a background, the input information may also include a patched image, a foreground mask, and a background mask, as previously described. Next, a process of acquiring a three-dimensional point cloud corresponding to each pixel of the two-dimensional image 110 by mathematical projection transformation as in the following equations (1) to (6) will be described in detail, taking a planar coordinate system as a pixel coordinate system and a three-dimensional coordinate system as a camera coordinate system as an example.

Wherein Z is_cIs height information of a pixel in the depth image, u and v are coordinate positions in a two-dimensional pixel coordinate system, u and v, respectively₀、v₀F is the focal length of the virtual camera, and the unit is mm; d_x、d_yIs the pixel size.

In this embodiment, the normalized focal length f in the x-axis and the y-axis shown in equations (3) and (4) can be obtained by mathematical transformation of equations (1) and (2)_xAnd f_y. To obtain the normalized focal length f_xAnd f_yThen, the coordinates X of the X-axis and Y-axis of the pixel in the camera coordinate system based on pinhole imaging can be obtained by equations (5) and (6) based on the principle of pinhole imaging_cAnd Y_c。

It should be noted that the mathematical transformation of equation (1) and equation (2) is merely exemplary, and those skilled in the art can adjust the mathematical transformation according to actual needs. And the coordinate X_cAnd Y_cThe determination of (3) and (4) is also illustrative, and those skilled in the art can adjust the equation to determine the normalized focal length in other ways according to actual needs, and the disclosure is not limited thereto.

It should be noted that the above equations (5) and (6) are u₀、v₀Obtaining X-and Y-axis coordinates for the image center_cAnd Y_cIn (1). Those skilled in the art can also use the position of any other pixel point to calculate the coordinate of the relevant pixel, which is not limited in this disclosure.

Further, the horizontal and vertical field angles of the virtual camera may be set toAndthen it is possible to obtain:

combining equations (7) and (8), the image pixel coordinate system can be converted to a camera coordinate system based on pinhole imaging, i.e. a specific rendering field angle is set, i.e. a point cloud in a three-dimensional camera coordinate system can be generated from pixels in a two-dimensional pixel coordinate system. The range of the depth map is 0-2¹⁶1, so to guarantee generality, the depth map range is mapped to 0 ~ 1. Thus, a three-dimensional image can be obtained from the two-dimensional image 110 by the simple procedure described above. Compared with a point cloud mode obtained based on hardware scanning, the point cloud is easy to lose (NAN value) due to inevitable deviation of hardware, and the point cloud obtained according to deep learning does not have the problem of point cloud loss, so that a three-dimensional image model with higher quality can be presented.

It should be noted that the manner of transforming the two-dimensional image 110 into the three-dimensional image is merely exemplary, and those skilled in the art may implement the transformation in any suitable manner, or implement the transformation by adjusting the above-mentioned parameters accordingly, and the disclosure is not limited thereto.

At block 230, a three-dimensional image for the two-dimensional image 110 is generated based on the point in the three-dimensional point cloud to which the target two-dimensional pixel corresponds and the set of neighboring points in the three-dimensional point cloud to which the set of pixels neighboring the target two-dimensional pixel correspond.

In this embodiment, the target two-dimensional pixel may be any pixel in the two-dimensional image 110, for example, may be a pixel X as shown in fig. 3. A group of pixels is a set of pixels adjacent to the target two-dimensional pixel, which may be, for example, a set of pixels a-H as shown in fig. 3. Because the two-dimensional pixels and the points in the three-dimensional point cloud have a one-to-one corresponding relationship, it can be understood that the points in the three-dimensional point cloud corresponding to the target two-dimensional pixel and the adjacent group of pixels have a certain position relationship in space. A three-dimensional image for the two-dimensional image 110 is generated based on the points in the three-dimensional point cloud corresponding to the target two-dimensional pixel and the set of neighboring points in the three-dimensional point cloud corresponding to a set of neighboring pixels.

Exemplary steps that block 230 further implements will be described below in conjunction with fig. 3. Fig. 3 shows a schematic diagram of a process 300 of generating a three-dimensional image based on a point cloud, in accordance with some embodiments of the present disclosure. In FIG. 3, the target two-dimensional pixel may be X, and a group of pixels adjacent to the target two-dimensional pixel may be a set of pixels A-H, as described above. Correspondingly, the set of neighboring points may be a set of points corresponding to the set of pixels A-H in the three-dimensional point cloud.

In some embodiments, a plane 301 for a target two-dimensional pixel X and the set of pixels may be acquired based on a point in the three-dimensional point cloud to which the target pixel corresponds and at least two points in the set of neighboring points, and the three-dimensional image for the two-dimensional image 110 may be generated based on the acquired plane mesh 301.

In this embodiment, the process of generating the planar mesh 301 may include encoding and tiling processes. That is, the point cloud in the three-dimensional coordinate system may be encoded and tiled based on the correspondence of the pixels and the point cloud, thereby generating a three-dimensional image model for the two-dimensional image. In this way, the three-dimensional image modeling can be realized in the mode of the planar grid 301, namely the three-dimensional modeling can be completed by one two-dimensional image without depending on extra hardware or performing field scanning, and the method has better practical value.

In this embodiment, referring to fig. 3, since only three points are generally needed to determine a plane, a tiling (i.e., a planarization) can be performed based on a point in the three-dimensional point cloud corresponding to the target two-dimensional pixel X and any two points in the three-dimensional point cloud corresponding to a set of pixels. In this case, the patch may be a triangular patch. Correspondingly, the planar mesh 301 may comprise at least one triangular mesh.

In one embodiment, and more particularly with reference to FIG. 3, in implementing a triangular patch, the pixels in the two-dimensional image 110 corresponding to two points in the selected set of adjacent points are adjacent. For example, after the target two-dimensional pixel X is determined, points in the 3D point cloud corresponding to the pixel a and the pixel B in the adjacent group of pixels may be selected, and based on the correspondence between the pixels and the point cloud, the pixel a, the pixel X, and the pixel B are tiled in the 3D point cloud to obtain the triangular patch 310.

Similarly, the pixel X, the pixel B, and the pixel C may be triangularly tiled to obtain a triangular patch 320, the pixel X, the pixel E, and the pixel C may be triangularly tiled to obtain a triangular patch 330, the pixel X, the pixel E, and the pixel H may be triangularly tiled to obtain a triangular patch 340, the pixel X, the pixel G, and the pixel H may be triangularly tiled to obtain a triangular patch 350, and so on, to obtain a complete planar mesh 301 composed of 8 triangular patches. Further, the above method is extended to each pixel of the whole two-dimensional image 110, and a three-dimensional image model can be obtained. In this way, a lossless three-dimensional image can be obtained, thereby greatly improving the interaction experience and the immersion experience of the user.

It should be noted that the above-mentioned triangular tiling process is only exemplary, and those skilled in the art can implement three-dimensional image modeling by any other suitable way, which is not limited by the present disclosure. For example, in the case of relatively low precision requirements, it may not be necessary to use a triangular patch to perform the modeling. That is, when selecting a group of pixels, adjacent pixels may not be necessary, and the planar grid 301 thus formed may be composed of relatively few patches. In this way, partial precision is sacrificed, but the calculation amount is greatly reduced, and the method can be suitable for some three-dimensional image models with low precision requirements.

In some embodiments, with continued reference to FIG. 3, the two-dimensional image 110 generally includes at least one of texture information or color information. In such an embodiment, at least one of the texture information or the color information may be rendered on the planar mesh 301 based on the correspondence relationship of the pixels of the two-dimensional image 110 and the three-dimensional point cloud midpoints, and the three-dimensional image for the two-dimensional image 110 may be rendered using the rendered planar mesh 301. The texture information is a visual feature reflecting a homogeneous phenomenon in an image, and the texture information reflects the surface structure organization arrangement attribute with slow change or periodic change on the surface of an object. Unlike image features such as gray scale, color, etc., textures are represented by the gray scale distribution of pixels and their surrounding spatial neighborhood. And the color information corresponds to the image characteristics such as the gray scale and the color of the image.

In this embodiment, the depth image based on the two-dimensional image 110 has a strict one-to-one correspondence relationship with the two-dimensional image 110, that is, it means that the three-dimensional point cloud has a one-to-one correspondence relationship with the texture information and the color information. Therefore, the three-dimensional image model obtained already can be rendered by at least one of texture information or color information, so that a complete three-dimensional image model is obtained. In fact, this step can be understood as a process of pasting at least one of texture information and color information onto a corresponding patch of the planar mesh 301, that is, a process of coloring and rendering the three-dimensional image model.

In the scheme of acquiring the point cloud based on hardware scanning, the point cloud has a one-to-many correspondence relationship with texture information and color information due to unavoidable deviation of hardware, the texture information and the point cloud information cannot be in complete one-to-one correspondence, and finally the texture quality is reduced. In this embodiment, due to the strong spatial correlation, the texture information and the color information are not lost, so as to obtain a high-quality three-dimensional image model that can completely represent the three-dimensional information of the two-dimensional image 110.

Fig. 4 illustrates a flow diagram of a process 400 of generating a three-dimensional video stream, according to some embodiments of the present disclosure. Process 400 may be implemented in computing device 120 as shown in FIG. 1, or in any other suitable computing device. For ease of discussion, process 400 will be described in conjunction with FIG. 1.

At block 410, a corresponding three-dimensional image is generated for the two-dimensional image 110 for each frame in the video stream. In some embodiments, the process of generating the two-dimensional image 110 into a three-dimensional image may be implemented according to the method shown in FIG. 2 or according to any other suitable method. It should be understood that when a three-dimensional image needs to be generated for a part of frames of a video stream, a part of a specific frame may also be selected for three-dimensional image modeling, which is not limited by the present disclosure.

At block 420, a three-dimensional video stream is generated using the generated three-dimensional image. In this way, a three-dimensional video stream can be further obtained on the basis of generating a three-dimensional image, and the immersive experience and the interactive experience of the user are further improved.

Fig. 5 illustrates a schematic diagram of an apparatus 500 for generating a three-dimensional image according to some embodiments of the present disclosure. The apparatus 500 includes an input information acquisition module 510, a three-dimensional point cloud acquisition module 520, and a three-dimensional image generation module 530.

The input information acquisition module 510 is configured to acquire input information for the two-dimensional image 110, the input information including at least depth information of the two-dimensional image 110. As previously mentioned, the input information may further comprise at least one of: a foreground mask of the two-dimensional image; a background mask of the two-dimensional image; and inpainted image information for the two-dimensional image.

The three-dimensional point cloud acquisition module 520 is configured to acquire a three-dimensional point cloud corresponding to each pixel of the two-dimensional image 110 using the two-dimensional image 110 and the input information.

In some embodiments, the three-dimensional point cloud acquisition module 520 may include a normalization processing module configured to normalize the two-dimensional image and the input information to obtain the three-dimensional point cloud.

In some embodiments, the three-dimensional point cloud obtaining module 520 may further include a three-dimensional coordinate system conversion module configured to convert a planar coordinate system in which the pixels of the two-dimensional image 110 are located into a three-dimensional coordinate system, and a three-dimensional point cloud generating module configured to generate a three-dimensional point cloud in the three-dimensional coordinate system based on the pixels of the two-dimensional image 110.

In some embodiments, the planar coordinate system may include at least one of a pixel coordinate system or an image coordinate system, and the three-dimensional coordinate system may include at least one of a camera coordinate system or a world coordinate system.

The three-dimensional image generation module 530 is configured to generate a three-dimensional image for the two-dimensional image 110 based on points in the three-dimensional point cloud corresponding to the target two-dimensional pixel and a set of neighboring points in the three-dimensional point cloud corresponding to a set of pixels neighboring the target two-dimensional pixel.

In some embodiments, the three-dimensional image generation module 530 may further include a planar mesh acquisition module configured to acquire a planar mesh for the target pixel and the set of pixels based on the point in the three-dimensional point cloud to which the target two-dimensional pixel corresponds and at least two points in the set of adjacent points, and a three-dimensional image generation sub-module configured to generate a three-dimensional image for the two-dimensional image 110 based on the acquired planar mesh.

In some embodiments, the pixels corresponding to at least two points in the set of neighboring points may be neighboring.

In some embodiments, the three-dimensional image generation module 530 further comprises a planar mesh rendering module configured to render at least one of texture information or color information on a planar mesh based on a correspondence of pixels of the two-dimensional image 110 to the three-dimensional point cloud midpoints, and a three-dimensional image rendering module configured to render a three-dimensional image for the two-dimensional image 110 using the rendered planar mesh.

Fig. 6 shows a schematic block diagram of an apparatus 600 for generating a three-dimensional video stream according to an embodiment of the present disclosure. The apparatus 600 comprises a second three-dimensional image generation module 610 and a three-dimensional video stream generation module 620. Apparatus 600 may be implemented in computing device 120 as shown in fig. 1, or in any other suitable device. For ease of discussion, process 600 will be described in conjunction with FIG. 1.

The second three-dimensional image generation module 610 is configured to generate a corresponding three-dimensional image for the two-dimensional image 110 of each frame in the video stream. The step of generating a three-dimensional image may be implemented using the apparatus 500 as described previously.

The three-dimensional video stream generation module 620 is configured to generate a three-dimensional video stream using the generated three-dimensional image. In this way, a three-dimensional video stream can be further obtained on the basis of generating a three-dimensional image, and the immersive experience and the interactive experience of the user are further improved.

Fig. 7 illustrates a block diagram of a computing device 700 capable of implementing multiple embodiments of the present disclosure. Device 700 may be used to implement computing device 120 of fig. 1. As shown, device 700 includes a Central Processing Unit (CPU)701 that may perform various appropriate actions and processes in accordance with computer program instructions stored in a Read Only Memory (ROM)702 or computer program instructions loaded from a storage unit 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The CPU701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

Various components in the device 700 are connected to the I/O interface 705, including: an input unit 706 such as a keyboard, a mouse, or the like; an output unit 707 such as various types of displays, speakers, and the like; a storage unit 708 such as a magnetic disk, optical disk, or the like; and a communication unit 709 such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

The processing unit 701 performs the various methods and processes described above, such as one or more of the processes 200 and 400. For example, in some embodiments, one or more of processes 200 and 400 may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708. In some embodiments, part or all of a computer program may be loaded onto and/or installed onto device 700 via ROM 702 and/or communications unit 709. When loaded into RAM 703 and executed by CPU701, may perform one or more steps of one or more of processes 200 and 400 described above. Alternatively, in other embodiments, CPU701 may be configured to perform one or more of processes 200 and 400 in any other suitable manner (e.g., by way of firmware).

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a load programmable logic device (CPLD), and the like.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

Further, while operations are depicted in a particular order, this should be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.