阅读说明：本技术 3d数据生成装置、3d数据再现装置、控制程序以及记录介质 (3D data generating device, 3D data reproducing device, control program, and recording medium ) 是由 德毛靖昭 于 2019-09-27 设计创作，主要内容包括：为了基于深度生成高清的3D模型,深度需要一定的分辨率,但在使用现有编解码器对深度图像进行编码的情况下,根据拍摄对象的大小、动作的不同,有时深度的动态范围宽,分辨率不足。输入表示一个或多个拍摄对象的三维形状的深度图像来生成3D数据的3D数据生成装置,具备：深度分割部,将上述深度图像分割为由矩形区域构成的多个部分深度图像；深度整合部,对上述多个部分深度图像进行封装,生成整合深度图像；深度图像编码部,对上述整合深度图像进行编码；以及附加信息编码部,对附加信息进行编码,该附加信息包括确定上述矩形区域的分割信息以及表示上述封装的信息。(In order to generate a high-definition 3D model based on depth, a certain resolution is required for depth, but when a depth image is encoded using a conventional codec, the dynamic range of depth may be wide and the resolution may be insufficient depending on the size and motion of an object to be captured. A3D data generation device for generating 3D data by inputting a depth image representing a three-dimensional shape of one or more subjects, the device comprising: a depth dividing unit configured to divide the depth image into a plurality of partial depth images each including a rectangular region; a depth integration unit for generating an integrated depth image by encapsulating the plurality of partial depth images; a depth image encoding unit that encodes the integrated depth image; and an additional information encoding unit that encodes additional information including division information for specifying the rectangular region and information indicating the package.)

1. A3D data generation device for generating 3D data by inputting a depth image representing a three-dimensional shape of an imaging target, comprising:

an integrated depth image generating unit that generates an integrated depth image in which at least two partial depth images are encapsulated, the partial depth images being partial depth images of rectangular regions constituting the depth image;

a depth image encoding unit that encodes the integrated depth image; and

and an additional information encoding unit that encodes segmentation information that specifies an upper left corner coordinate of the partial depth image on the depth image and encapsulation information that specifies an upper left corner coordinate of a region corresponding to the partial depth image on the integrated depth image.

2. The 3D data generation apparatus according to claim 1,

the additional information encoding unit encodes dynamic range information for specifying depth values in the partial depth image.

3. The 3D data generation apparatus according to claim 1,

the integrated depth image generating unit derives shape information indicating whether or not each pixel of the integrated depth image belongs to a specific imaging target.

4. The 3D data generation apparatus according to claim 1,

the integrated depth image generating unit packages partial depth images of the same subject to be captured into the same coding unit.

5. The 3D data generation apparatus according to claim 1,

the integrated depth image generating unit packages partial depth images having the same imaging direction into the same coding unit.

6. A3D data reproduction device for inputting 3D data and reproducing a three-dimensional shape of a photographic subject, comprising:

an integrated depth image generating unit that reconstructs an integrated depth image in which at least two partial depth images are encapsulated, the partial depth images being partial depth images of rectangular regions constituting the depth image;

a depth image decoding unit that decodes an integrated depth image included in the 3D data; and

and an additional information decoding unit configured to decode division information for specifying an upper left corner coordinate of the partial depth image in the depth image and package information for specifying an upper left corner coordinate of a region corresponding to the partial depth image in the integrated depth image.

7. A control program for causing a computer to function as the 3D data generating apparatus according to claim 1,

for causing a computer to function as the integrated depth image generating unit.

8. A computer-readable recording medium in which, among others,

a control program according to claim 7 is recorded.

Technical Field

One aspect of the present invention relates to a 3D data generation device, a 3D data generation method, a control program, and a recording medium that generate 3D data by inputting a depth image representing a three-dimensional shape of a photographic subject.

Background

In the field of CG (Computer Graphics), a method called dynamic fusion (dynamic fusion) for constructing a 3D model (three-dimensional model) by integrating input depths has been studied. The purpose of dynamic fusion is mainly to build a 3D model that removes noise in real time from the shot input depth. In dynamic fusion, the input depth acquired from the sensor is integrated into a general-purpose reference 3D model while compensating for the deformation of the three-dimensional shape. Thus, a precise 3D model can be generated from a low resolution and a high noise depth.

Patent document 1 discloses a technique of inputting a multi-viewpoint color image and a multi-viewpoint depth image corresponding to each pixel level, and outputting an image of an arbitrary viewpoint.

Documents of the prior art

Patent document

Patent document 1: japanese laid-open patent publication "Japanese unexamined patent publication No. 2013-30898"

Disclosure of Invention

Problems to be solved by the invention

In order to generate a high-definition 3D model based on depth, a certain resolution is required for depth, but when a depth image is encoded using a conventional codec, the dynamic range of depth may be wide and the resolution may be insufficient depending on the size and motion of an object to be captured.

Technical scheme

In order to solve the above problem, a 3D data generating device according to an aspect of the present invention is a 3D data generating device that generates 3D data by inputting a depth image representing a three-dimensional shape of one or a plurality of imaging subjects, and includes: a depth dividing unit configured to divide the depth image into a plurality of partial depth images each including a rectangular region; a depth integration unit for generating an integrated depth image by encapsulating the plurality of partial depth images; a depth image encoding unit that encodes the integrated depth image; and an additional information encoding unit that encodes additional information including division information for specifying the rectangular region and information indicating the package.

In order to solve the above problem, a 3D data playback device according to an aspect of the present invention is a 3D data playback device that inputs 3D data and plays back a three-dimensional shape of one or more objects, including: a depth image decoding unit that decodes an integrated depth image included in the 3D data; an additional information decoding unit that decodes additional information including information indicating a package of a plurality of partial depth images each including a rectangular region included in the integrated depth image and division information for specifying the rectangular region; a depth extraction unit that extracts a partial depth image from the decoded integrated depth image based on information indicating the capsule; and a depth coupling unit that couples the plurality of partial depth images based on the division information to reconstruct a depth image.

Advantageous effects

According to an aspect of the present invention, even when the dynamic range of the depth of a photographic subject is wide, 3D data with a small quantization error can be generated using an existing codec.

Drawings

Fig. 1 is a functional block diagram showing a configuration of a 3D data generating device according to embodiment 1 of the present invention.

Fig. 2 is a functional block diagram showing the internal configuration of the integrated depth image generator and the integrated color image generator according to embodiment 1 of the present invention.

Fig. 3 is a diagram showing an example of obtaining a depth image and a color image according to embodiment 1 of the present invention.

Fig. 4 is a diagram showing an example of a depth image output by the depth image acquisition unit and a color image output by the color image acquisition unit in embodiment 1 of the present invention.

Fig. 5 is a diagram showing an example of segmentation of a depth image according to embodiment 1 of the present invention.

Fig. 6 is a diagram showing an example of packing of a depth image and a color image according to embodiment 1 of the present invention.

Fig. 7 is a diagram showing an example of division of a color image according to embodiment 1 of the present invention.

Fig. 8 is a functional block diagram showing the configuration of a 3D data playback device according to embodiment 1 of the present invention.

Fig. 9 is a functional block diagram showing the internal configuration of the depth image reconstruction unit and the color image reconstruction unit according to embodiment 1 of the present invention.

Fig. 10 is a functional block diagram showing the configuration of a 3D data generating device according to embodiment 2 of the present invention.

Fig. 11 is a functional block diagram showing an internal configuration of the integrated depth image generating unit according to embodiment 2 of the present invention.

Fig. 12 is a functional block diagram showing the configuration of a 3D data playback device according to embodiment 2 of the present invention.

Fig. 13 is a functional block diagram showing an internal configuration of a depth image reconstruction unit according to embodiment 2 of the present invention.

Fig. 14 is a functional block diagram showing the configuration of a 3D data generating device according to embodiment 3 of the present invention.

Fig. 15 is a functional block diagram showing the internal configuration of the integrated depth image generator and the integrated color image generator according to embodiment 3 of the present invention.

Fig. 16 is a diagram showing an example of obtaining a depth image and a color image according to embodiment 3 of the present invention.

Fig. 17 is a diagram showing an example of packing of a depth image and a color image according to embodiment 3 of the present invention.

Fig. 18 is a diagram showing an example of packing of a depth image and a color image according to embodiment 3 of the present invention.

Fig. 19 is a functional block diagram showing the configuration of a 3D data playback device according to embodiment 3 of the present invention.

Detailed Description

Hereinafter, embodiments of the present invention will be described in detail.

< embodiment 1>

[ 3D data generating apparatus ]

First, a 3D data generating device according to embodiment 1 of the present invention will be described with reference to the drawings.

Fig. 1 is a functional block diagram showing a configuration of a 3D data generating device according to embodiment 1 of the present invention. The 3D data generating apparatus 1 is configured by a depth image acquiring unit 17, an integrated depth image generating unit 11, a depth image encoding unit 12, a color image acquiring unit 18, an integrated color image generating unit 14, a color image encoding unit 15, an additional information encoding unit 13, and a multiplexing unit 16.

The depth image acquiring unit 17 acquires depth data from a plurality of depth cameras and outputs a depth image to the integrated depth image generating unit 11.

The integrated depth image generating unit 11 generates a single integrated depth image by dividing and integrating (packaging) the plurality of depth images output from the depth image acquiring unit 17.

The depth image coding unit 12 performs compression coding on the integrated depth image input from the integrated depth image generating unit 11, and outputs depth coded data. For compression encoding, for example, HEVC (High Efficiency Video Coding) specified by ISO/IEC23008-2 can be used.

The color image acquisition unit 18 acquires color data from the plurality of color cameras and outputs a color image to the integrated color image generation unit 14.

The integrated color image generation section 14 generates a single integrated color image by dividing and integrating (packing) the plurality of color images output from the color image acquisition section 18.

The color image encoding section 15 performs compression encoding on the integrated color image input from the integrated color image generating section 14, and outputs color encoded data. For compression coding, for example, HEVC can be used.

The additional information encoding unit 13 encodes additional information necessary for reconstructing the original depth image from the integrated depth image generated by the integrated depth image generating unit 11 and additional information necessary for reconstructing the original color image from the integrated color image generated by the integrated color image generating unit 14, and outputs additional information encoded data. The details of the additional information will be described later.

The multiplexing unit 16 multiplexes the encoded data output from the depth image encoding unit 12, the color image encoding unit 15, and the additional information encoding unit 13, and outputs the multiplexed data as 3D data. For multiplexing, for example, ISOBMFF (ISO Base Media File Format: ISO Base Media File Format) specified by ISO/IEC 14496-12 can be used. The multiplexed 3D data can be recorded in various recording media such as a hard disk, an optical disk, and a nonvolatile memory, or streamed to a network. For Streaming, for example, MPEG-DASH (Dynamic Adaptive Streaming over HTTP: HTTP-based Dynamic Adaptive Streaming) specified by ISO/IEC 23009-1 can be used.

Fig. 2 (a) is a functional block diagram showing an internal configuration of the integrated depth image generating unit 11 according to embodiment 1 of the present invention. The integrated depth image generating unit 11 includes a depth dividing unit 111 and a depth integrating unit 113.

The depth dividing unit 111 divides the depth image output from the depth image acquiring unit 17 into a plurality of partial depth images each including a rectangular region. Specifically, a rectangular region is set for each subject included in the depth image, the depth image included in the rectangular region is output as a partial depth image, and the following division information is output.

[ example 1 of division information ]

Coordinates of the upper left corner of each rectangular region (origin at the upper left corner of the depth image)

Coordinates of the lower right corner of each rectangular region (origin at the upper left corner of the depth image)

Identifiers of photographic subjects included in each rectangular region

[ example 2 of information division ]

Coordinates of the upper left corner of each rectangular region (origin at the upper left corner of the depth image)

Width and height of each rectangular region

Identifiers of photographic subjects included in each rectangular region

The depth integration section 113 generates an integrated depth image by integrating (packaging) the plurality of partial depth images output from the depth dividing section 111 into a single image. Specifically, an integrated depth image in which all partial depth images are integrated is output, and the following package information is output.

[ example of encapsulating information 1 ]

Coordinates on the integrated depth image corresponding to the upper left corner of each partial depth image (origin at the upper left corner of the integrated depth image)

Coordinates on the integrated depth image corresponding to the lower right corner of each partial depth image (origin at the upper left corner of the integrated depth image)

Identifiers of photographic subjects included in each partial depth image

[ example 2 of encapsulating information ]

Coordinates on the integrated depth image corresponding to the upper left corner of each partial depth image (origin at the upper left corner of the integrated depth image)

Width and height of each partial depth image in the integrated depth image

Identifiers of photographic subjects included in each partial depth image

In the integrated color image generating section 14, a single integrated color image is generated by dividing and integrating (packaging) the color image output from the color image acquiring section 18, in the same manner as the integrated depth image generating section 11, based on the division information and the package information output from the integrated depth image generating section 11.

Fig. 3 is a diagram showing an example of obtaining a depth image and a color image according to embodiment 1 of the present invention. The case where 3 cameras C1, C2, and C3 are arranged for the subject a and the subject b, and each camera captures a depth image and a color image is shown.

Fig. 4 is a diagram showing an example of a depth image output by the depth image acquisition unit 17 and a color image output by the color image acquisition unit 18 in embodiment 1 of the present invention. G1, G2, and G3 in fig. 4 (a) are depth images acquired by cameras C1, C2, and C3, respectively. T1, T2, and T3 in fig. 4 (b) are color images acquired by the cameras C1, C2, and C3, respectively.

Here, the cameras C1, C2, and C3 can acquire depth values in the range of 0mm to 25000mm, and values obtained by quantizing the acquired depth values with 16 bits are stored in the pixel values of the depth images G1, G2, and G3 (for example, the depth values are stored in the Y component in the YUV 4: 2: 016 bit format). On the other hand, the color images T1, T2, and T3 store luminance (Y) and color differences (U, V) quantized with 8 bits (for example, as YUV 4: 2: 08 bit format).

Fig. 5 is a diagram showing an example of segmentation of a depth image according to embodiment 1 of the present invention. The depth dividing section 111 divides the depth image G1 into a partial depth image G1a including a rectangular region of the photographic subject a and a partial depth image G1b including a rectangular region of the photographic subject b. Likewise, the depth image G2 is segmented into partial depth images G2a and G2b, and the depth image G3 is segmented into partial depth images G3a and G3 b. The depth dividing unit 111 outputs the following division information.

[ division information of G1a ]

Coordinates of the upper left corner of the rectangular area: (X1a, Y1a)

Coordinates of the lower right corner of the rectangular area: (X1a + W1a, Y1a + H1a)

Identifier of the photographic subject included in the rectangular area: a is

[ segmentation information of G2a ]

Coordinates of the upper left corner of the rectangular area: (X2a, Y2a)

Coordinates of the lower right corner of the rectangular area: (X2a + W2a, Y2a + H2a)

Identifier of the photographic subject included in the rectangular area: a is

[ segmentation information of G3a ]

Coordinates of the upper left corner of the rectangular area: (X3a, Y3a)

Coordinates of the lower right corner of the rectangular area: (X3a + W3a, Y3a + H3a)

Identifier of the photographic subject included in the rectangular area: a is

[ division information of G1b ]

Coordinates of the upper left corner of the rectangular area: (X1b, Y1b)

Coordinates of the lower right corner of the rectangular area: (X1b + W1b, Y1b + H1b)

Identifier of the photographic subject included in the rectangular area: b

[ segmentation information of G2b ]

Coordinates of the upper left corner of the rectangular area: (X2b, Y2b)

Coordinates of the lower right corner of the rectangular area: (X2b + W2b, Y2b + H2b)

Identifier of the photographic subject included in the rectangular area: b

[ segmentation information of G3b ]

Coordinates of the upper left corner of the rectangular area: (X3b, Y3b)

Coordinates of the lower right corner of the rectangular area: (X3b + W3b, Y3b + H3b)

Identifier of the photographic subject included in the rectangular area: b

Fig. 6 (a) is a diagram showing an example of packaging of a partial depth image according to embodiment 1 of the present invention. The depth integration section 113 integrates (packages) the partial depth images G1a, G2a, G3a, G1b, G2b, G3b into a single image to generate an integrated depth image. The deep coupling section 113 outputs the following package information.

[ encapsulation information of G1a ]

Coordinates on the integrated depth image corresponding to the upper left corner of the partial depth image: (x1, y1)

Coordinates on the integrated depth image corresponding to the bottom right corner of the partial depth image: (x1', y1')

Identifiers of photographic subjects included in the partial depth image: a is

[ encapsulation information of G2a ]

Coordinates on the integrated depth image corresponding to the upper left corner of the partial depth image: (x2, y2)

Coordinates on the integrated depth image corresponding to the bottom right corner of the partial depth image: (x2', y2')

Identifiers of photographic subjects included in the partial depth image: a is

[ encapsulation information of G3a ]

Coordinates on the integrated depth image corresponding to the upper left corner of the partial depth image: (x3, y3)

Coordinates on the integrated depth image corresponding to the bottom right corner of the partial depth image: (x3', y3')

Identifiers of photographic subjects included in the partial depth image: a is

[ encapsulation information of G1b ]

Coordinates on the integrated depth image corresponding to the upper left corner of the partial depth image: (x4, y4)

Coordinates on the integrated depth image corresponding to the bottom right corner of the partial depth image: (x4', y4')

Identifiers of photographic subjects included in the partial depth image: b

[ encapsulation information of G2b ]

Coordinates on the integrated depth image corresponding to the upper left corner of the partial depth image: (x5, y5)

Coordinates on the integrated depth image corresponding to the bottom right corner of the partial depth image: (x5', y5')

Identifiers of photographic subjects included in the partial depth image: b

[ encapsulation information of G3b ]

Coordinates on the integrated depth image corresponding to the upper left corner of the partial depth image: (x6, y6)

Coordinates on the integrated depth image corresponding to the bottom right corner of the partial depth image: (x6', y6')

Identifiers of photographic subjects included in the partial depth image: b

And performing coding control on the background area of each part of the depth image in the integrated depth image based on the shape information. The shape information is information showing whether or not each pixel of the integrated depth image belongs to a subject (photographic subject), and for example, "1" is assigned in the case of a pixel belonging to a subject, and "0" is assigned in the case of a pixel not belonging to a subject. In the encoding process, for example, when all pixels in the CTU (coding tree unit) do not belong to the object, or when some pixels in the CTU do not belong to the object, a process such as encoding is performed after filling a region not belonging to the object with a pixel value of an edge of the object or a predetermined pixel value in a horizontal direction or a vertical direction. The depth coupling unit 113 outputs the shape information as package information.

Fig. 2 (b) is a functional block diagram showing an internal configuration of the integrated color image generating unit 14 according to embodiment 1 of the present invention. The integrated color image generating unit 14 includes a color dividing unit 141 and a color integrating unit 143.

Fig. 7 is a diagram showing an example of division of a color image according to embodiment 1 of the present invention. The color dividing section 141 divides the color image T1 into partial color images T1a and T1b according to the division information input from the integrated depth image generating section 11. Similarly, color image T2 is divided into partial color images T2a and T2b, and color image T3 is divided into partial color images T3a and T3 b.

Fig. 6 (b) is a diagram showing an example of packaging of a partial color image in embodiment 1 of the present invention. The color integrating section 143 integrates (encapsulates) the partial color images T1a, T2a, T3a, T1b, T2b, T3b into a single image according to the encapsulation information input from the integrated depth image generating section 11 to generate an integrated color image.

The background region of each partial color image in the integrated color image is subjected to encoding control based on the package information (shape information) input from the integrated depth image generating unit 11. For example, when all pixels in the CTU do not belong to the object, or when some pixels in the CTU do not belong to the object, a process such as encoding is performed after filling a region not belonging to the object with a pixel value of an edge of the object or a predetermined pixel value in the horizontal direction or the vertical direction.

The depth image coding unit 12 compression-codes the integrated depth image using the HEVC Main12 profile, and outputs the depth coded data to the multiplexing unit 16.

The color image coding unit 15 performs compression coding on the integrated color image using the HEVC Main profile, and outputs the color coded data to the multiplexing unit 16.

The additional information encoding unit 13 reversibly encodes the division information and the package information output from the integrated depth image generating unit 11 and information on the posture (position, direction, and the like in the three-dimensional space) of each camera, and outputs the encoded information to the multiplexing unit 16.

With the above configuration, the dynamic range of the depth values in each CTU constituting the partial depth image can be reduced, and the resolution in quantization can be improved. As a result, even when the dynamic range of depth is large due to the size and movement of the subject, the lack of resolution can be eliminated.

Further, compared to the case where depth images (G1, G2, and G3 in fig. 5 (a)) are directly coupled and encoded, the amount of generated code can be reduced by reducing the background region and reducing the image size.

Further, since it is sufficient to always transmit three streams of encoded data in which a depth image (fig. 6 (a)) is integrated, encoded data in which a color image (fig. 6 (b)) is integrated, and encoded data in which additional information is integrated, regardless of the number of cameras, there is an effect that the number of transmitted streams does not depend on the number of cameras.

Further, by determining the size and the number of divisions of the rectangular region by evaluating and optimizing the bit rate of the encoded data (depth + color + additional information), the coding distortion of the depth image, the coding distortion of the color image, and the like, higher-quality 3D data can be generated.

[ 3D data reproducing apparatus ]

Next, a 3D data playback device according to embodiment 1 of the present invention will be described with reference to the drawings.

Fig. 8 is a functional block diagram showing the configuration of a 3D data playback device according to embodiment 1 of the present invention. The 3D data playback device 2 includes a separation unit 26, a depth image decoding unit 22, a depth image reconstruction unit 21, an additional information decoding unit 23, a color image decoding unit 25, a color image reconstruction unit 24, a 3D model generation unit 27, a playback image synthesis unit 28, a playback viewpoint input unit 291, and a playback object selection unit 292.

The separation unit 26 separates the depth image coded data, the color image coded data, and the additional information coded data included in the input 3D data, and outputs the separated data to the depth image decoding unit 22, the color image decoding unit 25, and the additional information decoding unit 23, respectively.

The depth image decoding unit 22 decodes the HEVC-encoded depth image encoded data input from the separation unit 26. For example, the integrated depth image illustrated by (a) of fig. 6 is decoded.

The depth image reconstruction unit 21 reconstructs a depth image by extracting (decapsulating) and coupling desired partial depth images from the plurality of partial depth images included in the integrated depth image decoded by the depth image decoding unit 22 based on the additional information (segmentation information and encapsulation information) input from the additional information decoding unit 23.

The color image decoding unit 25 decodes the HEVC-encoded color image data input from the demultiplexing unit 26. For example, the integrated color image shown by (b) of fig. 6 is decoded.

The color image reconstruction unit 24 extracts a desired partial color image from the plurality of color images included in the integrated color image decoded by the color image decoding unit 25 based on the additional information (segmentation information, encapsulation information) input from the additional information decoding unit 23, thereby reconstructing the color image.

The additional information decoding unit 23 decodes additional information (segmentation information and encapsulation information) necessary for reconstructing a depth image and a color image from the additional information encoded data input from the separation unit 26.

The 3D model generator 27 generates a 3D model based on the plurality of depth images input from the depth image reconstructor 21. The 3D model is a model representing the three-dimensional shape of an imaging target, and one form thereof is a model represented by a mesh.

The reproduced image synthesizing unit 28 synthesizes a reproduced image from a reproduction viewpoint based on the 3D model generated by the 3D model generating unit 27, the color image reconstructed by the color image reconstructing unit, and reproduction viewpoint information (position, direction, and the like in a three-dimensional space) input by the user.

The playback viewpoint input portion 291 is an input portion for inputting a playback viewpoint (position and direction) in a three-dimensional space by a user.

The playback object selection unit 292 is a selection unit for selecting a desired playback object from a plurality of playback objects by the user.

Fig. 9 (a) is a functional block diagram showing an internal configuration of the depth image reconstruction unit 21 according to embodiment 1 of the present invention. The depth image reconstruction unit 21 includes a depth extraction unit 211 and a depth coupling unit 213.

The depth extracting section 211 extracts (decapsulates) a desired partial depth image from the plurality of partial depth images included in the integrated depth image based on the encapsulation information input from the additional information decoding section 23. For example, when the object a and the object b are selected as objects to be reproduced by the reproduction object selection unit 292, the partial depth images G1a, G2a, G3a, G1b, G2b, and G3b shown in fig. 5 are extracted and output to the depth coupling unit 213. Alternatively, when only the photographic subject b is selected, the partial depth images G1b, G2b, and G3b are extracted and output to the depth coupler.

The depth coupling unit 213 couples the partial depth images from the same viewpoint to the plurality of partial depth images based on the segmentation information input from the additional information decoding unit 23, reconstructs a depth image, and outputs the reconstructed depth image to the 3D model generation unit 27. For example, the depth images G1, G2, and G3 shown in fig. 4 (a) are output to the 3D model generator 27.

Fig. 9 (b) is a functional block diagram showing an internal configuration of the color image reconstruction unit 24 according to embodiment 1 of the present invention. The color image reconstruction unit 24 includes a color extraction unit 241 and a color coupling unit 243.

The color extraction section 241 extracts (unpacks) a desired partial color image from a plurality of partial color images included in the integrated color image based on the packing information input from the additional information decoding section 23. For example, when the object a and the object b are selected as objects to be reproduced by the reproduction object selection unit 292, the partial color images T1a, T2a, T3a, T1b, T2b, and T3b shown in fig. 7 are extracted and output to the color coupling unit 413. Alternatively, when only the subject b is selected, the partial color images T1b, T2b, and T3b are extracted and output to the color coupling unit.

The color coupling unit 243 couples the partial color images of the same viewpoint from the plurality of partial color images based on the division information input from the additional information decoding unit 23, reconstructs a color image, and outputs the color image to the reproduced image combining unit 28. For example, the color images T1, T2, and T3 shown in fig. 4 (b) are output to the reproduced image combining unit 28.

< embodiment 2>

[ 3D data generating apparatus ]

First, a 3D data generating device according to embodiment 2 of the present invention will be described with reference to the drawings. For convenience of explanation, members having the same functions as those described in the above embodiments are given the same reference numerals, and the explanation thereof will not be repeated.

Fig. 10 is a functional block diagram showing the configuration of a 3D data generating device according to embodiment 2 of the present invention. The 3D data generating device 3 includes a depth image acquiring unit 17, an integrated depth image generating unit 31, a depth image encoding unit 12, a color image acquiring unit 18, an integrated color image generating unit 14, an additional information encoding unit 33, and a multiplexing unit 16.

The integrated depth image generating section 31 generates a single integrated depth image by dividing, quantizing, and integrating (packaging) the plurality of depth images output from the depth image acquiring section 17.

The additional information encoding unit 33 encodes additional information necessary for reconstructing the original depth image from the integrated depth image generated by the integrated depth image generating unit 31 and additional information necessary for reconstructing the original color image from the integrated color image generated by the integrated color image generating unit 14, and outputs additional information encoded data. The details of the additional information will be described later.

Fig. 11 is a functional block diagram showing an internal configuration of the integrated depth image generating unit 31 according to embodiment 2 of the present invention. The integrated depth image generating unit 31 is composed of a depth dividing unit 111, a depth measuring unit 312, and a depth integrating unit 113.

When the resolution at the time of quantization is insufficient, such as when the dynamic range of the divided partial depth image is larger than a predetermined threshold (e.g., 600mm), the depth quantization unit 312 re-quantizes and outputs a partial depth image at a predetermined bit depth (e.g., 12 bits) in accordance with the dynamic range. The depth of the partial depth images G1a, G2a, and G3a shown in fig. 5 ranges from 1000mm to 2000mm, and this range is again linearly quantized with 12 bits. The partial depth images G1b, G2b, and G3b have depths in a range of 2000mm to 2500mm, and the input partial depth images are directly output. The depth quantization unit 312 outputs the minimum value and the maximum value of the value range of the depth of the quantized partial depth image as dynamic range information. For example, the following are output as the dynamic range information of the partial depth images G1a, G2a, G3 a.

[ dynamic Range information of G1a ]

Minimum of depth: 1000mm

Maximum of depth: 2000mm

[ dynamic Range information of G2a ]

Minimum of depth: 1000mm

Maximum of depth: 2000mm

[ dynamic Range information of G3a ]

Minimum of depth: 1000mm

Maximum of depth: 2000mm

With the above configuration, the resolution at the time of quantization can be improved for only a partial depth image with insufficient resolution in the segmentation. As a result, even when the dynamic range of depth is large due to the size and movement of the subject, the lack of resolution can be eliminated. For example, the resolution is about 6.1mm (═ 25000/2^12) in the case where the quantization is performed with 12 bits in the range of 0mm to 25000mm, and about 0.24mm ((2000 ^ 1000)/2^12) in the case where the quantization is performed with 12 bits in the range of 1000mm to 2000 mm. As a result, a higher definition 3D model can be generated on the playback side.

[ 3D data reproducing apparatus ]

Next, a 3D data playback device according to embodiment 2 of the present invention will be described with reference to the drawings. For convenience of explanation, members having the same functions as those described in the above embodiments are given the same reference numerals, and the explanation thereof will not be repeated.

Fig. 12 is a functional block diagram showing the configuration of a 3D data playback device according to embodiment 2 of the present invention. The 3D data playback device 2 includes a separation unit 26, a depth image decoding unit 22, a depth image reconstruction unit 41, an additional information decoding unit 43, a color image decoding unit 25, a color image reconstruction unit 24, a 3D model generation unit 27, a playback image synthesis unit 28, a playback viewpoint input unit 291, and a playback object selection unit 292.

The depth image reconstructing unit 41 reconstructs a depth image by extracting (decapsulating), inversely quantizing, and coupling a desired partial depth image from a plurality of partial depth images included in the integrated depth image decoded by the depth image decoding unit 22.

The additional information decoding unit 43 decodes additional information (segmentation information, encapsulation information, and dynamic range information) necessary for reconstructing a depth image and a color image from the additional information encoded data input from the separation unit 26.

Fig. 13 is a functional block diagram showing an internal configuration of the depth image reconstruction unit 41 according to embodiment 2 of the present invention. The depth image reconstruction unit 41 includes a depth extraction unit 211, a depth inverse quantization unit 412, and a depth coupling unit 213.

When the dynamic range information corresponding to the extracted partial depth image exists, the depth inverse quantization unit 412 inversely quantizes and outputs the partial depth image based on the dynamic range information. In the absence, the input partial depth image is directly output.

With the above configuration, the resolution at the time of quantization can be improved for only a partial depth image with insufficient resolution in the segmentation. As a result, the quantization error in encoding of the depth image can be reduced, and a higher definition 3D model can be generated.

< embodiment 3>

[ 3D data generating apparatus ]

First, a 3D data generating device according to embodiment 3 of the present invention will be described with reference to the drawings. For convenience of explanation, members having the same functions as those described in the above embodiments are given the same reference numerals, and the explanation thereof will not be repeated.

Fig. 14 is a functional block diagram showing the configuration of a 3D data generating device according to embodiment 3 of the present invention. The 3D data generating device 5 includes a depth image acquiring unit 17, an integrated depth image generating unit 51, a depth image encoding unit 12, a color image acquiring unit 18, an integrated color image generating unit 54, a color image encoding unit 15, an additional information encoding unit 13, a multiplexing unit 16, a depth image filtering unit 52, a color image filtering unit 53, and a reproduction target receiving unit 55.

The integrated depth image generator 51 divides the plurality of depth images output from the depth image acquirer 17, and generates a single integrated depth image by integrating (encapsulating) a partial depth image of a specific imaging target or a partial depth image in a specific imaging direction into a predetermined coding unit (e.g., a block of HEVC).

The integrated color image generating unit 54 divides the plurality of color images output from the color image acquiring unit 18 based on the division information and the encapsulation information output from the integrated depth image generating unit 51, and integrates (encapsulates) the partial color image of the specific imaging target or the partial color image in the specific imaging direction into a single integrated color image that falls within a predetermined coding unit (for example, a block of HEVC), in the same manner as the integrated depth image generating unit 51.

The depth image filtering unit 52 outputs a block including the reproduction target (the subject to be photographed, the photographing direction, and the like) specified by the reproduction target accepting unit 55, from the encoded data output from the depth image encoding unit 12. In the case where the reproduction object is not specified, all tiles are output.

The color image filtering unit 53 outputs a block including a reproduction target (an imaging target, an imaging direction, and the like) specified by the reproduction target accepting unit 55 among the encoded data output from the color image encoding unit 15. In the case where the reproduction object is not specified, all tiles are output.

The reproduction target reception unit 55 receives a user request for a reproduction target (for example, an imaging target a, an imaging target b, an imaging direction forward, an imaging direction backward, and the like).

Fig. 15 (a) is a functional block diagram showing an internal configuration of the integrated depth image generating unit 51 according to embodiment 3 of the present invention. The integrated depth image generating unit 51 includes a depth dividing unit 111 and a depth integrating unit 513.

The depth integration section 513 integrates (encapsulates) the partial depth image of the specific subject or the partial depth image of the specific shooting direction to fall into the same tile, thereby generating a single integrated depth image. The depth integration unit 513 outputs, as the package information, the identifiers of the imaging subjects and the imaging directions of the partial depth images included in the respective tiles, in addition to the package information in embodiment 1.

Fig. 15 (b) is a functional block diagram showing an internal configuration of the integrated color image generating unit 54 according to embodiment 3 of the present invention. The integrated color image generating unit 54 includes a color dividing unit 141 and a color integrating unit 543.

The color integration unit 543 integrates (encapsulates) the partial color image of the specific subject or the partial color image in the specific shooting direction so as to fall within the same tile, based on the encapsulation information input from the integrated depth image generation unit 51, thereby generating a single integrated color image.

Fig. 16 is a diagram showing an example of obtaining a depth image and a color image according to embodiment 3 of the present invention. The case where 5 cameras C1, C2, C3, C4, and C5 are arranged for the subject a and the subject b, and each camera captures a depth image and a color image is shown.

Fig. 17 (a) is a diagram showing an example of packaging of depth images in embodiment 3 of the present invention. In this example, the integrated depth image is encoded in two tiles according to the subject. The partial depth images G1a, G2a, G3a, G4a, and G5a of the photographic subject a captured by the cameras C1, C2, C3, C4, and C5 are packaged in the tile 1, the partial depth images G1b, G2b, G3b, G4b, and G5b of the photographic subject b captured by the cameras C1, C2, C3, C4, and C5 are packaged in the tile 2, and a single integrated depth image is output. Further, the depth integration section 513 outputs the following package information.

[ encapsulation information ]

Partial depth image comprised by tile 1: shooting object as a

Partial depth image comprised by tile 2: subject b

And performing coding control on the background area of each part of the depth image in the integrated depth image based on the shape information. The shape information is information showing whether or not each pixel of the integrated depth image belongs to a subject (photographic subject), and for example, "1" is assigned in the case of a pixel belonging to a subject, and "0" is assigned in the case of a pixel not belonging to a subject. In the encoding process, for example, when all pixels in the CTU (coding tree unit) do not belong to the object, or when some pixels in the CTU do not belong to the object, a process such as encoding is performed after filling a region not belonging to the object with a pixel value of an edge of the object or a predetermined pixel value in a horizontal direction or a vertical direction. The depth coupling section 513 outputs the shape information as package information.

Fig. 17 (b) is a diagram showing an example of encapsulation of a color image according to embodiment 3 of the present invention. Similarly to the integrated depth image, the partial color images T1a, T2a, T3a, T4a, and T5a of the subject a are packaged in the tile 1, and the partial color images T1b, T2b, T3b, T4b, and T5b of the subject b are packaged in the tile 2, and a single integrated color image is output.

The background region of each partial color image in the integrated color image is subjected to encoding control based on the package information (shape information) input from the integrated depth image generating unit 11. For example, when all pixels in the CTU do not belong to the object, or when some pixels in the CTU do not belong to the object, a process such as encoding is performed after filling a region not belonging to the object with a pixel value of an edge of the object or a predetermined pixel value in the horizontal direction or the vertical direction.

Fig. 18 (a) is a diagram showing another example of the package of the depth image according to embodiment 3 of the present invention. In this example, the integrated depth image is divided into two tiles according to the shooting direction for encoding. The partial depth images G1a, G2a, G3a, G1b, G2b, and G3b captured by the cameras C1, C2, and C3 from the front are packaged in tile 1, and the partial depth images G4a, G5a, G4b, and G5b captured by the cameras C4 and C5 from the rear are packaged in tile 2, and a single integrated depth image is output. Further, the depth integration section 513 outputs the following package information.

[ encapsulation information ]

Partial depth image comprised by tile 1: forward direction of shooting

Partial depth image comprised by tile 2: rear direction of the image

Fig. 18 (b) is a diagram showing another example of the package of the color image according to embodiment 3 of the present invention. Similarly to the integrated depth image, the partial color images T1a, T2a, T3a, T1b, T2b, and T3b captured from the front are packaged in the tile 1, and the partial color images T4a, T5a, T4b, and T5b captured from the rear are packaged in the tile 2, and a single integrated color image is output.

With the above configuration, the dynamic range of the depth values in each CTU constituting the partial depth image can be reduced, and the resolution in quantization can be improved. As a result, even when the dynamic range of depth is large due to the size and movement of the subject, the lack of resolution can be eliminated. In addition, when the user wants to reproduce only a specific subject or shooting direction, the 3D data necessary for reproduction can be efficiently transmitted even in a limited network bandwidth such as a mobile environment by transmitting only a tile including a partial depth image of the corresponding subject or shooting direction. On the reproduction side, only a part of the tiles need to be decoded, and thus the amount of processing required for decoding can be reduced. Furthermore, the depth image used for the generation of the 3D model is limited, and therefore the amount of processing required for the generation of the 3D model can be reduced.

In the above description, the coding unit is an HEVC block, but similar effects are also obtained even in other coding units such as HEVC slices.

[ 3D data reproducing apparatus ]

Next, a 3D data playback device according to embodiment 3 of the present invention will be described with reference to the drawings. For convenience of explanation, members having the same functions as those described in the above embodiments are given the same reference numerals, and the explanation thereof will not be repeated.

Fig. 19 is a functional block diagram showing the configuration of a 3D data playback device according to embodiment 3 of the present invention. The 3D data playback device 6 includes a separation unit 26, a depth image decoding unit 22, a depth image reconstruction unit 21, an additional information decoding unit 23, a color image decoding unit 25, a color image reconstruction unit 24, a 3D model generation unit 27, a playback image synthesis unit 28, a playback viewpoint input unit 291, a playback object selection unit 292, a depth image filtering unit 62, and a color image filtering unit 63.

The depth image filtering unit 62 outputs a tile including a partial depth image corresponding to the reproduction target (the subject or the direction of shooting) specified by the reproduction target selecting unit 292, from the encoded data output from the separating unit 26. For example, when a is designated as a subject, block 1 in fig. 17 (a) is output. Alternatively, when the rear side is designated as the shooting direction, block 2 in fig. 18 (a) is output. In the case where the reproduction object is not specified, all tiles are output.

Here, a method of decoding a portion of tiles in the case where tile 1 and tile 2 in the integrated depth image are stored in the same slice is explained.

Step 1: the playback target selection unit refers to the package information and acquires the tile number K (K1 or K2) of the specified playback target.

Step 2: the depth image filtering unit decodes the entry _ point _ offset _ minus1 syntax element in the slice header to obtain the byte length N of the encoded data of the tile 1.

And step 3: when K is 1, the depth image filtering unit outputs data up to N bytes of slice header and slice data. When K is 2, the depth image filtering section outputs a slice header and data after N +1 bytes of slice data.

And 4, step 4: the depth image decoding unit decodes slice data of the tile K.

The color image filtering unit 63 outputs a tile including a partial color image corresponding to the reproduction target (the subject or the direction of shooting) specified by the reproduction target selecting unit 292, from the encoded data output from the separating unit 26. For example, when a is designated as the subject, block 1 in fig. 17 (b) is output. Alternatively, when the rear side is designated as the shooting direction, block 2 in fig. 18 (b) is output. In the case where the reproduction object is not specified, all tiles are output.

Similarly, a method of decoding a portion of the tiles in the case where tile 1 and tile 2 in the integrated color image are stored in the same slice is described.

Step 1: the playback target selection unit refers to the package information and acquires the tile number K (K1 or K2) of the specified playback target.

Step 2: the color image filtering section decodes the entry _ point _ offset _ minus1 syntax element of the slice header to obtain the byte length N of the encoded data of tile 1.

And step 3: when K is 1, the color image filtering section outputs data up to N bytes of slice header and slice data. When K is 2, the color image filtering section outputs a slice header and data after N +1 bytes of slice data.

And 4, step 4: the color image decoding section decodes slice data of the tile K.

With the above configuration, it is possible to easily perform control of a reproduction target according to the processing capability of the terminal, such as enabling reproduction of all the subjects or the shooting directions by decoding all the tiles and generating an overall 3D model in a reproduction terminal having a high processing capability, enabling reproduction of only a specific subject or shooting direction by decoding only a part of the tiles and generating a part of the 3D model in a reproduction terminal having a low processing capability, and the like.

[ software-based implementation example ]

The control blocks (for example, the integrated depth image generating unit 11 and the integrated color image generating unit 14) of the 3D data generating apparatus 1 and the control blocks (for example, the depth image reconstructing unit 21 and the color image reconstructing unit 24) of the 3D data reproducing apparatus 2 may be implemented by logic circuits (hardware) formed in an integrated circuit (IC chip) or the like, or may be implemented by software.

In the latter case, the 3D data generation device 1 and the 3D data reproduction device 2 are provided with a computer that executes commands of a program that is software for realizing the respective functions. For example, the computer includes at least one processor (control device), and at least one computer-readable recording medium storing the program. In the computer, the processor reads a program from the recording medium and executes the program, thereby achieving the object of the present invention. The processor may be, for example, a Central Processing Unit (CPU). As the recording medium, a magnetic tape, a magnetic disk, a card, a semiconductor Memory, a programmable logic circuit, or the like can be used in addition to a "non-transitory tangible medium" such as a ROM (Read Only Memory) or the like. The program may be provided with a RAM (Random Access Memory) or the like for expanding the program. Further, the program may be supplied to the computer via an arbitrary transmission medium (a communication network, a broadcast wave, or the like) that can transmit the program. It should be noted that an aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

[ conclusion ]

A 3D data generating device according to claim 1 of the present invention is a 3D data generating device that generates 3D data by inputting a depth image representing a three-dimensional shape of one or a plurality of imaging subjects, and includes: a depth dividing unit configured to divide the depth image into a plurality of partial depth images each including a rectangular region; a depth integration unit for generating an integrated depth image by encapsulating the plurality of partial depth images; a depth image encoding unit that encodes the integrated depth image; and an additional information encoding unit that encodes additional information including division information for specifying the rectangular region and information indicating the package.

In the 3D data generating device according to claim 2 of the present invention, the additional information further includes information indicating a dynamic range of depth values in the partial depth image, and the 3D data generating device further includes a depth quantization unit configured to quantize the plurality of partial depth images based on the dynamic range.

In the 3D data generating device according to claim 3 of the present invention, the depth integration unit encapsulates partial depth images of the same subject to be captured into the same coding unit.

In the 3D data generating device according to claim 4 of the present invention, the depth integration unit encapsulates partial depth images having the same imaging direction into the same coding unit.

A 3D data playback device according to claim 5 of the present invention is a 3D data playback device that inputs 3D data and plays back a three-dimensional shape of one or more objects to be captured, and includes: a depth image decoding unit that decodes an integrated depth image included in the 3D data; an additional information decoding unit that decodes additional information including information indicating a package of a plurality of partial depth images each including a rectangular region included in the integrated depth image and division information for specifying the rectangular region; a depth extraction unit that extracts a partial depth image from the decoded integrated depth image based on information indicating the capsule; and a depth coupling unit that couples the plurality of partial depth images based on the division information to reconstruct a depth image.

In the 3D data playback device according to claim 6 of the present invention, the additional information further includes information indicating a dynamic range of depth values in the partial depth image, and the 3D data playback device further includes a depth inverse quantization unit configured to inversely quantize the plurality of partial depth images based on the dynamic range.

In the 3D data playback device according to claim 7 of the present invention, the partial depth images that are the same as the subject of imaging are encoded in the same coding unit in the 3D data.

In the 3D data playback device according to claim 8 of the present invention, the partial depth images having the same shooting direction are encoded in the same coding unit in the 3D data.

In this case, a control program for a 3D data generating apparatus in which the 3D data generating apparatus is realized by a computer by operating the computer as each unit (software element) provided in the 3D data generating apparatus, and a computer-readable recording medium in which the control program is recorded also fall within the scope of the present invention.

The present invention is not limited to the above embodiments, and various modifications can be made within the scope of the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments are also included in the technical scope of the present invention. Further, by combining the technical means disclosed in the respective embodiments, new technical features can be formed.

(cross-reference to related applications)

The present application addresses japanese patent applications filed on 2018, 9, 28: japanese patent application 2018-183903 claims the benefit of priority and includes the entire contents thereof in the present specification by reference thereto.

Description of the reference numerals

13D data generating device

11 integration depth image generating unit

111 depth division part

113 depth integration part

12-depth image encoding unit

13 additional information encoding unit

14 integral color image generating part

15 color image encoding section

16 multiplexing unit

17 depth image acquiring unit

18 color image acquiring section

23D data reproducing device

21 depth image reconstruction unit

211 depth extracting part

213 deep coupling part

22 depth image decoding unit

23 additional information decoding unit

24 color image reconstruction unit

25 color image decoding section

26 separating part

273D model generating unit

28 reproduced image synthesizing unit

291 reproduction viewpoint input unit

292 reproduction object selection part

33D data generating device

31 integration depth image generating unit

33 additional information encoding unit

312 depth measuring unit

43D data reproducing apparatus

41 depth image reconstruction unit

43 additional information decoding unit

413 depth inverse quantization part

53D data generating device

51 integration depth image generating unit

513 depth integration

54 integral color image generating part

543 color integration part

52 depth image filtering unit

53 color image filtering section

63D data reproducing device

62 depth image filtering unit

63 color image filtering section