阅读说明：本技术 估计单个图像的深度图的方法和设备 (Estimate the method and apparatus of the depth map of single image ) 是由 李斐 刘汝杰 于 2018-05-23 设计创作，主要内容包括：本发明公开了一种估计单个图像的深度图的方法和设备。该方法包括：a)获取标注有语义标签的单个图像；以及b)根据该图像及该图像的语义标签,估计该图像的深度图。(The invention discloses a kind of methods and apparatus of depth map for estimating single image.This method comprises: a) obtaining the single image for being labeled with semantic label；And the depth map of the image b) is estimated according to the image and the semantic label of the image.)

1. a kind of method for the depth map for estimating single image, comprising:

A) single image for being labeled with semantic label is obtained；And

B) according to the image and the semantic label of the image, estimate the depth map of the image.

2. the method for claim 1, wherein the step a) includes:

Obtain single image；

According to the image, the semantic label of the image is obtained.

3. the method as described in claim 1, further includes:

C) according to the image and estimated depth map, the semantic label of the image is obtained；

D) according to the image and semantic label obtained, estimate the depth map of the image.

4. method as claimed in claim 3, further includes:

Step c), d) is repeated, pre-determined number is reached.

5. the method for claim 1, wherein institute's semantic tags indicate the type in the region in the image.

6. method as claimed in claim 5, wherein the type in the region is between the depth of the pixel in the region Relationship has strong constraint.

7. method as claimed in claim 3, wherein above-mentioned steps b), c), d) respectively by deep neural network different from each other It executes, the deep neural network is separately or concurrently trained.

8. a kind of method for the semantic label for estimating single image, comprising:

A' the single image with depth map) is obtained；And

B') according to the image and the depth map of the image, estimate the semantic label of the image.

9. method according to claim 8, further includes:

C') according to the image and estimated semantic label, the depth map of the image is obtained；

D') according to the image and depth map obtained, estimate the semantic label of the image.

10. a kind of equipment for the depth map for estimating single image, comprising:

Acquisition device is configured as: obtaining the single image for being labeled with semantic label；And

Estimation device is configured as: according to the image and the semantic label of the image, estimating the depth map of the image.

Technical field

This invention relates generally to 3-D image process fields.Specifically, the present invention relates to one kind can estimate individually The method and apparatus of the depth map of image.

Background technique

In recent years, with the development of 3 dimension imaging technology, many relevant applications have been emerged in large numbers, as augmented reality, number are rich Object shop, 3 D-printing etc..The importance of 3 dimension imaging technology is three-dimensional reconstruction.Depth information is for three-dimensional reconstruction to pass It is important.In general, can be from single image, two images or more than two Image estimation depth.Wherein, estimate from single image Meter depth only needs an image, and estimated depth can be advantageously available for the computer visions such as Object identifying, Attitude estimation Using.But the Limited information that single image provides, the precision of estimation of Depth are to be improved.

Traditional method is using gradient data and multi-scale information come from single image estimating depth figure.However, image Relationship between Pixel Information and depth information is complex, and direct mapping between the two is difficult to learn.Therefore, it is necessary to utilize Additional information helps to improve the precision of estimation of Depth.

The present invention is directed to the precision from single image estimating depth is improved using semantic label information.

Summary of the invention

It has been given below about brief overview of the invention, in order to provide about the basic of certain aspects of the invention Understand.It should be appreciated that this summary is not an exhaustive overview of the invention.It is not intended to determine pass of the invention Key or pith, nor is it intended to limit the scope of the present invention.Its purpose only provides certain concepts in simplified form, Taking this as a prelude to a more detailed description discussed later.

The basic idea of the invention is that the semantic label of Pixel-level can indicate which class that each pixel belongs in image The region of type, some type of region, such as wall, the depth with relatively uniform or linear/even variation, therefore, these languages Adopted label has strong constraint to the depth of pixel.It can learn such constraint using deep neural network, and provide more quasi- True depth estimation result.In addition, inventor notice the influence between semantic label and depth be it is mutual, can also be from depth Degree figure, which sets out, estimates semantic label, and the accuracy of semantic label is improved by means of depth map.It is possible to further pass through depth Scheme to semantic label to arrive the continuous iteration of depth map again, further increases depth map/semantic label estimated accuracy.

To achieve the goals above, according to an aspect of the invention, there is provided a kind of depth map for estimating single image Method, be labeled with the single image of semantic label this method comprises: a) obtaining；And b) according to the image and the language of the image Adopted label estimates the depth map of the image.

According to another aspect of the present invention, a kind of equipment of depth map for estimating single image, the equipment packet are provided Include: acquisition device is configured as: obtaining the single image for being labeled with semantic label；And estimation device, it is configured as: according to The semantic label of the image and the image estimates the depth map of the image.

In accordance with a further aspect of the present invention, a kind of method of semantic label for estimating single image, this method packet are provided Include: a ') obtain the single image with depth map；And b ') according to the image and the depth map of the image, estimate the image Semantic label.

In addition, according to another aspect of the present invention, additionally providing a kind of storage medium.The storage medium includes that machine can The program code of reading, when executing said program code on information processing equipment, said program code makes at the information Equipment is managed to execute according to the above method of the present invention.

In addition, in accordance with a further aspect of the present invention, additionally providing a kind of program product.Described program product includes that machine can The instruction of execution, when executing described instruction on information processing equipment, described instruction executes the information processing equipment According to the above method of the present invention.

Detailed description of the invention

Referring to reference to the accompanying drawing to the explanation of the embodiment of the present invention, the invention will be more easily understood it is above and Other objects, features and advantages.Component in attached drawing is intended merely to show the principle of the present invention.In the accompanying drawings, identical or class As technical characteristic or component will be indicated using same or similar appended drawing reference.In attached drawing:

Fig. 1 shows the flow chart of the method for the depth map of the estimation single image of embodiment according to the present invention；

Fig. 2 shows the flow charts of the method for the depth map of estimation single image according to another embodiment of the present invention；

Fig. 3 shows the structural block diagram of the equipment of the depth map of the estimation single image of embodiment according to the present invention；

Fig. 4 shows the structure box of the equipment of the depth map of estimation single image according to another embodiment of the present invention Figure；

Fig. 5 shows the flow chart of the method for the semantic label of the estimation single image of embodiment according to the present invention；

Fig. 6 shows the process of the method for the semantic label of estimation single image according to another embodiment of the present invention Figure；And

Fig. 7 shows the schematic frame for the computer that can be used for implementing the method and apparatus of embodiment according to the present invention Figure.

Specific embodiment

Exemplary embodiment of the invention is described in detail hereinafter in connection with attached drawing.It rises for clarity and conciseness See, does not describe all features of actual implementation mode in the description.It should be understood, however, that developing any this reality Much decisions specific to embodiment must be made during embodiment, to realize the objectives of developer, For example, meeting restrictive condition those of related to system and business, and these restrictive conditions may be with embodiment It is different and change.In addition, it will also be appreciated that although development is likely to be extremely complex and time-consuming, to benefit For those skilled in the art of present disclosure, this development is only routine task.

Here, and also it should be noted is that, in order to avoid having obscured the present invention because of unnecessary details, in the accompanying drawings Illustrate only with closely related apparatus structure and/or processing step according to the solution of the present invention, and be omitted and the present invention The little other details of relationship.In addition, it may also be noted that being described in an attached drawing of the invention or a kind of embodiment Elements and features can be combined with elements and features shown in one or more other attached drawings or embodiment.

The stream of the method for the depth map of the estimation single image of embodiment according to the present invention is described below with reference to Fig. 1 Journey.

Fig. 1 shows the flow chart of the method for the depth map of the estimation single image of embodiment according to the present invention.Such as Fig. 1 Shown, this method comprises the following steps: obtaining the single image (step S1) for being labeled with semantic label；And according to the image and The semantic label of the image estimates the depth map (step S2) of the image.

Specifically, in step sl, the single image for being labeled with semantic label is obtained.

Semantic label indicates the type in the region in the image.Depth of the type in the region to the pixel in the region Relationship between degree has strong constraint.For example, the pixel of semantic label instruction wall, the depth value of the pixel of a face wall is unified (viewer for facing viewing image), or (inclination or the tiltedly viewer of opposite viewing image) of linear/even variation. Such semantic label just has strong constraint to the depth value of the pixel in indicated region, it is sufficient to for assisting to depth Estimation.Similar semantic label is, for example, ground, ceiling, road surface, facade of building etc..

As an implementation, the single image for being labeled with semantic label can be directly obtained.Its semantic label It is and single image ready-made semantic label provided together.

As another embodiment, step S1 may include: acquisition single image；According to the image, the image is obtained Semantic label.That is, semantic label here is that method of the invention oneself obtains.It may be manually to mark, Possibly by other processing as semantic segmentation generates.It all can be obtained herein from single image using known in the art The technology for obtaining semantic label is realized.

In step s 2, according to the image and the semantic label of the image, estimate the depth map of the image.

Step S2 is realized by trained first deep neural network.First deep neural network can input The semantic label of single image and the image, and export the depth map of the image.

Pass through the depth map using the training image and training image for being labeled with semantic label, the first depth nerve net of training Network.First deep neural network could be aware which semantic label has strong constraint for depth, how using by force by study Constraint, generates accurate depth value.

Fig. 2 shows the flow charts of the method for the depth map of estimation single image according to another embodiment of the present invention. As shown in Fig. 2, this method comprises the following steps: obtaining the single image (step S21) for being labeled with semantic label；According to the image And the semantic label of the image, estimate the depth map (step S22) of the image；According to the image and estimated depth map, obtain Obtain the semantic label (step S23) of the image；And according to the image and semantic label obtained, estimate the depth of the image Scheme (step S24).

Wherein, step S21, S22 is identical as step S1, S2 described referring to Fig.1 before.Step S22 is by the first depth mind Through network implementations.

The depth map that the present embodiment is obtained by step S22, using depth and semantic label mutually referring to, interact Principle obtains more accurate semantic label, and then utilize more accurate semantic label, with step S22 in step S23 Similarly, more accurate depth map is estimated again in step s 24.

As one can imagine can also continue to iteration after step S24 and execute following step:

I) according to the image and estimated depth map, the semantic label of the image is obtained；

I i) according to the image and semantic label obtained, estimate the depth map of the image.

The number of iterations is, for example, 1,2 time.

It should be noted that step S23 is realized by the second deep neural network.Second deep neural network utilizes training image, instruction The semantic label training for practicing the depth map, training image of image obtains.Second deep neural network by training study to how The semantic label of pixel in image is obtained according to image and its depth map.Second deep neural network and the first deep neural network Training is different, function is different, network parameter is different.

Step S24 is realized by third deep neural network.The first depth mind is similar in third deep neural network principle Through network, but due to its input be step S22 estimation depth map based on the semantic label estimated, training data It is different from the training data for realizing the first deep neural network of step S22, so, third deep neural network and the first depth The network parameter of neural network is different.

In successive ignition step i), i i) in the case where, execute each step i), i i) be depth different from each other Neural network.

The all of above deep neural network referred to can train simultaneously, can also be respectively trained.

The equipment of the depth map of the estimation single image of embodiment according to the present invention is described next, with reference to Fig. 3.

Fig. 3 shows the structural block diagram of the equipment of the depth map of the estimation single image of embodiment according to the present invention. As shown in figure 3, depth map estimation equipment 300 according to the present invention includes: the first acquisition device 31, it is configured as: obtains mark There is the single image of semantic label；And first estimation device 32, it is configured as: being marked according to the image and the semantic of the image Label, estimate the depth map of the image.

In one embodiment, first acquisition device 31 is further configured to: obtaining single image；According to the figure Picture obtains the semantic label of the image.

Fig. 4 shows the structure box of the equipment of the depth map of estimation single image according to another embodiment of the present invention Figure.As shown in figure 4, depth map estimation equipment 400 according to the present invention includes: the first acquisition device 41, it is configured as: obtains mark It is marked with the single image of semantic label；First estimation device 42, is configured as: according to the image and the semantic label of the image, Estimate the depth map of the image；Second acquisition device 43, is configured as: according to the image and as estimated by the first estimation device 42 Depth map, obtain the semantic label of the image；Second estimation device 44, is configured as: obtaining according to the image and by second The semantic label obtained of device 43, estimates the depth map of the image.

In one embodiment, depth map according to the present invention estimates equipment 400 further include: with the first acquisition device 41 and The different multiple acquisition device of second acquisition device 43, are configured as: according to the image and as estimated by prime estimation device Depth map obtains the semantic label of the image；The multiple estimations dress different from the first estimation device 42 and the second estimation device 44 It sets, is configured as: according to the image and by prime acquisition device semantic label obtained, estimating the depth map of the image.

In one embodiment, institute's semantic tags indicate the type in the region in the image.

In one embodiment, the type in the region has the relationship between the depth of the pixel in the region strong Constraint.

In one embodiment, described first, second, multiple acquisition device, described first, second, multiple estimation devices It is realized respectively by deep neural network different from each other.

In one embodiment, the deep neural network is separately or concurrently trained.

By institute in processing and method described above included in depth map according to the present invention estimation equipment 400 Including each step in processing it is similar, therefore for simplicity, omit the detailed description of these processing herein.

Interact since depth map and semantic label exist, can reference each other, therefore, for depth map Single image, can according to depth map estimate semantic label.

Fig. 5 shows the flow chart of the method for the semantic label of the estimation single image of embodiment according to the present invention.Such as Shown in Fig. 5, this method comprises the following steps: obtaining the single image (step S51) with depth map；And according to the image and The depth map of the image estimates the semantic label (step S52) of the image.

In one embodiment, step S51 may include: acquisition single image；According to the image, the depth of the image is obtained Degree figure.

In another embodiment, as shown in fig. 6, the method for the semantic label of estimation single image includes the following steps: Obtain the single image (step S61) with depth map；According to the image and the depth map of the image, the semanteme of the image is estimated Label (step S62)；According to the image and estimated semantic label, the depth map (step S63) of the image is obtained；According to this Image and depth map obtained estimate the semantic label (step S64) of the image.

In one embodiment, step S63, S64 is repeated, pre-determined number is reached.

In one embodiment, pre-determined number includes 1,2 time.

In one embodiment, institute's semantic tags indicate the type in the region in the image.

In one embodiment, above-mentioned steps S62, S63, S64 is executed by deep neural network different from each other respectively.

In one embodiment, above-mentioned all deep neural networks are separately or concurrently trained.

Correspondingly, a kind of equipment of semantic label for estimating single image is provided, comprising: acquiring unit is configured as: Obtain the single image with depth map；And estimation unit, it is configured as: according to the image and the depth map of the image, estimating Count the semantic label of the image.

In addition, it is still necessary to, it is noted that each component devices, unit can be by softwares, firmware, hard in above equipment here The mode of part or combinations thereof is configured.It configures workable specific means or mode is well known to those skilled in the art, In This is repeated no more.In the case where being realized by software or firmware, from storage medium or network to specialized hardware structure Computer (such as general purpose computer 700 shown in Fig. 7) installation constitutes the program of the software, which is being equipped with various journeys When sequence, it is able to carry out various functions etc..

Fig. 7 shows the schematic frame for the computer that can be used for implementing the method and apparatus of embodiment according to the present invention Figure.

In Fig. 7, central processing unit (CPU) 701 is according to the program stored in read-only memory (ROM) 702 or from depositing The program that storage part 708 is loaded into random access memory (RAM) 703 executes various processing.In RAM 703, also according to need Store the data required when CPU 701 executes various processing etc..CPU 701, ROM 702 and RAM 703 are via bus 704 are connected to each other.Input/output interface 705 is also connected to bus 704.

Components described below is connected to input/output interface 705: importation 706 (including keyboard, mouse etc.), output section Divide 707 (including display, such as cathode-ray tube (CRT), liquid crystal display (LCD) etc. and loudspeakers etc.), storage section 708 (including hard disks etc.), communications portion 709 (including network interface card such as LAN card, modem etc.).Communications portion 709 Communication process is executed via network such as internet.As needed, driver 710 can be connected to input/output interface 705. Detachable media 711 such as disk, CD, magneto-optic disk, semiconductor memory etc., which can according to need, is installed in driver On 710, so that the computer program read out is mounted to as needed in storage section 708.

It is such as removable from network such as internet or storage medium in the case where series of processes above-mentioned by software realization Unload the program that the installation of medium 711 constitutes software.

It will be understood by those of skill in the art that this storage medium be not limited to it is shown in Fig. 7 be wherein stored with program, Separately distribute with equipment to provide a user the detachable media 711 of program.The example of detachable media 711 includes disk (including floppy disk (registered trademark)), CD (including compact disc read-only memory (CD-ROM) and digital versatile disc (DVD)), magneto-optic disk (including mini-disk (MD) (registered trademark)) and semiconductor memory.Alternatively, storage medium can be ROM 702, storage section Hard disk for including in 708 etc., wherein computer program stored, and user is distributed to together with the equipment comprising them.

The present invention also proposes a kind of program product of instruction code for being stored with machine-readable.Described instruction code is by machine When device reads and executes, method that above-mentioned embodiment according to the present invention can be performed.

Correspondingly, it is also wrapped for carrying the storage medium of the program product of the above-mentioned instruction code for being stored with machine-readable It includes in disclosure of the invention.The storage medium includes but is not limited to floppy disk, CD, magneto-optic disk, storage card, memory stick etc. Deng.

In the description above to the specific embodiment of the invention, for the feature a kind of embodiment description and/or shown It can be used in one or more other embodiments in a manner of same or similar, with the feature in other embodiment It is combined, or the feature in substitution other embodiment.

It should be emphasized that term "comprises/comprising" refers to the presence of feature, element, step or component when using herein, but simultaneously It is not excluded for the presence or additional of one or more other features, element, step or component.

In addition, method of the invention be not limited to specifications described in time sequencing execute, can also according to it His time sequencing, concurrently or independently execute.Therefore, the execution sequence of method described in this specification is not to this hair Bright technical scope is construed as limiting.

Although being had been disclosed above by the description to specific embodiments of the present invention to the present invention, it answers The understanding, above-mentioned all embodiments and example are exemplary, and not restrictive.Those skilled in the art can be in institute Design is to various modifications of the invention, improvement or equivalent in attached spirit and scope of the claims.These modification, improve or Person's equivalent should also be as being to be considered as included in protection scope of the present invention.

Note:

1. a kind of method for the depth map for estimating single image, comprising:

A) single image for being labeled with semantic label is obtained；And

B) according to the image and the semantic label of the image, estimate the depth map of the image.

2. the method as described in note 1, wherein the step a) includes:

Obtain single image；

According to the image, the semantic label of the image is obtained.

3. the method as described in note 1, further includes:

C) according to the image and estimated depth map, the semantic label of the image is obtained；

D) according to the image and semantic label obtained, estimate the depth map of the image.

4. such as method described in note 3, further includes:

Step c), d) is repeated, pre-determined number is reached.

5. the method as described in note 1, wherein institute's semantic tags indicate the type in the region in the image.

6. the method as described in note 5, wherein the type in the region is between the depth of the pixel in the region Relationship has strong constraint.

7. such as method described in note 3, wherein above-mentioned steps b), c), d) respectively by deep neural network different from each other It executes.

8. the method as described in note 7, wherein the deep neural network is separately or concurrently trained.

9. a kind of equipment for the depth map for estimating single image, comprising:

First acquisition device, is configured as: obtaining the single image for being labeled with semantic label；And

First estimation device, is configured as: according to the image and the semantic label of the image, estimating the depth of the image Figure.

10. such as equipment described in note 9, wherein first acquisition device is further configured to:

Obtain single image；

According to the image, the semantic label of the image is obtained.

11. such as equipment described in note 9, further includes:

Second acquisition device, is configured as: according to the image and the depth map as estimated by the first estimation device, being somebody's turn to do The semantic label of image；

Second estimation device, is configured as: according to the image and by the second acquisition device semantic label obtained, estimation The depth map of the image.

12. the equipment as described in note 11, further includes:

The multiple acquisition device different from the first acquisition device and the second acquisition device, are configured as: according to the image and The depth map as estimated by prime estimation device obtains the semantic label of the image；

The multiple estimation devices different from the first estimation device and the second estimation device, are configured as: according to the image and By prime acquisition device semantic label obtained, the depth map of the image is estimated.

13. such as equipment described in note 9, wherein institute's semantic tags indicate the type in the region in the image.

14. the equipment as described in note 13, wherein the type in the region is between the depth of the pixel in the region Relationship have strong constraint.

15. the equipment as described in note 12, wherein described first, second, multiple acquisition device, it is described first, second, Multiple estimation devices are realized by deep neural network different from each other respectively.

16. the equipment as described in note 15, wherein the deep neural network is separately or concurrently trained.

17. a kind of method for the semantic label for estimating single image, comprising:

A' the single image with depth map) is obtained；And

B') according to the image and the depth map of the image, estimate the semantic label of the image.

18. such as method as stated in Note 17, further includes:

C') according to the image and estimated semantic label, the depth map of the image is obtained；

D') according to the image and depth map obtained, estimate the semantic label of the image.

19. the method as described in note 18, further includes:

Repeat step c '), d '), reach pre-determined number.

20. such as method as stated in Note 17, wherein institute's semantic tags indicate the type in the region in the image.