阅读说明：本技术 增强现实图像处理方法及装置、增强现实图像显示设备 (Augmented reality image processing method and device and augmented reality image display equipment ) 是由 张志扬 苏进 于勇 李琦 于 2019-09-03 设计创作，主要内容包括：本发明涉及图像处理领域,具体地涉及增强现实图像处理方法及装置、增强现实图像显示设备。该增强现实图像处理方法包括：采集现实世界的图像；接收针对所述图像的缩放操作；响应于所述缩放操作,对所述图像进行缩放处理以适应不同的视力水平,并对所述图像进行预畸形化处理；以及对被缩放和预畸形化的图像进行光学缩放处理,其中,所述预畸形化处理用于矫正在所述光学缩放处理时引起的光学畸形。该增强现实图像处理方法及装置、显示设备能够为低视力用户提供能适应其视力状况的AR体验,并且结构简单,体积轻巧,因而便于使用。(The invention relates to the field of image processing, in particular to an augmented reality image processing method and device and an augmented reality image display device. The augmented reality image processing method comprises the following steps: acquiring an image of the real world; receiving a zoom operation for the image; in response to the scaling operation, scaling the image to accommodate different vision levels and pre-morphing the image; and performing optical zoom processing on the zoomed and pre-deformated image, wherein the pre-deformatting processing is used for correcting optical deformities caused during the optical zoom processing. The augmented reality image processing method and device and the display equipment can provide AR experience which can adapt to the vision condition of a low-vision user, and are simple in structure, light in size and convenient to use.)

1. An augmented reality image processing method, characterized by comprising:

acquiring an image of the real world;

receiving a zoom operation for the image;

in response to the scaling operation, scaling the image to accommodate different vision levels and pre-morphing the image; and

the scaled and pre-morphed image is optically scaled,

wherein the pre-deformatting treatment is used to correct an optical deformity caused at the time of the optical zoom treatment.

2. The method of claim 1, wherein the scaling process comprises:

when the image is amplified, splitting an amplified pixel point in the image into a first preset number of amplified pixel points in the amplified image, wherein the pixel value of each pixel point in the first preset number of amplified pixel points is associated with the pixel value of the pixel point before amplification and the pixel value of the pixel point adjacent to the pixel point before amplification;

when the image is reduced, combining a second preset number of reduced pixel points in the image into one reduced pixel point in the reduced image, wherein the pixel value of the one reduced pixel point is associated with the second preset number of reduced pixel points and the pixel value of the adjacent pixel point of each reduced pixel point.

3. The method of claim 2,

the pixel value of each pixel point in the first preset number of amplified pixel points is obtained by performing weighted average on the pixel values of the pixel point before amplification and the pixel point adjacent to the pixel point before amplification through template convolution;

and the pixel value of the pixel point after being reduced is obtained by carrying out weighted average on the pixel value of the pixel point to be reduced and the pixel value of the adjacent pixel point of each pixel point to be reduced through template convolution.

4. The method of claim 3, wherein splitting an enlarged pixel point in the image into the first predetermined number of enlarged pixel points in the enlarged image is performed by the following equations one to four:

equation one:

ψ^↓(x)(2m,2n)＝p×x(m,n)+q×(x(m-1,n)+x(m,n-1))+r×x(m-1,n-1)

equation two:

ψ^↓(x)(2m+1,2n)＝p×x(m,n)+q×(x(m+1,n)+x(m,n-1))+r×x(m+1,n-1)

equation three:

ψ^↓(x)(2m,n+1)＝p×x(m,n)+q×(x(m,n+1)+x(m-1,n))+r×x(m-1,n+1)

equation four:

ψ^↓(x)(2m+1,2n+1)＝p×x(m,n)+q×(x(m+1,n)+x(m,n+1))+r×x(m+1,n+1)

wherein x (m, n) is the pixel value of the one enlarged pixel point, and x (m-1, n), x (m-1, n-1), x (m-1, n +1), x (m, n-1), x (m, n +1), x (m +1, n-1) and x (m +1, n +1) represent the onePixel values of adjacent pixels of the amplified pixel point, #^↓(x)(2m,2n)、ψ^↓(x)(2m+1,2n)、ψ^↓(x) (2m, n +1) and ψ^↓(x) (2m +1,2n +1) respectively represent pixel values of the first predetermined number of enlarged pixel points,

the merging of a second predetermined number of reduced pixel points in the image into one reduced pixel point in the reduced image is implemented by the following equation five:

equation five:

ψ^↑(x)(m,n)＝a×(x(2m,2n)+x(2m+1,2n)+x(2m+1,2n+1)+x(2m,2n+1))+b×(x(2m-1,2n)+x(2m-1,2n+1))+x(2m,2n-1)+x(2m-1,2n-1)+x(2m+2,2n)+x(2m+2,2n+1)+x(2m,2n+2)+x(2m+1,2n+2))+c×(x(2m-1,2n-1)+x(2m+2,2n-1)+x(2m+2,2n+2)+x(2m-1,2n+2))

wherein psi^↑(x) (m, n) represents a pixel value of the one reduced pixel, x (2m,2n), x (2m +1,2n +1) and x (2m,2n +1) are pixel values of the second predetermined number of reduced pixel points, x (2m-1,2n), x (2m-1,2n +1), x (2m,2n-1), x (2m-1,2n-1), x (2m +2,2n +1), x (2m,2n +2), x (2m +1,2n +2), (x (2m-1,2n-1), x (2m +2,2n +2) and x (2m-1,2n +2) are pixel values of adjacent pixels of each reduced pixel point,

m and n are natural numbers and indicate the positions of pixel points in the image, and a, b, c, p, q and r are convolution template coefficients respectively.

5. The method of claim 4, wherein the convolution template coefficients satisfy the following condition:

4ap+8bq+4ar＝1

2aq+2bq+2br+2cq＝0

ar+2bq+cp＝0

4a+8b+4c＝1

p+2q+r＝1。

6. an augmented reality image processing apparatus, characterized by comprising:

the image acquisition module is used for acquiring images of the real world;

a receiving module for receiving a zoom operation for the image;

the preprocessing module is used for responding to the zooming operation, zooming the image to adapt to different vision levels and carrying out pre-deformation processing on the image; and

an optical module for optically zooming the zoomed and pre-deformed image,

wherein the pre-deformatting treatment is used to correct an optical deformity caused at the time of the optical zoom treatment.

7. The apparatus of claim 6, wherein the scaling process comprises:

when the image is amplified, splitting an amplified pixel point in the image into a first preset number of amplified pixel points in the amplified image, wherein the pixel value of each pixel point in the first preset number of amplified pixel points is associated with the pixel value of the pixel point before amplification and the pixel value of the pixel point adjacent to the pixel point before amplification;

when the image is reduced, combining a second preset number of reduced pixel points in the image into one reduced pixel point in the reduced image, wherein the pixel value of the one reduced pixel point is associated with the second preset number of reduced pixel points and the pixel value of the adjacent pixel point of each reduced pixel point.

8. The apparatus of claim 7,

the pixel value of each of the first predetermined number of amplified pixels is obtained by performing weighted average on the pixel values of the one pre-amplification pixel and the adjacent pixel of the one pre-amplification pixel by template convolution,

and the pixel value of the pixel point after being reduced is obtained by carrying out weighted average on the pixel value of the pixel point to be reduced and the pixel value of the adjacent pixel point of each pixel point to be reduced through template convolution.

9. The method of claim 8, wherein splitting an enlarged pixel point in the image into the first predetermined number of enlarged pixel points in the enlarged image is performed by equations one through four as follows:

equation one:

ψ^↓(x)(2m,2n)＝p×x(m,n)+q×(x(m-1,n)+x(m,n-1))+r×x(m-1,n-1)

equation two:

ψ^↓(x)(2m+1,2n)＝p×x(m,n)+q×(x(m+1,n)+x(m,n-1))+r×x(m+1,n-1)

equation three:

ψ^↓(x)(2m,n+1)＝p×x(m,n)+q×(x(m,n+1)+x(m-1,n))+r×x(m-1,n+1)

equation four:

ψ^↓(x)(2m+1,2n+1)＝p×x(m,n)+q×(x(m+1,n)+x(m,n+1))+r×x(m+1,n+1)

wherein x (m, n) is a pixel value of the one enlarged pixel point, x (m-1, n), x (m-1, n-1), x (m-1, n +1), x (m, n-1), x (m, n +1), x (m +1, n-1), and x (m +1, n +1) denote pixel values of adjacent pixel points of the one enlarged pixel point, ψ^↓(x)(2m,2n)、ψ^↓(x)(2m+1,2n)、ψ^↓(x) (2m, n +1) and ψ^↓(x) (2m +1,2n +1) respectively represent pixel values of the first predetermined number of enlarged pixel points,

the merging of a second predetermined number of reduced pixel points in the image into one reduced pixel point in the reduced image is implemented by the following equation five:

equation five:

ψ^↑(x)(m,n)＝a×(x(2m,2n)+x(2m+1,2n)+x(2m+1,2n+1)+x(2m,2n+1))+b×(x(2m-1,2n)+x(2m-1,2n+1))+x(2m,2n-1)+x(2m-1,2n-1)+x(2m+2,2n)+x(2m+2,2n+1)+x(2m,2n+2)+x(2m+1,2n+2))+c×(x(2m-1,2n-1)+x(2m+2,2n-1)+x(2m+2,2n+2)+x(2m-1,2n+2))

wherein psi^↑(x) (m, n) represents the pixel value of the pixel point after the reduction, and x (2m,2n), x (2m +1,2n +1) and x (2m,2n +1) are allThe pixel values of the second predetermined number of reduced pixel points, x (2m-1,2n), x (2m-1,2n +1), x (2m,2n-1), x (2m-1,2n-1), x (2m +2,2n +1), x (2m,2n +2), x (2m +1,2n +2), (x (2m-1,2n-1), x (2m +2,2n +2), and x (2m-1,2n +2) respectively represent the pixel values of the adjacent pixel points of each of the reduced pixel points,

m and n are natural numbers and indicate the positions of pixel points in the image, and a, b, c, p, q and r are convolution template coefficients respectively.

10. The apparatus of claim 9, wherein the convolution template coefficients satisfy the following condition:

4ap+8bq+4ar＝1

2aq+2bq+2br+2cq＝0

ar+2bq+cp＝0

4a+8b+4c＝1

p+2q+r＝1。

11. an augmented reality image display apparatus, characterized by comprising:

the augmented reality image processing apparatus of any one of claims 6-10;

the operation module is used for carrying out zooming operation on the image; and

display means for presenting the image before zooming and pre-deformatting and the image after zooming and pre-deformatting.

12. A machine-readable storage medium having stored thereon instructions for causing a machine to perform the augmented reality image processing method of any one of claims 1-5.

Technical Field

The invention relates to the field of image processing, in particular to an augmented reality image processing method and device and an augmented reality image display device.

Background

Augmented Reality (AR) technology is a technology for fusing a virtual world and a real world by calculating the position and angle of an image in real time and superimposing a corresponding image, video and a 3D model on the image. The AR client can combine with the picture identification material directly stored in the local AR client to perform real-time image identification on the offline environment of the user, and corresponding display data are displayed in an enhanced mode according to the pre-configured display effect on the position of the identified specific offline target in the real scene.

The image quality Of the AR display device mainly depends on near-eye optics, and one Of the most important parameters for near-eye optical design is the Field angle (FOV), in an optical instrument, the lens Of the optical instrument is taken as a vertex, and the angle formed by two edges Of the maximum range through which the object image Of the measured object can pass is called the FOV. The size of the field angle determines the field of view of the optical instrument, with a larger field angle providing a larger field of view and a smaller optical magnification. On the one hand, the large field angle can bring a larger field of view, more contents are displayed, and more immersion experience is achieved. For a lightweight near-eye display device such as AR glasses, most FOVs do not exceed 40 degrees, for example, the FOV of Google Glass is tens of degrees, and the FOV of microsoft benchmarking product HoloLens reaches nearly 30 °.

In the case where the FOV is smaller than 40 °, the optical magnification of the conventional AR apparatus is not adjusted so much to obtain the effect of image display sharpness, and generally the magnification of image magnification is not more than 2 times. Patent application CN105843390A discloses an image zooming method and AR glasses based on the method, which relate to full-screen enlargement and zooming to the original size after the segmentation and extraction of the display image, but the patent application only discloses the extraction and zooming of the local image, and cannot improve the FOV.

Disclosure of Invention

Embodiments of the present invention provide an augmented reality image processing method and apparatus, and provide an augmented reality image display device using the method and apparatus, where the augmented reality image processing method and apparatus, and the display device can provide an AR experience that can adapt to the visual condition of a low-vision user, and are simple in structure, light in size, and therefore convenient to use.

In order to achieve the above object, an embodiment of the present invention provides an augmented reality image processing method, including: acquiring an image of the real world; receiving a zoom operation for the image; in response to the scaling operation, scaling the image to accommodate different vision levels and pre-morphing the image; and performing optical zoom processing on the zoomed and pre-deformated image, wherein the pre-deformatting processing is used for correcting optical deformities caused during the optical zoom processing.

Wherein the scaling process comprises: when the image is amplified, splitting an amplified pixel point in the image into a first preset number of amplified pixel points in the amplified image, wherein the pixel value of each pixel point in the first preset number of amplified pixel points is associated with the pixel value of the pixel point before amplification and the pixel value of the pixel point adjacent to the pixel point before amplification; when the image is reduced, combining a second preset number of reduced pixel points in the image into one reduced pixel point in the reduced image, wherein the pixel value of the one reduced pixel point is associated with the second preset number of reduced pixel points and the pixel value of the adjacent pixel point of each reduced pixel point.

The pixel value of each pixel point in the first preset number of amplified pixel points is obtained by performing weighted average on the pixel values of the pixel point before amplification and the pixel point adjacent to the pixel point before amplification through template convolution; and the pixel value of the pixel point after being reduced is obtained by carrying out weighted average on the pixel value of the pixel point to be reduced and the pixel value of the adjacent pixel point of each pixel point to be reduced through template convolution.

The splitting of one amplified pixel point in the image into a first predetermined number of amplified pixel points in the amplified image is realized by equations one to four described later. The merging of the second predetermined number of reduced pixel points in the image into one reduced pixel point in the reduced image is implemented by equation five described later.

According to another aspect of the present invention, there is also provided an augmented reality image processing apparatus, including: the image acquisition module is used for acquiring images of the real world; a receiving module for receiving a zoom operation for the image; the preprocessing module is used for responding to the zooming operation, zooming the image to adapt to different vision levels and carrying out pre-deformation processing on the image; and an optical module for performing optical zoom processing on the zoomed and pre-deformed image, wherein the pre-deformer processing is used for correcting optical deformities caused during the optical zoom processing.

Wherein the scaling process comprises: when the image is amplified, splitting an amplified pixel point in the image into a first preset number of amplified pixel points in the amplified image, wherein the pixel value of each pixel point in the first preset number of amplified pixel points is associated with the pixel value of the pixel point before amplification and the pixel value of the pixel point adjacent to the pixel point before amplification; when the image is reduced, combining a second preset number of reduced pixel points in the image into one reduced pixel point in the reduced image, wherein the pixel value of the one reduced pixel point is associated with the second preset number of reduced pixel points and the pixel value of the adjacent pixel point of each reduced pixel point.

The pixel value of each pixel point in the first predetermined number of amplified pixel points is obtained by performing weighted average on the pixel values of the pixel point before amplification and the pixel point adjacent to the pixel point before amplification through template convolution, and the pixel value of the pixel point after reduction is obtained by performing weighted average on the pixel values of the pixel point to be reduced and the pixel point adjacent to the pixel point to be reduced through template convolution.

Splitting an amplified pixel point in the image into a first preset number of amplified pixel points in the amplified image is realized through equations one to four. And combining the second predetermined number of reduced pixel points in the image into one reduced pixel point in the reduced image is realized by the equation five.

Wherein the convolution template coefficients satisfy the conditions already described above.

According to another aspect of the present invention, there is also provided an augmented reality image display apparatus, including: the augmented reality image processing apparatus of the present invention; the operation module is used for carrying out zooming operation on the image; and a display device for presenting the image before zooming and pre-deformatting and the image after zooming and pre-deformatting.

In another aspect, the present invention provides a machine-readable storage medium having stored thereon instructions for causing a machine to perform the augmented reality image processing method described herein.

Through the technical scheme, a user can zoom the image to the degree that the image can adapt to the vision level of the user, when the image is zoomed, the zoomed image is subjected to pre-deformanization according to the zoom degree, then the zoomed and pre-deformanized image is restored after being subjected to anti-deformanization and presented to two eyes, and partial image processing operation is converted into image processing by software through an optical element. Therefore, the AR experience can be provided for users with various vision levels, and the AR equipment is more convenient to use due to the simplified hardware structure.

Additional features and advantages of embodiments of the invention will be set forth in the detailed description which follows.

Drawings

The accompanying drawings, which are included to provide a further understanding of the embodiments of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the embodiments of the invention without limiting the embodiments of the invention. In the drawings:

fig. 1 is a flowchart of an augmented reality image processing method according to an embodiment of the present invention;

fig. 2 is a flowchart of an augmented reality image processing method according to another embodiment of the present invention;

fig. 3 is a flowchart of an augmented reality image processing method according to another embodiment of the present invention;

FIGS. 4 and 5 are schematic diagrams of a process for processing an image by an optical module used in an AR device in the prior art;

fig. 6 and 7 are schematic diagrams illustrating a process of processing an image using an optical module in an augmented reality image processing method and apparatus according to an embodiment of the present invention;

fig. 8 and 9 are diagrams showing examples of convolution template coefficients at the time of the enlargement operation and the reduction operation, respectively.

Fig. 10 is a block diagram of an augmented reality image processing apparatus according to an embodiment of the present invention;

fig. 11 is a block diagram of a configuration of an augmented reality image processing apparatus according to another embodiment of the present invention;

fig. 12 is a block diagram of an augmented reality image display apparatus according to an embodiment of the present invention; and

fig. 13 is a block diagram of a structure of an augmented reality image display apparatus according to another embodiment of the present invention.

Description of the reference numerals

1. 2, 3: optical module 10, 40: augmented reality image processing apparatus

11. 41: the image acquisition module 12: receiving module

13. 42: pre-processing module 14, 43: optical module

30. 50: operating modules 20, 60: display module

100. 200: augmented reality image processing apparatus

Detailed Description

The following detailed description of embodiments of the invention refers to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating embodiments of the invention, are given by way of illustration and explanation only, not limitation.

Fig. 1 is a flowchart of an augmented reality image processing method according to an embodiment of the present invention. As shown in fig. 1, the method comprises the steps of:

step S110, an image of the real world is acquired. The real-world image may be acquired by an imaging device such as a video camera or a still camera, may be a scene of daily life, or may be a photograph or a character. These images may also be pre-stored images, in which case the pre-stored images may be read from the storage space.

Step S120, receiving a zoom operation for the image. For low vision users, if the image is not clearly seen, the image may be scaled to adjust the image to a level that accommodates the user's vision level.

Step S130, in response to the zoom operation, performs zoom and pre-malformation processing on the image. Upon receiving the zoom operation, the image is zoomed to zoom the image to a degree desired by the user. During the scaling process (or before and after scaling), the image is pre-malformed.

In step S140, the scaled and pre-deformed image is optically scaled. In the optical zoom process, an image is distorted, and the optical deformity caused in the optical zoom process can be corrected in advance by the pre-deformity process. The pre-deformatting can be done by image processing techniques, thus eliminating the need to add elements to counteract the image deformity produced by the optical zoom process, thereby enabling the apparatus to be more simplified.

In another embodiment, the zoom operation may also be recorded and then an automatic zoom match is implemented. This is explained with particular reference to fig. 2.

Fig. 2 is a flowchart of an augmented reality image processing method according to another embodiment of the present invention. As shown in fig. 2, the method comprises the steps of:

step S210, an image of the real world is acquired.

Step S220, according to the recorded zoom operation, performing zoom processing on the image to adapt to different vision levels, and performing pre-deformation processing on the image.

The recorded zoom operation may be a pre-stored zoom operation after testing for a particular user, at which point the pre-stored zoom operation may be directly read for subsequent processing. In addition, a zooming operation of a specific user on the image can be received, and the zooming operation corresponding to the specific user is recorded as the recorded zooming operation. At this time, after the user only needs to manually perform the zooming operation during the first use, each use can be processed according to the recorded zooming operation during the first use without manual operation again, and when the user considers that the currently recorded zooming operation cannot adapt to the own eyesight level after using for a period of time, the user can manually perform the zooming operation again and then take the zooming operation as the recorded zooming operation.

In step S230, the scaled and pre-deformed image is subjected to an optical scaling process for correcting an optical deformation caused at the time of the optical scaling process.

In another preferred embodiment, the same AR device may be targeted to multiple users, at which point the recorded zoom operation may be associated with a particular user, whereby each user may obtain an AR experience tailored to his eyesight by his own identity when using the AR device. This is explained with particular reference to fig. 3.

Fig. 3 is a flowchart of an augmented reality image processing method according to another embodiment of the present invention. As shown in fig. 3, the method may include the steps of:

step S310, an image of the real world is acquired.

Step S320, receiving an identity of a specific user.

And step S330, zooming and pre-deforming the image according to the identity and the recorded zooming operation corresponding to the identity.

In step S340, the scaled and pre-deformed image is subjected to optical scaling processing.

With the embodiment shown in fig. 3, it may be implemented that one AR device corresponds to multiple users, and each user may input its own identity ID when using the AR device, so that the AR device may record a zoom operation for the user, and/or present an augmented reality image according to the recorded zoom operation for the user.

Further details of the implementation shown in fig. 2 and 3 may be found in relation to the description of the embodiment shown in fig. 1.

With the embodiments shown in fig. 1 to 3, a low-vision user can match his own vision by adjusting the size of the presented image, and thus even a low-vision user can have a good AR experience. Furthermore, the image can be further zoomed using the optical zoom process, thereby improving the FOV. The invention breaks through the design idea of pursuing definition in the traditional AR technology, performs scaling on the image with higher magnification to adapt to the vision level of the user, and realizes the matching of AR users with different vision levels by adopting the mode of sacrificing definition. Although scaling may reduce the image sharpness to some extent, this can be compensated for by providing a higher sharpness of the original image.

Fig. 4 and 5 are schematic diagrams of a process of processing an image by an optical unit used in an AR device in the related art. In the prior art, the original image is deformed due to optical scaling, so that extra optical devices of the equipment are required for carrying out anti-deformation processing. As shown in fig. 4, the image subjected to optical zoom looks almost free from deformity due to the additional provision of a convex lens (complex optical module 1). As shown in fig. 5, the simplified optical module 2 is used, and in this case, since sufficient inverse morphing processing is not performed, the image subjected to the optical zoom processing is morphed.

Fig. 6 and 7 are schematic diagrams illustrating a process of processing an image using an optical module in an augmented reality image processing method and apparatus according to an embodiment of the present invention. In fig. 6, the simplified optical module 2 similar to that in fig. 5 is used, and since the pre-deformation process is performed before the optical zoom process is performed, the edge deformation of the optically zoomed image is canceled by the pre-deformation process, and thus an image without deformation can be presented in the future by the simplified optical module 2.

As shown in fig. 7, a more simplified optical module 3 is used. At this time, the deformity generated by the optical zoom processing is not canceled (deformared) by any optical element, and the original image is pre-deformed to a greater extent than that of fig. 6 for a more serious degree of deformity of the optical module 3. Thereby, the deformities generated by the optical module 3 can be cancelled, thereby presenting an AR image without deformities.

Although fig. 6 and 7 do not show the scaling process in the schematic pre-morphing, the image may be scaled by an image processing technique before the optical scaling is performed. The degree of pre-deformity may be determined according to the degree of deformity generated when the zoomed image is subjected to the optical zoom process.

In the process of zooming, when the image is magnified, one magnified pixel point in the image can be split into a first preset number of magnified pixel points in the magnified image, and the pixel value of each pixel point in the first preset number of magnified pixel points is associated with the pixel value of the pixel point before magnification and the pixel value of the pixel point adjacent to the pixel point before magnification; when the image is reduced, combining a second preset number of reduced pixel points in the image into one reduced pixel point in the reduced image, wherein the pixel value of the one reduced pixel point is associated with the second preset number of reduced pixel points and the pixel value of the adjacent pixel point of each reduced pixel point.

Further, the enlargement and reduction processing may process the image to the target image through a plurality of stages of enlargement or reduction, each stage of enlargement and reduction being achieved by the method as described above. At this time, in order to maximize fidelity, it is preferable that the first predetermined number and the second predetermined number at each stage of the enlarging or reducing operation are the same value.

The pixel value of each pixel point in the first preset number of amplified pixel points is obtained by performing weighted average on the pixel values of the pixel point before amplification and the pixel point adjacent to the pixel point before amplification through template convolution; and the pixel value of the pixel point after being reduced is obtained by carrying out weighted average on the pixel value of the pixel point to be reduced and the pixel value of the adjacent pixel point of each pixel point to be reduced through template convolution.

Fig. 8 and 9 are diagrams showing examples of convolution template coefficients at the time of the enlargement operation and the reduction operation, respectively.

FIG. 8 shows a split of an enlarged pixel (m, n) into four enlarged pixels psi^↓(x)(2m,2n)、ψ^↓(x)(2m+1,2n)、ψ^↓(x) (2m,2n +1) and ψ^↓(x) An example of a convolution template when (2m +1,2 n-1).

The pixel value of the enlarged pixel point can be realized according to the pixel point (m, n) and 8 neighborhoods { (m-1, n-1), (m-1, n), (m-1, n +1), (m, n-1), (m, n +1), (m +1, n-1), (m +1, n), (m +1, n +1) } of the pixel point by the following equations one to four:

equation one:

ψ^↓(x)(2m,2n)＝p×x(m,n)+q×(x(m-1,n)+x(m,n-1))+r×x(m-1,n-1)

equation two:

ψ^↓(x)(2m+1,2n)＝p×x(m,n)+q×(x(m+1,n)+x(m,n-1))+r×x(m+1,n-1)

equation three:

ψ^↓(x)(2m,2n+1)＝p×x(m,n)+q×(x(m,n+1)+x(m-1,n))+r×x(m-1,n+1)

equation four:

ψ^↓(x)(2m+1,2n-1)＝p×x(m,n)+q×(x(m+1,n)+x(m,n+1))+r×x(m+1,n+1)

wherein x (m, n) is a pixel value of the one enlarged pixel point, x (m-1, n), x (m-1, n-1), x (m-1, n +1), x (m, n-1), x (m, n +1), x (m +1, n-1), and x (m +1, n +1) denote pixel values of adjacent pixel points of the one enlarged pixel point, ψ^↓(x)(2m,2n)、ψ^↓(x)(2m+1,2n)、ψ^↓(x) (2m,2n +1) and ψ^↓(x) (2m +1,2n-1) respectively represent pixel values of the first predetermined number of enlarged pixel points.

When the magnification ratio is large, multilayer amplification can be performed according to the amplification processing method, and one pixel point can be split into four pixel points each time.

Fig. 9 shows an example of a convolution module that merges four adjacent reduced pixel points (2m,2n), (2m +1,2n +1), (2m,2n +1) into one reduced pixel point (m, n) in the reduced image.

The pixel values of the merged reduced pixel points (m, n) can be implemented according to the four reduced pixel points and the pixel values of 16 adjacent pixel points of the four reduced pixel points by the following equation five:

equation five:

ψ^↑(x)(m,n)＝a×(x(2m,2n)+x(2m+1,2n)+x(2m+1,2n+1)+x(2m,2n+1))+b×(x(2m-1,2n)+x(2m-1,2n+1))+x(2m,2n-1)+x(2m-1,2n-1)+x(2m+2,2n)+x(2m+2,2n+1)+x(2m,2n+2)+x(2m+1,2n+2))+c×(x(2m-1,2n-1)+x(2m+2,2n-1)+x(2m+2,2n+2)+x(2m-1,2n+2))

wherein psi^↑(x) (m, n) represents the pixel value of the one reduced pixel point, x (2m,2n), x (2m +1,2n +1) and x (2m,2n +1) are the pixel values of the second predetermined number of reduced pixel points, x (2m-1,2n), x (2m-1,2n +1), x (2m,2n-1), x (2m-1,2n-1) are the pixel values of the second predetermined number of reduced pixel points, and x (2m-1,2n-1) is the pixel value of the one reduced pixel point) X (2m +2,2n), x (2m +2,2n +1), x (2m,2n +2), x (2m +1,2n +2), (x (2m-1,2n-1), x (2m +2,2n +2), and x (2m-1,2n +2) respectively represent the pixel values of the adjacent pixels of each reduced pixel.

In the above equations one to five, m and n are natural numbers and indicate the positions of pixel points in the image, and a, b, c, p, q and r are convolution template coefficients, respectively.

As a preferred example, the convolution template coefficients in the scaling process preferably satisfy the following condition:

4ap+8bq+4ar＝1

2aq+2bq+2br+2cq＝0

ar+2bq+cp＝0

4a+8b+4c＝1

p+2q+r＝1。

although the above exemplary description describes the process of splitting a pixel into four pixels and the process of combining four pixels into a pixel, those skilled in the art can expand the technical ideas described in the above detailed description and equations one to five, and these expanded schemes should be equivalent to the protection scope of the present invention.

Fig. 10 is a block diagram illustrating an augmented reality image processing apparatus according to an embodiment of the present invention. As shown in fig. 10, the apparatus 10 includes: an image acquisition module 11, configured to acquire an image of a real world; a receiving module 12, configured to receive a zoom operation for the image; a pre-processing module 13, responding to the zooming operation, zooming the image to adapt to different vision levels, and pre-deforming the image; and an optical module 14 for performing an optical zoom process on the zoomed and pre-deformatted image, wherein the pre-deformatting process is used to correct optical deformities caused at the time of the optical zoom process.

Fig. 11 is a block diagram of an augmented reality image processing apparatus according to another embodiment of the present invention. As shown in fig. 11, the apparatus 40 includes: an image acquisition module 41 for acquiring real-world images; a pre-processing module 42, which performs zooming processing on the image to adapt to different vision levels according to the recorded zooming operation, and performs pre-malformation processing on the image; and an optical module 43 for performing an optical zoom process on the zoomed and pre-deformatted image, wherein the pre-deformatting process is used to correct an optical deformity caused at the time of the optical zoom process.

In another preferred embodiment, the apparatus 40 may further include: a receiving module, configured to receive a zoom operation of a specific user for the image; and a recording module for recording the zoom operation corresponding to the specific user.

In another preferred embodiment, the recorded zoom operation is associated with a specific user, and the receiving module of the apparatus 40 is further configured to receive an identity of the specific user; the preprocessing module 42 is further configured to: and according to the identity identification and the recorded zooming operation corresponding to the identity identification, zooming the image to adapt to different vision levels, and performing pre-malformation treatment on the image.

Wherein the scaling process for the devices 10 and 40 may comprise: when the image is amplified, splitting an amplified pixel point in the image into a first preset number of amplified pixel points in the amplified image, wherein the pixel value of each pixel point in the first preset number of amplified pixel points is associated with the pixel value of the pixel point before amplification and the pixel value of the pixel point adjacent to the pixel point before amplification; when the image is reduced, combining a second preset number of reduced pixel points in the image into one reduced pixel point in the reduced image, wherein the pixel value of the one reduced pixel point is associated with the second preset number of reduced pixel points and the pixel value of the adjacent pixel point of each reduced pixel point.

The pixel value of each pixel point in the first predetermined number of amplified pixel points is obtained by performing weighted average on the pixel values of the pixel point before amplification and the pixel point adjacent to the pixel point before amplification through template convolution, and the pixel value of the pixel point after reduction is obtained by performing weighted average on the pixel values of the pixel point to be reduced and the pixel point adjacent to the pixel point to be reduced through template convolution.

Splitting an amplified pixel point in the image into a first preset number of amplified pixel points in the amplified image is realized through equations one to four. And combining the second predetermined number of reduced pixel points in the image into one reduced pixel point in the reduced image is realized by the equation five.

Wherein the convolution template coefficients preferably satisfy the conditions already described above.

For specific details regarding implementation of the augmented reality image processing apparatus, reference may be made to the above detailed description of the augmented reality image processing method.

Fig. 12 is a block diagram of an augmented reality image display apparatus according to an embodiment of the present invention. As shown in fig. 11, the apparatus 100 includes: the augmented reality image processing apparatus 10 of the present invention; an operation module 30, configured to perform a scaling operation on the image; and a display device 20 for presenting said image before zooming and pre-deformatting and said zoomed and pre-deformatted image.

Fig. 13 is a block diagram of a structure of an augmented reality image display apparatus according to another embodiment of the present invention. As shown in fig. 13, the apparatus 200 includes: an augmented reality image processing apparatus 40 according to the present invention; an operation module 50, configured to perform a scaling operation on the image; and a display device 60 for presenting said image before zooming and pre-deformatting and said zoomed and pre-deformatted image.

In a preferred embodiment, the operation module of the present invention may include a touch panel, for example, and the user may perform a zoom operation by operating the touch panel, and the zoom operation is received by the device 10 or 40 and then processed accordingly. The manipulation module may further include a touch pad and buttons, which may be used for a user to manually enter a zoom mode and determine a zoom result. For example, the user may start receiving a zoom operation of the user when pressing the button for the first time, and after zooming the image to a degree suitable for the user's eyesight and suitable definition through the touch panel, the user may press the button again, and at this time, the zoom magnification for the user is recorded, and the augmented reality image processing apparatus may perform zoom and pre-malformation processing according to the zoom magnification.

In another preferred embodiment, the operation module and the display module may be combined into a touch display screen, so that the user can directly perform a zoom operation on the touch display screen. In addition, the operation module may also be a voice module, in which the user may perform the zooming operation by voice, and the voice module converts the voice of the zooming operation into an operation instruction to transmit to the apparatus 10 or 40 when receiving the voice. The operation module may be additionally provided with other operation components, such as brightness adjustment and the like.

The following illustrates the steps of using the augmented reality image display device of the present invention:

first, a user wears natural-form glasses (AR glasses) including the enhanced image display device, the display module, and the operation module of the present invention;

the user faces the front of the head and the eyes to a real environment needing to be seen clearly;

the image acquisition unit acquires continuous images taking the natural sight center of the user as the center;

the image acquisition unit continuously acquires images along with the movement of the front face and eyes of the head of the user;

the sequential images are output to a display module (e.g., a light-transmissive near-eye display);

the user can automatically adjust the magnification of the image according to the self-demand (the method for automatically adjusting the magnification comprises finger touch, gesture control, voice command, key control and the like) to the optimal state of adapting to the vision level of the user;

the glasses (AR glasses) system records the magnification of the specific user and automatically applies the magnification in the subsequent use so as to avoid repeating the steps;

with the movement of the front face and eyes of the head of the user (glasses follow-up), the continuous images initially acquired by the image acquisition module are processed according to the magnification of the specific user and are output to the display module, so that the AR images adaptive to the strength level of the user are provided.

The invention can also record the zooming operation of a plurality of different specific scenes aiming at a specific user so as to be more conveniently selected by the user for application, and the plurality of different specific scenes can comprise: general indoor daily life scene, reading scene, listening scene, outdoor scene etc.. The recorded zoom operation may be associated with a particular scene, for example by a scene identification, whereby the current application scene may be selected in accordance with the scene identification. Or, different scene identifications can be presented to the user, so that the user can select the current application scene by himself, and then the recorded zooming operation associated with the scene can be called according to the scene selected by the user, so that zooming and pre-malformation processing are automatically carried out. In another embodiment, a scene identification may be entered by the user, invoking a corresponding recorded zoom operation. In addition, when the zoom operation for a specific scene of a specific user is recorded, the zoom operation for the scene may be recorded according to the method described above by first selecting or device-specifying the scene by the user.

Therefore, the invention not only can provide AR experience for users with different vision levels, but also can provide richer and more comfortable AR experience according to different life or learning scenes, and can improve the vision of the users to a certain extent.

Although the embodiments of the present invention have been described in detail with reference to the accompanying drawings, the embodiments of the present invention are not limited to the details of the above embodiments, and various simple modifications can be made to the technical solutions of the embodiments of the present invention within the technical idea of the embodiments of the present invention, and the simple modifications all belong to the protection scope of the embodiments of the present invention.

It should be noted that the various features described in the above embodiments may be combined in any suitable manner without departing from the scope of the invention. In order to avoid unnecessary repetition, the embodiments of the present invention do not describe every possible combination.

Those skilled in the art will understand that all or part of the steps in the method according to the above embodiments may be implemented by a program, which is stored in a storage medium and includes several instructions to enable a single chip, a chip, or a processor (processor) to execute all or part of the steps in the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

In addition, any combination of various different implementation manners of the embodiments of the present invention is also possible, and the embodiments of the present invention should be considered as disclosed in the embodiments of the present invention as long as the combination does not depart from the spirit of the embodiments of the present invention.