阅读说明：本技术 Vr图像的压缩传输方法、系统 (Compression transmission method and system for VR image ) 是由 于东壮 崔新宇 吴健 于 2021-06-22 设计创作，主要内容包括：本发明提供一种VR图像的压缩传输方法、系统,首先获取服务器待发送的视频图像,并根据压缩范围对所述视频图像进行定域压缩以形成第一压缩视频图像,而后对第一压缩视频图像进行视频编码处理以形成传输视频图像,再将该传输视频传输至客户端,并对该传输视频图像进行视频解码处理以将该传输视频图像还原为第二压缩视频图像,再对该第二压缩视频图像进行解压缩处理以形成展示视频图像,即使用d3d或opengl计算为介质,利用图像着色程序对边缘的压缩范围内的图像数据进行压缩,从而通过压缩边缘图像的分辨率来减小整体图像的分辨率,从而减小图像体积,提高传输速率,进而提高VR直播、VR串流的稳定性。(The invention provides a compression transmission method and a system of VR images, which firstly obtain a video image to be transmitted by a server, perform localized compression on the video image according to a compression range to form a first compressed video image, then perform video coding processing on the first compressed video image to form a transmission video image, transmit the transmission video to a client, perform video decoding processing on the transmission video image to restore the transmission video image to a second compressed video image, and then perform decompression processing on the second compressed video image to form a display video image, namely, using d3d or opengl calculation as a medium, compressing image data in a compression range of edges by using an image coloring program, thereby reducing the resolution of the whole image by compressing the resolution of the edge image, reducing the image volume and improving the transmission rate, and then improve the stability of VR live broadcast, VR series flow.)

1. A method for compressing and transmitting a VR image, comprising:

acquiring a video image to be sent by a server, and performing localized compression on the video image according to a compression range to form a first compressed video image; the compression range is determined by transmitting a sample image;

performing video coding processing on the first compressed video image to form a transmission video image;

transmitting the transmission video to a client, and carrying out video decoding processing on the transmission video image so as to restore the transmission video image into a second compressed video image;

and decompressing the second compressed video image to form a display video image.

2. The method for compressing and transmitting the VR image according to claim 1, wherein the process of determining the compression range from the transmission samples includes:

enabling the eyes of the user to correspond to the left lens barrel and the right lens barrel worn by the VR head through a guiding program;

capturing sample images marked with blind area range areas sent by the server through the left lens barrel and the right lens barrel;

adjusting the size of the sample image to cause the blind zone range region to exit the user's field of view.

3. The method for compressing and transmitting a VR image as set forth in claim 2, wherein the process of performing localized compression on the video image according to the compression range to form a first compressed video image comprises:

performing region locking on the video image according to the compression range to divide the video image into an edge region and a central region;

performing preset multiple compression sampling on the edge area through a first image coloring program to form a compressed edge area; real sampling is carried out on the central area to form a central restoration area;

and taking the compressed edge area and the central restoration area as a first compressed video image.

4. The method of compressed transmission of a VR image of claim 3,

the preset multiple is 2 times.

5. The method for compressed transmission of a VR image as in claim 3, further comprising, while performing video encoding processing on the first compressed video image to form a transmitted video image:

and storing the video image, and recording the compression range and the compression ratio of the first compressed video image in a custom information frame of the video image.

6. The method of compressed transmission of VR images of claim 5,

the video coding adopts h264 or h265 video.

7. The method of compressed transmission of a VR image of claim 3,

the second compressed video has the same resolution as the first compressed video.

8. The method for compressed transmission of a VR image as in claim 7, wherein the process of decompressing the second compressed video image to form a presentation video image comprises:

acquiring a compressed part and an original part of the second compressed video image; wherein the compressed portion is coincident with the compressed edge region; the original portion coincides with the central reduction zone;

performing differential sampling on the compressed part through a second image coloring program to restore the compressed part into a real edge area, and performing point-to-point sampling on the original part to form a real central area;

and fitting the real edge region and the real central region to form a display video image.

9. A compression transmission system for VR images, which implements the compression transmission method for VR images as claimed in any one of claims 1 to 8, comprising:

the edge compression module is used for acquiring a video image to be sent by the server and carrying out localized compression on the video image according to a compression range to form a first compressed video image; the compression range is determined by transmitting a sample image;

the video coding module is used for carrying out video coding processing on the first compressed video image to form a transmission video image;

the video decoding module is used for transmitting the transmission video to a client and carrying out video decoding processing on the transmission video image so as to restore the transmission video image into a second compressed video image;

and the video restoration module is used for decompressing the second compressed video image to form a display video image.

10. The system for compressed transmission of a VR image of claim 9, wherein the edge compression module includes an image shader, wherein,

and a first image shading program is arranged in the image shader and is used for carrying out preset multiple compression sampling on the edge area divided according to the compression range to form a compressed edge area.

Technical Field

The invention relates to the technical field of virtual reality, in particular to a compression transmission method and system for VR images.

Background

Due to advances in technology and the diversification of market demands, virtual reality systems are becoming more and more common and are used in many fields such as computer games, health and safety, industry and educational training. To name a few examples, hybrid virtual reality systems are being integrated into mobile communication devices, gaming machines, personal computers, movie theaters, theme parks, university laboratories, student classrooms, hospital exercise gyms, and other corners of life.

The core of VR streaming and VR live broadcasting is that VR images acquired from different places are transmitted to the local through coding transmission, unpacked, recombined, decoded and displayed again, so that the transmission flow is reduced, and the definition of the images is guaranteed. At present, video is compressed by h264 and h265 codes mostly, and although the effect is good, the problem of low transmission efficiency still exists due to overlarge image volume and overhigh resolution, so that the operation experience of VR live broadcast or VR streaming is influenced.

Therefore, a method and a system for compressing and transmitting VR images with reduced image and video transmission volume and improved video transmission rate are needed.

Disclosure of Invention

In view of the foregoing problems, an object of the present invention is to provide a method and a system for compressing and transmitting a VR image, so as to solve the problem that the operation experience of VR live broadcast or VR streaming is affected because the image size is too large and the resolution is too high, although the conventional video mostly uses h264 and h265 coding compression, and although the conventional video also has good effect, the transmission efficiency is still low.

The invention provides a compression transmission method of a VR image, which comprises the following steps:

acquiring a video image to be sent by a server, and performing localized compression on the video image according to a compression range to form a first compressed video image; the compression range is determined by transmitting a sample image;

performing video coding processing on the first compressed video image to form a transmission video image;

transmitting the transmission video to a client, and carrying out video decoding processing on the transmission video image so as to restore the transmission video image into a second compressed video image;

and decompressing the second compressed video image to form a display video image.

Preferably, the process of determining the compression range from the transmission samples comprises:

enabling the eyes of the user to correspond to the left lens barrel and the right lens barrel worn by the VR head through a guiding program;

capturing sample images marked with blind area range areas sent by the server through the left lens barrel and the right lens barrel;

adjusting the size of the sample image to cause the blind zone range region to exit the user's field of view.

Preferably, the process of performing localized compression on the video image according to the compression range to form a first compressed video image comprises:

performing region locking on the video image according to the compression range to divide the video image into an edge region and a central region;

performing preset multiple compression sampling on the edge area through a first image coloring program to form a compressed edge area; real sampling is carried out on the central area to form a central restoration area;

and taking the compressed edge area and the central restoration area as a first compressed video image.

Preferably, the preset multiple is 2 times.

Preferably, the video coding processing on the first compressed video image to form a transmission video image further comprises:

and storing the video image, and recording the compression range and the compression ratio of the first compressed video image in a custom information frame of the video image.

Preferably, the video coding employs h264 or h265 video.

Preferably, the second compressed video has the same resolution as the first compressed video.

Preferably, the process of decompressing the second compressed video image to form the presentation video image includes:

acquiring a compressed part and an original part of the second compressed video image; wherein the compressed portion is coincident with the compressed edge region; the original portion coincides with the central reduction zone;

performing differential sampling on the compressed part through a second image coloring program to restore the compressed part into a real edge area, and performing point-to-point sampling on the original part to form a real central area;

and fitting the real edge region and the real central region to form a display video image.

The invention also provides a compression and transmission system of the VR image, which realizes the compression and transmission method of the VR image, wherein the compression and transmission system comprises the following steps:

the edge compression module is used for acquiring a video image to be sent by the server and carrying out localized compression on the video image according to a compression range to form a first compressed video image; the compression range is determined by transmitting a sample image;

the video coding module is used for carrying out video coding processing on the first compressed video image to form a transmission video image;

the video decoding module is used for transmitting the transmission video to a client and carrying out video decoding processing on the transmission video image so as to restore the transmission video image into a second compressed video image;

and the video restoration module is used for decompressing the second compressed video image to form a display video image.

Preferably, the edge compression module comprises an image shader, wherein,

and a first image shading program is arranged in the image shader and is used for carrying out preset multiple compression sampling on the edge area divided according to the compression range to form a compressed edge area.

As can be seen from the above technical solutions, in the compression and transmission method and system for VR images provided by the present invention, a video image to be transmitted by a server is first obtained, and is locally compressed according to a compression range to form a first compressed video image, then the first compressed video image is subjected to video encoding processing to form a transmission video image, the transmission video image is transmitted to a client, and the transmission video image is subjected to video decoding processing to restore the transmission video image to a second compressed video image, and then the second compressed video image is subjected to decompression processing to form a display video image, even if d3d or opengl calculation is used as a medium, image data in a compression range of an edge is compressed by an image rendering program, so that a resolution of an entire image is reduced by compressing a resolution of the edge image, thereby reducing an image volume, and the transmission rate is improved, and the stability of VR live broadcast and VR streaming is further improved.

Drawings

Other objects and results of the present invention will become more apparent and more readily appreciated as the same becomes better understood by reference to the following specification taken in conjunction with the accompanying drawings. In the drawings:

fig. 1 is a flowchart of a method for compression transmission of a VR image according to an embodiment of the present invention;

fig. 2 is a schematic diagram of a compression range involved in a VR image compression transmission method according to an embodiment of the present invention;

fig. 3 is a schematic diagram of a compression transmission system of VR images according to an embodiment of the present invention.

Detailed Description

At present, video is compressed by h264 and h265 codes mostly, and although the effect is good, the problem of low transmission efficiency still exists due to overlarge image volume and overhigh resolution, so that the operation experience of VR live broadcast or VR streaming is influenced.

In view of the above problems, the present invention provides a method and a system for compressing and transmitting a VR image, and the following describes in detail a specific embodiment of the present invention with reference to the accompanying drawings.

In order to illustrate the compression and transmission method and system of the VR image provided by the present invention, fig. 1 exemplarily indicates the compression and transmission method of the VR image according to the embodiment of the present invention; fig. 2 shows an exemplary compression transmission system for VR images according to an embodiment of the present invention.

The following description of the exemplary embodiment(s) is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses. Techniques and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but are intended to be considered a part of the specification where appropriate.

As shown in fig. 1, the method for compressing and transmitting a VR image according to an embodiment of the present invention includes:

s1: acquiring a video image to be sent by a server, and performing localized compression on the video image according to a compression range to form a first compressed video image; the compression range is determined by transmitting the sample image;

s2: performing video coding processing on the first compressed video image to form a transmission video image;

s3: transmitting the transmission video to a client, and carrying out video decoding processing on the transmission video image so as to restore the transmission video image into a second compressed video image;

s4: and decompressing the second compressed video image to form a presentation video image.

As shown in fig. 1, step S1 is to acquire a video image to be sent by the server, and perform localized compression on the video image according to the compression range to form a first compressed video image; the compression range is determined by transmitting the sample image; wherein, S0: a process for determining a compression range from the transmitted samples, comprising:

s01: enabling the eyes of the user to correspond to the left lens barrel and the right lens barrel worn by the VR head through a guiding program;

s02: capturing sample images marked with blind area range areas sent by the server through the left lens cone and the right lens cone;

s04: adjusting the size of the sample image to make the blind area range region exit the visual field of the user.

The process of performing localized compression on the video image according to the compression range to form a first compressed video image comprises:

s11: performing region locking on the video image according to the compression range to divide the video image into an edge region and a central region;

s12: performing preset multiple compression sampling on the edge area through a first image coloring program to form a compressed edge area; real sampling is carried out on the central area to form a central restoration area;

s13: and taking the compressed edge area and the central restoration area as a first compressed video image.

Specifically, as shown in fig. 2, since the VR head wears the screen in a rectangular shape by using two approximately circular left and right barrels, the edge of the rectangular screen is not visible (black part in the figure) during normal wearing, and the image (black part) at the edge can be seen only when the line of sight is shifted as much as possible or the VR head wears the screen to move toward one side suddenly, at this time, due to abnormal line of sight or sudden movement, the display effect of the partial image is hardly normally felt, and therefore the resolution of the entire image can be reduced by compressing the resolution of the edge image, thereby reducing the image volume, so the compression range is determined first by step S0, in this embodiment, the video generation sender is the server, and the VR headset is the client, and after the server communicates with the client, the user normally wears the VR headset, and the bootstrap program of the client prompts the user: determining that the wearing is normal, clicking a confirmation button worn by the VR head after normal observation, and determining the left side and the right side of the edge; then, the server firstly presets a left and right position, such as the resolution of a video image 1920, presets positions 480 and 1440 (1920-480-1440), renders the video images to be pure red respectively with 0-480 and 1440-1920, and the other parts of the video images unchanged, transmits the images to the client for display, and records the resolution of the original images; at which point the client prompts the user to click on either button "+" or button "-" to increase or decrease the red area until the confirmation is clicked when the red area is seen only a little or not seen and does not affect the overall look and feel of the image. And the client sends a "+" or "-" instruction to the server, the server adjusts the initial boundary of the red part according to the instruction until the user clicks and confirms, the left and right compression boundaries are recorded, namely, the process is repeated on the upper and lower boundaries, and the compression range is obtained, namely, the finally confirmed red area is the compression range in the embodiment, and the video image to be sent by the server is compressed in a localized manner according to the compression range to form a first compressed video image.

In addition, in this embodiment, the preset multiple in step S1 is 2 times, specifically, in this embodiment, after the compression range is determined, a first image shading program of the image shader is written, in this embodiment, the first image shading program is a shader program, the shader program reconstructs a picture by using sampling, taking a video image with a resolution of 1920 × 1920 as an example, assuming that left, right, upper and lower compression boundaries are 480, and 2 times compression is adopted, a horizontal 0-480 part is an edge region, and samples are performed on every 2 pixels of the region, where odd-numbered lines sample basis points and even-numbered lines sample even-numbered points; 480-1440 part is central area, and one-to-one real sampling is carried out on the area; the first compressed video image formed after compression is reconstructed to 1440 x 1440 size, where the compressed portion is 240 for each of the left, right, top, bottom, and center regions for the middle uncompressed portion 960. After processing, the first compressed video image is changed to be 0.5625 times of the original video image, and the volume is obviously reduced.

In the embodiment shown in fig. 1, the step S2 is to perform a video coding process on the first compressed video image to form a transmission video image, and while performing the video coding process on the first compressed video image to form the transmission video image, the method further includes:

and saving the video image, and recording the compression range and the compression range of the first compressed video image in the custom information frame of the video image.

Specifically, in the process of transmitting the first compressed video image formed in step S1 to the client, video encoding is required to be performed on the first compressed video image, in this embodiment, the video encoding employs h264 or h265 video encoding, and if the first compressed video image is subjected to video encoding, such as h264 or h265 video, boundary information (compression range) and compression ratio are recorded in an SEI (custom information) frame at the same time, so as to restore the video image at a later stage. Meanwhile, if the original image is saved, the boundary information and the compression ratio can be recorded in the same folder or named at the beginning of the original resolution _ left-right boundary _ left-right compression ratio _ upper-lower boundary _ upper-lower compression ratio when the bitmap is named. Such as 1920#1920_480_2_480_2_ custom name.

As shown in fig. 1, step S3 is to transmit the transmission video to the client, and perform a video decoding process on the transmission video to restore the transmission video to a second compressed video, that is, after the first compressed video image with the compressed edge area is video-encoded, the second compressed video image needs to be video-decoded to obtain the decoded second compressed video image after being transmitted to the client, and in this embodiment, the decoded second compressed video image is identical to the first compressed video image before video encoding, that is, the second compressed video has the same resolution as the first compressed video image, so as to prevent video distortion.

As shown in fig. 1, step S4 is a process of decompressing a second compressed video image to form a presentation video image, where the process of decompressing the second compressed video image to form the presentation video image includes:

s41: acquiring a compressed part and an original part of the second compressed video image; wherein the compressed portion is coincident with the compressed edge region; the original portion coincides with the central reduction zone;

s42: performing differential sampling on the compressed part through a second image coloring program to restore the compressed part into a real edge area, and performing point-to-point sampling on the original part to form a real central area;

s43: fitting the real edge region and the real center region to form a display video image;

step S4 is to decompress the second compressed video image (which is identical to the first compressed video image) subjected to edge compression, specifically, during live streaming or live broadcast, the boundary information (compression range) and the original image resolution are already recorded, if the file stored in the decompression mode can be extracted from the SEI frame of the video stream or from picture naming, the decompression needs to write a second image shading program in the image shader, in this embodiment, the second image shading program is a shader program, the program reconstructs the picture by using sampling, that is, for the compressed portion, the pigments at two points are expanded from the original image by using differential sampling, that is, the sampler is set to be differential sampling, and the compressed portion is restored to the original size. Setting a sampler to point-to-point sampling for an original part (an image uncompressed part); wherein, the opengl or d3d technology is adopted as the medium for calling the loader program by the first image coloring program for compression and the second image coloring program for decompression, thereby ensuring the running speed.

Moreover, it should be noted that, in the process of real-time streaming or live broadcasting, the user can adjust the size of the compression range (boundary information) at any time, and then perform the localized compression in steps S1-S4 and the decompression operation for the localized compression according to the compression range, so that the transmission video is transmitted with a small resolution, the transmission rate is increased without affecting the image quality and definition of the video image, and the quality and stability of the real-time streaming or live broadcasting are ensured.

As described above, the compression and transmission method for VR images according to the present invention first obtains a video image to be transmitted from a server, performs localized compression on the video image according to a compression range to form a first compressed video image, performs video encoding processing on the first compressed video image to form a transmission video image, transmits the transmission video image to a client, performs video decoding processing on the transmission video image to restore the transmission video image to a second compressed video image, and performs decompression processing on the second compressed video image to form a display video image, i.e. using d3d or opengl as a medium, compresses image data in a compression range of an edge by using an image rendering program, so as to reduce the resolution of an entire image by compressing the resolution of the edge image, thereby reducing the image volume and improving the transmission rate, and then improve the stability of VR live broadcast, VR series flow.

As shown in fig. 3, the present invention further provides a compression and transmission system 100 for VR images, which implements the aforementioned compression and transmission method for VR images, wherein the compression and transmission system includes:

the edge compression module 101 is configured to obtain a video image to be sent by a server, and perform localized compression on the video image according to a compression range to form a first compressed video image; the compression range is determined by transmitting the sample image;

a video encoding module 102, configured to perform video encoding processing on the first compressed video image to form a transmission video image;

the video decoding module 103 is configured to transmit the transmission video to the client, and perform video decoding processing on the transmission video image to restore the transmission video image to a second compressed video image;

and the video restoration module 104 is configured to decompress the second compressed video image to form a display video image.

Wherein the edge compression module 101 comprises an image shader, wherein,

a first image shading program is arranged in the image shader and is used for carrying out preset multiple compression sampling on the edge area divided according to the compression range to form a compressed edge area;

the image shader is also included in the video restoration module 104, that is, the video restoration module also includes an image shader, and the image shader further includes a second image shading program, where the second image shading program is configured to perform differential sampling on a compressed portion in a second compressed video image to restore the compressed portion to a real edge region, and perform point-to-point sampling on an original portion in the second compressed video image to form a real center region, so as to form a display video image, so that the video image sent by the server is completely restored and displayed, thereby improving the transmission rate without affecting the stability of the video image.

It can be seen from the foregoing embodiments that, in the compression and transmission system for VR images provided by the present invention, first, a video image to be sent by a server is obtained through an edge compression module 101, and the video image is compressed in a localized manner according to a compression range to form a first compressed video image, then, the first compressed video image is subjected to video encoding processing through a video encoding module 102 to form a transmission video image, then, the transmission video is transmitted to a client through a video decoding module 103, and the transmission video image is subjected to video decoding processing to reduce the transmission video image into a second compressed video image, and then, the second compressed video image is decompressed through a video decompression module 104 to form a display video image, even if d3d or opengl is used as a medium, image data in the compression range of the edge is compressed by an image rendering program, therefore, the resolution of the whole image is reduced by compressing the resolution of the edge image, the image volume is reduced, the transmission rate is increased, and the stability of VR live broadcast and VR streaming is improved.

The compression transmission method and system of VR images proposed according to the present invention are described above by way of example with reference to the accompanying drawings. However, it should be understood by those skilled in the art that various modifications can be made to the VR image compression and transmission method and system provided by the present invention without departing from the scope of the present invention. Therefore, the scope of the present invention should be determined by the contents of the appended claims.