阅读说明：本技术 页面视频渲染方法及装置 (Page video rendering method and device ) 是由 王浩东 朱珍 于 2020-02-24 设计创作，主要内容包括：本发明公开一种页面视频渲染方法及装置。由主线程创建至少一个子线程,在播放页面视频时,当主线程接收到多个视频窗口的视频原始码流时,主线程调用子线程分别对每个视频窗口的视频原始码流进行视频解码,以获得每个视频窗口的图像数据；然后,主线程接收子线程回调的每个所述视频窗口的图像数据,当页面刷新时,主线程将多个所述视频窗口的图像数据合并提交至图形处理器。进而,主线程还将基于多个所述视频窗口的图像数据,向所述图形处理器合并提交绘制指令。本技术方案由于将视频解码的过程分配到子线程中,从而提高了多核CPU的利用率,并且通过合并提交图像数据以及合并绘制指令,降低了多视频窗口下的操作耗时及系统开销。(The invention discloses a page video rendering method and device. The method comprises the steps that at least one sub-thread is created by a main thread, when a page video is played and the main thread receives video original code streams of a plurality of video windows, the main thread calls the sub-threads to respectively carry out video decoding on the video original code streams of each video window so as to obtain image data of each video window; then, the main thread receives the image data of each video window called back by the sub-thread, and when the page is refreshed, the main thread merges and submits the image data of the video windows to the graphics processor. Furthermore, the main thread also merges and submits drawing instructions to the graphics processor based on the image data of the plurality of video windows. According to the technical scheme, the video decoding process is distributed to the sub-threads, so that the utilization rate of the multi-core CPU is improved, and the time consumption and the system overhead of operation under multiple video windows are reduced by combining and submitting image data and combining drawing instructions.)

1. A method for rendering a page video, comprising:

when a page video is played, if a main thread receives video original code streams of a plurality of video windows, the main thread calls a sub-thread to respectively perform video decoding on the video original code stream of each video window so as to obtain image data of each video window;

the main thread receives the image data of each video window called back by the sub-thread;

and when the page is refreshed, merging and submitting the image data of the video windows to a graphics processor by the main thread.

2. The method of claim 1, further comprising:

and the main thread merges and submits drawing instructions to the graphics processor based on the image data of the video windows.

3. The method of claim 1, wherein the calling of the sub-thread by the main thread for video decoding the video raw code stream of each video window respectively to obtain the image data of each video window comprises:

the main thread creates at least one sub-thread; wherein one of said child threads corresponds to one or more of said video windows;

the main thread calls each sub-thread to perform video decoding on the video original code streams of one or more corresponding video windows respectively so as to obtain image data of each video window respectively;

and adjusting the actual size of the corresponding image data based on the display size of each video window so as to enable the adjusted size of the image data to be matched with the display size of the corresponding video window.

4. The method of claim 1, further comprising, after the main thread receives image data for each of the video windows for a sub-thread callback:

the main thread determines the memory storage address of the image data corresponding to each video window according to the number sequence of each video window;

and storing the image data corresponding to each video window to the corresponding memory area according to the memory storage address.

5. The method of claim 1, wherein said main thread merging image data for a plurality of said video windows to a graphics processor when a page is refreshed comprises:

when the page is refreshed, if the image data corresponding to all the video windows are updated, the main thread merges the image data corresponding to all the video windows and submits the merged image data to the graphics processor.

6. The method of claim 4, wherein said main thread merging image data for a plurality of said video windows to a graphics processor when page refreshing comprises:

when a page is refreshed, if image data corresponding to a part of video windows are updated, determining a memory area to be selected based on a memory storage address corresponding to the video window with the updated image data; the image data in the memory area to be selected at least comprises image data of all updated video windows;

and merging and submitting the image data in the memory area to be selected to a graphic processor.

7. The method of claim 6, wherein the determining the memory area to be selected based on the memory storage address corresponding to the video window with the updated image data comprises:

taking the corresponding minimum memory storage address in the video window with the updated image data as the initial address of the memory area to be selected;

taking the corresponding maximum memory storage address in the video window with the updated image data as the end address of the memory area to be selected;

and determining the memory area to be selected according to the starting address and the ending address.

8. The method of claim 1, wherein the period of the page refresh is determined based on a minimum frame period of the frame periods of all video windows.

9. The method of claim 1, wherein the primary thread is a JS page thread and the secondary thread is a Webworker thread.

10. A page video rendering apparatus, comprising:

the video decoding processing module is used for calling the sub-thread to respectively carry out video decoding on the video original code stream of each video window to obtain the image data of each video window if the main thread receives the video original code streams of the plurality of video windows when the page video is played;

the image data receiving module is used for receiving the image data of each video window called back by the sub-thread by the main thread;

and the image data submitting module is used for merging and submitting the image data of the video windows to a graphics processor by the main thread when the page is refreshed.

11. The apparatus of claim 10, further comprising:

and the drawing instruction submitting module is used for merging and submitting the drawing instruction to the graphics processor by the main thread based on the image data of the video windows.

12. The apparatus of claim 10, wherein the video decoding processing module comprises:

a sub-thread creating unit for creating at least one sub-thread by the main thread; wherein one of said child threads corresponds to one or more of said video windows;

the video decoding processing unit is used for respectively calling each sub-thread by the main thread to respectively carry out video decoding on the video original code streams of one or more corresponding video windows so as to respectively obtain the image data of each video window;

and the size adjusting unit is used for adjusting the actual size of the corresponding image data based on the display size of each video window so as to enable the adjusted size of the image data to be matched with the display size of the corresponding video window.

13. The apparatus of claim 10, further comprising:

the memory storage address determining module is used for determining the memory storage address of the image data corresponding to each video window by the main thread according to the number sequence of each video window;

and the memory area determining module is used for storing the image data corresponding to each video window to the corresponding memory area according to the memory storage address.

14. The apparatus of claim 10, wherein the image data submission module comprises: and the first image data submitting unit is used for merging and submitting the image data corresponding to all the video windows to the graphics processor by the main thread if the image data corresponding to all the video windows are updated when the page is refreshed.

15. The apparatus of claim 13, wherein the image data submission module comprises: a second image data submission unit; the second image data submission unit includes:

the memory area determination subunit is used for determining a memory area to be selected based on a memory storage address corresponding to a video window with updated image data if the image data corresponding to a part of the video window is updated when a page is refreshed; the image data in the memory area to be selected at least comprises image data of all updated video windows;

and the image data submitting subunit is used for merging and submitting the image data in the memory area to be selected to the graphics processor.

16. The apparatus as claimed in claim 15, wherein the memory region to be selected determining subunit comprises:

a minimum memory storage address determining unit, configured to use a minimum memory storage address corresponding to a video window in which all the image data is updated as an initial address of the memory area to be selected;

a maximum memory storage address determining unit, configured to use a maximum memory storage address corresponding to the video window where all the image data is updated as an end address of the memory area to be selected;

and the memory area determining unit is used for determining the memory area to be selected according to the starting address and the ending address.

17. The apparatus of claim 10, wherein the period of the page refresh is determined based on a minimum frame period of the frame periods of all video windows.

18. The apparatus of claim 10, wherein the primary thread is a JS page thread and the secondary thread is a Webworker thread.

19. A computer-readable storage medium storing a computer program for executing the page video rendering method according to any one of claims 1 to 9.

20. An electronic device, the electronic device comprising:

a processor;

a memory for storing the processor-executable instructions;

the processor is used for reading the executable instruction from the memory and executing the instruction to realize the page video rendering method of any one of the claims 1-9.

[ technical field ] A method for producing a semiconductor device

The invention relates to the technical field of webpage video rendering, in particular to a webpage video rendering method and device.

[ background of the invention ]

The Web Graphics library (Web Graphics L library, WebG L for short) is a 3D drawing protocol, the drawing technical standard allows JavaScript and OpenG L ES 2.0 to be combined together, and by adding one JavaScript binding of OpenG L ES 2.0, WebG L can provide hardware 3D accelerated rendering for HTM L5 Canvas, so that Web developers can more smoothly display 3D scenes and models in a browser by means of a system display card, and can also create complex navigation and data visualization.

Fig. 1A is a schematic signaling timing diagram of a page video rendering method in the prior art.

Referring to fig. 1A, in a video playing process, a streaming media service 101 sends a video original code stream to a page, a JS (JavaScript) page thread 102 receives the video original code stream, performs video decoding, and sends decoded texture data and a drawing instruction to a Graphics processor 103(Graphics Processing Unit, GPU for short), and the GPU completes picture rendering based on the received texture data and the drawing instruction for executing a formulated window picture.

However, since a single thread in the operating system can only use one CPU core to perform calculation, in a scene where multiple windows play videos simultaneously in a page, the time for occupying the JS page thread in the process of playing the page videos will increase, resulting in an excessive load on the JS page thread.

FIG. 1B is a schematic diagram of the time consumption distribution of the thread of the JS page during the process of adopting the page video rendering method shown in FIG. 1A.

Referring to fig. 1B, taking an example that three windows exist in a page to play video, each window receives a frame of video original code stream, and performs three processes of video decoding, texture data submission and rendering instruction, time consumption of the three processes is denoted as a, B and C, respectively, and time consumption processes of JS page thread processing events (e.g., user input, timers, etc.), page service logic processing, idle state, and the like are denoted as D.

Specifically, when the JS page thread receives the video original code stream of window 1, the time-consuming processes corresponding to video decoding, texture data submission, and rendering instructions are denoted as a1, B1, and C1, respectively. After the time-consuming process D, when the JS page thread receives the video original code stream of the window 2, video decoding is performed, and time-consuming processes corresponding to submission of texture data and rendering instructions are denoted as a2, B2, and C2, respectively. After the time-consuming process D, when the JS page thread receives the video original code stream of the window 3, video decoding is performed, and time-consuming processes corresponding to submission of texture data and rendering instructions are denoted as a3, B3, and C3, respectively.

Therefore, when the number of windows is increased, the time of the JS page thread occupied by the page video playing process is linearly increased, so that the thread load is overlarge, the thread can be quickly fully loaded or even run in an overload mode, the video decoding rate cannot keep up with the rate of receiving video data, the page video frame rate is reduced, and the problems of user input events, page jamming and the like cannot be responded in time are solved.

[ summary of the invention ]

In view of this, embodiments of the present invention provide a method and an apparatus for rendering a page video, so as to solve the technical problem in the prior art that when there are multiple windows on a page to play a video, the page is stuck.

In one aspect, an embodiment of the present invention provides a page video rendering method, including: when a page video is played, if a main thread receives video original code streams of a plurality of video windows, the main thread calls a sub-thread to respectively perform video decoding on the video original code stream of each video window so as to obtain image data of each video window; the main thread receives the image data of each video window called back by the sub-thread; and when the page is refreshed, merging and submitting the image data of the video windows to a graphics processor by the main thread.

Optionally, the method further comprises: and the main thread merges and submits drawing instructions to the graphics processor based on the image data of the video windows.

Optionally, the calling, by the main thread, the sub-thread to respectively perform video decoding on the video original code stream of each video window to obtain the image data of each video window includes: the main thread creates at least one sub-thread; wherein one of said child threads corresponds to one or more of said video windows; the main thread calls each sub-thread to perform video decoding on the video original code streams of one or more corresponding video windows respectively so as to obtain image data of each video window respectively; and adjusting the actual size of the corresponding image data based on the display size of each video window so as to enable the adjusted size of the image data to be matched with the display size of the corresponding video window.

Optionally, after the main thread receives the image data of each video window called back by the sub-thread, the method further includes: the main thread determines the memory storage address of the image data corresponding to each video window according to the number sequence of each video window; and storing the image data corresponding to each video window to the corresponding memory area according to the memory storage address.

Optionally, when the page is refreshed, the merging and submitting the image data of the plurality of video windows to the graphics processor by the main thread includes: when the page is refreshed, if the image data corresponding to all the video windows are updated, the main thread merges the image data corresponding to all the video windows and submits the merged image data to the graphics processor.

Optionally, when the page is refreshed, the merging and submitting the image data of the plurality of video windows to the graphics processor by the main thread includes: when a page is refreshed, if image data corresponding to a part of video windows are updated, determining a memory area to be selected based on a memory storage address corresponding to the video window with the updated image data; the image data in the memory area to be selected at least comprises image data of all updated video windows; and merging and submitting the image data in the memory area to be selected to a graphic processor.

Optionally, the determining a memory area to be selected based on the memory storage address corresponding to the video window with the updated image data includes: taking the corresponding minimum memory storage address in the video window with the updated image data as the initial address of the memory area to be selected; taking the corresponding maximum memory storage address in the video window with the updated image data as the end address of the memory area to be selected; and determining the memory area to be selected according to the starting address and the ending address.

Optionally, the period of the page refresh is determined based on a minimum frame period among the frame periods of all the video windows.

Optionally, the main thread is a JS page thread, and the sub-thread is a Webworker thread.

On the other hand, an embodiment of the present invention further provides a page video rendering apparatus, including: the video decoding processing module is used for calling the sub-thread to respectively carry out video decoding on the video original code stream of each video window to obtain the image data of each video window if the main thread receives the video original code streams of the plurality of video windows when the page video is played; the image data receiving module is used for receiving the image data of each video window called back by the sub-thread by the main thread; and the image data submitting module is used for merging and submitting the image data of the video windows to a graphics processor by the main thread when the page is refreshed.

Compared with the prior art, the technical scheme at least has the following beneficial effects:

according to the page video rendering method provided by the embodiment of the invention, at least one sub-thread is created by a main thread, and when the main thread receives video original code streams of a plurality of video windows when a page video is played, the main thread calls the sub-thread to respectively perform video decoding on the video original code stream of each video window so as to obtain image data of each video window; then, the main thread receives the image data of each video window called back by the sub-thread, and when the page is refreshed, the main thread merges and submits the image data of the video windows to the graphics processor. Furthermore, the main thread also merges and submits drawing instructions to the graphics processor based on the image data of the plurality of video windows. According to the technical scheme, the video decoding process is distributed to the sub-threads, so that the utilization rate of the multi-core CPU is improved, and the time consumption and the system overhead of operation under multiple video windows are reduced by combining and submitting image data and combining drawing instructions.

[ description of the drawings ]

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1A is a schematic signaling timing diagram of a page video rendering method in the prior art;

FIG. 1B is a schematic diagram illustrating the time consumption distribution of a thread of JS page during the use of the method for rendering a video of a page shown in FIG. 1A;

fig. 2 is a flowchart illustrating a page video rendering method according to an embodiment of the present invention;

fig. 3 is a schematic diagram illustrating scaling of image data in a page video rendering method according to an embodiment of the present invention;

fig. 4 is a schematic diagram of time consumption distribution of a JS page thread and a Webworker thread in the page video rendering method according to the embodiment of the present invention;

fig. 5 is a flowchart illustrating a page video rendering method according to another embodiment of the present invention;

fig. 6 is a schematic diagram illustrating that the storage position of the image data of each video window in the memory is determined in the page video rendering method shown in fig. 5;

FIG. 7A is a diagram illustrating one embodiment of the JS page thread merging submission of image data to the graphics processor in the page video rendering method illustrated in FIG. 5;

FIG. 7B is a diagram illustrating another embodiment of the JS page thread merging submission of image data to the graphics processor in the page video rendering method illustrated in FIG. 5;

fig. 8 is a schematic structural diagram of an embodiment of a page video rendering apparatus according to an embodiment of the present invention.

[ detailed description ] embodiments

For better understanding of the technical solutions of the present invention, the following detailed descriptions of the embodiments of the present invention are provided with reference to the accompanying drawings.

It should be understood that the described embodiments are only some embodiments of the invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Fig. 2 is a flowchart illustrating a page video rendering method according to an embodiment of the present invention. Referring to fig. 2, the page video rendering method includes:

step 201, when a page video is played, if a main thread receives video original code streams of a plurality of video windows, the main thread calls a sub-thread to respectively perform video decoding on the video original code stream of each video window so as to obtain image data of each video window;

202, the main thread receives image data of each video window called back by the sub-thread;

step 203, when the page is refreshed, the main thread merges the image data of the plurality of video windows and submits the merged image data to a graphics processor.

It should be noted that, in this embodiment, the main thread uses a JavaScript page thread, and the sub-thread uses a Webworker thread as an example to describe a specific implementation.

Those skilled in the art will appreciate that since the execution of Javascript in a browser is single-threaded, whereas in an operating system a single thread can only use one CPU core to compute, the thread may soon run full or even overloaded. Therefore, the browser can block the response of the browser when executing JavaScript until the browser returns to a normal state after the execution is finished.

The Webworker thread in the HTM L5 aims to solve the problem that JavaScript does not have multiple threads in a browser environment, because Javascript executed by a single thread on a page is a main thread, the Webworker thread created by the JS page thread is a sub-thread, only code in the main thread can cause browser blockage when being executed, and the sub-thread can not.

In step 201, the video raw stream of the video window is provided by a Streaming Media (Streaming Media) service. As known to those skilled in the art, a streaming media service refers to a service that compresses a series of media data and transmits the compressed media data in a streaming manner in segments in a network to realize real-time transmission of media transmission manners such as audio, video, and image on the network for users to view. When the browser loads a page, the html code and the JS code are operated, each video window on the page starts to play a page video, and the JS page thread acquires the video original code stream of one or more video windows from the streaming media service. The transmission frame rates of the video original code streams corresponding to different video windows can be the same or different.

The main thread calls the sub-thread to respectively perform video decoding on the video original code stream of each video window so as to obtain the image data of each video window, and the method comprises the following steps:

step 2011, the main thread creates at least one sub-thread; wherein one of said child threads corresponds to one or more of said video windows;

step 2012, the main thread respectively calls each sub-thread to respectively perform video decoding on the video original code streams of the corresponding one or more video windows to respectively obtain image data of each video window;

step 2012, the actual size of the corresponding image data is adjusted based on the display size of each video window, so that the adjusted size of the image data matches the display size of the corresponding video window.

In this embodiment, before the page video is played, a Webworker thread is established by the JS page thread. Specifically, if the number of the video windows on the page is small (for example, the number of the video windows is 3), the number of the Webworker threads matched with the number of the video windows is established, that is, the number of the Webworker threads is the same as the number of the video windows on the page. Therefore, the Webworker threads can correspond to the video windows on the page one by one, and each Webworker thread performs video decoding on the video original code stream of one video window, so that the utilization rate of the multi-core CPU is improved. If the number of the video windows on the page is large (for example, the number of the video windows is 12), in order to avoid the situation that when multiple Webworker threads are simultaneously used for processing video decoding due to the fact that too many Webworker threads are established, and a multi-core CPU runs at full load, the number of the Webworker threads which is smaller than that of the video windows can be established, and thus one Webworker thread can correspond to one or more video windows.

The JS page thread calls each Webworker thread to respectively perform video decoding on the video original code stream of one or more corresponding video windows to respectively obtain image data of each video window, and the image data is stored in a texture mode in the WebG L rendering process, so that the image data is texture data.

And adjusting the actual size of the corresponding image data based on the display size of each video window so as to enable the adjusted size of the image data to be matched with the display size of the corresponding video window.

Fig. 3 is a schematic diagram illustrating scaling of image data in the page video rendering method according to the embodiment of the present invention.

Referring to fig. 3, for example, the page 31 includes 4 video windows, which are a video window 1, a video window 2, a video window 3, and a video window 4, the display sizes of the 4 video windows are different, and the actual sizes of image data obtained after the Webworker threads corresponding to the 4 video windows perform video decoding on the video original code stream are also different from each other. In order to enable the subsequent graphics processor to execute the instantiation drawing instruction according to the image data to render the video windows, the image data corresponding to each video window can be suitable for rendering in the corresponding video window, and the actual size of the corresponding image data needs to be adjusted according to the display size of each video window, so that the adjusted size of the image data is matched with the display size of the corresponding video window.

Continuing with fig. 3, for example, if the actual size of the image data 321 of the video window 1 is smaller than the display size of the video window 1, the actual size of the image data 321 of the video window 1 is enlarged according to the display size of the video window 1, so that the enlarged size of the image data 321 of the video window 1 matches the display size of the video window 1.

For another example, if the actual size of the image data 322 of the video window 2 is larger than the display size of the video window 2, the image data 322 of the video window 2 is reduced according to the display size of the video window 2, so that the reduced size of the image data 322 of the video window 2 matches the display size of the video window 2.

For another example, if the actual size of the image data 323 in the video window 3 is larger than the display size of the video window 3, the image data 323 in the video window 3 is reduced according to the display size of the video window 3, so that the reduced size of the image data 323 in the video window 3 matches the display size of the video window 3.

For another example, if the actual size of the image data 324 of the video window 4 is smaller than the display size of the video window 4, the actual size of the image data 324 of the video window 4 is enlarged according to the display size of the video window 4, so that the enlarged size of the image data 324 of the video window 4 matches the display size of the video window 4.

In practical applications, the image data may be reduced by using a downsampled (downsampled) or downsampled (downsampled) algorithm; the enlargement processing of the image data may be performed by an algorithm of upsampling (upsampling) or image interpolation (interpolating).

It should be noted that fig. 3 is only a schematic diagram, and in practical applications, the display size of the number of video windows on a page is not limited to that shown in fig. 3, and the actual size of the image data of each video window is not limited to that shown in fig. 3.

The main thread receives image data for each of the video windows of a sub-thread callback, as depicted in step 202.

Specifically, each Webworker thread recalls the image data which is processed and is subjected to size adjustment and corresponding to each video window to the JS page thread. And the JS page thread receives the video data which are subjected to size adjustment and correspond to each video window and are recalled by each Webworker thread.

When the page is refreshed, the main thread merges the image data of the plurality of video windows and submits the merged image data to the graphics processor, as in step 203.

Specifically, the page refresh period may be set by itself (for example, refreshing the page once every 25 to 50 ms), or determined according to the minimum frame period in the frame periods of all video windows on the page or the maximum frame rate of the original stream of the transmitted video (for example, if the minimum frame period is 10ms or the maximum frame rate is 100Hz, the page is refreshed once every 10 ms).

When the page is refreshed, the JS page thread merges and submits the stored image data of the multiple video windows to a Graphics Processing Unit (GPU for short). The merged rendering refers to that when the current page is refreshed, the JS page thread only calls the rendering interface once to render the stored image data of the plurality of video windows to the GPU together. Compared with the prior art, the method and the device can reduce the number of times of submitting interface calls, shorten the time of submitting the image data to the GPU and reduce the system overhead.

Further, in the process of playing the page video, different video windows on the page have different loading schedules, and only image data of part of the video windows are updated when a certain page is refreshed. Therefore, the JS page thread can merge and submit the image data of the video window with the update to the GPU. This can reduce the amount of image data that the JS page thread submits to the GPU at one time.

In this embodiment, the method further includes the following steps: and the main thread merges and submits drawing instructions to the graphics processor based on the image data of the video windows.

The method is characterized in that the JS page thread merges image data corresponding to a plurality of video windows and submits the merged image data to the GPU, and then merges drawing instructions corresponding to the video windows and submits the merged drawing instructions to the GPU, so that the GPU can execute the drawing instructions of the video windows at one time after receiving the image data and the drawing instructions of the video windows submitted by the JS page thread, and the number of times of context switching of WebG L and the number of times of calling a drawing interface are reduced by merging the rendering processes of the video windows, and particularly, the performance can be greatly improved when the number of the video windows is large.

Fig. 4 is a schematic diagram of time consumption distribution of a JS page thread and a Webworker thread in the page video rendering method according to the embodiment of the present invention.

Referring to fig. 4, it is exemplified that there are three video windows in the page to play the video, and the number of Webworker threads matches the number of the video windows. Different from the prior art, in the embodiment, the Webworker thread 1, the Webworker thread 2, and the Webworker thread 3 respectively perform video decoding on the video original code streams of the video window 1, the video window 2, and the video window 3, then call back the decoded image data to the JS page thread, and merge and submit the image data of the three video windows and the drawing instruction to the GPU by the JS page thread. Wherein, each Webworker thread receives each frame (for example, the nth frame, the N +1 th frame, the N +2 th frame, and the N +3 th frame shown in fig. 4) of the video original code stream corresponding to the video window to perform video decoding, and the time consuming processes of A, JS page thread merging and submitting the image data and the drawing instruction of the three video windows correspond to time consuming processes of B and C, JS page thread processing events (for example, user input, a timer, and the like), page service logic processing, an idle state, and the like, respectively, and are denoted as D.

It can be seen that the video decoding process a of the video original code stream of each video window, which takes a long time, is distributed to each Webworker thread, so that the JS page thread can greatly reduce the load. The JS page thread combines the image data and the drawing instruction of each video window respectively and then submits the combined image data and drawing instruction to the GPU, so that the number of times of interface submission calling can be reduced, the time of submitting the image data to the GPU is shortened, and the system overhead is reduced.

Fig. 5 is a flowchart illustrating a page video rendering method according to another embodiment of the present invention. Referring to fig. 5, the page video rendering method includes:

step 501, when a page video is played, if a main thread receives video original code streams of a plurality of video windows, the main thread calls a sub thread to respectively perform video decoding on the video original code stream of each video window so as to obtain image data of each video window.

And 502, the main thread receives the image data of each video window called back by the sub-thread.

Step 503, the main thread determines the memory storage address of the image data corresponding to each video window according to the number sequence of each video window.

And step 504, storing the image data corresponding to each video window to a corresponding memory area according to the memory storage address.

In step 5051, when the page is refreshed, if the image data corresponding to all the video windows are updated, the main thread merges the image data corresponding to all the video windows and submits the merged image data to the graphics processor.

Step 5052, when the page is refreshed, if the image data corresponding to a part of the video window is updated, determining a memory area to be selected based on the memory storage address corresponding to the video window with the updated image data; the image data in the memory area to be selected at least comprises the image data of all the updated video windows.

In step 5053, the image data in the memory area to be selected is merged and submitted to a graphics processor.

In this embodiment, the specific implementation process of step 501 and step 502 may refer to the detailed description of step 201 and step 202 in the embodiment described in fig. 2, and is not described herein again.

Unlike the embodiment shown in fig. 2, in this embodiment, a specific implementation process of the JS page thread (i.e., the main thread) storing the image data into the memory after receiving the image data recalled by each Webworker thread (i.e., the sub-thread) is further described, and a specific implementation process of how the JS page thread selects the image data from the memory and submits the image data to the graphics processor when the page is refreshed.

In step 503, the main thread determines the memory storage address of the image data corresponding to each of the video windows according to the number sequence of each of the video windows.

In step 504, the image data corresponding to each of the video windows is stored in the corresponding memory area according to the memory storage address.

Specifically, since the positions of the video windows on the same page are fixed, the video windows can be numbered, then a memory area for storing image data is allocated in the memory for each video window according to the numbering sequence of each video window, and in order to facilitate subsequent extraction of image data from the memory, the memory area allocated to each video window is in a continuous memory area.

Fig. 6 is a schematic diagram illustrating that the storage position of the image data of each video window in the memory is determined in the page video rendering method shown in fig. 5.

Referring to fig. 6, the page 61 includes 5 video windows, respectively video window 1, video window 2, video window 3, video window 4, and video window 5. Memory storage addresses are allocated in the memory 62 for each video window in accordance with the numbering sequence of the 5 video windows. For example, the memory storage address allocated to the memory 62 for the image data of the video window 1 is 0X00000005 to 0X0000000F corresponding to the memory area 1, the memory storage address allocated to the memory 62 for the image data of the video window 2 is 0X0000000F to 0X000002FF corresponding to the memory area 2, the memory storage address allocated to the memory 62 for the image data of the video window 3 is 0X000002FF to 0X00003FFF corresponding to the memory area 3, the memory storage address allocated to the memory 62 for the image data of the video window 4 is 0X00003FFF to 0X0003FFFF corresponding to the memory area 4, and the memory storage address allocated to the image data of the video window 5 in the memory 62 is 0X0003FFFF to 0X01FFFF corresponding to the memory area 5.

Then, the JS page thread stores the image data corresponding to the video window 1 in the memory area 1, stores the image data corresponding to the video window 2 in the memory area 2, stores the image data corresponding to the video window 3 in the memory area 3, stores the image data corresponding to the video window 4 in the memory area 4, and stores the image data corresponding to the video window 5 in the memory area 5.

As shown in step 5051, when the page is refreshed, if the image data corresponding to all the video windows are updated, the main thread merges the image data corresponding to all the video windows and submits the merged image data to the graphics processor.

FIG. 7A is a diagram illustrating one embodiment of the JS page thread merging submission of image data to the graphics processor in the method for page video rendering illustrated in FIG. 5.

Referring to fig. 7A, all video windows have updates to their corresponding image data (the video window with diagonal lines is the video window in which the image data has updates when the page is refreshed). The JS page thread takes the memory areas corresponding to all the video windows in the memory 72 as the memory area 721 to be selected, extracts image data from the memory area 721 to be selected, and submits the image data to the graphics processor, and the JS page thread merges and submits a drawing instruction to the graphics processor based on the image data of all the video windows, so that the graphics processor executes the drawing instruction on all the video windows to render all the video windows.

In step 5052, when the page is refreshed, if the image data corresponding to a part of the video window is updated, determining the memory area to be selected based on the memory storage address corresponding to the video window whose image data is updated; the image data in the memory area to be selected at least comprises the image data of all the updated video windows.

In step 5053, the image data in the to-be-selected memory area is merged and submitted to the graphics processor.

FIG. 7B is a diagram illustrating another embodiment of the JS page thread committing merged rendering of image data to the graphics processor in the method of page video rendering shown in FIG. 5.

Referring to fig. 7B, the image data corresponding to the video window 2 and the video window 4 is updated (the video window with the oblique lines is a video window in which the image data is updated when the page is refreshed), so that the to-be-selected memory area is determined according to the memory storage address corresponding to the video window in which the image data is updated.

Specifically, the corresponding minimum memory storage address in the video window where all the image data are updated is used as the starting address of the memory area to be selected.

With continued reference to fig. 7B, the image data corresponding to video window 2 and video window 4 is updated, and the minimum memory storage address 0X0000000F corresponding to video window 2 is used as the starting address of the memory region 721 to be selected.

And taking the maximum memory storage address corresponding to the video window with the updated image data as the end address of the memory area to be selected.

With reference to fig. 7B, the image data corresponding to the video window 2 and the video window 4 is updated, and the maximum memory storage address 0X0003FFFF corresponding to the video window 4 is used as the end address of the memory area 721 to be selected.

And determining the memory area to be selected according to the starting address and the ending address.

With continued reference to fig. 7B, the memory area to be selected is a memory area with memory storage addresses 0X 0000000F-0X 0003FFFF, including memory area 2, memory area 3, and memory area 4.

It can be seen that, according to the method for determining the memory area to be selected, for the situation that the image data corresponding to the partial video window is updated, in order to enable the JS page thread to select the image data in the whole memory area from the memory at one time, the determined memory area to be selected may include a memory area (for example, the memory area 3 in fig. 7B) where the image data corresponding to the video window is not updated.

And merging and submitting the image data in the memory space to be selected to a graphics processor, merging and submitting a drawing instruction to the graphics processor by the JS page thread based on the image data of all the video windows with updates, so that the graphics processor executes the drawing instruction on the video windows with updates to render the video windows with updates.

Fig. 8 is a schematic structural diagram of an embodiment of a page video rendering apparatus according to an embodiment of the present invention.

Referring to fig. 8, the apparatus 8 comprises: the video decoding processing module 801 is configured to, when a page video is played, if a main thread receives video original code streams of multiple video windows, invoke a sub thread to perform video decoding on the video original code stream of each video window respectively to obtain image data of each video window. An image data receiving module 802, configured to receive, by the main thread, image data of each video window called back by a sub-thread. An image data submitting module 803, configured to merge and submit the image data of the multiple video windows to a graphics processor by the main thread when the page is refreshed.

In this embodiment, the apparatus 8 further includes: a rendering instruction submitting module 804, configured to merge and submit the rendering instruction to the graphics processor by the main thread based on the image data of the plurality of video windows.

Specifically, the video decoding processing module 801 includes: a sub-thread creating unit 8011 for the main thread to create at least one sub-thread; wherein one of the child threads corresponds to one or more of the video windows. The video decoding processing unit 8012 is configured to respectively invoke each sub-thread by the main thread to respectively perform video decoding on the video original code streams of the corresponding one or more video windows, so as to respectively obtain image data of each video window. A size adjusting unit 8013, configured to adjust an actual size of the corresponding image data based on a display size of each of the video windows, so that the adjusted size of the image data matches the display size of the corresponding video window.

The device 8 further comprises: a memory storage address determining module 805, configured to determine, by the main thread, a memory storage address of image data corresponding to each video window according to the number sequence of each video window. A memory area determining module 806, configured to store the image data corresponding to each video window to a corresponding memory area according to the memory storage address.

The image data submission module 803 includes: the first image data submitting unit 8031 is configured to, when the page is refreshed, if the image data corresponding to all the video windows are updated, merge the image data corresponding to all the video windows and submit the merged image data to the graphics processor by the main thread.

The image data submission module 803 includes: a second image data submission unit 8032; the second image data submission unit 8032 includes: a to-be-selected memory area determining subunit 80321, configured to determine, when a page is refreshed, a to-be-selected memory area based on a memory storage address corresponding to a video window in which image data is updated if image data corresponding to a part of the video window is updated; the image data in the memory area to be selected at least comprises the image data of all the updated video windows.

An image data submitting subunit 80322, configured to merge and submit the image data in the to-be-selected memory area to the graphics processor.

The to-be-selected memory area determination subunit 80322 includes: a minimum memory storage address determining unit (not shown in the figure), configured to use a minimum memory storage address corresponding to the video window where all the image data is updated as an initial address of the memory area to be selected; a maximum memory storage address determining unit (not shown in the figure), configured to use a maximum memory storage address corresponding to the video window where all the image data is updated as an end address of the memory area to be selected; a memory area determining unit (not shown in the figure) configured to determine the memory area to be selected according to the start address and the end address.

An embodiment of the present invention further provides a computer-readable storage medium, where the storage medium stores a computer program, and the computer program is configured to execute each step in the above-mentioned embodiment of the page video rendering method.

An embodiment of the present invention further provides an electronic device, where the electronic device includes: a processor; a memory for storing the processor-executable instructions; the processor is configured to read the executable instruction from the memory and execute the instruction to implement each step in the above-mentioned embodiment of the page video rendering method.

The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the examples of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should be understood that the term "and/or" as used herein is merely one type of association that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.

The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination" or "in response to a detection", depending on the context. Similarly, the phrases "if determined" or "if detected (a stated condition or event)" may be interpreted as "when determined" or "in response to a determination" or "when detected (a stated condition or event)" or "in response to a detection (a stated condition or event)", depending on the context.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the embodiments provided in the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions in actual implementation, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.

The integrated unit implemented in the form of a software functional unit may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) or a Processor (Processor) to execute some steps of the methods according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.