阅读说明：本技术 一种全景视频传输方法及设备 (Panoramic video transmission method and device ) 是由 刘金朋 史东平 任子健 于 2021-08-30 设计创作，主要内容包括：本申请涉及全景视频技术领域,提供一种全景视频传输方法及设备,具体的,通过预设分块的数量开启相应数量的线程对全景视频帧进行分割,针对分割后的每个图像分块,定制了一个编码器,对该图像分块再次进行分割,对分割后的若干个分块切片同时进行编码,得到该图像分块的编码数据,调用相应的线程对该图像分块的编码数据进行封装,封装后发送给目标终端进行显示,通过为每个图像分块开启一个线程,可充分利用CPU多核处理能力,提高了图像分块的编码效率,由于图像分块的编码效率提高了,因此,减少了封装后数据的发送时延,提高了全景视频传输的实时性。(The application relates to the technical field of panoramic video, provides a panoramic video transmission method and equipment, and particularly, starting a corresponding number of threads to segment the panoramic video frame by presetting the number of the segments, customizing an encoder for each segmented image segment, dividing the image block again, coding a plurality of divided block slices simultaneously to obtain coded data of the image block, calling corresponding threads to package the coded data of the image block, sending the packaged coded data to a target terminal for display, by starting a thread for each image block, the multi-core processing capability of the CPU can be fully utilized, the coding efficiency of the image block is improved, because the coding efficiency of the image blocks is improved, the sending time delay of the data after packaging is reduced, and the real-time property of panoramic video transmission is improved.)

1. A method for transmitting panoramic video, comprising:

acquiring a data source and determining the type of the data source;

if the type of the data source is a panoramic video, starting threads with corresponding quantity according to a preset quantity of blocks for each panoramic video frame in the panoramic video frames to block the panoramic video frames, wherein each panoramic video frame is divided into a plurality of image blocks;

for each image block in each image block, dividing the image block according to a preset rule to obtain a plurality of block slices, and simultaneously coding the plurality of block slices to obtain coded data of the image block;

and respectively packaging the coded data of each image block, and transmitting the packaged file to a target terminal.

2. The method of claim 1, wherein the method further comprises:

and if the type of the data source is audio, directly packaging the audio, and sending the packaged audio to the target terminal.

3. The method of claim 1, wherein individual panoramic video frames in the panoramic video are obtained by:

and simultaneously decoding the panoramic video according to the preset number of threads to obtain each panoramic video frame, wherein the panoramic video frames are in YUV format.

4. The method of claim 1, wherein the encapsulation file contains a Uniform Resource Locator (URL) for the respective image partition, such that the target client retrieves the corresponding image partition according to the URL.

5. A method according to any of claims 1-4, characterized in that the employed coding standard comprises any of the H.26x standard, the MPEG standard, the source-coded AVS standard.

6. An electronic device comprising at least one external communication interface, a memory, a processor, characterized in that:

the external communication interface is connected with the processor through a bus and is configured to receive and transmit data;

the memory storage is connected with the processor through a bus and stores computer program instructions;

the processor is configured to perform the following operations in accordance with the computer program instructions:

acquiring a data source and determining the type of the data source;

if the type of the data source is a panoramic video, starting threads with corresponding quantity according to a preset quantity of blocks for each panoramic video frame in the panoramic video frames to block the panoramic video frames, wherein each panoramic video frame is divided into a plurality of image blocks;

for each image block in each image block, dividing the image block according to a preset rule to obtain a plurality of block slices, and simultaneously coding the plurality of block slices to obtain coded data of the image block;

and respectively packaging the coded data of each image block, and transmitting the packaged file to a target terminal.

7. The electronic device of claim 6, wherein the processor is further configured to:

and if the type of the data source is audio, directly packaging the audio, and sending the packaged audio to the target terminal.

8. The electronic device of claim 6, wherein the processor obtains individual panoramic video frames in the panoramic video by:

and simultaneously decoding the panoramic video according to the preset number of threads to obtain each panoramic video frame, wherein the panoramic video frames are in YUV format.

9. The electronic device of claim 6, wherein the encapsulation file contains a Uniform Resource Locator (URL) for the respective image tile, such that the target client retrieves the corresponding image tile according to the URL.

10. An electronic device as claimed in any one of claims 6 to 9, wherein the encoding standard employed comprises any one of the h.26x standard, the MPEG standard, the source-encoded AVS standard.

Technical Field

The present application relates to the field of panoramic video technologies, and in particular, to a panoramic video transmission method and apparatus.

Background

Panoramic video is a new multimedia form developed based on 360-degree panoramic images, and is converted into dynamic panoramic video by continuously playing a series of static panoramic images. The panoramic video is generally formed by splicing video images of all directions collected by a panoramic camera through software, is played by using a special player, projects a planar video into a 360-degree panoramic mode, and presents the planar video to a viewer with a full-surrounding space view of 360 degrees in the horizontal direction and 180 degrees in the vertical direction. The viewer can control the playing of the panoramic video in modes of head motion, eyeball motion, remote controller control and the like, so that the viewer can experience the experience of being personally on the scene. As a new heterogeneous multimedia Service, a panoramic video Service stream contains multiple data types such as audio, video, text, interaction, control command, etc., and has diversified Quality of Service (QoS) requirements.

Currently, panoramic video mostly adopts a full-view transmission scheme (also called a single-stream scheme). The full-view transmission scheme is that a 360-degree panoramic video is encoded into a single code stream and sent to a terminal, and the terminal performs full decoding on the single code stream to obtain the panoramic video and plays the panoramic video for a user to watch. However, since the full-view transmission scheme is to encode the whole panoramic video frame, the encoding efficiency is low, the amount of encoded data is large, the requirement on transmission bandwidth is high, the buffering time for playing the panoramic video by the terminal is long, and the decoding performance of the terminal is also high. Therefore, the full-view transmission scheme can only satisfy the low-resolution panoramic video transmission and playing.

Disclosure of Invention

The application provides a panoramic video transmission method and equipment, which are used for improving the coding efficiency of a panoramic video and further improving the real-time property of panoramic video transmission.

In a first aspect, an embodiment of the present application provides a method for transmitting a panoramic video, including:

acquiring a data source and determining the type of the data source;

if the type of the data source is a panoramic video, starting threads with corresponding quantity according to a preset quantity of blocks for each panoramic video frame in the panoramic video frames to block the panoramic video frames, wherein each panoramic video frame is divided into a plurality of image blocks;

for each image block in each image block, dividing the image block according to a preset rule to obtain a plurality of block slices, and simultaneously coding the plurality of block slices to obtain coded data of the image block;

and respectively packaging the coded data of each image block, and transmitting the packaged file to a target terminal.

In a second aspect, an embodiment of the present application provides an electronic device, including at least one external communication interface, a memory, and a processor:

the external communication interface is connected with the processor through a bus and is configured to receive and transmit data;

the memory storage is connected with the processor through a bus and stores computer program instructions;

the processor is configured to perform the following operations in accordance with the computer program instructions:

acquiring a data source and determining the type of the data source;

if the type of the data source is a panoramic video, starting threads with corresponding quantity according to a preset quantity of blocks for each panoramic video frame in the panoramic video frames to block the panoramic video frames, wherein each panoramic video frame is divided into a plurality of image blocks;

for each image block in each image block, dividing the image block according to a preset rule to obtain a plurality of block slices, and simultaneously coding the plurality of block slices to obtain coded data of the image block;

and respectively packaging the coded data of each image block, and transmitting the packaged file to a target terminal.

In a third aspect, an embodiment of the present application provides a computer-readable storage medium, where computer-executable instructions are stored in the computer-readable storage medium, and the computer-executable instructions are configured to enable a computer to execute a panoramic video transmission method provided in the embodiment of the present application.

In the embodiment of the application, for each panoramic video frame in the obtained panoramic video, the corresponding number of threads is started according to the preset number of the blocks, that is, each image block corresponds to one thread, and in the encoding process, the multiple threads are used for performing the blocking in parallel, so that the blocking efficiency is improved; each image block is divided into a plurality of block slices, the block slices contained in the image block are coded in parallel to obtain coded data of the image block, and the image block is further divided, so that the multithreading parallel Processing capability of a Central Processing Unit (CPU) can be fully utilized, and the coding efficiency of the image block is improved; furthermore, the coded data of each image block are respectively packaged, and the packaged file is transmitted to the target terminal.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to these drawings without inventive exercise.

Fig. 1 is a schematic diagram illustrating an application scenario provided by an embodiment of the present application;

fig. 2 is a schematic diagram illustrating a full-view image dividing manner provided by an embodiment of the present application;

fig. 3a schematically illustrates an image partitioning manner provided by an embodiment of the present application;

fig. 3b is a schematic diagram schematically illustrating another image block dividing manner provided by the embodiment of the present application;

fig. 4 is a flowchart illustrating a panoramic video transmission method provided by an embodiment of the present application;

fig. 5 schematically illustrates an encoding diagram of a panoramic video provided by an embodiment of the present application;

fig. 6 is a diagram illustrating an exemplary functional structure of a server provided in an embodiment of the present application;

fig. 7 is a diagram illustrating an electronic device hardware structure provided in an embodiment of the present application.

Detailed Description

To make the objects, embodiments and advantages of the present application clearer, the following description of exemplary embodiments of the present application will clearly and completely describe the exemplary embodiments of the present application with reference to the accompanying drawings in the exemplary embodiments of the present application, and it is to be understood that the described exemplary embodiments are only a part of the embodiments of the present application, and not all of the embodiments.

All other embodiments, which can be derived by a person skilled in the art from the exemplary embodiments described herein without inventive step, are intended to be within the scope of the claims appended hereto. In addition, while the disclosure herein has been presented in terms of one or more exemplary examples, it should be appreciated that aspects of the disclosure may be implemented solely as a complete embodiment.

It should be noted that the brief descriptions of the terms in the present application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of the present application. These terms should be understood in their ordinary and customary meaning unless otherwise indicated.

Furthermore, the terms "comprises" and "comprising," as well as any variations thereof, are intended to cover a non-exclusive inclusion, such that a product or device that comprises a list of elements is not necessarily limited to those elements explicitly listed, but may include other elements not expressly listed or inherent to such product or device.

The term "module" as used herein refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and/or software code that is capable of performing the functionality associated with that element.

Compared with the traditional video, the panoramic video has the characteristics of high resolution, large data volume and high code rate, the resolution of the panoramic video is continuously improved and gradually changes from 4K to 8K, even 12K and 16K, and the requirement on the bandwidth of a transmission network is higher.

Currently, panoramic video transmission mainly faces the following challenges:

1) higher network bandwidth: for a 2D panoramic video, the resolution of a full-view image is generally 8K, the transmission frame rate is 30FPS, the monocular resolution is 1920 × 1920 Pixels, the number of Pixels (PPD) that can be seen within a 1-Degree viewing angle is 21, and the network bandwidth requirement is about 100 Mbps; for a 3D panoramic video with better experience, the resolution of a full-view image is generally 24K, the transmission frame rate is 120FPS, the monocular resolution is 7680x7680 pixels, the PPD seen within a 1 degree view is 64, and the network bandwidth requirement is about 5 Gbps.

2) Strict transmission delay: when a panoramic video is viewed by using an immersive terminal such as a VR device (e.g., VR glasses, VR head-mounted display device), a visual field delay (MTP) cannot exceed 20ms for the case of head rotation, otherwise a head dizziness may be caused.

Because the encoding efficiency of the full-view transmission scheme is low, the requirement on network bandwidth is high, the transmission delay is long, and the full-view transmission scheme cannot be applied to playing of panoramic videos with high resolution, the FOV transmission scheme becomes a research direction of panoramic video transmission in recent years.

The FOV transmission scheme is a scheme for differentially transmitting a panoramic video based on a user view angle, and mainly focuses on the quality of a picture in a current user view angle area. The FOV transmission scheme carries out image blocking on a high-resolution panoramic video frame in space, the resolution and the coverage space range of each block are the same, then multi-code-rate coding is carried out on each image block, the code rate of each image block is the same, a plurality of video streams are generated, and then the video streams are sent to a terminal. The terminal displays a visual area of a user visual angle as an image block, and displays the visual area outside as a low-resolution full-visual-angle image. The FOV transmission scheme displays a portion outside the viewable area as a low-resolution full-view image, and thus, when the viewing angle changes, a user may see a low-definition full-view image, which causes a visual blur and reduces the user experience.

In order to reduce the phenomenon of visual blurring caused by visual angle change, another FOV transmission scheme samples a panoramic video frame to generate a plurality of full-view video frames with different resolutions, equally divides the full-view video frames with each resolution, performs multi-rate coding to generate a plurality of video streams, and sends the video streams to a terminal. The terminal displays the visual area of the user visual angle as the image blocks with high resolution, for the parts outside the visual area, the image blocks with different resolutions are transmitted according to the distance from the current viewpoint, and the closer the distance is, the higher the resolution of the transmitted image blocks is, so that when the field angle moves, the panoramic image with gradually transitional resolution is seen by the user, and the better user experience is improved. But the code rates of the image blocks with different resolutions are different, the data redundancy is higher, and the requirement on the stability of a transmission network is higher.

Compared with a full-view transmission scheme, the FOV transmission scheme is used for dividing a full-view video frame into a plurality of image blocks and then coding the image blocks, so that the coding efficiency is improved, the data volume of the image blocks is smaller than that of a complete panoramic video frame, and the requirement on network bandwidth is reduced. The coding efficiency of an image block is inversely related to the resolution, and the higher the resolution of the image block is, the lower the coding efficiency is. FOV transmission schemes to guarantee visual effect, the resolution of each image block needs to be guaranteed.

The application provides a panoramic video transmission method and equipment under the condition of ensuring the image block resolution, the multi-core processing capacity of a CPU is fully utilized, a thread is started for each image block, the image blocks are further divided, the granularity of coding is reduced, a plurality of block slices behind each image block are coded in parallel, the coding efficiency of a panoramic video is improved under the condition of ensuring the image block high resolution, and the real-time performance of the panoramic video transmission is further improved.

To describe the embodiments of the present application in detail, explanations are given below for the terms of the present application.

An MPD file: is an XML document describing the basic information of a multimedia fragment (image tiles in this application). The MPEG-DASH protocol specifies that the outermost layer of the ". MPD" file is a Period (Period), each Period represents a certain time Period, a complete DASH stream usually consists of multiple media representations (repetition), and consists of one or more periods, each Period contains one or more Adaptation sets (Adaptation sets), and each Adaptation Set includes one or more media content components and versions thereof with different bitrate levels. In fact, each playback corresponds to information such as determined bit rate, resolution, frame rate, etc., and each media content component is encoded into different media representations according to characteristics such as different bit rates, resolutions, etc. Each replication includes one or more media segments (segments), which are the basic units of the MPD file and all have their own unique Uniform Resource Locator (URL) address.

Embodiments of the present application are described in detail below with reference to the accompanying drawings.

Fig. 1 schematically illustrates an application scenario provided by an embodiment of the present application; as shown in fig. 1, in response to an audio/video playing request sent by the terminal 100, the server 200 acquires a data source from the repository 300, decodes the data source, determines the type of the data source, encodes and encapsulates the data source in a matching encoding manner according to the type of the data source, sends the encapsulated data to the terminal 100 in a bluetooth or WiFi manner, and displays the received data to a viewing user after the terminal 100 decodes the received data.

The terminal 100 may be a terminal having a panoramic video playing function, such as a VR head-mounted display device, VR glasses, a smart phone, a smart television, a notebook computer, and a desktop.

The server 200 may be an independent physical server, a server cluster or a distributed system (such as but not limited to the number of servers illustrated in fig. 1) formed by a plurality of physical servers, or a plurality of cloud servers providing basic services such as cloud service, cloud computing, cloud storage, cloud communication, middleware service, domain name service, security service, big data, artificial intelligence, and the like in the cloud service technology.

The repository 300 may be a MySQL database, a Redis database, or a MongoBD database, and is used to store resources of various data types such as audio, video, text, and pictures.

FIG. 2 is a schematic diagram illustrating a full-view image dividing manner provided by an embodiment of the present application; as shown in fig. 2, a full-view image with a resolution of M × N (pixels) is divided into 128 image blocks, and each image block has the same spatial range and the same resolution.

Each image block is further divided on an image block basis, see fig. 3a, into 4 block slices.

It should be noted that fig. 3a is only an example, and the number and the dividing manner of the partition slices in the embodiment of the present application are not limited, for example, the number may be 2 to 6, or the partitions may be divided vertically, as shown in fig. 3 b.

Fig. 4 is a flowchart illustrating a panoramic video transmission method provided by an embodiment of the present application; as shown in fig. 4, the process is executed by the server, and mainly includes the following steps:

s401: a data source is obtained and a type of the data source is determined.

In S401, the server receives an audio/video playing request sent by the target terminal, starts a data transmission application, obtains a data source from the repository, decodes the obtained data source, and determines the type of the data source.

S402: and determining whether the data source is panoramic video, if so, executing S403, and if so, executing S406.

In S402, there is a large difference between the data volumes of different types of data sources, and generally, for a panoramic video and an audio with the same playing duration, the data volume of the panoramic video is a multiple of the data volume of the audio, so that different encoding methods need to be adopted for the panoramic video and the audio.

S403: and starting threads with corresponding quantity to block the panoramic video frames according to the preset quantity of the blocks aiming at each panoramic video frame in the panoramic video frames, wherein each panoramic video frame is divided into a plurality of image blocks.

In S403, when the data source is a panoramic video, the panoramic video is decoded according to the number of preset threads to obtain each panoramic video frame in the YUV format, and the decoding speed of the panoramic video is increased by multi-thread parallel decoding.

And aiming at each panoramic video frame in each decoded video frame, starting threads with corresponding quantity according to the preset quantity of blocks, enabling each image block to correspond to one thread, blocking the panoramic video frame by using the multithreading parallel processing capacity of a CPU (Central processing Unit) and simultaneously dividing each panoramic video frame into a plurality of image blocks by a plurality of threads. For example, as shown in fig. 2, dividing a panoramic video frame into 16 × 8 — 128 blocks requires 128 threads to be started.

S404: the method comprises the steps of dividing image blocks according to a preset rule aiming at each image block in each image block to obtain a plurality of block slices, and simultaneously coding the plurality of block slices to obtain coded data of the image blocks.

In S404, for each divided image partition, a High Efficiency Video Coding (HEVC, i.e., h.265) encoder is customized to encode the image partition in YUV format. Specifically, the encoder divides the image blocks into a plurality of block slices (for example, 2 to 6) according to a preset rule, simultaneously encodes the plurality of block slices to obtain h.265 encoded data of the plurality of block slices, and splices the h.265 encoded data of the plurality of block slices to obtain encoded data of the image blocks.

It should be noted that, in the embodiment of the present application, there is no limitation on the encoding manner in S404, and the encoding manner may further include h.264, h.263, h.261(h.265, h.264, h.263, and h.261 are collectively referred to as h.26x Standard), MPEG (abbreviation of Moving Picture Experts Group) Standard, and Audio Video coding Standard (AVS) Standard.

S405: and respectively packaging the coded data of each image block, and transmitting the packaged file to a target terminal.

Since each image partition opens one thread, in S405, the multiple threads simultaneously encapsulate the encoded data of the corresponding image partition, and obtain a login file. The package file comprises an encoded media file and an MPD file, wherein the MPD file comprises URLs of all image blocks.

In S405, the server transmits the media file and the MPD file to the target terminal using a hypertext Transfer Protocol (HTTP), and the target terminal acquires corresponding image chunks according to URLs of the image chunks in the MPD file and displays the panoramic video after splicing.

It should be noted that, in the embodiment of the present application, there is no limitation requirement on the encoding standard of the encoder corresponding to the image partition, including but not limited to DASH format, MP4 format, and hls (http Live streaming) format.

S406: and directly packaging the audio, and sending the packaged audio to a target terminal.

In S406, the data size of the audio file is small, the server may directly package the audio and then send the audio to the target terminal, and the target terminal analyzes and decodes the audio, and plays the audio to the user after decoding.

Fig. 5 schematically illustrates an encoding diagram of a panoramic video provided by an embodiment of the present application; as shown in fig. 5, the server obtains a data source, decodes the data source, determines the type of the data source, puts the data source into a panoramic video buffer if the data source is a panoramic video, then segments a panoramic video frame according to a preset number of threads to obtain a plurality of image partitions, puts the image partitions into corresponding partition buffers, customizes an HEVC encoder in a thread corresponding to each image partition, first segments the corresponding image partition, simultaneously encodes a plurality of segmented slices, encodes a plurality of block slices, and then encapsulates the encoded block slices to obtain a corresponding MPD file.

The embodiment of the application can be applied to scenes with higher requirements on the playing time delay of the panoramic video, such as live scenes. Assuming that the resolution of the panoramic video is 7680 × 3840 (pixels), the frame rate is 30fps, each panoramic video frame is divided into 16 × 8 image blocks, each image block is divided into 4 block slices, and the coding efficiency of the panoramic video is 4 times that of the non-block slices.

According to the embodiment of the application, the threads of the corresponding quantity are started according to the quantity of the image blocks, each thread customizes an encoder for the corresponding image block, the image blocks are further divided, the encoding granularity is reduced, the multi-core processing capacity of a CPU is fully utilized, under the condition that the high resolution of the image blocks is ensured, a plurality of block slices behind each image block are encoded in parallel, the encoding efficiency of a panoramic video is improved, and the real-time performance of the panoramic video transmission is further improved.

Based on the same technical concept, the embodiment of the application provides a server, and the server can execute the panoramic video transmission method provided by the embodiment of the application and can achieve the same technical effect.

Referring to fig. 6, the server includes a data obtaining module 601, a blocking module 602, an encoding module 603, and an encapsulating module 604;

a data obtaining module 601, configured to obtain a data source and determine a type of the data source;

a blocking module 602, configured to, if the type of the data source is a panoramic video, start, according to a preset number of blocks, a corresponding number of threads to block the panoramic video frames for each panoramic video frame in the panoramic video frames, where each panoramic video frame is divided into a plurality of image blocks;

the encoding module 603 is configured to, for each image partition in each image partition, divide the image partition according to a preset rule to obtain a plurality of partition slices, and encode the plurality of partition slices to obtain encoded data of the image partition;

and the encapsulating module 604 is configured to encapsulate the encoded data of each image partition, and transmit the encapsulated file to the target terminal.

Optionally, the encapsulation module 604 is further configured to:

and if the type of the data source is audio, directly packaging the audio, and sending the packaged audio to the target terminal.

Optionally, the server further includes a decoding module 605, configured to:

and simultaneously decoding the panoramic video according to the preset number of threads to obtain each panoramic video frame, wherein the panoramic video frames are in YUV format.

Optionally, the package file includes URLs of the image blocks, so that the target client obtains the corresponding image blocks according to the URLs.

Optionally, the adopted coding standard comprises any one of h.26x standard, MPEG standard, AVS standard.

Based on the technical idea of the intended figure, the embodiment of the present application provides an electronic device, see fig. 7, which includes a processor 701, a memory 702, and at least one external communication interface 703; the processor 701, the memory 702, and the external communication interface 703 are connected by a bus 704.

The memory 702 stores computer program instructions, and the external communication interface 703 is used for transmitting and receiving data;

the processor 701 executes the computer program instructions in the memory 702 to implement the panoramic video transmission method discussed previously.

Fig. 7 illustrates an example of one processor 701, but the number of processors 701 is not limited in practice.

Based on the same technical concept, the embodiment of the present application also provides a computer storage medium storing computer instructions, which when run on a computer, cause the computer to execute the transmission method of the panoramic video as discussed above.

The storage medium may be a volatile storage medium (volatile memory), such as a random-access memory (RAM); the storage medium may also be a non-volatile storage medium (non-volatile memory), such as a read-only memory (rom), a flash memory, a hard disk (HDD) or a solid-state drive (SSD), or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to this. The storage medium may be a combination of the above.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.