阅读说明：本技术 内存管理方法以及相关装置 (Memory management method and related device ) 是由 周海林 曲波 曹斌 于 2018-06-28 设计创作，主要内容包括：本申请提供了内存管理方法以及相关装置,该方法包括：对码流进行解码得到第一图像帧,第一图像帧包括n个片；对第一图像帧进行压缩得到第二图像帧,第二图像帧包括n个经压缩后的片；在内存中为n个经压缩后的片分配帧缓冲区,并为n个经压缩后的片所占用的内存空间建立虚拟页面与物理页面的映射关系,每个经压缩后的片所占用的内存空间包括一个或多个物理页面,n个经压缩后的片所占用的多个虚拟页面的页号连续；最后,将n个经压缩后的片存储到帧缓冲区。实施本申请方案能够有效降低解码时的参考帧存取对内存的占用。(The application provides a memory management method and a related device, wherein the method comprises the following steps: decoding the code stream to obtain a first image frame, wherein the first image frame comprises n pieces; compressing the first image frame to obtain a second image frame, wherein the second image frame comprises n compressed slices; distributing frame buffers for the n compressed slices in a memory, establishing a mapping relation between a virtual page and a physical page for memory spaces occupied by the n compressed slices, wherein the memory space occupied by each compressed slice comprises one or more physical pages, and the page numbers of a plurality of virtual pages occupied by the n compressed slices are continuous; finally, the n compressed slices are stored in a frame buffer. By the adoption of the scheme, the occupation of the reference frame access to the memory during decoding can be effectively reduced.)

1. A method for memory management in video decoding, comprising:

decoding a code stream to obtain a first image frame, wherein the first image frame comprises n tiles, each tile represents an image of a region in the first image frame, and n is an integer greater than or equal to 2;

compressing the first image frame to obtain a second image frame, wherein the second image frame comprises n compressed slices;

distributing frame buffer for the n compressed slices in a memory;

establishing a mapping relation between virtual pages and physical pages for memory spaces occupied by the n compressed slices in the frame buffer area, wherein the memory space occupied by each compressed slice comprises one or more physical pages, and the page numbers of a plurality of virtual pages occupied by the n compressed slices are continuous under the mapping relation;

and respectively storing the n compressed slices into the frame buffer area.

2. The method of claim 1, wherein said allocating frame buffers in memory for the n compressed slices comprises:

allocating the frame buffer area for the n compressed slices in a memory block pool in a memory, wherein the frame buffer area comprises M memory blocks, each memory block comprises a plurality of physical pages, the physical pages correspond to a plurality of virtual pages with continuous page numbers, and the memory space occupied by the n compressed slices is less than or equal to the memory space corresponding to the M memory blocks;

the method further comprises the following steps: and recovering the unoccupied residual memory blocks in the frame buffer area to the memory block pool, wherein the residual memory blocks are used for subsequent frame buffer area allocation and slice storage.

3. The method according to claim 2, wherein said establishing a mapping relationship between virtual pages and physical pages for the memory space occupied by the n compressed slices in the frame buffer includes:

determining a page table corresponding to each memory block in the M memory blocks, where the page table corresponding to each memory block includes a mapping relationship between each virtual page and a physical page in the memory block;

and combining the page tables corresponding to the M memory blocks into a page table of the memory space occupied by the n compressed pieces.

4. The method of claim 1,

said allocating said frame buffers in memory for said n compressed slices, comprising:

allocating a virtual memory space to the n compressed slices, wherein the virtual memory space comprises a plurality of virtual pages with continuous page numbers; each compressed slice of the n compressed slices corresponds to one or more virtual pages;

respectively carrying out page table inquiry on the page numbers of the one or more virtual pages;

if the page table is inquired to have no mapping relation between the page numbers of the one or more virtual pages and the page numbers of the physical pages, the one or more physical pages are allocated to the one or more virtual pages in the memory.

5. The method of claim 4, wherein prior to said allocating the one or more physical pages for the one or more virtual pages in the memory, further comprising:

acquiring a bitmap bit map, wherein the bitmap comprises a plurality of bits, and each bit is used for indicating whether each physical page in a memory is occupied or not;

and finding the one or more unoccupied physical pages in the memory according to the bitmap.

6. The method according to claim 4 or 5, wherein the establishing a mapping relationship between a virtual page and a physical page for the memory space occupied by the n compressed slices in the frame buffer includes:

adding a page number of the one or more physical pages to the page table, such that the page table includes a mapping relationship between the one or more virtual pages and the one or more physical pages.

7. The method according to any one of claims 1 to 6,

the storing the n compressed slices into the frame buffer respectively includes:

storing the compressed slices, compressed information and header address information into the frame buffer area in a form of a data unit; each compressed slice corresponds to one piece of compression information, and the compression information is used for indicating the size of the memory space occupied by the corresponding compressed slice; the header address information is used to indicate a start address of the data unit.

8. The method of any of claims 1-7, wherein after storing the n compressed slices in the frame buffer, the method further comprises:

reading the n compressed slices from the frame buffer respectively;

decompressing the n compressed slices to obtain a decoded frame of a first image frame;

and further decoding the code stream by taking the decoded frame as a reference frame to obtain a third image frame.

9. An apparatus for memory management, the apparatus comprising a decoding module, a memory management module, and a compression module, wherein:

the decoding module is used for decoding the code stream to obtain a first image frame, wherein the first image frame comprises n tiles, each tile represents an image of an area in the first image frame, and n is an integer greater than or equal to 2;

the compression module is used for compressing the first image frame to obtain a second image frame, and the second image frame comprises n compressed slices;

the memory management module is used for allocating frame buffer for the n compressed slices in a memory; the processor is further configured to establish a mapping relationship between virtual pages and physical pages for memory spaces occupied by the n compressed slices in the frame buffer, where the memory space occupied by each compressed slice includes one or more physical pages, and the page numbers of the virtual pages occupied by the n compressed slices are continuous in the mapping relationship; and is further configured to store the n compressed slices in the frame buffer, respectively.

10. The apparatus of claim 9, wherein the memory management module is specifically configured to: allocating the frame buffer area for the n compressed slices in a memory block pool in a memory, wherein the frame buffer area comprises M memory blocks, each memory block comprises a plurality of physical pages, the physical pages correspond to a plurality of virtual pages with continuous page numbers, and the memory space occupied by the n compressed slices is less than or equal to the memory space corresponding to the M memory blocks;

the memory management module is further configured to: and recovering the unoccupied residual memory blocks in the frame buffer area to the memory block pool, wherein the residual memory blocks are used for subsequent frame buffer area allocation and slice storage.

11. The apparatus of claim 10, wherein the memory management module is specifically configured to:

determining a page table corresponding to each memory block in the M memory blocks, where the page table corresponding to each memory block includes a mapping relationship between each virtual page and a physical page in the memory block;

and combining the page tables corresponding to the M memory blocks into a page table of the memory space occupied by the n compressed pieces.

12. The apparatus of claim 9, wherein the memory management module is specifically configured to:

allocating a virtual memory space to the n compressed slices, wherein the virtual memory space comprises a plurality of virtual pages with continuous page numbers; each compressed slice of the n compressed slices corresponds to one or more virtual pages;

respectively carrying out page table inquiry on the page numbers of the one or more virtual pages;

if the page table is inquired to have no mapping relation between the page numbers of the one or more virtual pages and the page numbers of the physical pages, the one or more physical pages are allocated to the one or more virtual pages in the memory.

13. The apparatus of claim 12, wherein prior to said allocating the one or more physical pages to the one or more virtual pages in the memory, the memory management module is further configured to:

acquiring a bitmap bit map, wherein the bitmap comprises a plurality of bits, and each bit is used for indicating whether each physical page in a memory is occupied or not;

and finding the one or more unoccupied physical pages in the memory according to the bitmap.

14. The apparatus according to claim 12 or 13, wherein the memory management module is specifically configured to:

adding a page number of the one or more physical pages to the page table, such that the page table includes a mapping relationship between the one or more virtual pages and the one or more physical pages.

15. The apparatus according to any one of claims 9 to 14, wherein the memory management module is specifically configured to:

storing the compressed slices, compressed information and header address information into the frame buffer area in a form of a data unit; each compressed slice corresponds to one piece of compression information, and the compression information is used for indicating the size of the memory space occupied by the corresponding compressed slice; the header address information is used to indicate a start address of the data unit.

16. The method according to any of claims 9-15, wherein the apparatus further comprises a decompression module;

the memory management module is configured to, after storing the n compressed slices in the frame buffer, read the n compressed slices from the frame buffer;

the decompression module is used for decompressing the n compressed slices to obtain a decoding frame of a first image frame;

and the decoding module is also used for further decoding the code stream by taking the decoded frame as a reference frame to obtain a third image frame.

17. An apparatus for memory management, the apparatus comprising a processor and a memory, wherein:

the processor is configured to decode a code stream to obtain a first image frame, where the first image frame includes n tiles, each tile represents an image of a region in the first image frame, and n is an integer greater than or equal to 2;

compressing the first image frame to obtain a second image frame, wherein the second image frame comprises n compressed slices;

allocating frame buffer in the memory for the n compressed slices; the processor is further configured to establish a mapping relationship between virtual pages and physical pages for memory spaces occupied by the n compressed slices in the frame buffer, where the memory space occupied by each compressed slice includes one or more physical pages, and the page numbers of the virtual pages occupied by the n compressed slices are continuous in the mapping relationship; and is further configured to store the n compressed slices in the frame buffer, respectively.

18. The apparatus of claim 17, wherein the processor is specifically configured to: allocating the frame buffer area for the n compressed slices in a memory block pool in the memory, wherein the frame buffer area comprises M memory blocks, each memory block comprises a plurality of physical pages, the physical pages correspond to a plurality of virtual pages with continuous page numbers, and the memory space occupied by the n compressed slices is less than or equal to the memory space corresponding to the M memory blocks;

the processor is further configured to: and recovering the unoccupied residual memory blocks in the frame buffer area to the memory block pool, wherein the residual memory blocks are used for subsequent frame buffer area allocation and slice storage.

19. The apparatus of claim 18, wherein the processor is specifically configured to:

determining a page table corresponding to each memory block in the M memory blocks, where the page table corresponding to each memory block includes a mapping relationship between each virtual page and a physical page in the memory block;

and combining the page tables corresponding to the M memory blocks into a page table of the memory space occupied by the n compressed pieces.

20. The apparatus of claim 17, wherein the processor is specifically configured to:

allocating a virtual memory space to the n compressed slices, wherein the virtual memory space comprises a plurality of virtual pages with continuous page numbers; each compressed slice of the n compressed slices corresponds to one or more virtual pages;

respectively carrying out page table inquiry on the page numbers of the one or more virtual pages;

if the page table is inquired to have no mapping relation between the page numbers of the one or more virtual pages and the page numbers of the physical pages, the one or more physical pages are allocated to the one or more virtual pages in the memory.

21. The apparatus of claim 20, wherein prior to said allocating the one or more physical pages for the one or more virtual pages in the memory, the processor is further configured to:

acquiring a bitmap bit map, wherein the bitmap comprises a plurality of bits, and each bit is used for indicating whether each physical page in the memory is occupied or not;

and finding the one or more unoccupied physical pages in the memory according to the bitmap.

22. The apparatus according to claim 20 or 21, wherein the processor is specifically configured to:

adding a page number of the one or more physical pages to the page table, such that the page table includes a mapping relationship between the one or more virtual pages and the one or more physical pages.

23. The apparatus according to any of claims 17-22, wherein the processor is specifically configured to:

storing the compressed slices, compressed information and header address information into the frame buffer area in a form of a data unit; each compressed slice corresponds to one piece of compression information, and the compression information is used for indicating the size of the memory space occupied by the corresponding compressed slice; the header address information is used to indicate a start address of the data unit.

24. The method of any of claims 17-23, wherein the processor is further configured to:

after storing the n compressed slices in the frame buffer, respectively, reading the n compressed slices from the frame buffer, respectively;

decompressing the n compressed slices to obtain a decoded frame of a first image frame;

and further decoding the code stream by taking the decoded frame as a reference frame to obtain a third image frame.

25. A non-transitory computer-readable storage medium storing program code, wherein the program code is executable by a computing device to perform the method of any of claims 1-8.

Technical Field

The present invention relates to the field of video processing, and in particular, to a memory management method and related apparatus.

Background

A Double Data Rate SDRAM (DDR SDRAM), also called DDR memory, is an important component of a video processing system, and generally, the price of the DDR memory is equal to or even higher than that of a main chip. Therefore, the use of DDR is reduced on the premise of the same service experience, and the method is an effective way for reducing the cost of the video processing system.

In the media architecture, a frame buffer (also referred to as a frame buffer) of a Video Decoder (VDEC) occupies a major portion of the DDR memory, for example, 6 to 7 frames of image frames (such as reference frames) are stored in the frame buffer. Therefore, in order to reduce the memory consumption in video processing, compression techniques, including lossless compression and lossy compression, are usually introduced during the reference frame access.

The lossy compression utilizes the characteristic that human beings are insensitive to certain frequency components in images or sound waves, and certain information is lost in the data compression process, so the lossy compression can generally bring a larger image compression rate, but the lossy compression can introduce errors of images, and as the decoded reference frame has the reference characteristic, the errors are continuously accumulated and amplified after iteration of the reference frame, and finally, visually-perceivable image damage can be presented, so that the image Quality (PQ) is reduced, so the lossy compression technology is not generally adopted in the reference frame access process of a decoding link.

Lossless compression techniques are more commonly applied to reference frame access, which uses statistical redundancy of data for compression, and can fully recover the original data without causing image distortion at decompression. Therefore, lossless compression techniques are usually employed in the decoding process. Although lossless compression does not cause data damage, the compression rate of lossless compression is not reliable, for example, 4K video under lossless compression has the situation that the compression rate is only 20% or even cannot be compressed. Therefore, the existing lossless compression technology can not effectively save the memory occupation relatively. Therefore, how to effectively reduce the memory occupation of reference frame access during decoding becomes a problem.

Disclosure of Invention

Embodiments of the present invention provide a memory management method and a related apparatus, which can effectively reduce memory occupation caused by reference frame access during decoding.

In a first aspect, an embodiment of the present invention provides a memory management method in video decoding, where the method includes: a video decoder decodes a code stream to obtain a first image frame, where the first image frame includes n tiles (e.g., n same tiles), each tile represents an image of a rectangular region in the first image frame, and n is an integer greater than or equal to 2; compressing (e.g., lossless compression) the first image frame to obtain a second image frame, where the second image frame includes n compressed slices; allocating frame buffers for the n compressed slices in a memory; establishing a mapping relation between virtual pages and physical pages for memory spaces occupied by the n compressed slices in the frame buffer area, wherein the memory space occupied by each compressed slice comprises one or more physical pages, and the page numbers of a plurality of virtual pages occupied by the n compressed slices are continuous under the mapping relation; and respectively storing the n compressed slices into the frame buffer area.

It can be seen that, in the embodiment of the present invention, the video decoder may sequentially and compactly store each compressed (for example, lossless compression) slice into a memory area with continuous virtual addresses in the frame buffer, so as to realize aggregation of occupied memory space, where the page numbers of the physical pages occupied by the compressed slices may be discrete, but the page numbers of the virtual pages occupied by the compressed slices are continuous. Therefore, the embodiment of the invention can effectively reduce the occupation of the reference frame access to the memory during decoding.

Based on the first aspect, in a possible embodiment of the present invention, in order to fully utilize memory resources, the following two ways are designed to implement dynamic allocation of frame memory and implement fine-grained memory allocation or memory reclamation, where the two ways are respectively a memory reclamation mode and a pre-allocation mode (or an on-demand allocation mode), and the two ways are described below.

(1) In the memory allocation implementation mode adopting the memory reclamation mode, when physical memory allocation is performed, a small block of memory (for example, 1MB or 512KB) is allocated each time, so that the small block of memory is called a memory block, and a memory block pool of a decoding channel can be formed after multiple allocations. The size of the memory block pool may correspond to a minimum frame memory cost, for example, 15MB of memory space needs to be allocated to each frame buffer, and about 90MB is needed for a decoding channel of 4K video for 6 frame image frames (i.e., 15MB × 6 ═ 90 MB). In the decoding process, the frame buffer may be allocated to the n compressed slices in a memory block pool in a memory, where the frame buffer includes M memory blocks (for example, each memory block is 1MB, and then a 15MB frame buffer needs 15 memory blocks), each memory block includes multiple physical pages, and the multiple physical pages correspond to multiple virtual pages with continuous page numbers, but the physical pages corresponding to the virtual pages may be discrete (that is, the page numbers of the physical pages may be discontinuous). The memory space occupied by the n compressed pieces is less than or equal to the memory space corresponding to the M memory blocks. When n compressed pieces need to be actually stored in the memory, a mapping relation between a virtual address and a physical address is established, and then the n compressed pieces are compactly stored in the memory block of the frame buffer area. And finally, removing the unoccupied residual memory blocks in the frame buffer area according to the actual memory occupation amount, and recovering the unoccupied residual memory blocks to the memory block pool, wherein the residual memory blocks can be used for subsequent frame buffer area allocation and piece storage, so that the dynamic utilization of the memory space is realized, and the use efficiency of the memory is greatly improved.

The process of establishing the mapping relationship between the virtual page and the physical page in the memory space occupied by the n compressed slices in the frame buffer area in the embodiment of the present invention is as follows: and searching unoccupied memory blocks in the memory block pool, wherein the memory blocks respectively correspond to a page table, and the page table records the mapping relation between each virtual page and each physical page in the corresponding memory block. Then, when sufficient memory blocks (M memory blocks) are combined into a storage space of the size indicated by the container, for example, the virtual memory space is 15MB, several memory blocks of which the total amount is 15MB are selected to be combined into the physical memory space required by the container, for example, 15 memory blocks of 1MB are combined. Specifically, the page tables corresponding to the memory blocks with the total amount of 15MB may be combined into the page table of the frame buffer, so that the mapping relationship between the virtual address and the physical address in the frame buffer is actually established, thereby ensuring that the frame buffer has a real 15M physical memory

It can be seen that in the embodiment of the present invention, the compact memory arrangement after lossless compression of the image frame can be stored in the allocated frame buffer, and the fine allocation of the physical memory space is performed, so as to realize the collection and utilization of the fragmented memory, and after the video decoder finishes decoding one frame of image frame and stores the image frame in the frame buffer, the memory which is not used up in the frame buffer can be recovered, and the unoccupied free memory is released and merged into the decoding use of the next frame. As the lossless compression does not lose the image quality, the method can use less memory budget to actually obtain more frame buffer areas, thereby ensuring that each performance index is not damaged and achieving the purposes of saving bandwidth and frame memory on the premise of not losing the image quality.

(2) In a memory allocation implementation that employs a pre-allocation scheme, the frame buffer that may be allocated for an image frame prior to decoding includes only virtual memory space, which includes a plurality of virtual pages with consecutive page numbers, rather than physical memory space. That is, the page table established when the frame buffer is initially allocated only indicates one or more virtual pages, and does not indicate the mapping relationship between the virtual pages and the physical pages, unlike the former implementation, the on-demand allocation scheme does not involve memory reclamation but allocates virtual memory space in advance before allocating actual physical memory. For example, when 15 frame buffers are allocated for decoding, each frame buffer occupies 15MB of space, and a space of 15MB × 15 is allocated in the virtual memory space in a sufficient amount. However, for each of the 15 frame buffers, each virtual page corresponding to the frame buffer may not have a physical page mapped to it. When the data unit of the image frame needs to be stored in the memory, respectively performing page table query on the page numbers of the one or more virtual pages; if the page table is inquired to have no mapping relation between the page numbers of one or more virtual pages and the page numbers of the physical pages, the one or more physical pages are allocated to the one or more virtual pages in the memory, so that the actual physical pages are allocated according to the actual requirements of the data unit.

When one or more physical pages which are not occupied need to be searched in the memory, a bitmap (bit map) can be inquired, wherein the bitmap comprises a plurality of bits, each bit is used for indicating whether one physical page in the memory is occupied or not, and then the one or more physical pages which are not occupied are searched according to the bitmap. For example, each bit in the bitmap indicates an occupied status of one physical page, where "0" represents that the corresponding physical page is unoccupied, and "1" represents that the corresponding physical page is occupied. In the process of storing the compressed data unit into the frame buffer, when 4 physical pages which are not occupied need to be found, 4 physical pages indicated by 4 bits with the value of "0" can be found according to the bitmap.

In the embodiment of the invention, the process of establishing the mapping relation between the virtual page and the physical page of the frame buffer area comprises the following steps: when the compressed data unit needs to be stored in a frame buffer area, page table query is carried out according to the page numbers of one or more virtual pages corresponding to the data unit, if the page numbers of the physical pages corresponding to the page numbers of the one or more virtual pages cannot be queried, one or more physical pages which are not occupied are searched in a memory, and the page numbers of the one or more physical pages are added to a page table, so that each of the page numbers of the one or more virtual pages in the page table corresponds to the page number of one physical page respectively. Thus, the mapping relation between the virtual page and the physical page of the frame buffer area is established.

It can be seen that, the embodiment of the present invention realizes "occupation of a memory as needed" by improving an establishment mechanism of a page table, and after lossless compression, image frames are stored in a physical memory space allocated as needed through compact memory arrangement. In addition, when the page table is queried and the corresponding relation between the physical page and the virtual page is found to be lacked, the physical page can be automatically supplemented, so that the processing process of the lacked page is transparent to the media frame, the technical scheme of the embodiment of the invention can be suitable for the media frames in any forms, and the application and the user experience can be improved.

Based on the first aspect, in a possible implementation, the plurality of compressed slices, the plurality of compression information, and the header address information may be stored in the frame buffer in a form of one data unit during the storing. For example, an image frame forms data units in CTU units for data access, the image frame is divided into n pieces with the same data length, each data unit includes, for example, 3 pieces, each data unit further includes header address information indicating a start address of each data unit and compression information corresponding to each piece, the compression information indicating a compression mode/compressed length of each piece. As shown in fig. 7, a data unit 1 in an image frame may include header address information 1, compression information 1 of slice 1, compression information 2 of slice 2, compression information 3 of slice 3, and slice 1, slice 2, and slice 3. In the image compression process, the physical memory space is dynamically distributed according to the data volume required to be stored in the frame buffer area, and each compressed data unit is sequentially and compactly stored in the frame buffer area, so that the collection of occupied memory space is realized, and the mapping relation between each compressed data unit and a storage address is established. Thus, random access of each data unit can be realized by matching the original length of each slice. After the compressed image is completely stored in the frame buffer area, the unoccupied memory space in the frame buffer area can be released

Based on the first aspect, in a possible implementation manner, after the n compressed slices are respectively stored in a frame buffer of a memory, during a process of decoding a subsequent image, the method further includes: reading the n compressed slices from the frame buffer respectively; decompressing the n compressed slices to obtain a decoded frame of a first image frame; and further decoding the code stream by taking the decoded frame as a reference frame to obtain a third image frame. It should be noted that, if the compression process is lossless compression, the decoded frame is the first image frame; if the compression process is other compression methods (such as lossy compression) and the compression method causes partial image loss, the decoded frame is the first image frame with partial information loss.

In a second aspect, an embodiment of the present invention provides an apparatus for memory management, where the apparatus includes a decoding module, a memory management module, a compression module, and a decompression module, where: the decoding module is used for decoding the code stream to obtain a first image frame, the first image frame comprises n pieces, each piece represents an image of one area in the first image frame, and n is an integer greater than or equal to 2; the compression module is used for compressing the first image frame to obtain a second image frame, and the second image frame comprises n compressed slices; the memory management module is used for allocating frame buffer areas for the n compressed slices in the memory; the processor is further configured to establish a mapping relationship between virtual pages and physical pages for memory spaces occupied by the n compressed slices in the frame buffer, where the memory space occupied by each compressed slice includes one or more physical pages, and the page numbers of the virtual pages occupied by the n compressed slices are continuous in the mapping relationship; and is further configured to store the n compressed slices in the frame buffer, respectively.

It should be noted that the functional modules are specifically configured to implement the method described in the first aspect, and the implementation process of the related embodiment may also refer to the description of the first aspect, which is not described herein again.

In a third aspect, an embodiment of the present invention provides another apparatus for memory management, where the apparatus includes a processor and a memory, where: the processor is configured to decode a code stream to obtain a first image frame, where the first image frame includes n tiles, each tile represents an image of a region in the first image frame, and n is an integer greater than or equal to 2; compressing the first image frame to obtain a second image frame, wherein the second image frame comprises n compressed slices; allocating a frame buffer in the memory for the n compressed slices; the processor is further configured to establish a mapping relationship between virtual pages and physical pages for memory spaces occupied by the n compressed slices in the frame buffer, where the memory space occupied by each compressed slice includes one or more physical pages, and the page numbers of the virtual pages occupied by the n compressed slices are continuous in the mapping relationship; and is further configured to store the n compressed slices in the frame buffer, respectively.

It should be noted that the processor and the memory are specifically configured to implement the method described in the first aspect, and the implementation process of the related embodiment may also refer to the description of the first aspect, which is not described herein again.

In a fourth aspect, embodiments of the invention provide yet another non-transitory computer-readable storage medium; the computer readable storage medium is used for storing the implementation code of the method of the second aspect. The program code, when executed by a computing device, is for use by the user device in the method of the first aspect.

In a fifth aspect, an embodiment of the present invention provides a computer program product; the computer program product comprising program instructions which, when executed by a computing device, cause the controller to perform the method of the first aspect as set forth above. The computer program product may be a software installation package, which, in case it is required to use the method provided by any of the possible designs of the first aspect described above, may be downloaded and executed on a controller to implement the method of the first aspect.

It can be seen that the embodiment of the present invention can implement that the image frame is stored into the memory area with continuous virtual addresses in the frame buffer after being compressed (such as lossless compression), and the collection of the occupied memory area is implemented, so that the embodiment of the present invention can achieve the purpose of effectively reducing the memory occupied by the reference frame access during decoding. In addition, the memory recovery scheme provided by the embodiment of the invention can also recover the memory which is not used up in the frame buffer area, and the memory is combined with the decoding of the next frame for use, so that the effective utilization of the fragment memory is realized; the on-demand allocation scheme provided by the embodiment of the invention can improve the establishment mechanism of the page table, realizes the memory on-demand occupation and greatly improves the use efficiency of the memory.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments or the background art of the present invention, the drawings required to be used in the embodiments or the background art of the present invention will be described below.

Fig. 1 is a schematic block diagram of a video encoding and decoding system according to an embodiment of the present invention;

fig. 2 is a schematic block diagram of a video encoding and decoding device according to an embodiment of the present invention;

fig. 3 is a schematic block diagram of a destination device in a video codec system according to an embodiment of the present invention;

FIG. 4 is a diagram illustrating an image frame divided into a plurality of slices according to an embodiment of the present invention;

FIG. 5 is a diagram illustrating a random access implementation provided by the prior art;

fig. 6 is a schematic diagram of an implementation of random access according to an embodiment of the present invention;

FIG. 7 is a diagram illustrating a data format of a data unit according to an embodiment of the present invention;

fig. 8 is a schematic view of an application scenario of memory allocation according to an embodiment of the present invention;

fig. 9 is a schematic diagram of an application scenario for establishing a mapping relationship between a virtual page and a physical page according to an embodiment of the present invention;

fig. 10 is a schematic view of an application scenario of memory allocation according to another embodiment of the present invention;

fig. 11 is a schematic view of another application scenario for performing page fault processing by SMMU according to an embodiment of the present invention;

fig. 12 is a flowchart illustrating a memory management method according to an embodiment of the present invention;

fig. 13 is a flowchart illustrating another memory management method according to an embodiment of the present invention;

fig. 14 is a flowchart illustrating another memory management method according to an embodiment of the present invention.

Detailed Description

The embodiments of the present invention will be described below with reference to the drawings. The terminology used in the description of the embodiments of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

The following first describes a system architecture according to an embodiment of the present invention. Referring to fig. 1, fig. 1 is a schematic block diagram of a video codec system according to an embodiment of the present invention. As shown in fig. 1, the video codec system includes a source device 10a and a destination device 10 b. Source device 10a may generate encoded video data and send the codestream to destination device 10b, and thus, source device 10a may be referred to as a video encoding device or video encoding apparatus. Destination device 10b may decode the encoded video data generated by source device 10a and perform video display, and thus destination device 10b may be referred to as a video decoding device or video decoding apparatus. Source device 10a and destination device 10b may be examples of video codec devices or video codec apparatuses. Source device 10a and destination device 10b may comprise a wide range of devices, including desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, handsets such as smartphones, televisions, cameras, display devices, digital media players, video game consoles, in-vehicle computers, or the like.

In one example, source device 10a may access the storage medium via disk access or card access. The storage medium may comprise a variety of locally accessible data storage media such as DDR memory, Blu-ray disc, DVD, CD-ROM, flash memory, and the like.

Destination device 10b may receive the encoded video data from source device 10a via channel 17. Channel 17 may comprise one or more media and/or devices capable of moving encoded video data from source device 10a to destination device 10 b. In one example, channel 17 may include one or more communication media that enable source device 10a to transmit encoded video data directly to destination device 10b in real-time. In this example, source device 10a may modulate the encoded video data according to a communication standard (e.g., a wireless communication protocol), and may transmit the modulated video data to destination device 10 b. The one or more communication media may include wireless and/or wired communication media such as a Radio Frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (e.g., the internet). The one or more communication media may include routers, switches, base stations, or other apparatuses that facilitate communication from source device 10a to destination device 10 b.

In another example, channel 17 may comprise a storage medium that stores encoded video data generated by source device 10 a. In this example, destination device 10b may access the storage medium via disk access or card access. The storage medium may comprise a variety of locally accessible data storage media such as DDR memory, Blu-ray disc, DVD, CD-ROM, flash memory, and the like.

In another example, channel 17 may comprise a file server or another intermediate storage device that stores encoded video data generated by source device 10 a. In this example, destination device 10b may access encoded video data stored at a file server or other intermediate storage device via streaming or download. The file server may be of a type capable of storing encoded video data and transmitting the encoded video data to destination device 10 b. Example file servers include web servers (e.g., for a website), File Transfer Protocol (FTP) servers, Network Attached Storage (NAS) devices, and local disk drives. Destination device 10b may access the encoded video data via a standard data connection, such as an internet connection. Example types of data connections include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on a file server. The transmission of the encoded video data from the file server may be a streaming transmission, a download transmission, or a combination of both.

The technology of the embodiment of the invention is not limited to a wireless application scene, and can be applied to video coding and decoding supporting various multimedia applications such as the following applications by way of example: over-the-air television broadcasts, cable television transmissions, satellite television transmissions, streaming video transmissions (e.g., via the internet), encoding of video data stored on a data storage medium, decoding of video data stored on a data storage medium, or other applications. In some examples, a video codec system may be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and/or video telephony.

In the example of fig. 1, source device 10a includes a video data source 11, a video encoder 12, and a DDR memory 13. Furthermore, in some examples, source device 10a also includes an output interface (not shown in fig. 1), which may include a modulator/demodulator (modem) and/or a transmitter, for sending the encoded codestream to destination device 10b over channel 17.

Among other things, video data source 11 may include a video capture device (e.g., a video camera), a video archive containing previously captured video data, a video input interface to receive video data from a video content provider, and/or a computer graphics system for generating video data, or a combination of the aforementioned video data sources.

Video encoder 12 may encode video data from video data source 11, the video data comprising a plurality of frames of an image frame. In some examples, video encoder 12 may compress the encoded video data (e.g., reference frames) and store the compressed video data in DDR memory 13, and accordingly, video encoder 12 may decompress the reference frames in DDR memory 13 and use the reference frames to continue encoding other image frames.

In the example of fig. 1, destination device 10b includes video decoder 14, video display 15, and DDR memory 16. In some examples, destination device 10b may also include an input interface (not shown), which may include a receiver and/or a modem. The input interface may receive a codestream via a channel 17, which codestream comprises encoded video data.

The video decoder 14 may decode the encoded video data in the code stream to obtain decoded video data. In some examples, the video decoder 14 may compress the decoded video data and store the compressed video data in the DDR memory 16, and correspondingly, the video decoder 14 may also decompress the video data in the DDR memory 16, so that the decompressed image frame is used as a reference frame for the video decoder 14 to continue decoding subsequent code streams. In addition, when video display is required, the video data in the DDR memory 16 may be decompressed, and the decompressed video data may be input to the video display 15.

The video display 15 may be integral with the destination device 10b, or the video display 15 may be separately deployed external to the destination device 10 b. In general, the video display 15 may perform video display based on input video data. The video display 15 may include a variety of display devices, such as a Liquid Crystal Display (LCD), a plasma display, an Organic Light Emitting Diode (OLED) display, or other types of display devices.

The memory management method described in the embodiment of the present invention is mainly applied to the data compression/decompression process in the corresponding codec of the source device 10a and/or the destination device 10 b.

Referring to fig. 2, fig. 2 is a schematic block diagram of a video codec device 50 according to an embodiment of the present invention, where the video codec device 50 may be used to implement the functions of the source device 10a and/or the destination device 10 b. The apparatus 50 may be, for example, a mobile terminal of a wireless communication system or a user equipment of a wired communication system. It should be understood that embodiments of the present invention may be implemented within any electronic device or apparatus that may require encoding and decoding, or encoding or decoding, of video images.

As shown in fig. 2, apparatus 50 may include a processor 56 (or controller) to control apparatus 50, a memory 58, and a codec 54 corresponding to video encoder 12 of source device 10a or video decoder 14 of destination device 10b in fig. 1. The processor 56 may be configured to run an Operating System (OS), and the processor 56 may be connected to a storage 58, where the storage 58 is a storage medium such as a DDR memory (or DDR memory bank), a blu-ray disc, a DVD, a CD-ROM, a flash memory, etc., and in an embodiment of the present invention, the storage 58 may store data of a compressed image (e.g., an image frame after lossless compression), and/or may also store instructions for implementation on the processor 56, and the storage 58 corresponds to the DDR memory 13 of the source device 10a or the DDR memory 16 of the destination device 10b in fig. 1. The processor 56 may be connected to a codec 54 adapted to enable encoding and decoding of video data or auxiliary encoding and decoding controlled by the processor 56.

The device 50 may also include a display 32 in the form of a liquid crystal display, the display 32 may be any suitable display technology suitable for displaying images or video, corresponding to the video display 15 of the destination device 10 b. The apparatus 50 may also include a keypad 34. In other embodiments of the invention, any suitable data or user interface mechanism may be employed. For example, the user interface may be implemented as a virtual keyboard or a data entry system as part of a touch sensitive display. The device may include a microphone 36 or any suitable audio input, which may be a digital or analog signal input. The apparatus 50 may also include an audio output device which, in embodiments of the invention, may be any one of: headphones 38, speakers, or analog audio or digital audio output connections. The apparatus may also include an infrared port 42 for short-range line-of-sight communication with other devices. In other embodiments, the device 50 may also include any suitable short-range communication solution, such as a Bluetooth wireless connection or a USB/firewire wired connection.

Optionally, the apparatus 50 may further comprise a card reader 48 and a smart card 46, such as a UICC and UICC reader, for providing user information and adapted to provide authentication information for authenticating and authorizing a user at a network.

The apparatus 50 may further comprise a radio interface 52, the radio interface 52 being connected to the processor 56 and adapted to generate wireless communication signals, for example for communicating with a cellular communication network, a wireless communication system or a wireless local area network. The apparatus 50 may also include an antenna 44 connected to the radio interface 52 for transmitting radio frequency signals generated at the radio interface 52 to other apparatus(s) and for receiving radio frequency signals from other apparatus(s). In some embodiments of the present invention, the apparatus 50 may receive video data for encoding/decoding via a wireless or wired connection.

The memory management method described in the embodiment of the present invention is mainly applied to the data compression/decompression process in the corresponding codec of the device 50. The memory management scheme provided by the embodiment of the present invention is mainly described in detail below from the perspective of the decoding side, and it should be understood that the memory management scheme provided by the embodiment of the present invention may also be applied to the encoding side, that is, the memory management scheme on the encoding side may be implemented similarly with reference to the implementation scheme on the decoding side, because the decoding scheme may also be involved on the encoding side, and therefore, the scheme applicable to the decoding side mentioned in this embodiment may also be used, and will not be described in detail hereinafter.

Referring to fig. 3, fig. 3 shows a specific logical structure of a destination device in a video codec system, the destination device includes a video decoder 20, a video display 30, and a memory (including a frame buffer 22, which may also be referred to as a frame memory), such as a DDR memory. The video decoder 20 may be implemented by hardware as the codec 54 shown in the embodiment of fig. 2, or include the codec 54, or a part thereof, and the video decoder 20 includes the decoder 21 and the memory management module 23. In addition, the video decoder 20 may further include a compression module 24 and a decompression module 25, and in one possible embodiment, the compression module 24 and the decompression module 25 may be implemented as functional modules of the decoder; in yet another possible embodiment, the compression module 24 and the decompression module 25 may also be implemented independently of the decoder.

The frame buffer (frame buffer)22 is used for storing video data, and specifically, may store image frames compressed by the compression module 24 in a lossless compression manner. In one embodiment, the memory of the destination device may be implemented using DDR memory and the frame buffer 22 may be located in DDR memory, i.e., the frame buffer 22 indicates the allocated memory area for image storage in the memory. In one embodiment, the frame buffer 22 may be a direct image of the displayed image of the video display, where each storage unit of the frame buffer corresponds to a pixel on the screen, and the entire frame buffer corresponds to a frame of the image. The number of frame buffers 22 may be multiple, depending on the application scenario, and each frame buffer 22 may be used to dynamically store one reference frame for reference in subsequent decoding or image frames for video display.

It should be noted that the compression module 24 and the decompression module 25 of the present embodiment can use a lossless compression technique and a lossy compression technique, and achieve a better technical effect when using the lossless compression technique. The present embodiment is only described by taking the lossless compression technique as an example, but is not limited to the application of the embodiment scheme in the context of the lossy compression technique.

The memory management module 23 can allocate memory space (virtual memory space/physical memory space) to the frame buffer 22 through dynamic memory allocation, so as to realize effective collection and utilization of memory space or recycle of free memory space, thereby saving memory consumption.