阅读说明：本技术 一种自恢复的视频解码方法 (Self-recovery video decoding method ) 是由 高娟 于 2021-08-06 设计创作，主要内容包括：本发明一种自恢复的视频解码方法,首先,移植ffmpeg到海思平台,对ffmpeg进行源码修改使其适配海思芯片。其次,启动ffmpeg的动态库对网络传输的数据进行参数帧和图像帧的组合,获取完整图像信息,然后,建立过滤机制,动态筛选错误帧,分析编码规律,调整错误数据包内数据帧顺序,将可解码的数据包再传送给海思芯片的硬解码模块。最后,创建获取图像线程,从解码通道获取已解码的图像数据,启动自恢复脚本监控解码程序防止异常退出,从而完成解码任务。(The invention relates to a self-recovery video decoding method, which comprises the steps of firstly, transplanting ffmpeg to a Haisi platform, and modifying source codes of the ffmpeg to enable the ffmpeg to be adapted to a Haisi chip. Secondly, starting a dynamic library of the ffmpeg to combine parameter frames and image frames of data transmitted by the network to obtain complete image information, then establishing a filtering mechanism, dynamically screening error frames, analyzing a coding rule, adjusting the sequence of data frames in an error data packet, and transmitting the decodable data packet to a hard decoding module of the Haesi chip. And finally, establishing an image acquisition thread, acquiring decoded image data from the decoding channel, and starting a self-recovery script monitoring decoding program to prevent abnormal exit, thereby completing a decoding task.)

1. A method for self-healing video decoding, comprising the steps of:

1) the compiling attribute and parameter of the ffmpeg are configured, and the ffmpeg dynamic library is transplanted to the Haisi platform;

2) creating a network receiving task to receive original compressed data transmitted by a host;

3) creating a data packet acquiring thread, and acquiring an image data packet with a parameter-containing frame by using an ffmpeg dynamic library;

4) establishing a filtering mechanism, screening error frames, combining the error frames into a decodable data packet and sending the decodable data packet to a hard decoding module of the chip;

5) acquiring a decoding image in a decoding module;

6) and starting a self-recovery script monitoring decoding program.

2. A method for self-healing video decoding as recited in claim 1, wherein: step 6) the method for starting the self-recovery script monitoring decoding program specifically comprises the following steps:

61) setting a self-recovery script S09Decode under the system/etc/rcS/path;

62) acquiring a process number of decode _ video operation;

63) acquiring the running state of the decode _ video process by using a ps command;

64) if the running state of the decode _ video is the running state stillRunning, delaying for 5ms, and returning to step 62); executing the next step if decode _ video is in the exit state;

65) adding an exit decoding channel in an initialization function of decode, releasing a video cache block and releasing system resources;

66) entering a directory where the decode _ video is located by using a cd command, and executing a decode program;

67) return to step 62).

3. A method for self-healing video decoding as recited in claim 1, wherein: step 1) the ffmpeg dynamic library is stored in a Haisi platform, which specifically comprises the following steps:

11) firstly, configuring ffmpeg compiling attribute, and configuring parameters according to a platform type, a cpu type, a codec attribute, a format conversion attribute and a cross compiling attribute;

12) modify acodec.h file, add variable: the parameter frame length SEI _ len and the array SEI _ BUF are determined, and meanwhile, the parameter frame SIZE SEI _ BUF _ SIZE is determined according to the application layer requirements;

13) adding a parameter frame acquiring function in a decode _ nal _ sei _ prefix function in a hevc _ sei.c file: obtaining a parameter SIZE in a function, assigning the SIZE to SEI _ len, judging whether the SIZE is smaller than or equal to SEI _ BUF _ SIZE, copying data in a context parameter array gb to SEI _ BUF if a condition is met, wherein the copy length is the SIZE, and an array subscript i of the copied SEI _ BUF is an index of the gb array divided by 8, namely SEI _ BUF [ i ] ═ gb _ BUF [ index/8 ];

14) then, executing a configure command, and generating decoding libraries libavcodec, libavformat, libavutil and libswscale under the subfolder lib of the configuration folder;

15) and finally, copying the dynamic library to a/usr/lib path of a decoding board card.

4. A method for self-healing video decoding as recited in claim 1, wherein: step 2) the original compressed image transmitted by the receiving host specifically comprises:

21) firstly, acquiring a receiving ip and a port in a configuration file;

22) then, using the received ip address and the port number as parameters to create and bind the network socket, and after the creation is successful, performing step 23);

23) secondly, clearing the receiving buffer zone, and entering step 24) after waiting for receiving the image data sent by the network;

24) judging whether the length of the image data received this time is greater than zero, if so, carrying out the next step, otherwise, returning to the step 23);

25) judging whether the protocol frame header meets the protocol requirement, if not, discarding the frame; and if the image frame is matched with the image frame, storing the image frame in a data receiving area.

5. A method for self-healing video decoding as recited in claim 1, wherein: step 3) the creating and acquiring data packet thread specifically comprises the following steps:

321) acquiring an image data array first address pointer bufPtr and a length bufLen read from a network;

322) judging whether the current data length bufLen is larger than 0, if so, continuing the next step, and if not, exiting the decoding process of the data and waiting for the next data transmission of the network;

323) transmitting a data array first address pointer bufPtr and a length bufLen to a soft decoding module, segmenting data by frames by using a library function av _ parser _ parse2, recording and continuing the next step if a complete image frame data packet can be successfully obtained from the array, and quitting the decoding process if the complete image frame data packet cannot be obtained from the array, and waiting for the next image data array transmitted by the network;

324) storing the length ret of the image frame divided in the data array at the time, removing the length ret of the image frame divided at the time from the total length bufLen of the data array, and moving a first address pointer bufPtr forwards, wherein the moving times are equal to the length ret of the image frame data packet;

325) and putting the image complete data packet segmented this time into a queue to be decoded.

6. A method for self-healing video decoding as recited in claim 1, wherein: step 4) the method for establishing the filtering mechanism and screening the error frame specifically comprises the following steps:

41) judging the image type, analyzing the number of P frames contained in the image data packets of the slice number required by different types, and recording the data structure variable PSlicENumber of the channel;

42) judging whether the size of the current data packet is larger than the maximum number specified by the protocol frame, if so, exiting the filtering process of the data packet, returning to the step 41 after receiving a new image data packet, and otherwise, carrying out the next step;

43) circularly traversing the data in the data packet, sequentially judging whether the current frame accords with the h265 protocol frame header, if so, performing the next step, and if not, jumping to the step 49);

44) judging whether the current frame is a P frame or an I frame, if so, increasing the number temp _ PSlicENumber of the slots in the data packet transmitted by the channel at this time, and entering the next step; if not, jump to step 46);

45) if the temp _ PSlicENumber of the current data packet is equal to 2, judging whether p _ slice is 0 again, if so, setting p _ size to i, and updating and recording a slice flag p _ slice to 1;

46) judging whether the current frame type is SPS, if so, setting the mark top to be 1, otherwise, directly entering 48);

47) continuously judging whether the current frame is a parameter frame, if so, copying the position i in the current data packet array ptr to a data structure Idr _ buf, storing frame header information before the sei frame, recording the data Size Idr _ Size of the currently stored frame header as i, setting the first-appearing sps identifier entryFlag as 1, and resetting top to be 0; step 48) after completion;

48) judging whether the image data of the current channel needs to be inserted into an I frame, and the entryFlag is 1, if the two conditions are satisfied, judging whether the current frame type is a P frame, if the current frame type is the P frame, changing the byte content of the frame type of the data packet from 0x02 to 0x26, and changing the P frame to the I frame; otherwise, directly entering the step 49);

49) continuing to check whether the ptr in the data packet has other data to be traversed for query, if so, returning to the step 43), and if not, continuing to the next step;

410) judging whether the insertIDR mark and the entryFlag of the current channel are both 1, if so, continuing the next step, and if not, performing the step 413);

411) judging whether the current temp _ PSlicENumber is equal to the PSlicENumber which the image type of the channel should have, if so, then the next step, if not, discarding the data packet, and exiting the decoding process of the data packet; returning to step 41) after receiving a new image data packet;

412) copying the data of which the Size of the complete image frame in the current data packet is Size into Idr _ Buf, starting from the Idr _ Size of the Idr _ Buf at the copying position, increasing the Size of the Size by the Idr _ Size, copying the data of the Idr _ Buf into ptr again, wherein the copied data is Size;

413) judging whether the current temp _ PSlicENumber is equal to the PSlicENumber which the channel image type should have, if yes, carrying out the next step, and if not, carrying out the step 415);

414) judging whether p _ slice is 1, if so, assigning p _ size to size, and clearing p _ slice; otherwise, go directly to step 415);

415) recording the size of the image parameter frame of the current channel in seiSize, copying the parameter frame of the current image to a decodeArray of an image data structure to be decoded of the current channel, and taking the parameter number seiCount as a time stamp u64PTS of the image to be decoded;

416) inquiring the decoding state of the current channel, and saving the frame number of the decoded image in the state structure body to u32 DecodeSetreamFrames;

417) acquiring a first address ptr and a data length size of image data to be decoded of a current channel and a time stamp u64PTS, and sending the data to a hard decoding module in a streaming form;

418) circularly inquiring the working state of the decoder until a new decoded image is obtained;

419) calling a Haesi library function to inquire the channel state of the decoder;

420) judging whether the number of bytes needing to be decoded in the channel is 0, if so, entering the next step, and if not, delaying 1ms and jumping to a step 419);

421) whether the number of decoded picture frames of the decoder channel is not equal to the previously saved u32 decoderstreamframes or the insertIDR of the channel is marked as 1, if so, going to the next step, otherwise, going to step 423);

422) clearing the insrtIDR identifier, increasing the number of the parameter frames, judging whether seiCount is greater than the maximum SEI _ NUM specified by the protocol, clearing the parameter frame number identifier seiCount, and entering step 426); otherwise, directly entering the next step;

423) resetting the decoder, proceeding to step 424);

424) judging whether the decoder reset flag resetFlag is 0, if so, entering the next step, if not, continuing to wait for 1ms, and jumping to the step 423);

425) setting a reset state to 1, calling a library function to stop a video stream receiving task of a decoding channel, destroying the decoding channel, restarting the decoding channel, setting an insertIDR (idle IDR) inserted into an I frame to 1, setting the reset state to 0, and entering the next step;

426) and finishing the filtering process.

7. A method for self-healing video decoding as recited in claim 1, wherein: step 5) the method for obtaining the decoded image in the decoding module specifically comprises the following steps:

51) creating a decoding image obtaining thread;

52) entering an image acquisition cycle;

53) inquiring whether the reset state in the channel is 1, if so, delaying for 1ms, continuing to inquire the reset state, and if not, carrying out the next step;

54) calling a library function HI _ MPI _ VDEC _ GetFrame to acquire a memory address of image storage; if the failure is failed, jumping to the step 53), if the failure is successful, entering the next step;

55) setting a reset flag resetFlag to 1;

56) traversing parameter frames in the decodeArray structure, comparing whether the stored timestamp u64PTS is consistent with the u64PTS of the current decoding image or not, and if so, sending the image data;

57) creating a network sending task, and sending the data stored in the memory address in the step 54) to the host.

8. A processing apparatus, comprising:

a memory for storing a computer program;

a processor for calling and running the computer program from the memory to perform the method of any of claims 1 to 7.

9. A computer-readable storage medium, having stored thereon a computer program or instructions, which, when executed, implement the method of any one of claims 1 to 7.

10. A computer program product, characterized in that it comprises instructions which, when run on a computer, cause the computer to carry out the method of any one of claims 1 to 7.

Technical Field

The invention belongs to a video decoding technology under a linux system, and particularly relates to a self-recovery video decoding method.

Background

Hi3559AV100 is a professional 8K Ultra HD Mobile Camera SOC, provides digital video recording of 8K30/4K120 broadcast-level image quality, supports multi-channel Sensor input, supports H.265 coding output or video-level RAW data output, integrates high-performance ISP processing, and simultaneously adopts an advanced low-power-consumption process and a low-power-consumption architecture design, and provides excellent image processing capability for users. Hi3559AV100 supports the leading multi-channel 4K Sensor input in the industry, multi-channel ISP image processing, the high dynamic range technical standard of HDR10 and multi-channel panoramic hardware splicing. In support of 8K30/4K120 video recording, Hi3559AV100 provides hardened 6-Dof digital anti-shake, reducing reliance on mechanical holders.

However, the Hi3559AV100 belongs to the hard decoding category, and when the protocol frame does not completely conform to the decoding protocol or the number of error frames is large, the decoding efficiency is low or decoding is impossible, and the hard decoding module cannot extract the parameter information for the image information with parameter frames. In addition, there are cases where the hard decoding module may be abnormally exited due to a memory leak or a device error.

Disclosure of Invention

The technical problem solved by the invention is as follows: the method is designed and realized by aiming at the characteristics of Haisi Hi3559AV100 chips and combining the properties of a hard decoding module and the characteristics of soft decoding under the application scene that a system is linux and based on a self-recovery mechanism.

The technical solution of the invention is as follows:

in a first aspect, a method for decoding a self-healing video, comprises the steps of:

1) the compiling attribute and parameter of the ffmpeg are configured, and the ffmpeg dynamic library is transplanted to the Haisi platform;

2) creating a network receiving task to receive original compressed data transmitted by a host;

3) creating a data packet acquiring thread, and acquiring an image data packet with a parameter-containing frame by using an ffmpeg dynamic library;

4) establishing a filtering mechanism, screening error frames, combining the error frames into a decodable data packet and sending the decodable data packet to a hard decoding module of the chip;

5) acquiring a decoding image in a decoding module;

6) and starting a self-recovery script monitoring decoding program.

Optionally, the method for starting the self-recovery script monitoring decoding program in step 6) specifically includes:

61) setting a self-recovery script S09Decode under the system/etc/rcS/path;

62) acquiring a process number of decode _ video operation;

63) acquiring the running state of the decode _ video process by using a ps command;

64) if the running state of the decode _ video is the running state stillRunning, delaying for 5ms, and returning to step 62); executing the next step if decode _ video is in the exit state;

65) adding an exit decoding channel in an initialization function of decode, releasing a video cache block and releasing system resources;

66) entering a directory where the decode _ video is located by using a cd command, and executing a decode program;

67) return to step 62).

Optionally, the step 1) of dynamically storing ffmpeg into the haisi platform specifically includes:

11) firstly, configuring ffmpeg compiling attribute, and configuring parameters according to a platform type, a cpu type, a codec attribute, a format conversion attribute and a cross compiling attribute;

12) modify acodec.h file, add variable: the parameter frame length SEI _ len and the array SEI _ BUF are determined, and meanwhile, the parameter frame SIZE SEI _ BUF _ SIZE is determined according to the application layer requirements;

13) adding a parameter frame acquiring function in a decode _ nal _ sei _ prefix function in a hevc _ sei.c file: obtaining a parameter SIZE in a function, assigning the SIZE to SEI _ len, judging whether the SIZE is smaller than or equal to SEI _ BUF _ SIZE, copying data in a context parameter array gb to SEI _ BUF if a condition is met, wherein the copy length is the SIZE, and an array subscript i of the copied SEI _ BUF is an index of the gb array divided by 8, namely SEI _ BUF [ i ] ═ gb _ BUF [ index/8 ];

14) then, executing a configure command, and generating decoding libraries libavcodec, libavformat, libavutil and libswscale under the subfolder lib of the configuration folder;

15) and finally, copying the dynamic library to a/usr/lib path of a decoding board card.

Optionally, the receiving the original compressed image transmitted by the host in step 2) specifically includes:

21) firstly, acquiring a receiving ip and a port in a configuration file;

22) then, using the received ip address and the port number as parameters to create and bind the network socket, and after the creation is successful, performing step 23);

23) secondly, clearing the receiving buffer zone, and entering step 24) after waiting for receiving the image data sent by the network;

24) judging whether the length of the image data received this time is greater than zero, if so, carrying out the next step, otherwise, returning to the step 23);

25) judging whether the protocol frame header meets the protocol requirement, if not, discarding the frame; and if the image frame is matched with the image frame, storing the image frame in a data receiving area.

Optionally, the creating and acquiring a data packet thread in step 3) specifically includes:

321) acquiring an image data array first address pointer bufPtr and a length bufLen read from a network;

322) judging whether the current data length bufLen is larger than 0, if so, continuing the next step, and if not, exiting the decoding process of the data and waiting for the next data transmission of the network;

323) transmitting a data array first address pointer bufPtr and a length bufLen to a soft decoding module, segmenting data by frames by using a library function av _ parser _ parse2, recording and continuing the next step if a complete image frame data packet can be successfully obtained from the array, and quitting the decoding process if the complete image frame data packet cannot be obtained from the array, and waiting for the next image data array transmitted by the network;

324) storing the length ret of the image frame divided in the data array at the time, removing the length ret of the image frame divided at the time from the total length bufLen of the data array, and moving a first address pointer bufPtr forwards, wherein the moving times are equal to the length ret of the image frame data packet;

325) and putting the image complete data packet segmented this time into a queue to be decoded.

Optionally, the method for establishing a filtering mechanism and screening an error frame in step 4) specifically includes:

41) judging the image type, analyzing the number of P frames contained in the image data packets of the slice number required by different types, and recording the data structure variable PSlicENumber of the channel;

42) judging whether the size of the current data packet is larger than the maximum number specified by the protocol frame, if so, exiting the filtering process of the data packet, returning to the step 41 after receiving a new image data packet, and otherwise, carrying out the next step;

43) circularly traversing the data in the data packet, sequentially judging whether the current frame accords with the h265 protocol frame header, if so, performing the next step, and if not, jumping to the step 49);

44) judging whether the current frame is a P frame or an I frame, if so, increasing the number temp _ PSlicENumber of the slots in the data packet transmitted by the channel at this time, and entering the next step; if not, jump to step 46);

45) if the temp _ PSlicENumber of the current data packet is equal to 2, judging whether p _ slice is 0 again, if so, setting p _ size to i, and updating and recording a slice flag p _ slice to 1;

46) judging whether the current frame type is SPS, if so, setting the mark top to be 1, otherwise, directly entering 48);

47) continuously judging whether the current frame is a parameter frame, if so, copying the position i in the current data packet array ptr to a data structure Idr _ buf, storing frame header information before the sei frame, recording the data Size Idr _ Size of the currently stored frame header as i, setting the first-appearing sps identifier entryFlag as 1, and resetting top to be 0; step 48) after completion;

48) judging whether the image data of the current channel needs to be inserted into an I frame, and the entryFlag is 1, if the two conditions are satisfied, judging whether the current frame type is a P frame, if the current frame type is the P frame, changing the byte content of the frame type of the data packet from 0x02 to 0x26, and changing the P frame to the I frame; otherwise, directly entering the step 49);

49) continuing to check whether the ptr in the data packet has other data to be traversed for query, if so, returning to the step 43), and if not, continuing to the next step;

410) judging whether the insertIDR mark and the entryFlag of the current channel are both 1, if so, continuing the next step, and if not, performing the step 413);

411) judging whether the current temp _ PSlicENumber is equal to the PSlicENumber which the image type of the channel should have, if so, then the next step, if not, discarding the data packet, and exiting the decoding process of the data packet; returning to step 41) after receiving a new image data packet;

412) copying the data of which the Size of the complete image frame in the current data packet is Size into Idr _ Buf, starting from the Idr _ Size of the Idr _ Buf at the copying position, increasing the Size of the Size by the Idr _ Size, copying the data of the Idr _ Buf into ptr again, wherein the copied data is Size;

413) judging whether the current temp _ PSlicENumber is equal to the PSlicENumber which the channel image type should have, if yes, carrying out the next step, and if not, carrying out the step 415);

414) judging whether p _ slice is 1, if so, assigning p _ size to size, and clearing p _ slice; otherwise, go directly to step 415);

415) recording the size of the image parameter frame of the current channel in seiSize, copying the parameter frame of the current image to a decodeArray of an image data structure to be decoded of the current channel, and taking the parameter number seiCount as a time stamp u64PTS of the image to be decoded;

416) inquiring the decoding state of the current channel, and saving the frame number of the decoded image in the state structure body to u32 DecodeSetreamFrames;

417) acquiring a first address ptr and a data length size of image data to be decoded of a current channel and a time stamp u64PTS, and sending the data to a hard decoding module in a streaming form;

418) circularly inquiring the working state of the decoder until a new decoded image is obtained;

419) calling a Haesi library function to inquire the channel state of the decoder;

420) judging whether the number of bytes needing to be decoded in the channel is 0, if so, entering the next step, and if not, delaying 1ms and jumping to a step 419);

421) whether the number of decoded picture frames of the decoder channel is not equal to the previously saved u32 decoderstreamframes or the insertIDR of the channel is marked as 1, if so, going to the next step, otherwise, going to step 423);

422) clearing the insrtIDR identifier, increasing the number of the parameter frames, judging whether seiCount is greater than the maximum SEI _ NUM specified by the protocol, clearing the parameter frame number identifier seiCount, and entering step 426); otherwise, directly entering the next step;

423) resetting the decoder, proceeding to step 424);

424) judging whether the decoder reset flag resetFlag is 0, if so, entering the next step, if not, continuing to wait for 1ms, and jumping to the step 423);

425) setting a reset state to 1, calling a library function to stop a video stream receiving task of a decoding channel, destroying the decoding channel, restarting the decoding channel, setting an insertIDR (idle IDR) inserted into an I frame to 1, setting the reset state to 0, and entering the next step;

426) and finishing the filtering process.

Optionally, the method for acquiring a decoded image in the decoding module in step 5) specifically includes:

51) creating a decoding image obtaining thread;

52) entering an image acquisition cycle;

53) inquiring whether the reset state in the channel is 1, if so, delaying for 1ms, continuing to inquire the reset state, and if not, carrying out the next step;

54) calling a library function HI _ MPI _ VDEC _ GetFrame to acquire a memory address of image storage; if the failure is failed, jumping to the step 53), if the failure is successful, entering the next step;

55) setting a reset flag resetFlag to 1;

56) traversing parameter frames in the decodeArray structure, comparing whether the stored timestamp u64PTS is consistent with the u64PTS of the current decoding image or not, and if so, sending the image data;

57) creating a network sending task, and sending the data stored in the memory address in the step 54) to the host.

In a second aspect, a processing apparatus comprises:

a memory for storing a computer program;

a processor for calling and running the computer program from the memory to perform the method of the first aspect.

A computer readable storage medium having stored thereon a computer program or instructions which, when executed, implement the method of the first aspect.

A computer program product comprising instructions for causing a computer to perform the method of the first aspect when the computer program product is run on a computer.

Compared with the prior art, the invention has the advantages that:

the invention adopts an ffmpeg decoding library to obtain parameter frame information and data frame information in an original frame, obtain a complete image frame data packet, establish a filtering mechanism to screen error frames, adjust the frame sequence in the data packet, transmit a decodable complete data packet to a hard decoding module to complete a decoding function, and monitor abnormal exit of a decoding program module by using a self-recovery mechanism, thereby ensuring continuous decoding.

Drawings

Fig. 1 is a flowchart illustrating a self-recovery video decoding method according to the present invention.

Detailed Description

In order to solve the video decoding problem in the linux system, a method of combining ffmpeg soft decoding and haisi chip module hard decoding based on a filtering mechanism is adopted in combination with fig. 1. The present invention is further described below.

1) Transplanting ffmpeg to Haisi platform

First, the ffmpeg compile attribute is configured. Then, the ffmpeg source code is modified to have the function of returning to the parameter frame, and the ffmpeg source code is cross-coded. And finally, obtaining the ffmpeg decoding dynamic library and copying the dynamic library to a Haima development board.

11) First, ffmpeg compiling attribute is configured, and parameters are configured according to the platform type, the cpu type, the codec attribute, the format conversion attribute and the cross compiling attribute.

12) Modify acodec.h file, add variable: the parameter frame length SEI _ len and the array SEI _ BUF determine the parameter frame SIZE SEI _ BUF _ SIZE according to the application layer requirements;

13) adding a parameter frame acquiring function in a decode _ nal _ sei _ prefix function in a hevc _ sei.c file: obtaining a parameter SIZE in a function, assigning the SIZE to SEI _ len, judging whether the SIZE is smaller than or equal to SEI _ BUF _ SIZE, copying data in a context parameter array gb to SEI _ BUF if a condition is met, wherein the length of the copy is the SIZE, and an array subscript i of the copied SEI _ BUF is an index of the gb array divided by 8, namely SEI _ BUF [ i ] ═ gb _ BUF [ index/8 ];

14) then, executing a configure command, and generating decoding libraries libavcodec, libavformat, libavutil and libswscale under the subfolder lib of the configuration folder;

15) and finally, copying the dynamic library to a/usr/lib path of a decoding board card.

2) Creating a network receiving thread to receive original compressed image data sent by a host

Firstly, acquiring the information of ip and port number in transmission according to the configuration file. Then, image data waiting for network transmission is blocked.

21) First, the receiving ip and port in the configuration file are obtained.

22) Then, using the received ip address and the port number as parameters to create and bind the network socket, and after the creation is successful, performing step 23);

23) secondly, clearing the receiving buffer zone, and entering step 24) after waiting for receiving the image data sent by the network;

24) and judging whether the length of the image data received this time is greater than zero, if so, carrying out the next step, and otherwise, returning to the step 23).

25) Judging whether the protocol frame header meets the protocol requirement, if not, discarding the frame; and if the image frame is matched with the image frame, storing the image frame in a data receiving area.

3) Obtaining a complete compressed image data packet containing a parameter frame using an ffmpeg dynamic library

First, the ffmpeg usage environment is initialized. Second, an get packet thread is created. The full packet of the compressed image containing the parametric frame is then obtained using the ffmpeg dynamic library.

31) Initializing decoding library usage environment

First, a decoder type is set, and an HEVC (h265) type decoder is adopted. Then, the context environment of the decoder is initialized, and the image frame storage space is dynamically applied.

32) Creating an ffmpeg acquisition data packet thread, wherein the thread comprises the following specific flows:

321) acquiring an image data array first address pointer bufPtr and a length bufLen read from a network;

322) judging whether the current data length bufLen is larger than 0, if so, continuing the next step, and if not, exiting the decoding process of the data and waiting for the next data transmission of the network;

323) transmitting a data array first address pointer bufPtr and a length bufLen to a soft decoding module, segmenting data by frames by using a library function av _ parser _ parse2, recording and continuing the next step if a complete image frame data packet can be successfully obtained from the array, and quitting the decoding process if the complete image frame data packet cannot be obtained from the array, and waiting for the next image data array transmitted by the network;

324) storing the length ret of the image frame divided in the data array at the time, removing the length ret of the image frame divided at the time from the total length bufLen of the data array, and moving a first address pointer bufPtr forwards, wherein the moving times are equal to the length ret of the image frame data packet;

325) and putting the image complete data packet segmented this time into a queue to be decoded.

4) Filtering the complete data packet and sending the complete data packet to a decoding module

And establishing a filtering mechanism, analyzing the number and the type of frames in the data packet, and storing the number and the type of protocol frames required by different images according to a compression principle and image type classification. At the same time, the image data is reprocessed according to the error type, thereby combining decodable data packets to the hard decoding module. The method comprises the following specific steps:

41) judging the image type, analyzing the number of P frames contained in the image data packets of the slice number required by different types), and recording the data structure variable PSlicENumber in the channel;

42) judging whether the size of the current data packet is larger than the maximum number specified by the protocol frame, if so, exiting the filtering process of the data packet, returning to the step 41 after receiving a new image data packet, and otherwise, carrying out the next step;

43) circularly traversing the data in the data packet, wherein the traversal is from 0 to (size-3), the size is the size of the data packet, whether the current frame accords with the h265 protocol frame header (namely, the three continuous bytes are 000001) is judged in a classified mode, if so, the next step is carried out, and if not, the step 49 is skipped;

44) judging whether the current frame is a P frame or an I frame, if so, increasing the number temp _ PSlicENumber of the slots in the data packet transmitted by the channel at this time, and entering the next step; if not, jump to step 46);

45) if the temp _ psicenumber of the current data packet is equal to 2, judging whether the current slice is the first slice of the current data packet, namely whether p _ slice is 0, if so, recording the position i of the traversal byte in the ptr of the current data packet, namely setting p _ size to be i, updating and recording a slice flag p _ slice to be 1, and indicating that one slice already exists;

46) judging whether the current frame type is SPS, if the current frame type is SPS, setting the mark top to be 1 and entering step 47), otherwise, directly entering step 48);

47) judging whether top is 1, if so, continuously judging whether the current frame is a parameter frame (sei frame), if so, copying the i position in the current data packet array ptr to a data structure Idr _ buf, storing frame header information (vps, pps, sps frames) before the sei frame, recording the data Size Idr _ Size of the currently stored frame header as i, marking entryFlag as 1 when the sps appears for the first time, and resetting the top to be 0; step 48) after completion;

48) judging whether the image data of the current channel needs to be inserted with an I frame (identified by an insertiDR), and the entryFlag is 1, if the two conditions are satisfied, judging whether the current frame type is a P frame, if the current frame type is the P frame, changing the byte content of the frame type of the data packet from 0x02 to 0x26, namely changing the P frame to the I frame; otherwise, directly entering the step 49);

49) continuing to check whether the ptr in the data packet has other data to be traversed for query, if so, returning to the step 43), and if not, continuing to the next step;

410) judging whether the insertIDR mark and the entryFlag of the current channel are both 1, if so, continuing the next step, and if not, performing the step 413);

411) judging whether the current temp _ PSlicENumber is equal to the PSlicENumber which the image type of the channel should have, if so, then the next step, if not, discarding the data packet, and exiting the decoding process of the data packet; returning to step 41) after receiving a new image data packet;

412) copying data with the Size of a complete image frame in a current data packet into Idr _ Buf, starting from the Idr _ Size of the Idr _ Buf at the copying position, increasing the Size of the Size by the Idr _ Size (namely the Size of the current data packet needs to be inserted into frame header information stored before), copying the data of the Idr _ Buf into ptr again, wherein the copied data Size is the Size;

413) judging whether the current temp _ PSlicENumber is equal to the PSlicENumber which the channel image type should have, if yes, carrying out the next step, and if not, carrying out the step 415);

414) judging whether p _ slice is 1, if so, assigning p _ size to size (namely, taking only), clearing p _ slice and entering step 415); otherwise, go directly to step 415);

415) recording the size of the image parameter frame of the current channel in seiSize, copying the parameter frame of the current image to a decodeArray of an image data structure to be decoded of the current channel, and taking the parameter number seiCount as a time stamp u64PTS of the image to be decoded;

416) inquiring the decoding state of the current channel, and saving the frame number of the decoded image in the state structure body to u32 DecodeSetreamFrames;

417) acquiring a first address ptr and a data length size of image data to be decoded of a current channel and a time stamp u64PTS, and sending the data to a decoding module in a streaming form;

418) circularly inquiring the working state of the decoder until a new decoded image is obtained;

419) calling a Haesi library function to inquire the channel state of the decoder;

420) judging whether the number of bytes needing to be decoded in the channel is 0, if so, entering the next step, and if not, delaying 1ms and jumping to a step 419);

421) whether the number of decoded picture frames of the decoder channel is not equal to the previously saved u32 decoderstreamframes or the insertIDR (I frame needs to be inserted) of the channel is marked as 1, if yes, the next step is carried out, and if not, the step 423 is carried out;

422) clearing the insrtIDR identifier, increasing the number of the parameter frames (seiCount is increased by 1), judging if seiCount is larger than the maximum SEI _ NUM specified by the protocol, clearing the parameter frame number flag seiCount, and entering step 426); otherwise, directly entering the next step;

423) resetting the decoder, proceeding to step 424);

424) judging whether the decoder reset flag resetFlag is 0, if so, entering the next step, if not, continuing to wait for 1ms, and jumping to the step 423);

425) setting a reset state to 1, calling a library function to stop a video stream receiving task of a decoding channel, destroying the decoding channel, restarting the decoding channel, setting an insertIDR (idle IDR) inserted into an I frame to 1, setting the reset state to 0, and entering the next step;

426) ending the filtering process;

5) capturing images within a decoder channel

And establishing a task of acquiring a decoded image, reading the working state of a decoder in real time, calling a library function to acquire the decoded image in a decoding channel, and sending the decoded image data to a host through a network sending task. The method comprises the following specific steps:

51) creating a decoding image obtaining thread;

52) entering an image acquisition cycle;

53) inquiring whether the reset state in the channel is 1, if so, delaying for 1ms, continuing to inquire the reset state, and if not, carrying out the next step;

54) calling a library function HI _ MPI _ VDEC _ GetFrame to acquire a memory address of image storage, jumping to the step 53 if the library function HI _ MPI _ VDEC _ GetFrame fails, and entering the next step if the library function HI _ MPI _ VDEC _ GetFrame succeeds;

55) setting a reset flag resetFlag to 1;

56) traversing parameter frames in the decodeArray structure, comparing whether the stored timestamp u64PTS is consistent with the u64PTS of the current decoding image or not, and if so, sending the image data;

57) creating a network sending task, and sending the data stored in the memory address in the step 54) to the host.

6) Starting self-recovery process monitoring decoding program

And setting a self-recovery script under a self-running path of the system, and monitoring the running state of the decoding program in real time by the content of the script. And if the decoding program is monitored to be abnormally exited, automatically recovering the decoding program. And simultaneously, stopping the thread of the decoding channel for sending the video stream before the program is restarted, exiting the video buffer block, and releasing system resources. The method comprises the following specific steps:

61) setting a self-recovery script S09Decode under the system/etc/rcS/path;

62) acquiring a process number of decode _ video operation;

63) acquiring the running state of the decode _ video process by using a ps command;

64) if the running state of the decode _ video is the running state stillRunning, delaying for 5ms, and returning to step 62); executing the next step if decode _ video is in the exit state;

65) adding an exit decoding channel in an initialization function of decode, releasing a video cache block and releasing system resources;

66) entering a directory where the decode _ video is located by using a cd command, and executing a decode program;

67) return to step 62).

The step 1) of the invention carries out the configuration of the decoding environment on the board, transplants the ffmpeg of the soft decoding library to adapt to the Haisi platform, and modifies the ffmpeg at the same time, so that the returned image data is provided with the parameter frame. Step 2) performing frame header analysis on the original compressed data, and preliminarily filtering error frames which do not meet the protocol requirements; step 3) a usage environment is created for ffmpeg and a decoder with a start requirement is specified, and protocol analysis can be performed on the original data. Meanwhile, original data are circularly obtained from a data area received by the network, a library function is called to obtain the length of a data packet which can be combined into a complete image frame with a parameter frame, and the length is removed from the data area received by the network and the process is circularly repeated until the data area has no data. And 4) establishing a filtering mechanism, dynamically screening error frames which do not accord with the protocol regulation, analyzing the coding rule, adjusting the sequence of the data packet image frames, and dynamically modifying the frame types according to the characteristics of the coder to combine the error frames into a decodable image data packet. Meanwhile, the state of the decoder is monitored in real time, and the operation of resetting the decoding channel is started according to the requirement of the current data packet, so that the decoding module of the decoding channel continuously works. And 5) establishing a data acquisition thread, starting a library function according to the reset state to acquire the image in the decoding channel, modifying the reset state after the data so that the working state of the decoder can be updated in real time in the step 4, and sending the decoded image to the host through a network sending task. And 6) starting a script for monitoring the decoding program according to the self-running script path characteristics of the linux system, judging the running state of the process in which the decoding program is located in real time, releasing resources when the decoding program is abnormally exited, and starting the decoding program again, so that the decoding is continuously carried out.

By utilizing the technical scheme and adopting the operation steps, the video decoding and transmission problems under the linux system can be realized, and the method is verified by an algorithm and is tested experimentally. The result shows that the scheme can adopt a soft decoding method to form a complete data packet containing image frames and parameter frames for solving the problem of image decoding, adopts a filtering mechanism to dynamically screen error frames which do not conform to the protocol type to obtain decodable image data packets, and then uses a hard decoding module of a chip to complete the decoding process, thereby effectively reducing the decoding time.

Although the present invention has been described with reference to the preferred embodiments, it is not intended to limit the present invention, and those skilled in the art can make variations and modifications of the present invention without departing from the spirit and scope of the present invention by using the methods and technical contents disclosed above.

Those skilled in the art will appreciate that those matters not described in detail in the present specification are well known in the art.