阅读说明：本技术 一种音视频流处理方法、装置、电子设备及存储介质 (Audio and video stream processing method and device, electronic equipment and storage medium ) 是由 昝晓飞 于 2020-11-30 设计创作，主要内容包括：本公开是关于一种音视频流处理方法、装置、电子设备及存储介质,涉及通信技术领域,用以解决相关技术中生成短视频时,操作繁琐,浪费时间的问题。本公开首先接收用户编辑指令,然后基于用户编辑指令包括音频编辑指令且不包括视频编辑指令,解复用原始音视频流,得到原始音频压缩数据,再基于音频编辑指令编辑该原始音频压缩数据,得到目标音频文件,最后基于原始音视频流和目标音频文件,得到目标音视频流,由于当用户编辑指令包括音频编辑指令,且不包括视频编辑指令时,只对原始音频压缩数据进行处理,最终得到目标音视频流,无需对视频数据进行处理,操作简单,节省时间,从而能够提高生成短视频的效率。(The disclosure relates to an audio and video stream processing method and device, electronic equipment and a storage medium, relates to the technical field of communication, and aims to solve the problems of complex operation and time waste when short videos are generated in the related technology. The method comprises the steps of firstly receiving a user editing instruction, then demultiplexing an original audio and video stream based on the user editing instruction including an audio editing instruction but not including a video editing instruction to obtain original audio compressed data, editing the original audio compressed data based on the audio editing instruction to obtain a target audio file, and finally obtaining the target audio and video stream based on the original audio and video stream and the target audio file.)

1. An audio/video stream processing method, characterized by comprising:

receiving a user editing instruction;

demultiplexing the original audio and video stream based on the user editing instruction including the audio editing instruction and not including the video editing instruction to obtain original audio compression data;

editing the original audio compression data based on the audio editing instruction to obtain a target audio file;

and obtaining a target audio and video stream based on the original audio and video stream and the target audio file.

2. The method of claim 1, wherein said editing the original audio compressed data based on the audio editing instructions to obtain a target audio file comprises:

decoding the original audio compressed data to obtain original audio data;

editing the original audio data based on the audio editing instruction, and determining target audio data;

carrying out coding operation on the target audio data to obtain target audio compressed data;

and multiplexing the target audio compressed data, and determining the target audio file.

3. The method of claim 2, wherein the editing the original audio data comprises performing a voicing process, a volume adjustment process, or a mixing process on the original audio data.

4. The method of claim 1, wherein obtaining a target audio-video stream based on the original audio-video stream and the target audio file comprises:

demultiplexing the original audio and video stream and the target audio file at the same time to obtain original video compressed data and target audio compressed data;

and multiplexing the original video compression data and the target audio compression data to obtain the target audio and video stream.

5. An audio-video stream processing apparatus, characterized by comprising:

a receiving unit configured to receive a user editing instruction;

the demultiplexing unit is configured to execute demultiplexing of the original audio and video stream based on that the user editing instruction includes an audio editing instruction and does not include a video editing instruction, so as to obtain original audio compression data;

the editing unit is configured to edit the original audio compression data based on the audio editing instruction to obtain a target audio file;

and the processing unit is configured to execute the audio and video data based on the original audio and video data and the target audio file to obtain a target audio and video data.

6. The apparatus of claim 5, wherein the editing unit is configured to perform:

decoding the original audio compressed data to obtain original audio data;

editing the original audio data based on the audio editing instruction, and determining target audio data;

carrying out coding operation on the target audio data to obtain target audio compressed data;

and multiplexing the target audio compressed data, and determining the target audio file.

7. The apparatus of claim 6, wherein said editing said original audio data comprises some or all of:

performing sound changing treatment;

adjusting the volume;

and (5) mixing sound processing.

8. The apparatus of claim 5, wherein the processing unit is configured to perform:

demultiplexing the original audio and video stream and the target audio file at the same time to obtain original video compressed data and target audio compressed data;

and multiplexing the original video compression data and the target audio compression data to obtain the target audio and video stream.

9. An electronic device, comprising:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the audio-video stream processing method according to any one of claims 1 to 4.

10. A storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the audio-video stream processing method according to any one of claims 1 to 4.

Technical Field

The present invention relates to the field of communications technologies, and in particular, to an audio/video stream processing method and apparatus, an electronic device, and a storage medium.

Background

When a short video is generated, a user can perform editing operation on the original short video, wherein the editing operation can comprise two types of operation, one of which is to edit video data in the short video, such as operations of adding a filter, beautifying, adding a special effect and the like; the second is to edit the audio data in the short video, for example, to change the audio, adjust the volume, add music, and so on.

If the final short video is to be generated, a series of operations such as decoding and encoding are required to be performed on the audio data and the video data, respectively, and finally the final short video is synthesized.

The method for generating the short video is complex in operation and wastes time.

Disclosure of Invention

The present disclosure provides an audio and video stream processing method, an apparatus, an electronic device, and a storage medium, which are used to solve the problems of complex operation and time waste when generating a short video in the related art.

According to a first aspect of the embodiments of the present disclosure, there is provided an audio and video stream processing method, including:

receiving a user editing instruction;

demultiplexing the original audio and video stream based on the user editing instruction including the audio editing instruction and not including the video editing instruction to obtain original audio compression data;

editing the original audio compression data based on the audio editing instruction to obtain a target audio file;

and obtaining a target audio and video stream based on the original audio and video stream and the target audio file.

In an optional implementation manner, the editing the original audio compressed data based on the audio editing instruction to obtain a target audio file includes:

decoding the original audio compressed data to obtain original audio data;

editing the original audio data based on the audio editing instruction, and determining target audio data;

carrying out coding operation on the target audio data to obtain target audio compressed data;

and multiplexing the target audio compressed data, and determining the target audio file.

In an alternative embodiment, the editing the original audio data comprises some or all of the following:

performing sound changing treatment;

adjusting the volume;

and (5) mixing sound processing.

In an optional implementation manner, the obtaining a target audio/video stream based on the original audio/video stream and the target audio file includes:

demultiplexing the original audio and video stream and the target audio file at the same time to obtain original video compressed data and target audio compressed data;

and multiplexing the original video compression data and the target audio compression data to obtain the target audio and video stream.

According to a second aspect of the embodiments of the present disclosure, there is provided an audio/video stream processing apparatus including:

a receiving unit configured to receive a user editing instruction;

the demultiplexing unit is configured to execute demultiplexing of the original audio and video stream based on that the user editing instruction includes an audio editing instruction and does not include a video editing instruction, so as to obtain original audio compression data;

the editing unit is configured to edit the original audio compression data based on the audio editing instruction to obtain a target audio file;

and the processing unit is configured to execute the audio and video data based on the original audio and video data and the target audio file to obtain a target audio and video data.

In an alternative embodiment, the editing unit is specifically configured to perform:

decoding the original audio compressed data to obtain original audio data;

editing the original audio data based on the audio editing instruction, and determining target audio data;

carrying out coding operation on the target audio data to obtain target audio compressed data;

and multiplexing the target audio compressed data, and determining the target audio file.

In an alternative embodiment, the editing the original audio data comprises some or all of the following:

performing sound changing treatment;

adjusting the volume;

and (5) mixing sound processing.

In an alternative embodiment, the processing unit is specifically configured to perform:

demultiplexing the original audio and video stream and the target audio file at the same time to obtain original video compressed data and target audio compressed data;

and multiplexing the original video compression data and the target audio compression data to obtain the target audio and video stream.

According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the audio-video stream processing method according to any one of the first aspect.

According to a fourth aspect of embodiments of the present disclosure, there is provided a storage medium, where instructions, when executed by a processor of an electronic device, enable the electronic device to perform the audio/video stream processing method according to any one of the first aspect.

The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:

because the user editing instruction comprises the audio editing instruction and does not comprise the video editing instruction, only the original audio compressed data is processed to obtain the target audio file, and finally the target audio and video stream is obtained based on the original video stream and the target audio file, so that the audio data does not need to be processed, the video data does not need to be processed, the operation is simple, the time is saved, and the efficiency of generating the short video can be improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.

FIG. 1 is a schematic flow diagram illustrating the generation of a short video according to an exemplary embodiment;

fig. 2 is a flow diagram illustrating an audio video stream processing method according to an exemplary embodiment;

fig. 3 is a full flow diagram illustrating an audio video stream processing method according to an exemplary embodiment;

fig. 4 is a block diagram illustrating an audio video stream processing apparatus according to an exemplary embodiment;

FIG. 5 is a block diagram illustrating an electronic device in accordance with an exemplary embodiment;

fig. 6 is a block diagram illustrating a terminal device according to an example embodiment.

Detailed Description

In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

Some of the words that appear in the text are explained below:

1. the term "and/or" in the embodiments of the present disclosure describes an association relationship of associated objects, and means that there may be three relationships, for example, a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

2. The term "electronic device" in the embodiments of the present disclosure may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

3. decode: and decoding, namely decoding the video/audio compression coded data into uncompressed video/audio original data.

4. An encode: the encoding, the reverse process of decode, compresses the uncompressed video/audio raw data into encoded data, thereby reducing the data amount. The video coding is to compress pixel data (RGB, YUV and the like) into a video code stream so as to reduce the data volume of the video; audio coded audio sample data (PCM (Pulse Code Modulation, etc.)) is compressed into an audio Code stream, thereby reducing the data amount of audio.

5. A muxer: multiplexing, i.e. video-audio multiplexer. Video compression data (e.g., h.264) and Audio compression data (e.g., AAC (Advanced Audio Coding)) are incorporated into a package format data (e.g., MKV), which does not involve encoding and decoding.

6. remuxer: the video and audio compression code stream is directly obtained from a file with one packaging format without the video and audio coding and decoding, and then is packaged into a file with another packaging format, and certainly, two or more video stream files and audio stream files can be packaged into one file.

7. demuxer: demultiplexing, i.e. video-audio separator. The video compression data (e.g., h.264) and the audio compression data (e.g., AAC) in the package format data (e.g., mp4) are separated, and encoding and decoding are not involved in this process.

The application scenario described in the embodiment of the present disclosure is for more clearly illustrating the technical solution of the embodiment of the present disclosure, and does not form a limitation on the technical solution provided in the embodiment of the present disclosure, and as a person having ordinary skill in the art knows, with the occurrence of a new application scenario, the technical solution provided in the embodiment of the present disclosure is also applicable to similar technical problems. Wherein, in the description of the present disclosure, unless otherwise indicated, "plurality" means.

Before uploading the short video, the short video is usually required to be generated, and in the process of generating the short video, if the user does not edit the original short video, such as editing the video data and/or editing the audio data, the original short video can be directly uploaded, that is, the original picture is uploaded.

However, if the user edits the original short video, for example, the sound in the original short video is adjusted in volume, the audio part and the video part in the original short video need to be demultiplexed, demux, decoded, encoded, encode, and multiplexed to generate the final short video. In the above process of generating short video, the most time consuming phases are decoding the video part and encoding the encode part.

Fig. 1 is a schematic flow chart of a method for generating a short video, as shown in fig. 1, including the following steps.

S101, inputting original audio and video stream;

s102, receiving a user editing instruction, wherein the user editing instruction comprises a video editing instruction or an audio editing instruction;

s103, carrying out demultiplexing demuxer operation on the original audio and video stream to obtain audio compression data and video compression data;

s104, performing decoding operation on the audio compressed data to obtain audio pcm original data, and performing decoding operation on the video data to obtain video original data;

s105, preprocessing the audio pcm original data and the video original data according to a user editing instruction;

s106, carrying out encoding encode operation on the preprocessed video data to obtain processed video compressed data, and carrying out encoding encode operation on the audio data to obtain audio compressed data;

s107, multiplexing muxer operation is carried out on the processed video compression data and the processed audio compression data to obtain target audio and video streams;

and S108, uploading the target audio and video stream.

On the basis of the audio/video stream processing method described in fig. 1, fig. 2 is a flowchart of another audio/video stream processing method according to an exemplary embodiment, and in some embodiments, the audio/video stream processing method may be implemented on a server or a terminal. The audio and video stream processing method comprises the following steps.

In S21, a user edit instruction is received.

In S22, based on that the user edit instruction includes an audio edit instruction and does not include a video edit instruction, demultiplexing the original audio/video stream to obtain original audio/video compressed data.

In some embodiments, the user editing instructions may include: only audio editing instructions are included, only video editing instructions are included, and audio editing instructions and video editing instructions are included at the same time. The audio-video stream processing method described according to the embodiment of the present disclosure is applied when the user editing instruction includes only the audio editing instruction and does not include the video editing instruction. Wherein the audio editing instructions and the video editing instructions are determined according to user editing instructions.

In some embodiments, the audio editing instruction refers to an instruction for performing an editing operation on an audio stream in the audio-video stream. The audio editing instructions may include an adjust volume instruction, an add music instruction, or a retain the audio stream instruction. In some embodiments, the audio instructions may be such that, for example, the adjust volume instructions may be used to adjust only the volume of the original audio, but not to mute (mute); the add music instruction may implement retaining the original audio and adding one or more pieces of music, or not retaining the original audio and adding a plurality of pieces of music, based on the instruction information of the add music instruction. The preservation of the audiovisual instructions may be used to achieve preservation of the original audio without muting.

In some embodiments, the original audio/video stream may be an audio/video stream that is input to the terminal and has not been subjected to any processing, for example, the audio/video stream that is acquired by the terminal through a camera may be an audio/video stream stored in the terminal, and may also be an audio/video stream stored in the server.

In S23, the original audio compressed data is edited based on the audio editing instruction, resulting in a target audio file.

In some embodiments, the original audio compressed data is edited based on the audio editing instruction, and the original audio compressed data may be decoded to obtain original audio data, then the original audio data is edited based on the audio editing instruction to determine target audio data, then the target audio data is encoded to obtain target audio compressed data, and finally the target audio compressed data is multiplexed to determine a target audio file.

In some embodiments, the editing the original audio data based on the audio editing instructions comprises: and performing sound changing processing, volume adjusting processing or sound mixing processing on the original audio data based on the audio editing instruction. For example, when the audio editing instruction includes adjusting the volume of the original audio only, but not muting, the volume adjustment processing is performed on the original audio data.

In S24, a target audio/video stream is obtained based on the original audio/video stream and the target audio file.

In specific implementation, the original audio and video stream and the target audio file can be demultiplexed at the same time to obtain original video compressed data and target audio compressed data, and then the original video compressed data and the target audio compressed data are multiplexed to obtain the target audio and video stream.

For ease of understanding, two specific examples are described below.

For example, a terminal acquires an original audio and video stream through a camera, acquires a user editing instruction, determines that the user editing instruction only comprises a tuning instruction, demultiplexes the original audio and video stream by the terminal to obtain original audio compressed data and original video compressed data, decodes the original audio compressed data by the terminal to obtain original audio data, tunes the original audio data by the terminal, encodes the tuned audio data to obtain tuned audio compressed data, multiplexes the tuned audio compressed data to obtain a processed audio file, demultiplexes the original audio and video stream and the processed audio file to obtain original video compressed data and target audio compressed data, multiplexes the target audio compressed data and the original video compressed data to obtain a target audio and video stream, and uploading the target audio and video stream (the audio and video stream can be called as short video) by the terminal. For example, the terminal uploads the edited short video to the express platform.

For another example, the server stores original audio and video streams, the server obtains a user editing instruction through a terminal, the server determines that the user editing instruction only comprises a tuning instruction, the server performs demultiplexing operation on the original audio and video streams to obtain original audio compressed data and original video compressed data, the server performs decoding operation on the original audio compressed data to obtain original audio data, the server tunes the original audio data, performs coding operation on the tuned audio data to obtain tuned audio compressed data, performs multiplexing operation on the tuned audio compressed data to obtain a processed audio file, demultiplexes the original audio and video streams and the processed audio file to obtain original video compressed data and target audio compressed data, and performs multiplexing operation on the target audio compressed data and the original video compressed data, and obtaining a target audio and video stream, and using the target audio and video stream to cover the original audio and video stream which is locally stored by the server. For example, after editing the short video in the express platform in the audio/video stream (the audio/video stream may be referred to as a short video) displayed on the express platform, the edited short video is used to cover the short video before editing.

By adopting the method for generating the target audio and video stream, only the audio data is processed, and the video data does not need to be decoded and encoded, so that the time can be saved, and meanwhile, the video quality can be improved compared with the prior art because the video data is not decoded and encoded.

Fig. 3 is a full flow chart illustrating an audio-video stream processing method according to an exemplary embodiment, as shown in fig. 3, including the following steps.

S301, inputting original audio and video stream;

s302, receiving a user editing instruction which only comprises an audio editing instruction and does not comprise a video editing instruction;

s303, carrying out demultiplexing demuxer operation on the original audio and video stream to obtain original audio compressed data and original video compressed data;

s304, decoding the original audio compression data to obtain original audio pcm data;

s305, editing (preprocessing) the original audio pcm data according to a user editing instruction;

s306, carrying out encoding encode operation on the edited audio data to obtain processed audio compressed data;

s307, multiplexing muxer operation is carried out on the processed audio compression data to obtain a processed audio MP4 file;

s308, carrying out demultiplexing demuxer operation on the processed audio MP4 file and the original audio and video stream to obtain target audio compression data and original video compression data;

s309, multiplexing the target audio compressed data and the original video compressed data by a muxer to obtain a target audio and video stream;

and S310, uploading the target audio and video stream.

Fig. 4 is a block diagram illustrating an audio video stream processing apparatus 400 according to an exemplary embodiment. Referring to fig. 4, the apparatus includes a receiving unit 401, a demultiplexing unit 402, an editing unit 403, and a processing unit 404.

A receiving unit 401 configured to receive a user editing instruction;

a demultiplexing unit 402 configured to demultiplex the original audio and video stream to obtain original audio compressed data, based on that the user editing instruction includes an audio editing instruction and does not include a video editing instruction;

an editing unit 403 configured to edit the original audio compressed data based on the audio editing instruction, resulting in a target audio file;

a processing unit 404 configured to execute a target audio/video stream based on the original audio/video stream and the target audio file.

In an alternative embodiment, the editing unit 403 is specifically configured to perform:

decoding the original audio compressed data to obtain original audio data;

editing the original audio data based on the audio editing instruction, and determining target audio data;

carrying out coding operation on the target audio data to obtain target audio compressed data;

and multiplexing the target audio compressed data, and determining the target audio file.

In an alternative embodiment, the editing the original audio data comprises some or all of the following:

performing sound changing treatment;

adjusting the volume;

and (5) mixing sound processing.

In an alternative embodiment, the processing unit 404 is specifically configured to perform:

demultiplexing the original audio and video stream and the target audio file at the same time to obtain original video compressed data and target audio compressed data;

and multiplexing the original video compression data and the target audio compression data to obtain the target audio and video stream.

Fig. 5 is a block diagram illustrating an electronic device 500 according to an example embodiment, the apparatus comprising:

a processor 501;

a memory 502 for storing instructions executable by the processor 501;

the processor 501 is configured to execute instructions to implement any resource processing progress display method in the embodiments of the present disclosure.

In an exemplary embodiment, a storage medium comprising instructions, such as the memory 502 comprising instructions, executable by the processor 501 of the electronic device 500 to perform the above-described method is also provided. Alternatively, the storage medium may be a non-transitory computer readable storage medium, for example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

In an embodiment of the present disclosure, a terminal device is further provided, and a structure of the terminal device is shown in fig. 6, where an embodiment of the present disclosure provides a display terminal 600 for a resource processing progress, including: radio Frequency (RF) circuit 610, power supply 620, processor 630, memory 640, input unit 650, display unit 660, camera 670, communication interface 680, and Wireless Fidelity (Wi-Fi) module 690. Those skilled in the art will appreciate that the configuration of the terminal shown in fig. 6 is not intended to be limiting, and that embodiments of the present disclosure provide terminals that include more or less components than those shown, or that combine certain components, or that are arranged in different components.

The following describes the various components of the terminal 600 in detail with reference to fig. 6:

the RF circuitry 610 may be used for receiving and transmitting data during a communication or conversation. Specifically, the RF circuit 610 sends the downlink data of the base station to the processor 630 for processing; and in addition, sending the uplink data to be sent to the base station. In general, RF circuit 610 includes, but is not limited to, an antenna, at least one Amplifier, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like.

In addition, the RF circuit 610 may also communicate with networks and other terminals through wireless communication. The wireless communication may use any communication standard or protocol, including but not limited to Global System for Mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

Wi-Fi technology belongs to short distance wireless transmission technology, and terminal 600 can connect with Access Point (AP) through Wi-Fi module 690, thereby realizing Access of data network. Wi-Fi module 690 may be used for receiving and transmitting data during communication.

Terminal 600 may be physically connected to other terminals via communication interface 680. Optionally, the communication interface 680 is connected to a communication interface of another terminal through a cable, so as to implement data transmission between the terminal 600 and the other terminal.

Since the terminal 600 can implement a communication service and send information to other contacts in the embodiment of the present disclosure, the terminal 600 needs to have a data transmission function, that is, the terminal 600 needs to include a communication module inside. Although fig. 6 illustrates communication modules such as RF circuitry 610, a Wi-Fi module 690, and a communication interface 680, it is to be understood that at least one of the above-described components or other communication modules (e.g., bluetooth modules) for enabling communications may be present in the terminal 600 for data transmission.

For example, when the terminal 600 is a mobile phone, the terminal 600 may include the RF circuit 610 and may further include the Wi-Fi module 690; when the terminal 600 is a computer, the terminal 600 may include a communication interface 680 and may also include a Wi-Fi module 690; when the terminal 600 is a tablet computer, the terminal 600 may include a Wi-Fi module.

The memory 640 may be used to store software programs and modules. The processor 630 executes various functional applications and data processing of the terminal 600 by executing software programs and modules stored in the memory 640, and after the processor 630 executes program codes in the memory 640, part or all of the processes in fig. 2 or fig. 3 of the embodiments of the present disclosure may be implemented.

Alternatively, the memory 640 may mainly include a program storage area and a data storage area. The storage program area can store an operating system, various application programs (such as communication application), a face recognition module and the like; the storage data area may store data (such as various multimedia files like pictures, video files, etc., and face information templates) created according to the use of the terminal, etc.

Further, the memory 640 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

The input unit 650 may be used to receive numeric or character information input by a user and generate key signal inputs related to user settings and function control of the terminal 1300.

Alternatively, the input unit 650 may include a touch panel 651 and other input terminals 652.

The touch panel 651, also referred to as a touch screen, may collect touch operations of a user (e.g., operations of the user on or near the touch panel 651 by using any suitable object or accessory such as a finger, a stylus, etc.) thereon or nearby, and drive the corresponding connecting device according to a preset program. Alternatively, the touch panel 651 may include two parts of a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 630, and can receive and execute commands sent by the processor 630. In addition, the touch panel 651 may be implemented in various types, such as resistive, capacitive, infrared, and surface acoustic wave.

Alternatively, other input terminals 652 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

The display unit 660 may be used to display information input by the user or information provided to the user and various menus of the terminal 600. The display unit 660 is a display system of the terminal 600, and is used for presenting an interface and implementing human-computer interaction.

The display unit 660 may include a display panel 661. Alternatively, the Display panel 661 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like.

Further, the touch panel 651 may cover the display panel 661, and when the touch panel 651 detects a touch operation thereon or nearby, it is transmitted to the processor 630 to determine the type of the touch event, and then the processor 630 provides a corresponding visual output on the display panel 661 according to the type of the touch event.

Although in fig. 6, the touch panel 651 and the display panel 661 are implemented as two separate components to implement the input and output functions of the terminal 600, in some embodiments, the touch panel 651 and the display panel 661 may be integrated to implement the input and output functions of the terminal 600.

The processor 630 is a control center of the terminal 600, connects the respective components using various interfaces and lines, performs various functions of the terminal 600 and processes data by operating or executing software programs and/or modules stored in the memory 640 and calling data stored in the memory 640, thereby implementing various terminal-based services.

The present application is described above with reference to block diagrams and/or flowchart illustrations of methods, apparatus (systems) and/or computer program products according to embodiments of the application. It will be understood that one block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, and/or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer and/or other programmable data processing apparatus, create means for implementing the functions/acts specified in the block diagrams and/or flowchart block or blocks.

Accordingly, the subject application may also be embodied in hardware and/or in software (including firmware, resident software, micro-code, etc.). Furthermore, the present application may take the form of a computer program product on a computer-usable or computer-readable storage medium having computer-usable or computer-readable program code embodied in the medium for use by or in connection with an instruction execution system. In the context of this application, a computer-usable or computer-readable medium may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.