阅读说明：本技术 图像处理方法和装置、存储介质、电子设备 (Image processing method and device, storage medium and electronic equipment ) 是由 陈岩 于 2018-07-16 设计创作，主要内容包括：本申请涉及一种图像处理方法和装置、电子设备、存储介质,应用于电子设备,电子设备支持富执行环境REE和可信任执行环境TEE。从REE中获取相机标定结果；通过REE和TEE之间的数据通道将所获取的相机标定结果传输至TEE中；在TEE中对相机标定结果进行解析,得到解析结果；将解析结果通过REE和TEE之间的数据通道传输至REE中,在REE中根据解析结果对拍摄的图像进行处理。对相机标定结果进行解析的整个过程都是在REE中进行的,即都是在安全的环境中进行的,避免了数据在解析过程中被篡改和盗取的风险。(The application relates to an image processing method and device, an electronic device and a storage medium, which are applied to the electronic device, wherein the electronic device supports a rich execution environment REE and a trusted execution environment TEE. Acquiring a camera calibration result from the REE; transmitting the acquired camera calibration result to the TEE through a data channel between the REE and the TEE; analyzing the camera calibration result in the TEE to obtain an analysis result; and transmitting the analysis result to the REE through a data channel between the REE and the TEE, and processing the shot image according to the analysis result in the REE. The whole process of analyzing the camera calibration result is carried out in the REE, namely, the camera calibration result is carried out in a safe environment, so that the risk of tampering and stealing data in the analyzing process is avoided.)

1. An image processing method applied to an electronic device supporting a Rich Execution Environment (REE) and a Trusted Execution Environment (TEE), the method comprising:

acquiring a camera calibration result from the REE;

transmitting the acquired camera calibration result to the TEE through a data channel between the REE and the TEE;

analyzing the camera calibration result in the TEE to obtain an analysis result;

and transmitting the analysis result to the REE through a data channel between the REE and the TEE, and processing the shot image in the REE according to the analysis result.

2. The method of claim 1, wherein resolving the camera calibration results in the TEE to obtain resolved results comprises:

and analyzing the camera calibration result in the TEE to obtain distortion correction information and binocular calibration information.

3. The method of claim 2, wherein transmitting the parsing result to the REE via a data channel between the REE and the TEE, wherein processing the image according to the parsing result in the REE comprises:

and transmitting the distortion correction information and the binocular calibration information to the REE through a data channel between the REE and the TEE, and reconstructing and beautifying the shot image in the REE according to the distortion correction information and the binocular calibration information.

4. The method of claim 2, wherein transmitting the parsing result to the REE via a data channel between the REE and the TEE, wherein processing the image according to the parsing result in the REE further comprises:

and transmitting the distortion correction information and the binocular calibration information to the REE through a data channel between the REE and the TEE, and correcting depth information of the shot image in the REE according to the distortion correction information and the binocular calibration information.

5. The method of claim 1, prior to obtaining camera calibration results from the REE, comprising:

calibrating the camera in the TEE to obtain a camera calibration result;

and storing the camera calibration result in a safe area.

6. The method according to claim 1, wherein said analyzing the camera calibration result in the TEE, after obtaining the analysis result, comprises:

acquiring a shot image;

processing the shot image in the TEE according to the analysis result to obtain a processing result;

transmitting the processing result to the REE through a data channel between the REE and the TEE.

7. A control method applied to an electronic device supporting a rich execution environment, REE, and a secure chip, the method comprising:

acquiring a camera calibration result from the REE;

transmitting the acquired camera calibration result to a security chip through a data channel between the REE and the security chip;

analyzing the camera calibration result in the security chip to obtain an analysis result;

and transmitting the analysis result to the REE through a data channel between the REE and a security chip, and processing the shot image in the REE according to the analysis result.

8. An image processing apparatus, characterized in that the apparatus comprises:

the camera calibration result acquisition module is used for acquiring a camera calibration result from the REE;

the transmission module is used for transmitting the acquired camera calibration result to the TEE through a data channel between the REE and the TEE;

the analysis module is used for analyzing the camera calibration result in the TEE to obtain an analysis result;

and the image processing module is used for transmitting the analysis result to the REE through a data channel between the REE and the TEE, and processing the shot image in the REE according to the analysis result.

9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the image processing method according to any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, the memory having stored thereon a computer program operable on the processor, wherein the processor, when executing the computer program, performs the steps of the image processing method of any of claims 1 to 7.

Technical Field

The present application relates to the field of computer technologies, and in particular, to an image processing method and apparatus, a storage medium, and an electronic device.

Background

A conventional smart terminal provides a common execution environment, i.e., Rich Execution Environment (REE), which can be used to run a wide variety of general-purpose operating systems. While great flexibility and functionality is provided by such an execution environment, smart terminals are beginning to face a wide variety of security threats to a great extent. In the era of increasing emphasis on network security, how to improve security in the information processing process has become a problem to be solved urgently.

Disclosure of Invention

The embodiment of the application provides an image processing method and device, a storage medium and electronic equipment, which can improve information security in the image processing process.

An image processing method applied to an electronic device supporting a Rich Execution Environment (REE) and a Trusted Execution Environment (TEE), the method comprising:

acquiring a camera calibration result from the REE;

transmitting the acquired camera calibration result to the TEE through a data channel between the REE and the TEE;

analyzing the camera calibration result in the TEE to obtain an analysis result;

and transmitting the analysis result to the REE through a data channel between the REE and the TEE, and processing the shot image in the REE according to the analysis result.

An image processing apparatus, the apparatus comprising:

the camera calibration result acquisition module is used for acquiring a camera calibration result from the REE;

the transmission module is used for transmitting the acquired camera calibration result to the TEE through a data channel between the REE and the TEE;

the analysis module is used for analyzing the camera calibration result in the TEE to obtain an analysis result;

and the image processing module is used for transmitting the analysis result to the REE through a data channel between the REE and the TEE, and processing the shot image in the REE according to the analysis result.

A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the image processing method as described above.

An electronic device comprising a memory and a processor, the memory having stored thereon a computer program operable on the processor to, when executed by the processor, perform the steps of the image processing method as described above.

The image processing method and device, the storage medium and the electronic equipment are applied to the electronic equipment, and the electronic equipment supports a rich execution environment REE and a trusted execution environment TEE. Acquiring a camera calibration result from the REE; transmitting the acquired camera calibration result to the TEE through a data channel between the REE and the TEE; analyzing the camera calibration result in the TEE to obtain an analysis result; and transmitting the analysis result to the REE through a data channel between the REE and the TEE, and processing the shot image according to the analysis result in the REE. The whole process of analyzing the camera calibration result is carried out in the REE, namely, the camera calibration result is carried out in a safe environment, so that the risk of tampering and stealing data in the analyzing process is avoided.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a diagram of the internal structure of an electronic device in one embodiment;

FIG. 2 is a flow diagram of a method of image processing in one embodiment;

FIG. 3 is a schematic diagram of camera calibration result parsing and transmission in a REE environment and camera calibration result parsing and transmission in a TEE environment;

FIG. 4 is a flowchart of an image processing method in another embodiment;

FIG. 5 is a flowchart of an image processing method in yet another embodiment;

FIG. 6 is a diagram showing a configuration of an image processing apparatus according to an embodiment;

FIG. 7 is a schematic diagram showing a configuration of an image processing apparatus according to another embodiment;

FIG. 8 is a schematic diagram showing a configuration of an image processing apparatus according to still another embodiment;

FIG. 9 is a schematic diagram of an image processing circuit in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

Fig. 1 is a schematic diagram of an internal structure of an electronic device in one embodiment. As shown in fig. 1, the electronic device includes a processor, a memory, and a network interface connected by a system bus. Wherein, the processor is used for providing calculation and control capability and supporting the operation of the whole electronic equipment. The memory is used for storing data, programs and the like, and the memory stores at least one computer program which can be executed by the processor to realize the scene recognition method suitable for the electronic device provided in the embodiment of the application. The Memory may include a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random-Access-Memory (RAM). For example, in one embodiment, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by a processor to implement an image processing method provided in the following embodiments. The internal memory provides a cached execution environment for the operating system computer programs in the non-volatile storage medium. The network interface may be an ethernet card or a wireless network card, etc. for communicating with an external electronic device. The electronic device may be a mobile phone, a tablet computer, or a personal digital assistant or a wearable device, etc.

In one embodiment, as shown in fig. 2, an image processing method is provided and applied to an electronic device, which supports a rich execution environment REE and a trusted execution environment TEE, and is described by taking the application of the method to the electronic device in fig. 1 as an example, including:

step 220, a camera calibration result is obtained from the REE.

The Rich Execution Environment REE is called Rich Execution Environment, and general mobile phone software and software on a computer PC can run in the Environment, but the REE often has the problem of information leakage. This time, a new concept has emerged, Trusted Execution Environment (TEE), which is a feature that coexists with rich Execution environments and is dedicated to providing a secure area for devices to execute Trusted code. To achieve true security, the TEE execution environment must ensure that all code executing in the TEE itself must be highly reliable. The TEE has the following characteristics:

firstly, protected by a hardware mechanism: the TEE is isolated from the REE, and the REE can only communicate with the TEE through a specific entrance; secondly, high performance: TEE run-time uses the full performance of the CPU (Exclusive); thirdly, a quick communication mechanism: the TEE can access the internal memory of the REE, and the REE cannot access the TEE internal memory protected by hardware; and fourthly, multiple Trusted Applications (TAs) can be simultaneously operated in the TEE. Therefore, based on the characteristics of the TEE, if the TEE is used in the fields of mobile payment, fingerprint identification, face unlocking and the like, the information security is greatly improved.

Camera calibration (Camera calibration) is simply a process of converting a world coordinate system into an image coordinate system, that is, a process of finding a final projection matrix. The basic coordinate systems involved in camera calibration include: a world coordinate system (world coordinate system), a camera coordinate system (camera coordinate system), and an image coordinate system (image coordinate system), wherein the image coordinate system includes an image physical coordinate system and an image pixel coordinate system.

The camera calibration aims to obtain the internal parameters, the external parameters and the distortion coefficients of the camera. Namely, the camera calibration result comprises the internal parameters of the camera, the external parameters of the camera and the distortion coefficient. After camera calibration is performed on a camera of the electronic device, a camera calibration result is obtained. The camera calibration results are typically stored in hardware in the electronic device for later retrieval. And acquiring a camera calibration result in the REE.

Step 240, transmitting the obtained camera calibration result to the TEE through a data channel between the REE and the TEE.

A specific data channel exists between the two environments of REE and TEE, so that data meeting the conditions can be transmitted between the two environments of REE and TEE. And transmitting the acquired camera calibration result to the TEE through a data channel between the REE and the TEE. Therefore, the camera calibration result is analyzed in the TEE in the subsequent process, all data information generated in the whole analysis process and the analysis process are operated in the TEE environment, the data security is ensured, and the risk of data attack is reduced.

And step 260, analyzing the camera calibration result in the TEE to obtain an analysis result.

And analyzing the camera calibration result in the TEE environment to obtain distortion correction information and binocular calibration information, namely rotation and translation information.

And step 280, transmitting the analysis result to the REE through a data channel between the REE and the TEE, and processing the shot image according to the analysis result in the REE.

And transmitting the distortion correction information and the binocular calibration information acquired after analysis to the REE through a data channel between the REE and the TEE. Applications that need to use the parsing result may include face unlocking, 3D beauty, and other applications that need to further process the image using the parsing result. Of course, some hardware using the parsed result may be included, for example, a 3D camera Processor (3D camera Processor). Because the analysis result is obtained after being analyzed in the TEE environment and is transmitted to the REE through a data channel between the REE and the TEE, and the analysis result is obtained and used by applications such as human face unlocking and 3D facial beautification or hardware such as a 3D camera processor in the REE environment, the accuracy and the safety of the data of the analysis result input into the applications and the hardware are guaranteed in the whole process.

In the embodiment of the application, the electronic device supports a rich execution environment REE and a trusted execution environment TEE. Acquiring a camera calibration result from the REE; transmitting the acquired camera calibration result to the TEE through a data channel between the REE and the TEE; analyzing the camera calibration result in the TEE to obtain an analysis result; and transmitting the analysis result to the REE through a data channel between the REE and the TEE, and processing the shot image according to the analysis result in the REE. The whole process of analyzing the camera calibration result is carried out in the REE, namely, the camera calibration result is carried out in a safe environment, so that the risk of tampering and stealing data in the analyzing process is avoided.

In one embodiment, resolving the camera calibration result in the TEE to obtain a resolved result includes:

and analyzing the camera calibration result in the TEE to obtain distortion correction information and binocular calibration information.

In particular, the imaging process of the camera is essentially a transformation of the coordinate system. Firstly, converting points in the space from a world coordinate system to a camera coordinate system, then projecting the points to an imaging plane (an image physical coordinate system), and finally converting data on the imaging plane to an image pixel coordinate system. But distortion is introduced due to the manufacturing accuracy of the lens of the camera in the electronic device and the deviation of the assembly process, resulting in distortion of the original image. The distortion of the lens is classified into radial distortion and tangential distortion. Distortion correction, the distortion coefficient of the camera is firstly known, and then the original image is transformed in a way opposite to the distortion, so that the distortion can be eliminated. The distortion coefficients have been obtained at the time of the aforementioned camera calibration.

The binocular calibration is the calibration of a binocular camera, and the calibration of the binocular camera not only needs to obtain the internal parameters of each camera, but also needs to measure the relative position between the two cameras (namely, the translation vector t and the rotation matrix R of the right camera relative to the left camera). And analyzing the camera calibration result in the TEE to obtain distortion correction information and binocular calibration information.

As shown in fig. 3(a), a camera calibration result is obtained in an REE environment for a conventional method, and is analyzed, and then the analyzed result is transmitted to a specific application or hardware for subsequent image processing.

As shown in fig. 3(b), in the method provided in this embodiment, the camera calibration result is obtained from the REE, and the obtained camera calibration result is transmitted to the TEE through the data channel between the REE and the TEE. And analyzing the camera calibration result in the TEE to obtain an analysis result, transmitting the analysis result to the REE through a data channel between the REE and the TEE, and processing the shot image in the REE by specific application or hardware according to the analysis result.

In the embodiment of the present application, the camera calibration result includes an internal parameter of the camera, an external parameter of the camera, and a distortion coefficient. When the image processing needs to be performed on the shot original image subsequently, the camera calibration result needs to be analyzed firstly. The traditional analysis process is directly carried out in the REE environment, the safety in the REE environment is not high, therefore, the process of analyzing the camera calibration result is placed in the TEE environment to be executed in the embodiment of the application, and the safety coefficient in the TEE environment is higher than that of the REE environment, so that the safety in the analysis process of the camera calibration result is ensured to a certain extent, and the camera calibration result is prevented from being mistakenly generated in the analysis process, so that the subsequent processing of the original image is influenced.

In one embodiment, the transmitting the parsing result to the REE through a data channel between the REE and the TEE, and the processing the image according to the parsing result in the REE includes:

and transmitting the distortion correction information and the binocular calibration information to the REE through a data channel between the REE and the TEE, and reconstructing and beautifying the shot image in the REE according to the distortion correction information and the binocular calibration information.

Specifically, after the distortion correction information and the binocular calibration information are obtained by analyzing the camera calibration result in the TEE environment, the distortion correction information and the binocular calibration information may be transmitted to the REE through a data channel between the REE and the TEE. Applications that need to use the distortion correction information and the binocular calibration information can be directly used in the REE environment, and for example, applications such as face unlocking, 3D beauty, 3D face payment authentication, and the like can directly use the distortion correction information and the binocular calibration information to further process the image. For example, applications such as face unlocking, 3D face beautification, 3D face payment authentication, and the like can reconstruct and beautify the photographed original image.

In the embodiment of the application, the whole process of analyzing the camera calibration result is performed in the REE, namely, the camera calibration result is performed in a safe environment, so that the risk of tampering and stealing data in the analyzing process is avoided. And then, the camera calibration result is transmitted from the TEE to the REE for use by a third party application and hardware equipment, so that the safety of the camera calibration result in the using and outputting processes is improved.

In one embodiment, the method includes transmitting the parsing result to the REE through a data channel between the REE and the TEE, and processing the image according to the parsing result in the REE, further including:

and transmitting the distortion correction information and the binocular calibration information to the REE through a data channel between the REE and the TEE, and correcting the depth information of the shot image in the REE according to the distortion correction information and the binocular calibration information.

In the embodiment of the application, after the shot original image is corrected according to the distortion correction information and the binocular calibration information, the corrected image is subjected to depth information calculation. Therefore, the depth information calculated directly according to the original image is corrected, and more accurate depth information is obtained.

In one embodiment, as shown in fig. 4, before obtaining the camera calibration result from the REE, the method includes:

step 420, calibrating the camera in the TEE to obtain a camera calibration result;

step 440, storing the camera calibration result in a safe area.

Specifically, the process of calibrating the camera to obtain the camera calibration result is also implemented in the TEE environment. And acquiring images of the calibration plate taken by the camera at different angles from the REE, and transmitting the acquired images to the TEE through a data channel between the REE and the TEE. And calibrating the camera in the TEE according to the acquired image to obtain a camera calibration result, outputting the camera calibration result to the REE through a data channel between the REE and the TEE, and storing the camera calibration result in a safe area. The secure area may be a persistence partition, where the persistence partition is generally used to store protected information, such as an account number and a password, used by an frp (factory reset) function, and may be prevented from being cleared after the factory setting is restored. Of course, the secure area may be other storage space.

In this embodiment, the original data, the intermediate data, and the calibration result in the camera calibration process are all run in the REE environment, and the camera calibration result is stored in a persistent partition, which is a storage space that is not used even when the electronic device is booted. Therefore, under the conditions that a user frequently uses similar applications with higher safety requirements such as face unlocking, 3D face beautifying, 3D face payment authentication and the like daily and electronic equipment is refreshed, the information safety in the camera calibration process, the camera calibration result analysis and the subsequent recalculation process is ensured, and data tampering and stealing caused by possible attack are avoided.

In one embodiment, resolving the camera calibration result in the TEE to obtain a resolved result includes:

acquiring a shot image;

processing the shot image in the TEE according to the analysis result to obtain a processing result;

and transmitting the processing result to the REE through a data channel between the REE and the TEE.

In this embodiment, the captured original image is acquired, the captured original image is processed in the TEE according to the analysis result to obtain a processing result, and the processing result is transmitted to the TEE through a data channel between the TEE and the REE. The shot original image is processed according to the analysis result, and the process of obtaining the processing result is also operated in the TEE environment, so that the data security is further improved. For example, for the application of face unlocking, after an original image of a photographed face is acquired, a camera calibration result is analyzed, and then, the processes of reconstructing and correcting the original image according to distortion correction information and binocular calibration information obtained by the analysis are all executed in the TEE environment. And finally, only the result of face unlocking needs to be output, for example, whether unlocking is successful or failed.

In one embodiment, there is also provided a control method, as shown in fig. 5, applied to an electronic device, the electronic device supporting a rich execution environment REE and a secure chip, the method including:

step 520, a camera calibration result is obtained from the REE.

And 540, transmitting the acquired camera calibration result to the security chip through a data channel between the REE and the security chip.

Se (secure element) is a security chip, and is a microcomputer, and functions such as data secure storage, encryption and decryption operations are realized through the security chip and a Chip Operating System (COS). There is a specific data path between the REE and SE so that data satisfying the condition can be transmitted between the REE and SE. The acquired camera calibration results are transmitted to the SE through a data channel between the REE and the SE. Therefore, the camera calibration result is analyzed in the SE in the subsequent process, all data information generated in the whole analysis process and the analysis process are operated in the TEE environment, the data security is ensured, and the risk of data attack is reduced.

And 560, analyzing the camera calibration result in the security chip to obtain an analysis result.

And analyzing the camera calibration result in the SE to obtain distortion correction information and binocular calibration information, namely rotation and translation information.

And 580, transmitting the analysis result to the REE through a data channel between the REE and the security chip, and processing the shot image according to the analysis result in the REE.

And transmitting the distortion correction information and the binocular calibration information acquired after analysis to the REE through a data channel between the REE and the SE. Applications that need to use the parsing result may include face unlocking, 3D beauty, and other applications that need to further process the image using the parsing result. Of course, some hardware using the parsed result may be included, for example, a 3D camera Processor (3D camera Processor). Because the analysis result is obtained after the analysis is carried out in the SE, and is transmitted to the REE through a data channel between the REE and the SE, and is used by applications such as face unlocking and 3D beauty or hardware such as a 3D camera processor in the REE environment, the accuracy and the safety of the data of the analysis result input into the applications and the hardware are ensured in the whole process.

In the embodiment of the application, the SE is used as hardware, the security of the SE is obviously higher than that of the TEE, and certainly higher than that of the REE, so that the whole process of analyzing the camera calibration result is performed in the SE, namely, in a secure environment, and the risk of tampering and stealing data in the analysis process is avoided.

In one embodiment, as shown in fig. 6, there is provided an image processing apparatus 600 including: a camera calibration result obtaining module 620, a transmission module 640, an analysis module 660, and an image processing module 680. Wherein the content of the first and second substances,

a camera calibration result obtaining module 620, configured to obtain a camera calibration result from the REE;

a transmission module 640, configured to transmit the obtained camera calibration result to the TEE through a data channel between the REE and the TEE;

the analysis module 660 is configured to analyze the camera calibration result in the TEE to obtain an analysis result;

and the image processing module 680 is configured to transmit the analysis result to the REE through a data channel between the REE and the TEE, and process the captured image in the REE according to the analysis result.

In an embodiment, the parsing module 660 is further configured to parse the camera calibration result in the TEE to obtain distortion correction information and binocular calibration information.

In one embodiment, the image processing module 680 is further configured to transmit the distortion correction information and the binocular calibration information to the REE through a data channel between the REE and the TEE, and reconstruct and beautify the captured image according to the distortion correction information and the binocular calibration information in the REE.

In one embodiment, the image processing module 680 is further configured to transmit the distortion correction information and the binocular calibration information to the REE via a data channel between the REE and the TEE, where the REE corrects the depth information of the captured image according to the distortion correction information and the binocular calibration information.

In one embodiment, as shown in fig. 7, there is provided an image processing apparatus 600 further comprising:

the camera calibration module 610 is configured to calibrate a camera in the TEE to obtain a camera calibration result; and storing the camera calibration result in a safe area.

In one embodiment, the image processing module 780 is further configured to obtain the captured image; processing the shot image in the TEE according to the analysis result to obtain a processing result; and transmitting the processing result to the REE through a data channel between the REE and the TEE.

In one embodiment, as shown in fig. 8, there is also provided an image processing apparatus 800 comprising: a camera calibration result obtaining module 820, a transmission module 840, an analysis module 860 and an image processing module 880. Wherein the content of the first and second substances,

a camera calibration result obtaining module 820, configured to obtain a camera calibration result from the REE;

a transmission module 840, configured to transmit the obtained camera calibration result to the security chip through a data channel between the REE and the security chip;

the analysis module 860 is used for analyzing the camera calibration result in the security chip to obtain an analysis result;

and the image processing module 880 is configured to transmit the analysis result to the REE through a data channel between the REE and the security chip, and process the captured image in the REE according to the analysis result.

The division of the modules in the image processing apparatus is only for illustration, and in other embodiments, the image processing apparatus may be divided into different modules as needed to complete all or part of the functions of the image processing apparatus.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when being executed by a processor, implements the steps of the image processing method provided by the above embodiments.

In one embodiment, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the steps of the image processing method provided in the above embodiments are implemented.

The embodiments of the present application also provide a computer program product, which when run on a computer, causes the computer to execute the steps of the image processing method provided in the foregoing embodiments.

The embodiment of the application also provides the electronic equipment. The electronic device may be any terminal device including a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a vehicle-mounted computer, a wearable device, and the like, taking the electronic device as the mobile phone as an example: the electronic device includes therein an Image processing circuit, which may be implemented using hardware and/or software components, and may include various processing units defining an ISP (Image signal processing) pipeline. FIG. 9 is a schematic diagram of an image processing circuit in one embodiment. As shown in fig. 9, for convenience of explanation, only aspects of the image processing technique related to the embodiments of the present application are shown.

As shown in fig. 9, the image processing circuit includes a first ISP processor 930, a second ISP processor 940 and a control logic 950. The first camera 910 includes one or more first lenses 912 and a first image sensor 914. First image sensor 914 may include an array of color filters (e.g., Bayer filters), and first image sensor 914 may acquire light intensity and wavelength information captured with each imaging pixel of first image sensor 914 and provide a set of image data that may be processed by first ISP processor 930. The second camera 920 includes one or more second lenses 922 and a second image sensor 924. The second image sensor 924 may include a color filter array (e.g., a Bayer filter), and the second image sensor 924 may acquire the light intensity and wavelength information captured with each imaged pixel of the second image sensor 924 and provide a set of image data that may be processed by the second ISP processor 940.

The first image collected by the first camera 910 is transmitted to the first ISP processor 930 for processing, after the first ISP processor 930 processes the first image, the statistical data (such as brightness of the image, contrast value of the image, color of the image, etc.) of the first image may be sent to the control logic 950, and the control logic 950 may determine the control parameter of the first camera 910 according to the statistical data, so that the first camera 99 may perform operations such as auto focus, auto exposure, etc. according to the control parameter. The first image may be stored in the image memory 960 after being processed by the first ISP processor 930, and the first ISP processor 930 may also read the image stored in the image memory 960 to process the image. In addition, the first image may be directly transmitted to the display 970 for display after being processed by the ISP processor 930, or the display 970 may read the image in the image memory 960 for display.

Wherein the first ISP processor 930 processes the image data pixel by pixel in a plurality of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the first ISP processor 930 may perform one or more image processing operations on the image data, collecting statistical information about the image data. Wherein the image processing operations may be performed with the same or different bit depth calculation accuracy.

The image Memory 960 may be a portion of a Memory device, a storage device, or a separate dedicated Memory within an electronic device, and may include a DMA (Direct Memory Access) feature.

Upon receiving an interface from first image sensor 914, first ISP processor 930 may perform one or more image processing operations, such as temporal filtering. The processed image data may be sent to image memory 960 for additional processing before being displayed. The first ISP processor 930 receives the processed data from the image memory 960 and performs image data processing in RGB and YCbCr color spaces on the processed data. The image data processed by the first ISP processor 930 may be output to a display 970 for viewing by a user and/or further processed by a Graphics Processing Unit (GPU). Further, the output of the first ISP processor 930 may also be sent to an image memory 960, and the display 970 may read image data from the image memory 960. In one embodiment, image memory 960 may be configured to implement one or more frame buffers.

The statistics determined by the first ISP processor 930 may be sent to the control logic 950. For example, the statistical data may include first image sensor 914 statistics such as auto-exposure, auto-white balance, auto-focus, flicker detection, black level compensation, first lens 912 shading correction, and the like. The control logic 950 may include a processor and/or microcontroller that executes one or more routines (e.g., firmware) that may determine control parameters of the first camera 910 and control parameters of the first ISP processor 930 based on the received statistical data. For example, the control parameters of the first camera 910 may include gain, integration time of exposure control, anti-shake parameters, flash control parameters, first lens 912 control parameters (e.g., focal length for focusing or zooming), or a combination of these parameters, and the like. The ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (e.g., during RGB processing), as well as first lens 912 shading correction parameters.

Similarly, a second image acquired by the second camera 920 is transmitted to the second ISP processor 940 for processing, after the second ISP processor 940 processes the first image, the second ISP processor 940 may send statistical data (such as brightness of the image, contrast value of the image, color of the image, and the like) of the second image to the control logic 950, and the control logic 950 may determine control parameters of the second camera 920 according to the statistical data, so that the second camera 920 may perform operations such as auto-focus and auto-exposure according to the control parameters. The second image may be stored in the image memory 960 after being processed by the second ISP processor 940, and the second ISP processor 940 may also read the image stored in the image memory 960 to process the image. In addition, the second image may be directly transmitted to the display 970 for display after being processed by the ISP processor 940, or the display 970 may read the image in the image memory 960 for display. The second camera 920 and the second ISP processor 940 may also implement the processes as described for the first camera 910 and the first ISP processor 930.

The following steps are performed to implement the image processing method using the image processing technique of fig. 9.

Any reference to memory, storage, database, or other medium used herein may include non-volatile and/or volatile memory. Suitable non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and bus dynamic RAM (RDRAM).

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.