阅读说明：本技术 基于卡尔曼滤波的视觉目标处理方法、装置、设备及介质 (Visual target processing method, device, equipment and medium based on Kalman filtering ) 是由 朱耀宇 马雷 郑雅菁 张羽 曹岗 黄铁军 于 2021-09-06 设计创作，主要内容包括：本申请公开了一种基于卡尔曼滤波的视觉目标处理方法、装置、设备及介质。该方法包括：将视觉目标的状态向量分解为两个分向量；根据两个分向量各自所对应的状态误差协方差矩阵,并行地对两个分向量进行卡尔曼滤波处理,分别对应得到一个滤波后分向量以及一个滤波后状态误差协方差矩阵,所述状态误差协方差矩阵为块对角矩阵；合并两个滤波后分向量得到视觉目标的滤波后状态向量。本申请的方法使相关矩阵简化为块对角矩阵并且并行地处理,降低了计算量,提高了计算效率。(The application discloses a visual target processing method, a visual target processing device and a visual target processing medium based on Kalman filtering. The method comprises the following steps: decomposing the state vector of the visual target into two component vectors; performing Kalman filtering processing on the two vectors in parallel according to state error covariance matrixes corresponding to the two vectors respectively, and correspondingly obtaining a filtered vector and a filtered state error covariance matrix respectively, wherein the state error covariance matrix is a block diagonal matrix; and combining the two filtered sub-vectors to obtain a filtered state vector of the visual target. The method simplifies the correlation matrix into the block diagonal matrix and processes the block diagonal matrix in parallel, reduces the calculation amount and improves the calculation efficiency.)

1. A visual target processing method based on Kalman filtering is characterized by comprising the following steps:

decomposing the state vector of the visual target into two component vectors;

performing Kalman filtering processing on the two partial vectors in parallel according to state error covariance matrixes corresponding to the two partial vectors respectively, and correspondingly obtaining a filtered partial vector and a filtered state error covariance matrix respectively, wherein the state error covariance matrix is a block diagonal matrix;

and combining the two filtered sub-vectors to obtain a filtered state vector of the visual target.

2. The method according to claim 1, wherein performing kalman filtering on the two eigenvectors in parallel according to the state error covariance matrices corresponding to the two eigenvectors respectively to obtain a filtered eigenvector and a filtered state error covariance matrix respectively comprises:

acquiring an updated state component vector of the first component vector by using the state transition matrix;

calculating an updated state error covariance matrix by using the state transition matrix, the state noise covariance matrix and the state error covariance matrix of the first component vector;

calculating a Kalman gain matrix by using the updated state error covariance matrix, the state observation matrix and the observation noise covariance matrix;

calculating a filtered component vector by using the updated state component vector, the Kalman gain matrix, the state observation matrix and a pre-acquired state observation value;

calculating a filtered state error covariance matrix by using the updated state error covariance matrix, the Kalman gain matrix and the state observation matrix;

wherein the first component vector is any one of the two component vectors; the state transition matrix, the observation noise covariance matrix, the state noise covariance matrix, and the state observation matrix are all preset.

3. The method of claim 2, wherein obtaining the updated state component vector of the first component vector using the state transition matrix comprises:

and multiplying the first component vector by the state transition matrix to obtain an updated state component vector.

4. The method of claim 2, wherein the updated state error covariance matrix is calculated using the state transition matrix, the state noise covariance matrix, and the state error covariance matrix of the first component vector, and wherein the calculation formula is:

the updated state error covariance matrix is state transition matrix x state error covariance matrix x state transition matrix transpose + state noise covariance matrix.

5. The method of claim 2, wherein the updated state error covariance matrix, state observation matrix, and observation noise covariance matrix are used to calculate a kalman gain matrix, wherein the calculation formula is:

kalman gain matrix (updated state error covariance matrix x state observation matrix transpose matrix x (state observation matrix x updated state error covariance matrix x state observation matrix transpose matrix + observation noise covariance matrix)^-1。

6. The method of claim 2, wherein the filtered component vectors are calculated using the updated state component vectors, the kalman gain matrix, the state observation matrix, and pre-acquired state observations, and wherein the formula is:

the filtered state partition vector is the updated state partition vector + kalman gain matrix x (state observation-state observation matrix x updated state partition vector).

7. The method of claim 2, wherein computing a filtered state error covariance matrix using the updated state error covariance matrix, the kalman gain matrix, and the state observation matrix comprises:

and (3) after filtering, the state error covariance matrix is the updated state error covariance matrix-Kalman gain matrix multiplied by the state observation matrix multiplied by the updated state error covariance matrix.

8. The method of claim 1, further comprising:

and respectively updating the state vector and the state error covariance matrix by using the filtered state vector and the filtered state error covariance matrix, and turning to decompose the state vector of the visual target into two component vectors until a preset end condition is reached.

9. A visual target processing device based on Kalman filtering is characterized by comprising:

the decomposition module is used for decomposing the state vector of the visual target into two component vectors;

the parallel processing module is used for performing Kalman filtering processing on the two vectors in parallel according to state error covariance matrixes corresponding to the two vectors respectively, and obtaining a filtered vector and a filtered state error covariance matrix correspondingly respectively, wherein the state error covariance matrix is a block diagonal matrix;

and the merging module is used for merging the two filtered sub-vectors to obtain a filtered state vector of the visual target.

10. The apparatus of claim 9, further comprising:

and the updating module is used for respectively updating the state vector and the state error covariance matrix by using the filtered state vector and the filtered state error covariance matrix, and turning to decompose the state vector of the visual target into two component vectors until a preset finishing condition is reached.

11. The apparatus of claim 9 or 10, wherein the parallel processing module comprises:

a state memory for storing the two partial vectors;

the state error covariance memory is used for storing state error covariance matrixes corresponding to the two vectors respectively;

the Kalman state updating circuit is used for receiving the two vectors from the state memory and the two state error covariance matrixes from the state error covariance memory, performing Kalman filtering processing on the two vectors in parallel, and respectively and correspondingly obtaining a filtered vector and a filtered state error covariance matrix;

the state memory is also used for storing two partial vectors which are obtained from the Kalman state updating circuit after Kalman filtering processing;

the state error covariance memory is further configured to store the two filtered state error covariance matrices from the kalman state update circuit.

12. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the kalman filter based visual target processing method of any one of claims 1-8.

13. A computer-readable storage medium, on which a computer program is stored, characterized in that the program is executed by a processor to implement the visual object processing method based on kalman filtering according to any one of claims 1 to 8.

Technical Field

The application relates to the technical field of computer vision, in particular to a visual target processing method, a device, equipment and a medium based on Kalman filtering.

Background

The Kalman filtering iteratively estimates a target observation matrix by calculating a system state matrix, and then completes tasks of estimation, prediction and the like of a target. The Kalman filtering technology is widely applied to the field of visual target tracking, and the development of the visual target tracking technology is powerfully promoted. The kalman filtering tracking technique involves matrix multiplication and inversion operations. The matrix operation has high calculation complexity and large resource occupation, and when the matrix dimension is increased, the calculation delay or the resource consumption can be rapidly increased. It is therefore necessary to optimize the calculation method of kalman filtering.

Disclosure of Invention

The application aims to provide a visual target processing method, a visual target processing device, a visual target processing equipment and a visual target processing medium based on Kalman filtering. The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed embodiments. This summary is not an extensive overview and is intended to neither identify key/critical elements nor delineate the scope of such embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.

According to an aspect of an embodiment of the present application, there is provided a visual target processing method based on kalman filtering, including:

decomposing the state vector of the visual target into two component vectors;

performing Kalman filtering processing on the two partial vectors in parallel according to state error covariance matrixes corresponding to the two partial vectors respectively, and correspondingly obtaining a filtered partial vector and a filtered state error covariance matrix respectively, wherein the state error covariance matrix is a block diagonal matrix;

and combining the two filtered sub-vectors to obtain a filtered state vector of the visual target.

In some embodiments of the present application, the performing kalman filtering processing on the two eigenvectors in parallel according to the state error covariance matrices corresponding to the two eigenvectors respectively to obtain a filtered eigenvector and a filtered state error covariance matrix respectively includes:

acquiring an updated state component vector of the first component vector by using the state transition matrix;

calculating an updated state error covariance matrix by using the state transition matrix, the state noise covariance matrix and the state error covariance matrix of the first component vector;

calculating a Kalman gain matrix by using the updated state error covariance matrix, the state observation matrix and the observation noise covariance matrix;

calculating a filtered component vector by using the updated state component vector, the Kalman gain matrix, the state observation matrix and a pre-acquired state observation value;

calculating a filtered state error covariance matrix by using the updated state error covariance matrix, the Kalman gain matrix and the state observation matrix;

wherein the first component vector is any one of the two component vectors; the state transition matrix, the observation noise covariance matrix, the state noise covariance matrix, and the state observation matrix are all preset.

In some embodiments of the present application, the obtaining the updated state component vector of the first component vector by using the state transition matrix includes:

and multiplying the first component vector by the state transition matrix to obtain an updated state component vector.

In some embodiments of the present application, the updated state error covariance matrix is calculated by using the state transition matrix, the state noise covariance matrix, and the state error covariance matrix of the first component vector, and the calculation formula is:

the updated state error covariance matrix is state transition matrix x state error covariance matrix x state transition matrix transpose + state noise covariance matrix.

In some embodiments of the present application, the kalman gain matrix is calculated by using the updated state error covariance matrix, the state observation matrix, and the observation noise covariance matrix, and the calculation formula is:

kalman gain matrix (updated state error covariance matrix x state observation matrix transpose matrix x (state observation matrix x updated state error covariance matrix x state observation matrix transpose matrix + observation noise covariance matrix)^-1。

In some embodiments of the present application, the updated state eigenvector, the kalman gain matrix, the state observation matrix, and the state observation value obtained in advance are used to calculate a filtered eigenvector, where the calculation formula is:

the filtered state partition vector is the updated state partition vector + kalman gain matrix x (state observation-state observation matrix x updated state partition vector).

In some embodiments of the present application, said calculating a filtered state error covariance matrix using said updated state error covariance matrix, said kalman gain matrix, and said state observation matrix comprises:

and (3) after filtering, the state error covariance matrix is the updated state error covariance matrix-Kalman gain matrix multiplied by the state observation matrix multiplied by the updated state error covariance matrix.

In some embodiments of the present application, the method further comprises:

and respectively updating the state vector and the state error covariance matrix by using the filtered state vector and the filtered state error covariance matrix, and turning to decompose the state vector of the visual target into two component vectors until a preset end condition is reached.

According to another aspect of the embodiments of the present application, there is provided a visual target processing apparatus based on kalman filtering, including:

the decomposition module is used for decomposing the state vector of the visual target into two component vectors;

the parallel processing module is used for performing Kalman filtering processing on the two vectors in parallel according to state error covariance matrixes corresponding to the two vectors respectively, and obtaining a filtered vector and a filtered state error covariance matrix correspondingly respectively, wherein the state error covariance matrix is a block diagonal matrix;

and the merging module is used for merging the two filtered sub-vectors to obtain a filtered state vector of the visual target.

In some embodiments of the present application, the apparatus further comprises:

and the updating module is used for respectively updating the state vector and the state error covariance matrix by using the filtered state vector and the filtered state error covariance matrix, and turning to decompose the state vector of the visual target into two component vectors until a preset finishing condition is reached.

In some embodiments of the present application, the parallel processing module comprises:

a state memory for storing the two partial vectors;

the state error covariance memory is used for storing state error covariance matrixes corresponding to the two vectors respectively;

the Kalman state updating circuit is used for receiving the two vectors from the state memory and the two state error covariance matrixes from the state error covariance memory, performing Kalman filtering processing on the two vectors in parallel, and respectively and correspondingly obtaining a filtered vector and a filtered state error covariance matrix;

the state memory is also used for storing two partial vectors which are obtained from the Kalman state updating circuit after Kalman filtering processing;

the state error covariance memory is further configured to store the two filtered state error covariance matrices from the kalman state update circuit.

According to another aspect of the embodiments of the present application, there is provided an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any one of the above-mentioned kalman filter-based visual target processing methods.

According to another aspect of embodiments of the present application, there is provided a computer-readable storage medium having a computer program stored thereon, where the computer program is executed by a processor to implement any one of the above-mentioned visual target processing methods based on kalman filtering.

The technical scheme provided by one aspect of the embodiment of the application can have the following beneficial effects:

according to the visual target processing method based on Kalman filtering, the state vector of the visual target is decomposed into two component vectors, the correlation matrix is simplified into the block diagonal matrix according to the state error covariance matrix corresponding to the two component vectors, and Kalman filtering processing is performed on the two component vectors in parallel, so that the calculated amount is reduced, the calculation parallelism is improved, the calculation complexity is reduced, and the calculation efficiency is improved.

Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the embodiments of the application, or may be learned by the practice of the embodiments. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 illustrates a Kalman filtering parallel computation flow diagram of one embodiment of the present application;

FIG. 2 illustrates a flow diagram of a visual target processing method based on Kalman filtering according to an embodiment of the present application;

FIG. 3 shows a flowchart of one implementation of step S20 of the embodiment shown in FIG. 2;

FIG. 4 shows a block diagram of a Kalman filtering based visual target processing apparatus according to an embodiment of the present application;

FIG. 5 illustrates a Kalman state observation parallel computation circuit architecture diagram of one embodiment of the present application;

FIG. 6 illustrates a Kalman state observation circuit pipeline computation architecture diagram of one embodiment of the present application;

FIG. 7 illustrates a Kalman state update parallel computing circuit architecture diagram of one embodiment of the present application;

FIG. 8 illustrates a Kalman state update circuit pipeline computation architecture diagram of one embodiment of the present application;

FIG. 9 shows a block diagram of an electronic device of an embodiment of the present application;

FIG. 10 shows a computer-readable storage medium schematic of an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is further described with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It will be understood by those within the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

In a two-dimensional visual sequence, a visual target can be modeled as a point, and the variables to be estimated include four variables of lateral coordinates, longitudinal coordinates, lateral velocity, and longitudinal velocity. The state variable of the target can be represented as a column vector x of four rows and one column; the covariance matrix P of the target state is a 4x4 matrix.

The prediction formula of the kalman filter can be expressed as formula (1):

wherein Q is a state noise covariance matrix (state noise matrix for short) of 4x4, F represents a target state transition matrix, k is an index of the kth time step (or iteration step), and F_kRepresenting the target state transition matrix at the kth time step (time kth),is represented by F_kThe transposed matrix of (2). x'_kFor the updated target state vector, P, of Kalman filtering_k' is the updated state error covariance matrix. x is the number of_k-1Is the optimal estimate of the target state (i.e., the output value of the kalman filter at time k-1) at time k-1 (time step k-1). Q_kRepresenting the state noise covariance matrix at time k.

Assuming that R is an observation noise covariance matrix (abbreviated as an observation noise matrix), H is a transfer matrix from a target state to an observation value (abbreviated as an observation matrix), and K is a kalman gain matrix, the optimal estimation and error correction method of the kalman filter is as shown in formula (2):

wherein z is_kIs the view of the target state at the k-th timeAnd (6) measuring. x'_kUpdated target state vector, x, for Kalman filtering_kFor optimal estimation of the target state at the kth time, P_kIs a corrected error covariance matrix. P'_kRepresenting the updated error covariance matrix. In a two-dimensional visual target tracking task, z_kThe target position detected by the detection algorithm, or the target position estimated by other algorithms besides the kalman filter, or the target position marked by human, etc. can be used.

F_k、Q_k、H_k、R_kThe four parameters are kept unchanged in the iteration process, and initial values of the four parameters can be set according to a specific application scene and can be obtained by debugging according to the specific application scene, namely, the four parameters are set according to experience. Noise Q in state prediction_kAnd observation noise R_kIn the indeterminate case, Q_k、R_kThe optimal value can be set as white gaussian noise, salt and pepper noise or other noise, or can be selected by multiple attempts.

In equations (1) and (2), all matrices are 4x4 in size (i.e., 4 rows and 4 columns) and all vectors are 4x1 in size (i.e., 4 rows and 1 column). Formula (2) also involves matrix inversion operations, which are computationally complex.

For two-dimensional object motion, the embodiment of the present application assumes that there is no correlation (i.e. the correlation coefficient is zero) between the two motion directions of the object, and then the corresponding covariance matrix can be divided into four matrix blocks with the size of 2 × 2, and the matrix block on the main diagonal is valid data, and the matrix block on the sub diagonal is an all-zero matrix. Through decoupling and matrix blocking operation, the embodiment of the application converts high-dimensional matrix multiplication and inversion operation into low-dimensional matrix multiplication and inversion operation, thereby reducing the calculation amount and achieving the effect of improving the calculation efficiency. In addition, the parallel circuit computing scheme provided by the embodiment of the application realizes the method, and the Kalman filtering speed and the energy efficiency can be further improved. The error caused by matrix decomposition is partially compensated by the noise covariance matrix (the embodiment of the application compensates by setting the initial value), and the other part is transmitted to the output end to influence the prediction result. In the applications of space experiments, spacecraft motion tracking and prediction and the like, the assumption is established, so that the scheme of the embodiment has great practical value.

The overall architecture of the kalman filter matrix blocking calculation method based on the uncorrelated hypothesis, which is proposed by the embodiment of the present application, is shown in fig. 1. And decomposing the initial state of the target into two blocks, respectively sending the two blocks into two paths of Kalman filtering modules with the same architecture, and finally combining the two paths of calculation results to output the overall state of the target. Meanwhile, the target state output at this time step is also the target state input at the next time step.

As shown in fig. 2, an embodiment of the present application provides a visual target processing method based on kalman filtering, including the following steps:

and S10, decomposing the state vector of the visual target into two component vectors.

The state vector of the visual target includes four variables of lateral coordinates, longitudinal coordinates, lateral velocity, and longitudinal velocity. The state variable of the target can be represented as a column vector x of four rows and one column; the covariance matrix P of the target state is a 4x4 matrix.

The state vector of the visual target can be decomposed into two components, namely a transverse component vector and a longitudinal component vector, and can also be decomposed into two components, namely a position component vector and a speed component vector.

For example, the state vector x of the visual target is (x)₁,v₁,x₂,v₂)^TX can be decomposed into two components of two rows and one column to obtain transverse componentAnd longitudinal componentOr x can be decomposed into two vectors of two rows and one column to obtain position vectorsAnd component of velocity componentThe present embodiment is described by taking the decomposition into the horizontal component vector and the vertical component vector as an example.

S20, according to the state error covariance matrixes corresponding to the two partial vectors respectively, Kalman filtering processing is conducted on the two partial vectors in parallel, a filtered partial vector and a filtered state error covariance matrix are obtained correspondingly, and the state error covariance matrix is a block diagonal matrix.

As shown in fig. 3, in some embodiments, step S20 includes:

s201, acquiring an updated state component vector of the first component vector by using the state transition matrix;

specifically, the updated state component vector is obtained by multiplying the state transition matrix by the first component vector. The first component vector is any one of the two component vectors and is represented ask is the index of the kth time step (or iteration step), or k may also be referred to as the number of iterations.

Updated state vector A state transition matrix representing the first component vector.

S202, calculating an updated state error covariance matrix by using the state transition matrix, the state noise covariance matrix and the state error covariance matrix of the first component vector;

specifically, the updated state error covariance matrix ═ state transition matrix × state error covariance matrix × state transition matrix transpose matrix + state noise covariance matrix.

Representing the updated state error covariance matrix,representing a state noise covariance matrix corresponding to the first component vector.RepresentsThe transposed matrix of (2).

S203, calculating a Kalman gain matrix by using the updated state error covariance matrix, the state observation matrix and the observation noise covariance matrix;

specifically, the kalman gain matrix is the updated state error covariance matrix × the state observation matrix transpose (the state observation matrix × the updated state error covariance matrix × the state observation matrix transpose + the observation noise covariance matrix)^-1。

A representation of a kalman gain matrix is given,representing a state observation matrix corresponding to the first component vector,representing an observed noise covariance matrix corresponding to the first component vector.RepresentsThe state observation matrix of (2).

S204, calculating a filtered component vector by using the updated state component vector, the Kalman gain matrix, the state observation matrix and a pre-acquired state observation value;

specifically, the post-filter segment vector is the post-update state segment vector + kalman gain matrix x (state observation-state observation matrix × post-update state segment vector).

Wherein z is_kThe state observed value of the target at the k-th time is acquired in advance.

S205, calculating a filtered state error covariance matrix by using the updated state error covariance matrix, the Kalman gain matrix and the state observation matrix;

specifically, the post-filter state error covariance matrix — the post-update state error covariance matrix — the kalman gain matrix × the state observation matrix × the post-update state error covariance matrix.

The state transition matrix, the observation noise covariance matrix, the state noise covariance matrix and the state observation matrix are all preset.

And S30, combining the two filtered sub-vectors to obtain a filtered state vector of the visual target.

And combining the two filtered partial vectors to form a vector, namely obtaining the filtered state vector of the visual target.

According to the visual target processing method based on Kalman filtering, the state vector of the visual target is decomposed into two component vectors, the correlation matrix is simplified into the block diagonal matrix according to the state error covariance matrix corresponding to the two component vectors, and Kalman filtering processing is performed on the two component vectors in parallel, so that the calculated amount is reduced, the calculation parallelism is improved, the calculation complexity is reduced, and the calculation efficiency is improved.

In some embodiments, the method of this embodiment further includes step S40, updating the state vector and the state error covariance matrix with the filtered state vector and the filtered state error covariance matrix, respectively, and turning to step S10 until reaching a preset end condition.

The preset ending condition may be set according to actual needs, for example, may be set as the number of updates, and when the preset number of updates is reached, the processing procedure is ended.

After decomposing the state vector x of the visual target into two vectors of two rows and one column, since there is no correlation between the two directions (horizontal and vertical), the state transition matrix F of the target can be simplified to a block diagonal matrix, i.e. the remaining elements are zero except for the four elements at the top left and the four elements at the bottom right. Similarly, under the assumption of the uncorrelated equation in this embodiment, the state error covariance matrix P of the target can also be simplified to be a block diagonal matrix. So equation (1) is converted into equation (3). Wherein the content of the first and second substances,a vector of two rows and one column represents a transverse or longitudinal (position and velocity) variable.To representError covariance matrix of the two variables. To preserve the properties of the block-diagonal matrix of P, the present embodiment further assumes that the state noise covariance matrix Q is also a block-diagonal matrix.

In the formula (3), the first and second groups,is an effective matrix block of the matrix Q. From the above assumptions, equation (2) can be further simplified to equation (4).

Based on the above assumptions, the embodiment of the present application further assumes that the transition process from the target state to the observed value is only related to the corresponding direction, and there is no correlation between the two directions, so H ∈ R^4x4Also simplified as a block diagonal matrix.Is one of two valid matrix blocks of H. Combining the above assumptions, the principle of matrix calculation can be known as follows: in the formula (2)Is also a block diagonal matrix. In order to further reduce the amount of calculation, the embodiment of the present application further assumes that the observation noise has irrelevancy between the two directions, i.e., the correlation result is zero. Then R degenerates into a block diagonal matrix.Is a valid data block in R.An active matrix block of K. Based on the above analysis, the 4x4 matrix inversion in equation (2) can be decomposed into two 2x2 matrix inverses. Therefore, the calculation amount of the matrix inverse operation of the Kalman filter is greatly reduced.Is a vector of position observations and velocity observations in one of the directions。

In this embodiment, the following are provided:

H_kis also fixed in the iterative updating process and is set asI.e., identity matrix, meaning: target State value (i.e., x 'in equation (2))'_k) And the observed value (i.e. z in equation (2))_k) Have the same reliability and direct correspondence.

F_kIs a fixed value and remains unchanged during the iteration process. F_kThe method is a modeling of motion, and in a short time, a target can be assumed to be in uniform linear motion, and then the initialization value at the moment isIn thatIn the upper right corner, the "1" can vary according to the motion assumption.

Specifically, a visual target processing method proposed by another embodiment of the present application includes the following steps:

the first step is as follows: initializing two state error covariance matricesAndinitializing two state transition matricesAndinitializing two observed noise covariance matricesAndinitializing two state noise covariance matricesAndinitializing a state error covariance matrix 1, a state transition matrix 2, an observation noise covariance matrix 3, a state noise covariance matrix 4 and a state observation transition matrix 5;

the second step is that: obtaining a target state vector x ═ x (x) for the visual target from the output of the detector or tracker at a previous time instant₁,v₁,x₂,v₂)^TAnd decomposing x into two vectors of two rows and one column to obtain transverse componentAnd longitudinal component

The third step: and according to a formula (3), performing Kalman filter state updating calculation on the two paths of target state vectors, the state noise covariance matrix, the state error covariance matrix and the state transition matrix in parallel to obtain an updated target state vector and an updated state error covariance matrix.

The fourth step: according to the formula (4), respectively calculating Kalman gain (namely a first formula of the formula (4)) for a state transition matrix from two paths of states to observation, an observation noise covariance matrix and an updated state error covariance matrix, estimating an optimal target state (namely a second formula of the formula (4)) by using the Kalman gain matrix and a target state observation vector, and finally correcting the state error covariance matrix (namely a third formula of the formula (4)).

The fifth step: and combining the two paths of target state vectors, outputting a tracking result, returning to the second step, and continuously executing the process until the tracking task is finished.

By the above irrelevance assumption, the calculation amount (or the calculation resource demand amount) of the kalman filter operation is greatly reduced. Under the condition of weak correlation, the formula (3) and the formula (4) hold. The method improves the calculation speed by utilizing the advantages of CPU multithreading or GPU multicore on a general calculation platform. The method is decoupled according to two dimensions of speed and position, and makes noises between the two irrelevant, and can also achieve the effects of replacing the whole matrix operation with the matrix block operation, thereby reducing the calculation amount, increasing the calculation parallelism and improving the calculation efficiency.

According to the visual target processing method based on Kalman filtering, the target state is decoupled according to the motion direction, so that a state error covariance matrix, a state transition matrix, a noise covariance matrix and a state-to-observation transition matrix are simplified into a block diagonal matrix, operations such as 4x4 matrix multiplication and inversion are simplified into multiplication and inversion operations of two 2x2 matrixes, the calculation amount can be reduced by partitioning the matrixes, and the calculation parallelism is improved.

Referring to fig. 4, another embodiment of the present application provides a kalman filter-based visual target processing apparatus, including:

the decomposition module is used for decomposing the state vector of the visual target into two component vectors;

the parallel processing module is used for performing Kalman filtering processing on the two vectors in parallel according to the state error covariance matrixes corresponding to the two vectors respectively, and correspondingly obtaining a filtered vector and a filtered state error covariance matrix respectively;

and the merging module is used for merging the two filtered sub-vectors to obtain a filtered state vector of the visual target.

In some embodiments, the apparatus of this embodiment further includes an updating module, configured to update the state vector and the state error covariance matrix with the filtered state vector and the filtered state error covariance matrix, respectively, and turn to the decomposition of the state vector of the visual target into two component vectors, which is stopped until a preset end condition is reached.

In certain embodiments, the visual target processing apparatus further comprises a storage circuit for storing the eigenvector, the updated state eigenvector, the state transition matrix, the state noise covariance matrix, the state error covariance matrix, the updated state error covariance matrix, the state observation matrix, the observation noise covariance matrix, the kalman gain matrix, the state observations, the filtered eigenvector, and the filtered state error covariance matrix.

In some embodiments, a parallel processing module comprises:

the first calculation unit is used for acquiring the updated state component vector of the first component vector by using the state transition matrix;

the second calculation unit is used for calculating an updated state error covariance matrix by using the state transition matrix, the state noise covariance matrix and the state error covariance matrix of the first component vector;

the third calculation unit is used for calculating a Kalman gain matrix by using the updated state error covariance matrix, the state observation matrix and the observation noise covariance matrix;

a fourth calculating unit, configured to calculate a filtered component vector using the updated state component vector, the kalman gain matrix, the state observation matrix, and a state observation value acquired in advance;

a fifth calculating unit, configured to calculate a filtered state error covariance matrix using the updated state error covariance matrix, the kalman gain matrix, and the state observation matrix;

wherein the first component vector is any one of the two component vectors; the state transition matrix, the observation noise covariance matrix, the state noise covariance matrix, and the state observation matrix are all preset.

In some embodiments, a parallel processing module comprises:

a state memory for storing the two partial vectors;

the state error covariance memory is used for storing state error covariance matrixes corresponding to the two vectors respectively;

the Kalman state updating circuit is used for receiving the two vectors from the state memory and the two state error covariance matrixes from the state error covariance memory, performing Kalman filtering processing on the two vectors in parallel, and respectively and correspondingly obtaining a filtered vector and a filtered state error covariance matrix;

the state memory is also used for storing two partial vectors which are obtained from the Kalman state updating circuit after Kalman filtering processing;

the state error covariance memory is further configured to store the two filtered state error covariance matrices from the kalman state update circuit.

Specifically, in an implementation manner of this embodiment, referring to fig. 5 to 8, the parallel processing module includes a state error covariance memory ram _ p, a buffer _ rp1, a buffer _ rp2, a state memory ram _ x, a buffer _ rx1, a buffer _ rx2, a buffer _ ox1, and a buffer _ ox 2;

the state error covariance matrices corresponding to the two vectors are stored in the state error covariance memory ram _ p in fig. 7, and are read out by the control logic before the kalman filtering process is started, and sent to the buffer _ rp1 and the buffer _ rp 2;

the two vectors are stored in the state memory ram _ x of fig. 7, and the control logic reads out the two vectors before the kalman filtering process is started, and sends the two vectors to the buffer _ rx1 and the buffer _ rx 2.

The filtered partial vectors are firstly buffered in a buffer _ ox1 and a buffer _ ox2 respectively and then sent into a state memory ram _ x under the control of a control logic;

the Kalman state update circuit card Kalman filtering in FIG. 7 obtains filtered state error covariance matrices, which are buffered in buffer _ op1 and buffer _ op2, respectively, and then fed into a state error covariance memory ram _ p under the control of control logic.

The first calculation unit is used for acquiring the updated state component vector of the first component vector by using the state transition matrix.

The first component vector is stored in buffer _ rx1 or buffer _ rx2 in fig. 7.

The state transition matrix is stored in the state update transition matrix memory IF1 or the state update transition matrix memory IF2 in fig. 7.

The control logic reads the state transition matrix into buffer _ if1 or buffer _ if 2. The buffer _ if1 and the buffer _ if2 in fig. 7 correspond to the buffer _ if buffer in fig. 8.

The updated state component vectors are buffered in the buffer _ ox1 and the buffer _ ox2 in fig. 7, respectively, and then input into the updated state memory ram _ x.

The second calculation unit is used for calculating an updated state error covariance matrix by using the state transition matrix, the state noise covariance matrix and the state error covariance matrix of the first component vector.

The state error covariance matrix before updating is stored in the state error covariance memory ram _ p in fig. 7, and then buffered in the buffer _ rp1 and the buffer _ rp2, and then respectively input into the kalman state update circuit #1 and the kalman state update circuit #2 modules. The structures of the kalman state updating circuit #1 and the kalman state updating circuit #2 are completely consistent, and the kalman state updating circuit is uniformly expressed below.

The details of the kalman state update circuit in fig. 7 are shown in fig. 8. The buffer _ rp in fig. 8 corresponds to the buffer _ rp1 or the buffer _ rp2 in fig. 7; similarly, buffer _ rx in FIG. 8 corresponds to buffer _ rx1 or buffer _ rx2 in FIG. 7.

The memory of the state noise covariance matrix is the state noise covariance memory IQ1 or the state noise covariance memory IQ2 in fig. 7, the state noise covariance matrix is buffered by the buffer _ IQ1 and the buffer _ IQ2 and then input to the kalman state updating circuit, and the buffer _ IQ in fig. 8 corresponds to the buffer _ IQ1 or the buffer _ IQ2 in fig. 7.

The updated state error covariance matrix is buffered by the buffer _ op1 and the buffer _ op2 in fig. 7 and then input into the updated error covariance memory ram _ p. The buffer _ op in FIG. 8 corresponds to buffer _ op1 or buffer _ op2 in FIG. 7.

And the third calculation unit is used for calculating a Kalman gain matrix by using the updated state error covariance matrix, the state observation matrix and the observation noise covariance matrix.

The observation noise covariance matrix is stored in an observation noise covariance memory IR1 in fig. 5, buffered by a buffer _ IR1, and then input to a kalman state observation circuit # 1; similarly, the observation noise covariance matrix of another component vector is stored in the observation noise covariance memory IR2 of fig. 5, buffered by the buffer _ IR2, and then input to the kalman state observation circuit # 2. The buffer _ ir1 and the buffer _ ir2 in fig. 5 are collectively denoted as buffer _ ir in fig. 6.

The state observation matrix is stored in the state-to-observation transition matrix memory IH1 in fig. 5, and is input to the kalman state observation circuit #1 after being buffered in the buffer _ IH1, and the state observation matrix of the other vector component is stored in the state-to-observation transition matrix memory IH2, and is input to the kalman state observation circuit #2 after being buffered in the buffer _ IH 2. Buffer _ ih1 and buffer _ ih2 are collectively shown as buffer _ ih in fig. 6.

In fig. 5, the memory corresponding to the updated state error covariance matrix is an error covariance memory ram _ p, which is equivalent to the updated error covariance memory ram _ p in fig. 7, and is buffered by a buffer _ rp1 and a buffer _ rp2, and then input to the kalman state observation circuit #1 and the kalman state observation circuit # 2. The buffer _ rp1 and the buffer _ rp2 are represented in fig. 6 as buffer _ rp. The circuit details of the kalman state observation circuit #1 and the kalman state observation circuit #2 in fig. 5 are shown in fig. 6.

The fourth calculating unit is configured to calculate a filtered component vector by using the updated state component vector, the kalman gain matrix, the state observation matrix, and a state observation value acquired in advance.

The updated state component vector corresponds to buffer _ rx1 and buffer _ rx2 in fig. 5, and also corresponds to buffer _ rx in fig. 6. The kalman gain matrix is the value of the k0 buffer in fig. 6. The observed value of the state of the target at the k-th time is the value of buffer _ zs 0. The filtered vector is the value of buffer _ ox in fig. 6.

And the fifth calculation unit is used for calculating a filtered state error covariance matrix by using the updated state error covariance matrix, the Kalman gain matrix and the state observation matrix.

The updated state error covariance matrix is the value of buffer _ rp in fig. 6. The filtered state error covariance matrix is the value of buffer _ op in the buffer of fig. 6. The buffer _ op and the buffer _ ox in fig. 6 correspond to the buffer _ op1, the buffer _ op2, the buffer _ ox1 and the buffer _ ox2 in fig. 5, respectively, and the corrected error covariance memory ram _ p and the optimal state estimation memory ram _ x are input under the control of the control logic.

The updating module is used for respectively updating the state vector and the state error covariance matrix by using the filtered state vector and the filtered state error covariance matrix.

The updated error covariance memory ram _ p in fig. 7 is equivalent to the error covariance memory ram _ p in fig. 5; the updated state memory ram _ x in fig. 7 is equivalent to the state memory ram _ x in fig. 5; the corrected error covariance memory ram _ p in fig. 5 is equivalent to the error covariance memory ram _ p in fig. 7, and the optimum state estimation memory ram _ x in fig. 5 is equivalent to the state memory ram _ x in fig. 7.

For multiple visual targets, one embodiment of the present application matches a set of Kalman system matrices for each target. These matrix data are stored in ram. In order to fully utilize parallelism and improve calculation energy efficiency of the embodiment of the present application, the embodiment of the present application provides a kalman filter parallel calculation circuit shown in fig. 5 to implement the function shown in formula (4). For one target, the control circuit reads two matrixes with the size of 2x2 from the error covariance memory, and the matrixes are respectively sent to the two Kalman filtering circuits after being buffered by buffer _ rp. The state error covariance matrices fed into the two calculation circuits are not necessarily the same, and may be different. Similarly, the control circuit reads the data of the state memory twice, and the data are respectively sent to the two Kalman state observation circuit calculation modules after being cached by the buffer _ rx. The transverse coordinate memory and the longitudinal coordinate memory store the observed value at the current moment, and the data are respectively sent to the first Kalman state observation circuit calculation module and the second Kalman state observation circuit calculation module after being cached by buffer _ cx and buffer _ cy. The circuits of the two Kalman state observation computation modules are identical, but the input data is not necessarily the same. Similarly, the control circuit buffers the data of the observation noise covariance memory through the buffer _ ir and the data of the observation transfer matrix memory from the state to the buffer _ ih and then respectively sends the buffered data to the Kalman state observation circuit module. Each state observation circuit module outputs the corrected state error covariance matrix buffer _ op and the state optimal estimation vector buffer _ ox, and the state error covariance matrix buffer _ op and the state optimal estimation vector buffer _ ox are respectively written into an output covariance memory and an output state memory under the drive of the control circuit. In practical applications, the observation noise covariance memories IR1 and IR2 may be combined into the same memory; similarly, state-to-observation transfer matrix memories IH1 and IH2 may be combined into the same memory; similarly, the input state memory, the output state memory, the input error covariance memory, and the output error covariance memory may be replaced with two memories.

The kalman filter state to output (i.e., equation (4)) computation circuit architecture is shown in fig. 6. The whole module is divided into three parts of a control circuit, an arithmetic circuit and a register circuitAnd (4) partial. The control circuit controls the operation of each sub-component, and the embodiment of the application omits the connection between the control circuit and each sub-component for the sake of drawing clarity. In the figure, the position of the upper end of the main shaft,shown on the left side of the operator symbol,shown to the right of the corresponding operator. Addition and subtraction are independent of inversion sign and position, so they are not labeled.

The buffer _ ih matrix is transposed and then sent into a buffer _ iht buffer, and then the buffer _ iht buffer matrix is multiplied by the buffer _ rp matrix, so that the result is tmp 0. tmp0 feeds into two branches: one of the two branches is multiplied by buffer _ ih, then added with the matrix of buffer _ ir, and multiplied by the matrix tmp0 of the other branch after inversion operation, and finally the result is sent to a k0 buffer. The buffer _ rp matrix is subtracted by the product of k0 and the three matrices, buffer _ ih and buffer _ rp, and the difference is fed into the buffer _ op buffer.

In addition, the data of the buffer _ cx is sent to the buffer _ z0 for buffering, then the product of the buffer _ ih and the buffer _ rx is subtracted from the buffer _ z0, the obtained difference value is multiplied by k0, and then the product is added with the buffer _ rx and sent to the buffer _ ox for outputting. In fig. 6, inv denotes the inverse operation of the 2 × 2 matrix. Note that in order to increase the clock frequency of the circuit, one or more registers/buffers may be added between or inside the operators, and the registers/buffers are omitted in the embodiments of the present application for image clarity.

The parallel computing circuit architecture of equation (3) is shown in fig. 7. The control circuit reads the state error covariance matrix stored in blocks from the state error memory, and the state error covariance matrix is respectively sent to two completely consistent Kalman state updating circuits after being cached; similarly, the state vector is also sent to two Kalman state updating circuits after being blocked. Meanwhile, the control circuit respectively sends the state updating transition matrix and the state noise covariance matrix into the two Kalman state updating circuits through caching. And finally, after caching the calculation result of the Kalman state updating circuit, respectively writing back the calculation result to the error covariance memory and the state memory.

The computing architecture of the kalman state update circuit sub-module shown in fig. 7 is shown in fig. 8. The circuit expands each operator in the formula (3) into a calculation circuit, thereby conveniently executing pipelining operation and greatly improving the calculation efficiency. The buffer _ if is transposed and buffered in the buffer _ ift, then the buffer _ ift is multiplied by the buffer _ rp, the product is multiplied by the buffer _ if, and then added with the buffer _ iq and sent into the buffer _ op buffer. The product of the buffer _ rx and the buffer _ if is directly fed into the buffer _ ox buffer. The whole calculation process works under the control of the control circuit.

The Kalman filtering calculation circuit in the embodiment of the application can realize the parallelization calculation scheme, and can remove calculation components of general processors such as instruction fetching, decoding, out-of-order execution, cache caching and the like, so that the energy consumption in the calculation process is saved.

Another embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the kalman filter-based visual target processing method according to any one of the above embodiments.

As shown in fig. 9, the electronic device 10 may include: the system comprises a processor 100, a memory 101, a bus 102 and a communication interface 103, wherein the processor 100, the communication interface 103 and the memory 101 are connected through the bus 102; the memory 101 stores a computer program that can be executed on the processor 100, and the processor 100 executes the computer program to perform the method provided by any of the foregoing embodiments of the present application.

The Memory 101 may include a high-speed Random Access Memory (RAM), and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used.

The bus 102 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 101 is used for storing a program, and the processor 100 executes the program after receiving an execution instruction, and the method disclosed in any of the foregoing embodiments of the present application may be applied to the processor 100, or implemented by the processor 100.

Processor 100 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 100. The Processor 100 may be a general-purpose Processor, and may include a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101 and completes the steps of the method in combination with the hardware.

The electronic device provided by the embodiment of the application and the method provided by the embodiment of the application have the same inventive concept and have the same beneficial effects as the method adopted, operated or realized by the electronic device.

Another embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, where the program is executed by a processor to implement the kalman filter based visual target processing method of any one of the above embodiments. Referring to fig. 10, a computer-readable storage medium is shown as an optical disc 20, on which a computer program (i.e., a program product) is stored, and when the computer program is executed by a processor, the computer program performs the method provided by any of the foregoing embodiments.

It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory, or other optical and magnetic storage media, which are not described in detail herein.

The computer-readable storage medium provided by the above-mentioned embodiments of the present application and the method provided by the embodiments of the present application have the same advantages as the method adopted, executed or implemented by the application program stored in the computer-readable storage medium.

It should be noted that:

the term "module" is not intended to be limited to a particular physical form. Depending on the particular application, a module may be implemented as hardware, firmware, software, and/or combinations thereof. Furthermore, different modules may share common components or even be implemented by the same component. There may or may not be clear boundaries between the various modules.

The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose devices may also be used with the examples based on this disclosure. The required structure for constructing such a device will be apparent from the description above. In addition, this application is not directed to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present application as described herein, and any descriptions of specific languages are provided above to disclose the best modes of the present application.

It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

The above-mentioned embodiments only express the embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.