阅读说明：本技术 确定可移动的设备的位置 (Determining the position of a mobile device ) 是由 王森 罗纳德·克拉克 尼基·特里戈尼 于 2018-01-17 设计创作，主要内容包括：一种确定包括摄像头的可移动的设备的位置的计算机执行的方法。该方法包括以下步骤：使用摄像头捕获一段时间内的图像序列；对于来自图像序列的连续图像对,使用第一神经网络确定表明设备的运动的特征,该设备的运动在捕获该图像对的第一图像的时间与捕获该图像对的第二图像的时间之间；对于连续图像序列,使用第二神经网络从由第一神经网络确定的特征确定表明设备的位置的特征；以及对于连续图像序列,从由第二神经网络确定的特征确定设备的位置。(A computer-implemented method of determining a position of a movable device including a camera. The method comprises the following steps: capturing a sequence of images over a period of time using a camera; for successive pairs of images from the sequence of images, determining features indicative of motion of the device using a first neural network, the motion of the device being between a time of capturing a first image of the pair of images and a time of capturing a second image of the pair of images; for a sequence of consecutive images, determining features indicative of a location of the device from the features determined by the first neural network using the second neural network; and for a sequence of consecutive images, determining a location of the device from the features determined by the second neural network.)

1. A computer-implemented method of determining a position of a mobile device including a camera, comprising the steps of:

capturing a sequence of images over a period of time using the camera;

determining, using a first neural network, features indicative of motion of the device for successive pairs of images from the sequence of images, the motion of the device being between a time of capturing a first image of the pair of images and a time of capturing a second image of the pair of images;

for a sequence of consecutive images, determining features indicative of a location of the device from features determined by the first neural network using a second neural network; and

for a sequence of consecutive images, the location of the device is determined from the features determined by the second neural network.

2. The method of claim 1, wherein the orientation of the device is determined in addition to the location of the device.

3. Method according to claim 1 or 2, characterized in that the images of the image sequence are monocular images.

4. The method of any preceding claim, wherein the first neural network is a convolutional neural network.

5. The method of any preceding claim, wherein the second neural network is a recurrent neural network.

6. The method of claim 5, wherein the second neural network is a long-short term memory neural network.

7. The method according to any of the preceding claims, further comprising the steps of: for each pair of successively captured images, relative position information and orientation information of the device is determined from the features determined by the second neural network.

8. The method of claim 7, wherein the step of determining the location of the device comprises: integrating the relative position information and the orientation information determined from the features determined by the second neural network.

9. The method according to claim 7 or 8, further comprising the steps of: for each pair of successively captured images, respective uncertainty information of the relative position information and the orientation information is determined.

10. The method according to any of the preceding claims, wherein each image of the sequence of images has associated with it respective position information (and orientation information), and the method further comprises the steps of: the first and second neural networks are trained using the respective location information (and orientation information).

11. The method of any preceding claim, wherein the device is an autonomous robot.

12. A mobile device, comprising:

a memory;

a processor;

a camera;

wherein the apparatus is arranged to:

capturing a sequence of images over a period of time using the camera;

for successive image pairs from the sequence of images, determining features indicative of motion of the device using a first neural network provided by the processor, the motion of the device being between a time of capturing a first image of the image pair and a time of capturing a second image of the image pair;

for a sequence of consecutive images, determining features indicative of a location of the device from features determined by the first neural network using a second neural network provided by the processor; and

for a sequence of consecutive images, the location of the device is determined from the features determined by the second neural network.

13. A device as claimed in claim 12, characterized in that the device is arranged to determine the orientation of the device in addition to the position of the device.

14. Device according to claim 12 or 13, characterized in that the images of the image sequence are monocular images.

15. The apparatus of any one of claims 12 to 14, wherein the first neural network is a convolutional neural network.

16. The apparatus of any one of claims 12 to 15, wherein the second neural network is a recurrent neural network.

17. The apparatus of claim 16, wherein the second neural network is a long-short term memory neural network.

18. The apparatus of any of claims 12 to 17, further arranged to: for each pair of successively captured images, relative position information and orientation information of the device is determined from the features determined by the second neural network.

19. The apparatus of claim 18, further arranged to: determining a location of the device by integrating the relative location information and the orientation information determined from the features determined by the second neural network.

20. The apparatus of claim 18 or 19, further arranged to: for each pair of successively captured images, respective uncertainty information of the relative position information and the orientation information is determined.

21. A computer program product arranged to implement the method of any one of claims 1 to 11 when executed on a mobile device.

22. A computer program product arranged to provide a transportable device according to any one of claims 12 to 20 when executed on a transportable device.

Technical Field

The invention relates to determining the position of a movable device. More particularly, but not exclusively, the invention relates to determining the position of a moveable device from images captured by a camera of the moveable device using a neural network.

The invention is particularly, but not exclusively, applicable where the movable apparatus is an autonomous robot. However, the invention is also applicable to other types of mobile and wearable devices, such as mobile phones, smart watches, etc.

"location" as discussed herein may refer to an absolute location, such as a location on earth defined by latitude and longitude, of a mobile device, and may also refer to a relative location with respect to another location (e.g., a distance and direction of the mobile device from an initial starting location). The determination of position also often includes a determination of orientation, for example relative to the absolute value of the earth's magnetic field, and relative to an initial orientation by a certain amount of rotation.

Background

It is desirable to be able to determine the location of a mobile device in situations where GPS signals are not available. This is especially true for autonomous robots, to allow accurate navigation. A known method is to use images from a camera in order to determine the position. However, such systems often require very accurate calibration of the camera if position is to be determined reliably. Conventional visual ranging techniques include sparse methods (including several steps including detection and matching of features, motion estimation and optimization), and direct methods (including steps of motion estimation and optimization). Such techniques tend to require accurate camera calibration and often fail in poorly textured environments (i.e., where there are few features) or when the camera that is capturing the image is rotating rapidly. Additionally, while such systems are generally capable of determining the shape of the path traveled, they are generally unable to estimate a dimension, i.e., the actual distance traveled.

Alternatively, it is known to use neural networks to process images from cameras in order to determine position. "DeepVO: a Deep Learning method for Monocular Visual ranging (published 18.11.2016 in preprint website (arXiv: 1611.06069)) discloses such a system. However, there are various problems with known systems using neural networks. They often need to be trained on the specific environment in which they are to be used and therefore cannot be used in new environments without first being properly trained.

The present invention seeks to alleviate the above problems. Alternatively and/or additionally, the present invention seeks to provide an improved method of determining the position of a moveable device.

Disclosure of Invention

According to a first aspect of the present invention there is provided a computer-implemented method of determining the position of a moveable device comprising a camera, the method comprising the steps of:

capturing a sequence of images over a period of time using a camera;

for successive pairs of images from the sequence of images, determining features indicative of motion of the device using a first neural network, the motion of the device being between a time of capturing a first image of the pair of images and a time of capturing a second image of the pair of images;

for a sequence of consecutive images, determining features indicative of a location of the device from the features determined by the first neural network using the second neural network; and

for a sequence of consecutive images, the location of the device is determined from the features determined by the second neural network.

By using a combination of two neural networks, it has been found that a much more robust and more reliable position determination is possible. In particular, the first neural network may be trained to most efficiently determine features from the images that indicate motion implied by differences between the images, which motion depends only on the two images and does not depend on historical information (e.g., previously determined locations). However, the second neural network may be trained at the same time to most efficiently determine the location of the mobile device from the features determined by the first neural network for which historical information (e.g., previously determined locations) is very useful. By splitting the processing into two neural networks in this way, training for both step-wise motion and overall position can be effectively achieved. Furthermore, two neural networks may be trained simultaneously by training the overall system, so in particular the first neural network may be trained to determine any motion features that are best for operation of the overall system, rather than training the first neural network to determine motion features having preselected attributes that may not actually be the best type of features to use.

Preferably, in addition to determining the location of the device, the orientation of the device is also determined. Thus, the "pose" of the device is determined.

Preferably, the images of the image sequence are monocular images.

Advantageously, the first neural network is a convolutional neural network. This type of neural network is particularly suitable for operating on data with a large number of parameters, such as image data.

Advantageously, the second neural network is a recurrent neural network. In this case, preferably, the second neural network is a long-short term memory neural network. Recurrent neural networks, particularly of the long-short term memory type, are particularly suitable for operating on time-dependent data.

Preferably, the method further comprises the step of determining, for each pair of successively captured images, relative position information and orientation information of the device from the features determined by the second neural network. In this case, preferably, the step of determining the location of the device comprises integrating relative location information and orientation information determined from the features determined by the second neural network. In other words, the position of the device is determined on a range from continuous motion estimation.

Advantageously, the method further comprises the step of determining, for each pair of successively captured images, respective uncertainty information of the relative position information and the orientation information. Uncertainty information may be used with the pose information as an input to a Simultaneous Localization And Mapping (SLAM) algorithm.

Each image of the sequence of images may have associated with it respective location information, and the method may further comprise the step of training the first and second neural networks using the respective location information. Preferably, each image has also associated with it orientation information.

The device may be an autonomous robot. The device may alternatively be a mobile phone, a wearable device or any other suitable mobile device.

According to a second aspect of the present invention, there is provided a mobile device comprising:

a memory;

a processor;

a camera;

wherein the apparatus is arranged to:

capturing a sequence of images over a period of time using a camera;

for successive pairs of images from the sequence of images, determining features indicative of motion of the device between a time of capturing a first image of the pair of images and a time of capturing a second image of the pair of images using a first neural network provided by the processor;

for a sequence of consecutive images, determining features indicative of a location from the features determined by the first neural network using a second neural network provided by the processor; and

for a sequence of consecutive images, the location of the device is determined from the features determined by the second neural network.

Preferably, the device is arranged to determine the orientation of the device in addition to the position of the device.

Preferably, the images of the image sequence are monocular images.

Advantageously, the first neural network is a convolutional neural network.

Advantageously, the second neural network is a recurrent neural network. In this case, preferably, the second neural network is a long-short term memory neural network.

Preferably, the device is further arranged to determine, for each pair of successively captured images, relative position information and orientation information of the device from the features determined by the second neural network. In this case, preferably, the device is arranged to determine the position of the device by integrating relative position information and orientation information determined from the features determined by the second neural network.

Advantageously, the device is further arranged to determine, for each pair of successively captured images, respective uncertainty information for the relative position information and the orientation information.

According to a third aspect of the present invention there is provided a computer program product arranged to implement any of the methods described above when executed on a removable device.

According to a fourth aspect of the present invention there is provided a computer program product arranged to provide any of the removable devices described above when executed on a removable device.

It will of course be appreciated that features described in relation to one aspect of the invention may be incorporated into other aspects of the invention. For example, the method of the invention may incorporate any of the features described with reference to the mobile device of the invention, and vice versa.

Drawings

Embodiments of the invention will now be described, by way of example only, with reference to the accompanying schematic drawings in which:

FIG. 1 is a schematic illustration of an autonomous robot according to an embodiment of the invention;

FIG. 2 is a flow chart illustrating the operation of the autonomous robot of FIG. 1 to estimate its position;

FIG. 3 is a schematic diagram illustrating the process of FIG. 2; and

fig. 4 is a schematic diagram of an LSTM used in the process of fig. 2 and 3.

Detailed Description

A schematic view of an autonomous robot according to an embodiment of the invention is shown in fig. 1. The autonomous robot 1 comprises a processor 2. It should be understood that in various embodiments, processor 2 may be a single processor system, a dual processor system, or any other suitable processor system. The processor 2 is in communication with the camera 3 and a memory 4, the memory 4 storing images (and other content) captured by the camera 3.

The operation of the autonomous robot 1 to determine its position will now be described with reference to the flowchart of fig. 2. This process is also schematically illustrated in fig. 3. At each time step, the autonomous robot 1 determines its position based on the information currently available to it. Fig. 3 shows three consecutive time steps t, t +1 and t +2, wherein for each time step the "Pose (position)" of the autonomous robot 1 is determined, which is respectively the position _t、Pose _t+1And Pose _t+2Where the pose is a combination of position information and orientation information (i.e. the direction the autonomous robot is facing).

First, pairs of images continuously captured by the camera 3 are obtained (step 21, part 31 of fig. 3). Each image is then pre-processed by subtracting the average RGB channel values from the example image set (step 22, part 32 of fig. 3). The set of images may be, for example, images of a trained autonomous robot 1 as described in detail below. Further, the size of the image is adjusted to a multiple of 64. However, such pre-processing is optional and may not occur in other embodiments. As can be seen in FIG. 3, for time step t, the initial image pair produces a pre-processed image RGB _tAnd RGB _t+1And for time step t +1, the preprocessed image is RGB _t+1And RGB _t+2And so on for other time steps.

A Convolutional Neural Network (CNN) implemented by the processor 2 and the memory 4 acquires the pair of preprocessed images and uses the pair of preprocessed images to determine features (step 23, part 33 of fig. 3). The CNN determines the features from its training, which has been performed as described below.

CNN is a neural network that incorporates convolutional layers in a network structure, and thus, in sharp contrast to the fully connected layers used in other types of neural networks, CNN can exploit the spatial regularity of data. This means that the number of parameters required for CNN is significantly reduced, allowing them to operate on high dimensional input (e.g. raw image data). In CNN, multiple convolution operations are applied at each convolution layer to determine multiple features from the output map of the previous layer. The filter kernel convolved with the mapping is learned during training, as described, for example, in [38 ].

CNN takes as input the tensor generated by stacking the preprocessed pairs of consecutive images. CNN consists of 9 convolutional layers, each followed by a rectifying Linear Unit (ReLU) nonlinear activation, except for the last convolutional layer, giving a total of 17 layers. The layer configuration is as follows:

the size of the receptive field in the network is gradually reduced from 7 x 7 to 5 x 5 and then to 3 x 3 to capture the small features of interest. The zero padding is introduced either to adapt the configuration of the receptive field or to preserve the spatial dimension of the tensor after convolution. The number of channels (i.e., the number of filters used for feature detection) is increased to learn various features.

In this embodiment, CNN has 5500 thousand trainable weights, but it should be understood that in other embodiments, a different number of weights may be used.

The feature from the final layer (i.e., Conv6) is then the output of the CNN.

Next, a Recurrent Neural Network (RNN) takes the features generated by the CNN and determines the motion features from them (step 24, LSTM box of section 34 of FIG. 3). Similar to the CNN, the RNN does this according to its training, as will be described in detail below.

The RNN is a neural network where the layers operate on inputs, but also on delayed versions of hidden layers and/or outputs. In this way, RNNs have internal states that they can use as "memory" to track past inputs and corresponding decisions.

In the present embodiment, an RNN with a Long Short-Term Memory (LTSM) architecture (with variations in it) is used, as shown in FIG. 4, where ⊙ represents the element-by-element product, and

representing the addition of two vectors. The contents of the memory cell are stored in c _tIn (1). The input gate controls how the current time step input enters the memorized content. Forget door f _tThe control signals 0 to 1 clear the memory cells as needed by generating control signals 0 to 1 to determine when the memory cells should be cleared. Finally, an output gate o _tIt is determined whether the contents of the memory cell should be used at the current time step. The operation of the RNN is described by the following equation:

i _t＝σ(W _xix _t+W _hih _t-1+W _cic _t-1+b _i)

f _t＝σ(W _xfx _t+W _hfh _t-1+W _cfc _t-1+b _f)

z _t＝tanh(W _xcx _t+W _hch _t-1+b _c)

c _t＝f _t⊙c _t-1+i _t⊙z _t

o _t＝act(W _xox _t+W _hoh _t-1+W _coc _t+b _o)

h _t＝o _t⊙tanh(c _t)。

parameter W _i，jAnd b _iThe operation of the RNN is fully parameterized and learned during training. The recursive hidden layer allows the network to take advantage of the temporal regularity of the input data to improve its performance.

Although in the conventional LSTM model the hidden state is only continued from the previous time step, in the present embodiment the gesture determined for the previous time step is fed directly to the RNN as input. This can be seen in FIG. 3, where the gesture for a time step is fed to the LSTM box for the next time step. The reason for this is that for position estimation, the output is essentially an integral of the successive shifts at each time step. Therefore, the determined posture of the previous time step is particularly important.

In this embodiment, the LSTM has two layers with 2000 cells, but it should be understood that in other embodiments, a different number of layers and cells may be used.

Next, the motion features determined by the (high-dimensional) RNN are passed to a fully connected layer (step 25) which outputs low-dimensional features (at least 6 features for pose, at least 6 features for uncertainty, and possibly more features for each if a hybrid gaussian model is used to estimate pose and uncertainty).

Next, the low-dimensional features from the fully connected layer are passed to the SE (3) layer (step 26, SE3 box of section 34 of FIG. 3). SE (3) integrates continuous motion features for each time step in order to determine the position of the autonomous robot 1 (in fact the Pose, e.g. the position of time step t) at each time step _t)。

SE3 is a special euclidean group whose elements are the transformation matrix consisting of rotation and translation vectors from a special orthogonal group SO 3:

generating the transform estimate belonging to SE3 is not straightforward, since the SO3 component needs to be an orthogonal matrix. However, the Lie Algebra (Lie Algebra) SE3 of SE3 may be described by components that are not subject to orthogonal constraints:

the conversion between SE3 and SE3 may then be done using exponential mapping:

exp:se3→SE3。

in an alternative embodiment, a rotated quaternion representation is used instead of a matrix representation. Specifically, the ω component is converted to a vector:

w _t＝[0，ω _x，ω _y，ω _z]

the gradients of these quantities can then be calculated using only simple linear algebraic operations. Furthermore, expensive Eigenvalue Decompensation (Eigenvalue Decompensation) required for computing the exponential mapping is avoided.

Thus, in this way, the autonomous robot 1 uses the images from the camera 3 to estimate its position, in particular its pose.

Estimating position from continuous sensor measurements (i.e., odometry) is inevitably subject to drift. Therefore, it is often used in conjunction with a ring closure, map matching, or pose graph optimization method to create a Simultaneous localization and Mapping (SLAM) system. A key aspect of integrating odometry measurements into such systems is the availability of uncertainty estimates.

To provide such an estimate, the output of the fully connected layer is used (before the SE (3) layer). The estimates produced by the fully connected layer are compared to ground truth pose information from the training data, resulting in an error distribution in pose (position and orientation). The maximum likelihood method is then used to train a prediction of the mixture of gaussian distributions representing the uncertainty.

To operate, it is of course necessary to train the neural network, which is done by providing test data and a cost function to be minimized. The training of the CNN and RNN of the autonomous robot 1 as now described, in effect, both are trained simultaneously.

As described above, the system of the present embodiment estimates both pose and uncertainty. The test data will be a sequence of images having a "ground truth" pose (i.e. correct pose). The trained cost function consists of two parts, the first part relating to pose estimation and the second part to uncertainty estimation. For pose estimation, the first part of the cost function trains the system to minimize the difference between the estimated pose and the ground truth pose. For uncertainty estimation, a second part of the cost function trains the system by comparing the output of the neural network to the pose labels. Training is then done by back-propagation through time to adjust the weights of the CNN and the RNN to optimally minimize the result of the cost function.

In this way, it can be seen that the CNN is trained to provide the most appropriate features for input to the RNN, and at the same time the RNN is trained to most accurately determine the pose (and its uncertainty) of the autonomous robot 1 from these features (and previous determinations). CNNs are not specifically trained to best provide any particular type of feature or feature with any particular attribute; rather, it is simply trained to provide the best features for the operation of the overall system. However, in some embodiments, to speed up the initial training process, the CNN is initially trained in isolation (or otherwise provided with weights having such a training effect) to provide features indicative of motion between successive images. This provides an initial state for the CNN, which is then further optimally trained when the system as a whole is trained.

While the invention has been described and illustrated with reference to specific embodiments, those skilled in the art will appreciate that there are numerous variations of the invention itself that are not specifically illustrated herein.

In the foregoing description, integers or elements having known, obvious or foreseeable equivalents are mentioned, which equivalents are herein incorporated as if individually set forth. Reference should be made to the claims for determining the true scope of the present invention, which should be construed so as to encompass any such equivalents. The reader will also appreciate that integers or features of the invention that are described as preferred, advantageous, convenient or the like are optional and do not limit the scope of the independent claims. Moreover, it will be understood that alternative integers or features, which may be beneficial in some embodiments of the invention, may not be desirable and may therefore not be present in other embodiments.