阅读说明：本技术 一种登机桥停靠方法 (Boarding bridge parking method ) 是由 李剑思 林姝含 郑文涛 于 2019-10-16 设计创作，主要内容包括：本发明涉及一种登机桥停靠方法,包括停靠阶段和停靠准备阶段,所述停靠阶段通过设置在登机桥前端的双目摄像机识别和跟踪飞机舱门,依据飞机舱门左下角点、右下角点及两者连线的中点的第一本地坐标进行登机桥停靠运动的控制,实现飞机舱门底边与登机桥头端底边的中点对齐和相互贴合,在停靠准备阶段,以前向摄像机采集包含停靠区域的场景图像,在场景图像上人工指定停靠起点位置,将停靠起点位置的图像坐标系坐标转换为第二本地坐标系中的地面坐标,依据停靠起点位置的地面坐标控制登机桥运动至停靠起点位置。本发明基本上实现登机桥的自动停靠,且设备简单,显示直观,有助于提高可靠性且符合操作习惯。(The invention relates to a boarding bridge parking method, which comprises a parking stage and a parking preparation stage, wherein in the parking stage, a binocular camera arranged at the front end of a boarding bridge is used for identifying and tracking an airplane cabin door, the control of the parking motion of the boarding bridge is carried out according to a left lower angular point, a right lower angular point and a first local coordinate of a midpoint of a connecting line of the left lower angular point and the right lower angular point of the airplane cabin door, the alignment and the mutual attachment of the bottom edge of the airplane cabin door and the midpoint of the bottom edge of the head end of the boarding bridge are realized, in the parking preparation stage, a scene image containing a parking area is collected by a forward camera, a parking starting point position is manually specified on the scene image, the image coordinate system coordinate of the parking starting point position is converted into a ground coordinate in a second local coordinate system, and. The invention basically realizes the automatic stop of the boarding bridge, has simple equipment and visual display, is beneficial to improving the reliability and conforms to the operation habit.)

1. a method for parking a boarding bridge comprises a parking stage, wherein a binocular camera is arranged at the head end of the boarding bridge, the visual field range of the binocular camera can always cover an airplane cabin door in the moving process of the boarding bridge, a first local coordinate system of the boarding bridge is constructed, the origin of the first local coordinate system is the middle point of the bottom edge of the head end of the boarding bridge, a plane with the Z being 0 is a plane which comprises an abutting surface abutting the bottom of the head end of the boarding bridge and the bottom of the airplane cabin door and is vertical to the ground, a plane with the Y being 0 is a plane where the ground of the boarding bridge is located, a plane with the X being 0 is a plane which passes through the origin and is vertical to the plane with the Y being 0 and the plane with the Z being 0, external parameter calibration is carried out on each camera of the binocular camera, the position coordinate and the attitude angle of each camera of the binocular camera at the calibration time in the first local coordinate system are obtained and are used as the constant position coordinate and attitude angle of each camera of the binocular camera in the first local coordinate system, in the parking stage, video images containing the airplane cabin door are collected through cameras of the binocular camera, automatic detection and tracking of the airplane cabin door are carried out according to the video images, three target points, namely a left lower angular point, a right lower angular point and a bottom edge midpoint of the cabin door, of the airplane cabin door are identified in real time through feature identification and stereo matching, coordinates of the three target points in a first local coordinate system are obtained through calculation, a parking motion mode of the boarding bridge is determined or adjusted in real time according to the coordinates of the three target points in the first local coordinate system until the bottom edge midpoint of the cabin door of the airplane is attached to or leaves a set abutting gap with the bottom edge of the head end of the boarding bridge, and the left lower angular point and the right lower angular point of the cabin door are attached to or leave a set abutting gap with the bottom edge of the.

2. The method of claim 1, wherein the docking phase docking motion and its control comprises any one or more of:

a) rotating: based on the included angle between the connecting line between the left lower corner point and the right lower corner point of the cabin door and the bottom edge of the head end of the boarding bridge, the boarding bridge is rotated in the horizontal plane, so that the boarding bridge and the cabin door tend to be consistent;

b) and (3) vertical movement: lifting operation is carried out on the boarding bridge in the vertical direction based on the height difference between the middle point of the connecting line of the left lower angular point and the right lower angular point of the cabin door and the bottom edge of the head end of the boarding bridge, so that the heights of the boarding bridge and the cabin door tend to be consistent;

c) and (3) transverse movement: and based on the difference between the transverse position of the midpoint of the connecting line of the left lower angular point and the right lower angular point of the cabin door and the transverse position of the midpoint of the bottom edge of the head end of the boarding bridge, carrying out left-right horizontal movement operation on the boarding bridge, and enabling the boarding bridge and the boarding bridge to tend to be consistent.

d) Straight forward (along the Z axis): based on the distance between the longitudinal position of the midpoint of the connecting line of the left lower angular point and the right lower angular point of the cabin door and the longitudinal position of the midpoint of the bottom edge of the head end of the boarding bridge, the boarding bridge is moved in the front-back horizontal direction to make the two tend to be consistent, namely when the Z-axis coordinate of the midpoint of the connecting line of the left lower angular point and the right lower angular point of the cabin door is Z-axis coordinate₂Or z₂-d is greater than 0, the bridge performs a forward movement operation, when z is₂Or z₂No further forward movement when d is equal to 0, where d is the set abutment gap.

3. Method according to claim 2, characterized in that the speed of movement of the bridge during the docking phase, in particular the speed of movement in the Z-direction, is controlled in dependence on the distance between the bridge and the door of the aircraft, the lower the speed being the closer to the door of the aircraft.

4. The method according to any one of claims 1 to 3, further comprising or not comprising a parking preparation phase in which the boarding bridge is moved from the waiting position to the parking start position with the boarding bridge nose toward the aircraft door.

5. The method according to claim 4, wherein a forward camera is provided at the front of the boarding bridge, the forward camera is calibrated at a starting position (e.g. a waiting position or any other position used for related operations) of a parking preparation stage, a second local coordinate system is established with a Y-axis directly in front of the head end of a control room of the boarding bridge, an X-axis perpendicular to the Y-axis in a horizontal plane, and a Z-axis perpendicular to the ground plane, the forward camera captures a scene image including a parking area, the parking starting position is manually specified on the scene image, and the ground coordinate of the parking starting position is calculated according to the following formula:

wherein the content of the first and second substances,

m＝[uv]^Tan image coordinate matrix for a stop start position, wherein (u, v) are image coordinate system coordinates for the stop start position,

M＝[xyz]^Ta second local coordinate matrix for a stop starting location, wherein (x, y, z) is a second local coordinate system coordinate for the specified stop starting location,

for the internal parameter matrix of the forward-facing camera, α and β are the scale factors of the image coordinates in the horizontal and vertical directions, respectively, γ is the deviation factor of the horizontal axis from the vertical axis of the image, (u, v) are the image coordinates of the optical center of the forward-facing camera,

R＝[R_xR_yR_z]is the attitude angle matrix of the forward facing camera, where R_x、R_yRz are the attitude angles of the forward facing camera with respect to the axes of the second local coordinate system X, Y, Z,

T＝[T_xT_yT_z]is the second local coordinate matrix of the forward facing camera, where (T)_x,T_y,T_z) Is the second local coordinate system coordinate of the forward-facing camera.

6. The method of claim 5, wherein the origin of the second local coordinate system is set as a contact point of any one of the driving wheels under the boarding bridge control room.

7. The method of claim 5, wherein the image coordinates of the optical center of the forward-facing camera are determined in accordance with the following:

(u,v)＝(u₀,v₀)

wherein (u)₀,v₀) And calibrating the optical center image coordinates of the forward camera at the moment for the forward camera.

8. The method of claim 5, wherein the forward camera pose angle matrix is determined according to:

R＝R₀＝[R_x0R_y0R_z0]

wherein R is₀For forward direction cameraForward camera attitude angle matrix, R, at a calibrated time_x0、R_y0、R_zoThe attitude angles of the forward cameras relative to the axes of the second local coordinate system X, Y, Z at the time of the forward camera are calibrated for the forward cameras, respectively.

9. The method of claim 5, wherein the second local coordinate matrix for the forward-facing camera is determined in accordance with:

T_x＝T_x0，T_y＝T_y0，T_z＝T_z0+(h-h₀)

wherein (T)_x0,T_y0,T_z0) The coordinates of a second local coordinate system of the forward camera at the time of calibration of the forward camera, h is the relative height of the boarding bridge or the control room, h₀The relative height of the boarding bridge or control room at the moment is calibrated for the forward facing camera.

10. The method according to claim 9, wherein the relative height of the bridge or cabin is collected by a respective height sensor as the height of the bridge or cabin relative to a respective reference height.

Technical Field

The invention relates to a boarding bridge parking method.

Background

The boarding bridge is also called an air bridge or an airplane corridor bridge, is a facility in an airport terminal building, extends from a boarding door to an airplane cabin door, and is convenient for passengers to get in and out of a cabin. Under any weather conditions, the boarding bridge can ensure that passengers can conveniently board and leave the airplane without being exposed to the sun and rain, and simultaneously, the operation efficiency of the airport can be improved.

The head end of a typical boarding bridge is fixed on a middle shaft at a boarding door, a bridge body can move left and right, and the head end and the tail end can be lifted and stretched, so that the boarding bridge is suitable for various airplanes. The tail end has a control room for controlling the movement of the bridge body, and a bellows is extended outward to closely engage the cabin door, so that the boarding bridge is moved from the waiting position to the cabin door before the passengers are loaded and unloaded, and the bellows is closely engaged with the cabin door, which is called the boarding bridge parking or landing against the boarding bridge. And after the boarding and the alighting and other subsequent processes are finished, the boarding bridge is moved to the original waiting position, namely the boarding bridge is evacuated or the corridor bridge is withdrawn.

At present, the boarding bridge stopping and withdrawing are mainly completed manually by boarding bridge operators in a control room, the boarding bridge operators are combined with visual and handle operations, high operation skills are needed, and the operation process is troublesome.

Although many attempts have been made to achieve automatic docking of boarding bridges, no practical application has been found so far. However, the continuous development of computer vision technology and automation control technology provides possibility for the automation of boarding bridge parking, so that there is a need to develop a practical, reliable and convenient boarding bridge automatic parking technology in due time.

One technical route is that infrared sensors are arranged at the bottom end of a boarding bridge and below an airplane cabin door for measuring the relative distance between the boarding bridge and an airplane and controlling the boarding bridge to stop, but because the infrared sensors corresponding to the boarding bridge are required to be arranged on each airplane needing to be stopped automatically, the feasibility is not high, the universality is poor, and the difficulty of popularization and implementation is high; the other technical route is that image data of corresponding cabin doors are collected aiming at airplane models, the distance between an airplane and a boarding bridge is calculated through computer vision technology and model data comparison, the boarding bridge is automatically controlled to stop, but because data collection needs to be carried out aiming at the cabin doors of various types of airplanes corresponding to each boarding bridge, the workload is large, the data is difficult to update, and practical application is hindered.

Disclosure of Invention

In order to overcome the defects in the prior art, the invention provides a boarding bridge parking method, which basically realizes the automatic parking of the boarding bridge, has simple equipment and visual display, is beneficial to improving the reliability and conforms to the operation habit.

The technical scheme of the invention is that the boarding bridge parking method comprises a parking stage, wherein a binocular camera is arranged at the head end of the boarding bridge, the visual field range of the binocular camera can always cover an airplane cabin door in the moving process of the boarding bridge, a first local coordinate system of the boarding bridge is constructed, the origin of the first local coordinate system is the middle point of the bottom edge of the head end of the boarding bridge, a plane with the value of Z being 0 is a plane which comprises an abutting surface abutting the bottom of the head end of the boarding bridge and the bottom of the airplane cabin door and is vertical to the ground (or a horizontal plane), a plane with the value of Y being 0 is a plane where the ground of the boarding bridge is located, a plane with the value of X being 0 is a plane which passes through the origin and is vertical to the plane with the value of Y being 0 and the plane with the value of Z being 0, external parameter calibration is carried out on each camera of the binocular camera, and the position coordinate and the attitude angle of each camera of the binocular, since the positions and postures of the cameras of the binocular camera on the boarding bridge are kept unchanged in the process, and the external parameters are also unchanged, the positions and postures (the position coordinates and the posture angles of the cameras of the binocular camera in the first local coordinate system at the calibration time) can be used as the constant position coordinates and posture angles of the cameras of the binocular camera in the first local coordinate system, and the changed coordinate positions and posture angles can be re-calibrated or calculated according to a specific changing mode when the first local coordinate system coordinates or the posture angles of the cameras of the binocular camera are changed manually in the subsequent data processing process, and in the parking stage, video images containing airplane cabin doors are collected (obtained by shooting) through the cameras of the binocular camera, and the automatic detection and tracking of the airplane cabin doors are carried out according to the video images, through feature recognition and stereo matching, three target points, namely a left lower angular point, a right lower angular point and a bottom edge midpoint of a cabin door of an airplane, are recognized in real time, coordinates of the three target points in a first local coordinate system are obtained through calculation, a parking motion mode of the boarding bridge is determined or adjusted in real time according to the coordinates of the three target points in the first local coordinate system until the bottom edge midpoint of the cabin door of the airplane is attached to the bottom edge midpoint of the head end of the boarding bridge or a set abutting gap is reserved, the left lower angular point and the right lower angular point of the cabin door are attached to the bottom edge of the head end of the boarding bridge or a set abutting gap is reserved, namely, in an ideal state, the first local coordinate system coordinate of the middle point of the bottom edge of the cabin door is (0,0,0) or (0,0, d), the Y-axis coordinate of the lower left corner point and the lower right corner point is 0, the Z-axis coordinate is 0 or d, wherein d is a set abutting gap.

The docking motion of the docking stage and the control mode thereof comprise any one or more of the following modes:

a) rotating: based on the included angle between the connecting line between the left lower corner point and the right lower corner point of the cabin door and the bottom edge (X axis) of the head end of the boarding bridge, the boarding bridge is rotated in the horizontal plane, so that the boarding bridge and the cabin door tend to be consistent;

b) vertical movement (movement along Y axis): based on the height difference between the middle point of the connecting line of the left lower angular point and the right lower angular point of the cabin door and the bottom edge of the head end of the boarding bridge (namely the middle point of the connecting line of the left lower angular point and the right lower angular point of the cabin door is in the Y-axis coordinate Y of the first local coordinate system₂) Lifting the boarding bridge in the vertical direction to make the heights of the boarding bridge consistent, namely when the Y-axis coordinate Y of the midpoint of the connecting line of the left lower angular point and the right lower angular point of the cabin door₂When the height is more than 0, the boarding bridge performs the lifting operation, and when y₂When less than 0, the boarding bridge performs a lowering operation, when y₂No vertical movement is performed at 0;

c) lateral movement (movement along the X axis): based on the difference between the transverse position of the midpoint of the connecting line of the left lower angular point and the right lower angular point of the cabin door and the transverse position of the midpoint of the bottom edge of the boarding bridge head end (namely, the midpoint of the connecting line of the left lower angular point and the right lower angular point of the cabin door is in the X-axis coordinate X of the first local coordinate system₂) Moving the aerobridge in the horizontal direction to make the two directions consistent, that is, when the X-axis coordinate X of the middle point of the connecting line of the left lower angular point and the right lower angular point of the hatch door₂When the value is more than 0, the boarding bridge executes left shift operation, and when x is greater than 0, the boarding bridge executes left shift operation₂When the number is less than 0, the boarding bridge executes right shift operation, and when x is less than 0₂When 0, no lateral movement is performed.

d) Straight forward (in the Z-axis direction): based on the distance between the longitudinal position of the midpoint of the connecting line of the left lower angular point and the right lower angular point of the cabin door and the longitudinal position of the midpoint of the bottom edge of the boarding bridge head end (namely, the middle of the connecting line of the left lower angular point and the right lower angular point of the cabin door)Z-axis coordinate Z of point in first local coordinate system₂When an abutting gap d is set, z is₂D) moving the boarding bridge back and forth horizontally (longitudinally) to make the boarding bridge consistent, namely, when the Z-axis coordinate Z of the middle point of the connecting line of the left lower angular point and the right lower angular point of the cabin door₂Or z₂-d is greater than 0, the bridge performs a forward movement operation, when z is₂Or z₂No further forward movement when d is equal to 0, where d is the set abutment gap.

The movement modes can be alternately implemented according to actual needs, and can also be implemented by repeating the operation from coarse to fine for multiple times (multiple cycles).

The speed of movement of the bridge during the docking phase, in particular the speed of movement in the Z direction (direction of the Z axis of the first local coordinate system), is controlled in dependence on the distance between the bridge and the aircraft door, the lower the speed is the closer to the aircraft door, for example, when the docking movement is performed in several (multi-cycle) cycles, the higher the speed can be used initially, the slower the speed can be used for the last time.

When the waiting position of the boarding bridge is far away from the airplane cabin door, the airplane cabin door cannot be or is inconvenient to be identified and positioned by adopting the binocular camera, the boarding bridge can be moved to a manually set stopping starting point position firstly, then the stopping starting point position is used as a stopping starting point for stopping, and the process can be regarded as a stopping preparation stage. Since the waiting position of the boarding bridge is usually far from the aircraft door, the parking preparation phase is usually included, but the possibility of not needing to perform the parking preparation phase is not excluded, and the parking preparation phase may not be considered as one phase of the parking method from a management perspective.

Thus, the present invention may or may not include a dock preparation phase. In the preparation stage of parking, the boarding bridge is moved from the waiting position to the starting position of parking and the head end of the boarding bridge is towards the cabin door of the airplane, so that the implementation of the subsequent parking stage is facilitated.

The docking preparation phase may be carried out using any suitable prior art technique, for example, by manual operation.

It can be understood that, although the binocular camera is related to or mainly used in the parking stage, since the calibration of the camera is based on the first local coordinate system and is not dependent on the position of the boarding bridge, the setting, calibration and the like of the binocular camera can be performed in advance without waiting for the parking stage.

The movement of the boarding bridge in the preparation phase for parking can be carried out in an automatically controlled manner. For example, one preferred embodiment is: the front part of the boarding bridge (for example, the upper part of a cab) is provided with a forward camera, the calibration of the forward camera is carried out at the starting position of a parking preparation stage (for example, a waiting position or any other position used for related operations), the right front of the head end of a control room of the boarding bridge is taken as an axis Y, the direction vertical to the axis Y in the horizontal plane is taken as an axis X, establishing a second local coordinate system by taking the vertical ground direction as a Z axis, wherein the second local coordinate system can determine the origin by taking the contact point of any driving wheel below a boarding bridge control room as the origin or by other modes convenient for calculation or control, calibrating a forward camera, acquiring (shooting) a scene image containing a parking area by the forward camera, manually appointing a stop starting point position on a corresponding scene image (image picture), and calculating and obtaining the ground coordinates (XY plane coordinates of a second local coordinate system) of the stop starting point position according to the following formula:

wherein the content of the first and second substances,

is m ═ uv]^TThe homogeneous matrix of (a) is,

is M ═ xyz]^TThe homogeneous matrix of (a) is,

m＝[uv]^Tan image coordinate matrix for a stop start position, wherein (u, v) are image coordinate system coordinates for the stop start position,

M＝[xyz]^Tfor a second book in the stop starting positionA ground coordinate matrix, wherein (x, y, z) is the second local coordinate system coordinates of the specified stop start location,

for the internal parameter matrix of the forward camera, α and β are the scale factors of the image coordinates in the horizontal and vertical directions, respectively, γ is the deviation coefficient of the horizontal and vertical axes of the image, and (u, v) is the image coordinate of the optical center of the forward camera. The image coordinates of the optical center of the forward-facing camera can be determined using any of the prior art techniques, for example, in the following manner:

(u,v)＝(u₀,v₀)，(u₀,v₀) Calibrating the optical center image coordinates of the forward camera at the moment for the forward camera;

R＝[R_xR_yR_z]is the attitude angle matrix of the forward facing camera, where R_x、R_yRz are the attitude angles of the forward facing camera with respect to the axes of the second local coordinate system X, Y, Z, respectively. The attitude angle matrix of the forward-facing camera may be determined using any of the existing techniques, for example, in the following manner:

R＝R₀＝[R_x0R_y0R_z0]，R₀forward camera attitude angle matrix, R, for time calibration of a forward camera_x0、R_y0、R_zoThe attitude angles of the forward cameras relative to the axes of the second local coordinate system X, Y, Z at the times are respectively calibrated for the forward cameras, and are constant relative to the second local coordinate system due to the forward cameras being fixedly mounted above the control room or other suitable location.

T is the second local coordinate matrix of the forward camera, T ═ T_xT_yT_z]Wherein (T)_x,T_y,T_z) The second local coordinate system coordinates (real-time coordinates) for the forward-facing camera. The second local coordinate matrix of the forward-facing camera may be determined using any of the prior art techniques, for example, in the following manner:

T_x＝T_x0，T_y＝T_y0，T_z＝T_z0+(h-h₀) Wherein (T)_x0,T_y0,T_z0) The coordinates of a second local coordinate system of the forward camera at the time of calibration of the forward camera, h is the relative height of the boarding bridge or the control room, h₀The relative height of the boarding bridge or control room at the moment is calibrated for the forward facing camera. The relative height of the bridge or cabin, which is the height of the bridge or cabin relative to the corresponding reference height, can be determined using any known technique, and can be typically collected by a corresponding height sensor, for example.

After the boarding bridge reaches a stop starting point, determining a plane azimuth angle for attitude adjustment according to the actual attitude and the standard stop attitude, determining a height difference for front end height adjustment according to corresponding airplane cabin door information, and adjusting a front end angle and a height position according to the plane angle and the height difference to reach the standard stop attitude.

The relevant working process of the parking preparation stage is as follows: the method comprises the steps that a fixed forward camera is installed above a boarding bridge control room, a camera shooting area covers a boarding bridge parking area, the forward camera can be calibrated in advance to obtain internal parameters (posture, focal length, distortion coefficient and the like) of the forward camera, external parameters of the forward camera are calibrated under a second local coordinate system, the height of the forward camera is obtained in real time through a sensor inside the boarding bridge during work (the control room can be lifted and lowered, so the height is changed) so as to calculate the change of the external parameters (only the height or Z-axis coordinate change) in real time, the corresponding relation between the image coordinate of the forward camera and the three-dimensional space position is established based on the internal parameters and the external parameters of the forward camera, an operator clicks a desired parking position on an image, a system automatically calculates the parking position of the boarding bridge on the ground, and conditions are provided for automatic parking, the change of the stop position (relative position) can be tracked and monitored/calculated in real time, the stop position is transmitted to the corresponding control and driving device, and the boarding bridge is moved from the waiting position to the stop starting position through the control and driving device.

In the invention, in the docking stage, each camera of a binocular camera is calibrated on the basis of a first local coordinate system of the boarding bridge to obtain corresponding external parameters, the airplane door is identified and tracked in real time based on an image target identification technology and a binocular mapping technology according to the internal parameters and the external parameters of each camera of the binocular camera, the motion control of the boarding bridge is carried out according to the position coordinates of a relevant target point of the airplane door on the first local coordinate system, additional sensors are not required to be installed at the position of the airplane door and on the boarding bridge or the calibration of the airplane door of each model is not required, and the manual intervention on a target frame for tracking is allowed, the required equipment and data processing process are simple, the positioning is accurate and reliable, and the operation and monitoring habits of people are met, when the docking position of the boarding bridge is far away from the airplane door, the boarding bridge can be moved to any docking starting point suitable for implementing the docking stage by setting an automatic control and implemented docking preparation stage, the position of the stop starting point is designated by an operator in a video image of a corresponding camera, the method is simple and convenient, only a camera and a simple computing unit need to be additionally arranged, the position of the stop starting point can be continuously tracked in the moving process of the boarding bridge by adopting the existing target tracking technology, the accurate position of the airplane is obtained without adopting an automatic means, the amount of movement in the vertical direction of the forward camera is obtained by adopting an original height sensor of the boarding bridge or a control room, and the end point (the position of the stop starting point) in the stage is limited on the horizontal ground, so that the image coordinate of the corresponding position can be mapped to the only one point in a three-dimensional coordinate system, and the problem of how to convert the image coordinate of the image into the specific position under the three-dimensional coordinate system in the moving occasion.

Drawings

FIG. 1 is a flow chart of the work involved in the docking phase of the present invention;

fig. 2 is a positioning flowchart involved in the landing preparation phase of the present invention.

Detailed Description

Referring to fig. 1, the invention provides a method for stopping a boarding bridge, which comprises the steps of installing 1 binocular camera at the head of the boarding bridge, setting a first local coordinate system, calibrating and correcting the parameters of the binocular camera in advance to obtain internal parameters and external parameters of the binocular camera, identifying an airplane cabin door based on an image target identification technology and a binocular mapping technology, calculating the stopping position of the boarding bridge on the ground (an XZ plane of the first local coordinate system) in real time according to the airplane cabin door, driving the boarding bridge to move to the stopping position through a driving system according to the stopping position, and finally finishing automatic stopping, namely, the head end of the boarding bridge is abutted to the airplane cabin door.

The method specifically comprises the following steps:

1) binocular camera setup

1 binocular camera is arranged at the bridge head of the boarding bridge, and the arrangement position and the arrangement mode of the binocular camera ensure that the complete cabin door of the airplane can be shot in the whole motion process of the boarding bridge in the parking stage.

2) Binocular camera calibration

Calibrating internal parameters of a left camera and a right camera of the binocular camera by using auxiliary tools such as a calibration plate in advance to obtain an internal parameter matrix A of each camera of the binocular camera₁、A₂：

Wherein alpha is₁、β₁And alpha₂、β₂Scale factors, gamma, of the image coordinates of the two cameras in the horizontal and vertical directions, respectively₁And gamma₂Deviation coefficients of horizontal axis and vertical axis of the images obtained by the two cameras respectively, (u)₁₀,v₁₀) And (u)₂₀,v₂₀) Respectively the image coordinates of the optical centers of the two cameras.

According to actual needs, the corresponding internal parameter matrix A can be used₁、A₂And respectively carrying out image correction on the two cameras, and carrying out subsequent target identification, image target positioning and the like on the corrected images.

Any suitable prior art may be used to accomplish the above calibration and correction^[1]。

And calibrating external parameters of each camera of the binocular camera set on site. To include the bottom of the aerobridge head end and flyA plane which is close to a contact surface at the bottom of the cabin door and is vertical to the ground is used as a plane Z which is 0, the ground of the boarding bridge is used as a plane Y which is 0, a plane which passes through the middle point of the bottom edge of the head end of the boarding bridge and is vertical to the plane Y which is 0 and a plane Z which is 0 are used as planes X which are 0 to construct a first local coordinate system, two cameras of the camera are calibrated in the first local coordinate system, and a coordinate matrix T of the two cameras in the first local coordinate system is obtained₁₀＝[T_1x0T_1y0T_1z0]、T₂₀＝[T_2x0T_2y0T_2z0]And an attitude angle matrix R of the two cameras in the first local coordinate system₁₀＝[R_1x0R_1y0R_1z0]、R₂₀＝[R_2x0R_2y0R_2z0]. Wherein (T)_1x0,T_1y0,T_1z0) And (T)_2x0,T_2y0,T_2z0) First local coordinate system coordinates, R, of two cameras, respectively_1x0And R_1x0Respectively the attitude angles, R, of the two cameras relative to the X axis of the first local coordinate system_1y0And R_1y0Respectively the attitude angles, R, of the two cameras relative to the Y axis of the first local coordinate system_1z0And R_1z0The attitude angles of the two cameras relative to the Z axis of the first local coordinate system are respectively.

3) Automatic cabin door detection and manual confirmation

Transmitting images shot by the binocular camera to a monitoring terminal and a video analysis server of a monitoring person, wherein the video analysis server detects the images according to any suitable existing target detection algorithm^[2-4]And obtaining the position of the cabin door target frame and the position of the cabin door target frame in the image based on the general cabin door detection model obtained by pre-training, and superposing and displaying the cabin door target frame on the image picture of the monitoring terminal according to the position.

If the door target frame does not completely contain all door images, the monitoring personnel will perform manual intervention to manually adjust the size and/or area of the door target frame to completely contain the door images.

After confirming that the door target frame contains the complete door image, the monitoring personnel confirms that the door automatic detection step is completed, and then the next step is carried out.

4) Hatch door tracking and feature point extraction

Respectively tracking a cabin door target in a target frame in image pictures of two cameras of a binocular camera to obtain a cabin door tracking frame, segmenting an image in the cabin door tracking frame, identifying characteristic points, identifying three target points, namely a left lower angular point, a right lower angular point and a bottom edge midpoint of a cabin door, and adopting an EPnP algorithm^[5,6]Two groups of characteristic points respectively from the two camera images are matched to obtain the positions (x) of the three target points in the first local coordinate system_i，y_i，z_i) Where i ∈ [1,2,3 ]](which may be considered as the number of each of the three feature points). Any suitable prior art may be used for the above treatments^[7-12]。

5) Boarding bridge movement

The purpose of the movement of the boarding bridge is to ensure that the bottom edge of the head end of the boarding bridge coincides with the connecting line of the left lower corner point and the right lower corner point of the airplane cabin door, and the middle point of the bottom edge of the head end of the boarding bridge coincides with the middle point of the connecting line of the left lower corner point and the right lower corner point of the airplane cabin door. According to the actual situation, any one or more of the following moving modes can be included:

a) rotating: based on the included angle between the connecting line between the left lower corner point and the right lower corner point of the cabin door and the bottom edge (X axis) of the head end of the boarding bridge, the boarding bridge is rotated in the horizontal plane, so that the boarding bridge and the cabin door tend to be consistent;

b) vertical movement (movement along the Y axis of the first local coordinate system): based on the height difference between the middle point of the connecting line of the left lower angular point and the right lower angular point of the cabin door and the bottom edge of the head end of the boarding bridge (namely the Y-axis coordinate Y of the middle point of the connecting line of the left lower angular point and the right lower angular point of the cabin door₂) Lifting the boarding bridge in the vertical direction to make the heights of the boarding bridge consistent, namely when the Y-axis coordinate Y of the first local coordinate system of the midpoint of the connecting line of the left lower angular point and the right lower angular point of the cabin door is in line with the Y-axis coordinate Y of the first local coordinate system of the midpoint of the connecting line of the left lower angular point and the right lower angular point of the cabin door₂When the height is more than 0, the boarding bridge performs the lifting operation, and when y₂When less than 0, the boarding bridge performs a lowering operation, when y₂No vertical movement is performed at 0;

c) lateral movement (movement along the X axis of the first local coordinate system):based on the difference between the transverse position of the midpoint of the connecting line of the left lower angular point and the right lower angular point of the cabin door and the transverse position of the midpoint of the bottom edge of the boarding bridge head end (namely, the X-axis coordinate X of the midpoint of the connecting line of the left lower angular point and the right lower angular point of the cabin door₂) Moving the aerobridge in the horizontal direction to make the aerobridge and the aerobridge consistent, that is, when the X-axis coordinate X of the first local coordinate system of the middle point of the connecting line of the left lower angular point and the right lower angular point of the hatch door is in line with each other₂When the value is more than 0, the boarding bridge executes left shift operation, and when x is greater than 0, the boarding bridge executes left shift operation₂When the number is less than 0, the boarding bridge executes right shift operation, and when x is less than 0₂When 0, no lateral movement is performed.

d) Straight forward (forward along the Z-axis of the first local coordinate system): based on the difference between the longitudinal position of the midpoint of the connecting line of the left lower angular point and the right lower angular point of the cabin door and the longitudinal position of the midpoint of the bottom edge of the boarding bridge head end (namely, the Z-axis coordinate Z of the first local coordinate system of the midpoint of the connecting line of the left lower angular point and the right lower angular point of the cabin door₂When an abutting gap d is set, z is₂D) moving the boarding bridge back and forth horizontally (longitudinally) to make the boarding bridge consistent, namely, when the Z-axis coordinate Z of the middle point of the connecting line of the left lower angular point and the right lower angular point of the cabin door₂Or z₂-d is greater than 0, the bridge performs a forward movement operation, when z is₂Or z₂No further forward movement when d is equal to 0, where d is the set abutment gap.

During the docking process, the above steps 4) and 5) can be repeated.

With the gradual reduction of the distance between the boarding bridge and the cabin door, the moving speed of the boarding bridge is gradually reduced until the cabin door is stopped.

Since the above-mentioned parking manner is based on that the boarding bridge is already parked at a position close to the corresponding airplane door, which may be referred to as a parking starting position, and in many cases, the waiting position of the boarding bridge is far from the parking position of the airplane, it is necessary to move the boarding bridge to the parking starting position by other manners, and then to implement the above-mentioned method, the process of moving and controlling the boarding bridge from the parking starting position to the parking position may be referred to as a parking stage, and the process of moving and controlling the boarding bridge from the waiting position to the parking starting position may be referred to as a parking preparation stage.

The boarding bridge can be moved from the waiting position to the stop start position by the following method.

Referring to fig. 2, the boarding bridge is moved from the waiting position to the stop start position by the following steps:

1) forward camera setup

The forward-facing camera can be usually arranged outside the boarding bridge control room, such as above the top of the control room, and is fixed relative to the control room, and the horizontal field angle and the pitch angle of the forward-facing camera are set so as to ensure that when the boarding bridge is in a waiting area, a video picture shot by the forward-facing camera can cover a stop starting point position (area), namely an area where a driving wheel below the boarding bridge control room is stopped when the driving wheel stops.

2) Forward camera calibration

The forward camera can be calibrated by adopting the prior art to obtain external parameters such as the position and the posture of the forward camera and internal parameters such as an optical center and a focal length.

Without loss of generality, firstly, a second local coordinate system is established by taking the contact point of a driving wheel below a boarding bridge control room as an origin (one driving wheel is selected when a plurality of driving wheels exist), taking the right front of the head end of the control room as a Y axis, taking the direction vertical to the Y axis in a horizontal plane as an X axis, and taking the direction vertical to the ground as a Z axis.

Moving a driving wheel for determining the origin of a coordinate system below a control room of the boarding bridge to a known position on the ground, fixing the control room of the boarding bridge at a certain height, reading and recording the height h of the boarding bridge at the moment from a system sensor of the control room₀(note that in general, this height is the relative height of a point in the control room).

Placing a calibration plate on the ground or selecting calibration points whose positions can be measured, manually or automatically obtaining image coordinates of these points, and adopting known method to make calibration^[1]And obtaining the internal parameters and the external parameters of the forward camera.

The internal parameter matrix a is represented by formula (1):

wherein (u)₀,v₀) Coordinates representing the optical center of the forward-facing camera, α and β represent scale factors of the image coordinates in the horizontal and vertical directions, respectively, and γ represents a deviation coefficient of the horizontal and vertical axes of the image. For convenience, the forward camera imaging distortion is not considered here, and when it is desired to correct for the imaging distortion, the corresponding prior art may be followed.

The extrinsic parameters include the position T of the forward-facing camera in the second local coordinate₀＝[T_x0T_y0T_z0]And attitude angle R₀＝[R_x0R_y0R_z0]。

3) Image coordinates of an image are correlated with the location of a ground point

Since the forward-facing camera is fixed with respect to the control room, other parameters than the height are not changed while the boarding bridge is moved and lifted. And a height T_zCan be obtained by reading the data of the height sensor, i.e.

T_z＝T_z0+(h-h₀) (3)

Wherein h is₀Initial height, T, of boarding bridge recorded for calibration_z0Is the initial height of the forward camera during calibration, and h is the current height of the boarding bridge.

Therefore, when the boarding bridge moves and ascends and descends, the external parameter of the forward camera can be expressed as an attitude angle R ═ R₀And a three-dimensional position T, where T_x0,T_y0,T_z0The initial coordinate position of the forward camera on the axis of the second local coordinate system X, Y, Z is calibrated, so that at any time in the boarding bridge moving machine and lifting process, in the second local coordinate system which is constructed by taking the contact point of the driving wheel below the boarding bridge control room as the origin, the internal parameter A, the external parameter attitude angle R and the external parameter three-dimensional space coordinate T of the forward camera are known or can be obtained by calculation.

The forward-facing camera image coordinate matrix may be expressed as m ═ u v]^TThe three-dimensional point coordinate matrix in the second local coordinate system may be represented as M ═ x y z]^TBy adding a new element

And

and

the relationship between them can be expressed by equation (4):

the ground plane referred to herein may be represented as a plane with z equal to 0 in the second local coordinate system, and therefore, selecting any pixel point on a given digital image may calculate its spatial coordinate [ x y0] under the constraint of z equal to 0 from equation (4).

4) Boarding bridge parking position setting based on position association

The picture (scene image) of the forward camera video is displayed on an operation panel in front of an operator, the operator moves a cursor to the position of a target pixel point in the picture through a mouse or a direction button, after the selection is clicked, the system calculates the actual position on the airport surface corresponding to the position of the pixel point according to a formula (4) to be used as the target position of a corresponding driving wheel below a boarding bridge control room, the target position is transmitted to a control system, and the control system finishes the horizontal movement of the boarding bridge through a driving device of the control system.

In addition, according to the relative relationship between the set stop starting point position (or the actual position when the boarding bridge reaches the stop starting point position) and the standard stop position of the corresponding airplane, the horizontal azimuth angle which the boarding bridge should have when the right front direction of the boarding bridge is perpendicular to the airplane (or the horizontal azimuth angle between the actual position when the boarding bridge reaches the stop starting point position and the standard stop attitude) can be calculated, and the height of the passenger door can be obtained according to airplane model data provided by the system. And driving a boarding bridge front end rotating device and a lifting device according to the obtained horizontal azimuth angle and the passenger cabin door height. Through the operation, the boarding bridge reaches the standard parking attitude at the parking starting point, the front end of the boarding bridge is aligned to the cabin door, the height of the boarding bridge is consistent with that of the cabin door, and only a small safety distance exists between the front end of the boarding bridge and the cabin door, so that the boarding bridge can be conveniently parked finally.

In this specification, when used to distinguish between parameters of different cameras, subscript 1 denotes one of the binocular cameras, subscript 2 denotes the other of the binocular cameras, and there is no corresponding subscript for the parameters of the forward camera.

The binocular camera referred to in the specification includes any image acquisition device having two cameras capable of being used for binocular positioning or ranging, and may be an integrated binocular camera having two cameras, or a combination of two cameras.

The image coordinate system referred to in the present specification means a coordinate system for an image, and includes a so-called image coordinate system and a so-called image coordinate system.

Reference documents:

[1]Zhang Z.A Flexible New Technique for Camera Calibration[M].2000.

[2]He K,Gkioxari Georgia,Dollar Piotr,et al.Mask R-CNN[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2017,PP(99):1-1.

[3]Redmon J,Farhadi A.YOLOv3:An Incremental Improvement[J].2018.

[4]Zhao Q,Sheng T,Wang Y,et al.M2Det:A Single-Shot Object Detector based on Multi-Level Feature Pyramid Network[J].2018.

[5]Hartley R I,Sturm P.Triangulation[J].Comput Vision Image Understanding,1995,68(2):146-157.

[6]Lepetit V,Francesc Moreno-Noguer,Pascal Fua.EPnP:An AccurateO(n)Solution to the PnP Problem[J].International Journal of Computer Vision,2009,81(2):155-166.

[7]Wu Y,Lim,Jongwoo,Yang,Ming-Hsuan.Object Tracking Benchmark[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2015,37(9):1834-1848.

[8]Kristan M,et al.The Visual Object TrackingVOT2013 Challenge Results[C]//IEEE International Conference on Computer Vision Workshops.2014.

[9]Chen L C,Papandreou G,Kokkinos I,et al.Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs[J].Computer Science,2014(4):357-361.

[10]Chen L C,Zhu Y,Papandreou G,et al.Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation[J].2018.

[11]Duda R O,Hart,Peter E.Use of the Hough transformation to detect lines and curves in pictures[J].Cacm,1972,15(1):11-15.

[12]Melkman A A.On-Line Construction of the Convex Hull of a Simple Polyline[M]//On-line construction of the convex hull of a simple polyline.,1987:11-12.