阅读说明：本技术 紧固件安装状态检测方法、装置、电子设备及介质 (Fastener installation state detection method and device, electronic equipment and medium ) 是由 汪顺利 左正新 王飞亚 卢丹 王诗源 沈珉杰 王云霞 于 2020-12-30 设计创作，主要内容包括：本申请实施例公开了一种紧固件安装状态检测方法、装置、电子设备及介质。该方法包括：通过位于至少三个方位的图像采集器对紧固件所在的设备表面区域进行图像采集,得到至少三个采集图像；基于多目视觉定位算法,根据所述至少三个采集图像,确定所述紧固件的实际三维坐标；根据所述紧固件的实际三维坐标,以及预先建立模型中紧固件的理论三维坐标,确定所述紧固件的安装状态。上述方案能够基于机器视觉确定紧固件安装的三维信息,从而精确地确定紧固件的安装位置,并通过与预先建立模型中紧固件位置的比对,从而快速、准确地确定紧固件是否存在移位、错装和漏装等问题,实现高精度、高稳定性的紧固件安装状态检测。(The embodiment of the application discloses a method and a device for detecting the installation state of a fastener, electronic equipment and a medium. The method comprises the following steps: acquiring images of the surface area of the equipment where the fastener is located by image collectors located in at least three directions to obtain at least three acquired images; determining the actual three-dimensional coordinates of the fastener according to the at least three collected images based on a multi-view visual positioning algorithm; and determining the installation state of the fastener according to the actual three-dimensional coordinates of the fastener and the theoretical three-dimensional coordinates of the fastener in the pre-established model. According to the scheme, the three-dimensional information of the installation of the fastener can be determined based on machine vision, so that the installation position of the fastener is accurately determined, the position of the fastener is compared with the position of the fastener in the pre-established model, the problems of displacement, misloading, neglected loading and the like of the fastener are quickly and accurately determined, and the installation state detection of the fastener with high precision and high stability is realized.)

1. A method of detecting a fastener installation condition, the method comprising:

acquiring images of the surface area of the equipment where the fastener is located by image collectors located in at least three directions to obtain at least three acquired images;

determining the actual three-dimensional coordinates of the fastener according to the at least three collected images based on a multi-view visual positioning algorithm;

and determining the installation state of the fastener according to the actual three-dimensional coordinates of the fastener and the theoretical three-dimensional coordinates of the fastener in the pre-established model.

2. The method of claim 1, wherein determining actual three-dimensional coordinates of the fastener from the at least three captured images based on a multi-view visual positioning algorithm comprises:

performing feature extraction and identification on at least three acquired images, and determining two-dimensional coordinates of the fastener in the at least three acquired images;

and determining the actual three-dimensional coordinates of the fastener according to the two-dimensional coordinates and the matching relation of the fastener in the at least three acquired images and the acquisition visual angles of the at least three acquired images.

3. The method according to claim 2, wherein the determining of the matching relationship comprises:

based on a visual analysis algorithm, determining the matching relation of the fasteners in at least three acquired images according to internal parameters and external parameters of at least three image collectors, epipolar geometric constraints among the at least three image collectors and two-dimensional coordinates of the fasteners.

4. The method of claim 1, wherein prior to determining the installation state of the fastener based on the actual three-dimensional coordinates of the fastener and the theoretical three-dimensional coordinates of the fastener in the pre-established model, the method further comprises:

splicing collected images obtained by collecting images aiming at different equipment surface areas, and performing three-dimensional reconstruction on the equipment surface to obtain an actual three-dimensional structure;

correspondingly, determining the installation state of the fastener according to the actual three-dimensional coordinates of the fastener and the theoretical three-dimensional coordinates of the fastener in the pre-established model comprises the following steps:

and determining the installation state of the fastener according to the actual three-dimensional coordinates of the fastener in the actual three-dimensional structure and the theoretical three-dimensional coordinates of the fastener in the pre-established model.

5. The method of claim 1, wherein the actual three-dimensional coordinates of the fastener are coordinates in an image collector coordinate system;

accordingly, before determining the installation state of the fastener according to the actual three-dimensional coordinates of the fastener and the theoretical three-dimensional coordinates of the fastener in the pre-established model, the method further comprises:

detecting the characteristic points on the surface of the equipment where the fastening piece is located through an image collector, and determining the coordinates of the characteristic points in a coordinate system of the image collector;

determining a coordinate conversion relation according to the coordinates of the feature points in the image collector coordinate system and the coordinates of the feature points in a pre-established model coordinate system;

and converting the actual three-dimensional coordinates of the fastener in the image collector coordinate system into the actual three-dimensional coordinates of the fastener in a pre-established model coordinate system according to the coordinate conversion relation.

6. The method of claim 5, wherein determining a coordinate transformation relationship according to the coordinates of the feature points in the image collector coordinate system and the coordinates of the feature points in the pre-established model coordinate system comprises:

matching the characteristic points on the surface of the equipment acquired by the image acquisition device with the characteristic points in the pre-established model;

and determining a coordinate conversion relation according to the coordinate of the successfully matched feature point in the image collector coordinate system and the coordinate in the pre-established model coordinate system.

7. The method of claim 5 or 6, wherein determining the installation state of the fastener based on the actual three-dimensional coordinates of the fastener and the theoretical three-dimensional coordinates of the fastener in the pre-established model comprises:

and comparing the actual three-dimensional coordinates of the fastener in a pre-established model coordinate system with the theoretical three-dimensional coordinates of the fastener in the pre-established model to determine the installation state of the fastener.

8. A fastener installation state detection device, characterized in that the device comprises:

the acquisition image acquisition module is used for acquiring images of the surface area of the equipment where the fastener is located through the image acquisition devices located in at least three directions to obtain at least three acquired images;

the actual three-dimensional coordinate determination module is used for determining the actual three-dimensional coordinate of the fastener according to the at least three acquired images based on a multi-view visual positioning algorithm;

and the mounting state determining module is used for determining the mounting state of the fastener according to the actual three-dimensional coordinates of the fastener and theoretical three-dimensional coordinates of the fastener in a pre-established model.

9. An electronic device, characterized in that the electronic device comprises:

one or more processors;

a memory for storing one or more programs;

when executed by the one or more processors, cause the one or more processors to implement the fastener installation status detection method of any one of claims 1-7.

10. A computer-readable storage medium on which a computer program is stored, the program, when executed by a processor, implementing the fastener installation state detection method according to any one of claims 1 to 7.

Technical Field

The embodiment of the application relates to the technical field of part detection, in particular to a method and a device for detecting the installation state of a fastener, electronic equipment and a medium.

Background

During the assembly of an aircraft, there is a great deal of drilling and fastener installation work. Due to the complex shape of the airplane, a plurality of hole making and fastener installation work needs to be completed manually. The installation of fasteners is usually checked in the links of component assembly, acceptance and the like, the number of the fasteners on one passenger plane is as many as hundreds of thousands of groups, and the structural size of the airplane is very large, so that the detection workload is very large.

The traditional method for detecting the fastener is manually completed, has low detection efficiency and difficult control of precision, is time-consuming and labor-consuming especially for large-batch detection, and is difficult to adapt to the development requirement of modern aircraft assembly. And the detection result is influenced by factors such as personal experience, mental state, fatigue degree and the like of detection personnel, lacks stability and objectivity, and is easy to cause problems such as missing detection, false detection and the like. The traditional two-dimensional image-based industrial detection method can meet the requirement of general defect detection, but the close association with the spatial position relation cannot be realized due to the lack of three-dimensional information.

Disclosure of Invention

The embodiment of the invention provides a method and a device for detecting the installation state of a fastener, electronic equipment and a medium, which are used for improving the detection efficiency and accuracy of the installation state of the fastener.

In one embodiment, the present application provides a method for detecting a fastener installation state, including:

acquiring images of the surface area of the equipment where the fastener is located by image collectors located in at least three directions to obtain at least three acquired images;

determining the actual three-dimensional coordinates of the fastener according to the at least three collected images based on a multi-view visual positioning algorithm;

and determining the installation state of the fastener according to the actual three-dimensional coordinates of the fastener and the theoretical three-dimensional coordinates of the fastener in the pre-established model.

In another embodiment, the present application further provides a fastener installation state detection apparatus, including:

the acquisition image acquisition module is used for acquiring images of the surface area of the equipment where the fastener is located through the image acquisition devices located in at least three directions to obtain at least three acquired images;

the actual three-dimensional coordinate determination module is used for determining the actual three-dimensional coordinate of the fastener according to the at least three acquired images based on a multi-view visual positioning algorithm;

and the mounting state determining module is used for determining the mounting state of the fastener according to the actual three-dimensional coordinates of the fastener and theoretical three-dimensional coordinates of the fastener in a pre-established model.

In another embodiment, an embodiment of the present application further provides an electronic device, including: one or more processors;

a memory for storing one or more programs;

when the one or more programs are executed by the one or more processors, the one or more processors implement the fastener installation state detection method according to any one of the embodiments of the present application.

In yet another embodiment, the present application further provides a computer-readable storage medium on which a computer program is stored, the program, when executed by a processor, implementing the fastener installation state detection method according to any one of the embodiments of the present application.

In the embodiment of the application, the image acquisition device positioned in at least three directions is used for acquiring the image of the surface area of the equipment where the fastener is positioned to obtain at least three acquired images; and determining the actual three-dimensional coordinate of the fastener according to the at least three acquired images based on a multi-view visual positioning algorithm, thereby accurately determining the three-dimensional coordinate position of the fastener on the surface of the equipment, determining the installation state of the fastener according to the actual three-dimensional coordinate of the fastener and the theoretical three-dimensional coordinate of the fastener in the pre-established model, thereby quickly and efficiently determining the installation state of the fastener, and automatically and accurately detecting whether the installation of the fastener has the problems of dislocation, misloading, neglected loading and the like.

Drawings

FIG. 1 is a flow chart of a method for detecting a fastener installation status according to one embodiment of the present invention;

FIG. 2 is a flow chart of a method of detecting a fastener installation condition according to another embodiment of the present invention;

FIG. 3 is a flow chart of the triangular camera monitoring provided by the present invention;

FIG. 4 is a flow chart of an algorithm provided by the present invention;

FIG. 5 is a schematic diagram of an original image provided by the present invention;

FIG. 6 is a schematic diagram of an image after threshold binarization processing according to the present invention;

FIG. 7 is a schematic view of a filtered image provided by the present invention;

FIG. 8 is a schematic diagram of a final inspection image provided by the present invention;

FIG. 9 is a schematic diagram of dynamic 3D coordinates provided by the present invention;

FIG. 10 is a schematic diagram of the movement of a camera provided by the present invention;

FIG. 11 is a schematic diagram of two adjacent frames according to the present invention;

FIG. 12 is a diagram illustrating a digital-to-analog file before deduplication provided by the present invention;

FIG. 13 is a schematic diagram of a de-duplicated digital-to-analog file according to the present invention;

FIG. 14 is a diagram of a standard digital-to-analog file provided by the present invention;

fig. 15 is a schematic structural view of a fastener installation state detection apparatus according to an embodiment of the present invention;

fig. 16 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.

Fig. 1 is a flowchart of a method for detecting a fastener installation state according to an embodiment of the present invention. The method for detecting the installation state of the fastener provided by the embodiment can be applied to the situation of detecting the installation state of the fastener. Typically, the method may be applied to the inspection of fasteners installed on the surface of an aircraft skin. The method may be specifically executed by a fastener installation state detection device, where the device may be implemented by software and/or hardware, and the device may be integrated in an electronic device capable of implementing the fastener installation state detection method provided in the embodiments of the present application. The method for monitoring the installation state of the fastener provided by the embodiment of the application can be applied to detection of the fastener on any equipment surface, and for convenience of visual description, the installation state of the fastener on the surface of an airplane is detected. Referring to fig. 1, the method of the embodiment of the present application specifically includes:

s110, image acquisition is carried out on the surface area of the equipment where the fastener is located through image collectors located in at least three directions, and at least three acquired images are obtained.

In embodiments of the present application, the equipment surface may be an aircraft skin surface. The surface of the aircraft skin is provided with a hole for inserting a fastener, and the fastener is arranged in the hole. The surface areas of the equipment collected by the image collectors in at least three directions are the same area. The image collector in at least three positions collects the image of the surface area of the equipment where the fastener is located, so that the multi-directional information of the surface area of the equipment where the fastener is located can be obtained, and the position of the fastener can be determined more accurately.

In the embodiment of the application, the image acquisition of the surface area of the equipment where the fastener is located can be performed through the image acquisition devices in at least three directions, so that a group of acquired images can be obtained. The image collectors in at least three directions can also be used for collecting the traversing images of the surface of the aircraft skin to obtain a plurality of groups of collected images so as to detect and analyze the mounting state of the fasteners on the surface of the aircraft skin.

And S120, determining the actual three-dimensional coordinates of the fastener according to the at least three acquired images based on a multi-view visual positioning algorithm.

Illustratively, at least three collected images are collected from three positions, depth information of the fastener can be determined through the images at different positions, and then the actual three-dimensional coordinates of the fastener can be determined according to the coordinates of the fastener in the at least three collected images and the depth information.

Specifically, at least three collected images can be matched pairwise to determine a matching relationship, and then the actual three-dimensional coordinates of the fastener in the image collection system are determined according to the matching relationship and the geometric constraint conditions based on a multi-view visual positioning algorithm.

S130, determining the installation state of the fastener according to the actual three-dimensional coordinates of the fastener and the theoretical three-dimensional coordinates of the fastener in the pre-established model.

Illustratively, an aircraft skin surface model is previously established based on theoretical specifications of the aircraft and installation specifications of fasteners, the previously established model includes fasteners that are correctly installed in accordance with theory, and theoretical three-dimensional coordinates of the fasteners that are correctly installed in accordance with theory are known. According to the comparison between the actual three-dimensional coordinates of the fastener detected actually and the theoretical three-dimensional coordinates of the fastener in the pre-established model, the installation state of the fastener can be known, and problems such as dislocation, neglected loading, misloading and the like exist. By comparison with the theoretical three-dimensional coordinates of the fasteners in the pre-established model, the distance of the fasteners from the edge, the spacing of the fasteners, the offset distance of the fasteners, etc. in the case of misalignment can also be determined.

In the embodiment of the application, the image acquisition device positioned in at least three directions is used for acquiring the image of the surface area of the equipment where the fastener is positioned to obtain at least three acquired images; and determining the actual three-dimensional coordinate of the fastener according to the at least three acquired images based on a multi-view visual positioning algorithm, thereby accurately determining the three-dimensional coordinate position of the fastener on the surface of the equipment, determining the installation state of the fastener according to the actual three-dimensional coordinate of the fastener and the theoretical three-dimensional coordinate of the fastener in the pre-established model, thereby quickly and efficiently determining the installation state of the fastener, and automatically and accurately detecting whether the installation of the fastener has the problems of dislocation, misloading, neglected loading and the like.

Fig. 2 is a flowchart of a method for detecting a fastener installation state according to another embodiment of the present invention. For further optimization of the embodiments, details which are not described in detail in the embodiments are described in the embodiments. Referring to fig. 2, the method for detecting the installation state of the fastener provided by the embodiment may include:

s210, image acquisition is carried out on the surface area of the equipment where the fastener is located through image collectors located in at least three directions, and at least three acquired images are obtained.

S220, performing feature extraction and recognition on the at least three acquired images, and determining two-dimensional coordinates of the fastener in the at least three acquired images.

For example, at least three captured images are obtained by capturing images of the surface area of the device where the fastener is located, and therefore, it is necessary to extract and identify the features of the images corresponding to the area, so as to identify the fastener in the captured images and determine the two-dimensional coordinates of the fastener in the image capturing system.

S230, determining the actual three-dimensional coordinates of the fastener according to the two-dimensional coordinates and the matching relation of the fastener in the at least three acquired images and the acquisition visual angles of the at least three acquired images.

Illustratively, at least three collected images can reflect the position information and the specification information of the fastener from multiple directions, and more accurate depth information can be embodied. And determining the same fastener according to the matching relationship, and positioning the actual position of the fastener according to the two-dimensional coordinates of the fastener and the acquisition visual angles of the at least three image collectors, thereby determining the actual three-dimensional coordinates of the fastener.

In this embodiment of the present application, the determining process of the matching relationship includes: based on a visual analysis algorithm, determining the matching relation of the fasteners in at least three acquired images according to internal parameters and external parameters of at least three image collectors, epipolar geometric constraints among the at least three image collectors and two-dimensional coordinates of the fasteners.

The internal reference and the external reference of the image collector can be adjusted in advance according to actual conditions. For example, at least three captured images may be matched two by two, and the matching relationship of the at least three captured images may be determined for the same fastener.

S240, splicing collected images obtained by collecting images of different equipment surface areas, and performing three-dimensional reconstruction on the equipment surface to obtain an actual three-dimensional structure.

For example, the installation state of the fastener on the surface of the whole aircraft skin generally needs to be detected, so that image collectors located in at least three directions can be used for performing traversal collection on the surface of the aircraft skin to obtain multiple groups of collected images, then the collected images belonging to different aircraft surface areas are spliced, and three-dimensional reconstruction is performed based on depth information to obtain an actual three-dimensional structure. Correspondingly, determining the installation state of the fastener according to the actual three-dimensional coordinates of the fastener and the theoretical three-dimensional coordinates of the fastener in the pre-established model comprises the following steps: and determining the installation state of the fastener according to the actual three-dimensional coordinates of the fastener in the actual three-dimensional structure and the theoretical three-dimensional coordinates of the fastener in the pre-established model.

And S250, detecting the characteristic points on the surface of the equipment where the fasteners are located through the image collector, and determining the coordinates of the characteristic points in the coordinate system of the image collector.

The characteristic points are inherent characteristic points on the surface of the equipment, such as logo marks, component marks, color marks and the like, and the characteristic points on the surface of the equipment where the fastener is located are detected through the image collectors located in at least three directions, so that three-dimensional coordinates of the characteristic points are calculated. The three-dimensional coordinates of the feature points and the three-dimensional coordinates of the fasteners are calculated in the same manner.

And S260, determining a coordinate conversion relation according to the coordinates of the feature points in the image collector coordinate system and the coordinates of the feature points in a pre-established model coordinate system.

Since the actual three-dimensional coordinates of the detected fastener need to be compared with the theoretical three-dimensional coordinates of the fastener in the pre-established model to determine the installation state of the fastener, the actual three-dimensional coordinates of the fastener and the theoretical three-dimensional coordinates of the fastener in the pre-established model need to be converted into the same coordinate system for comparison, and the reference is provided. In the embodiment of the application, the coordinate conversion relationship can be determined according to the coordinates of the feature points in the image collector coordinate system and the coordinates of the feature points in the pre-established model coordinate system. For example, for the same feature point, a relational equation is established according to the three-dimensional coordinate of the feature point in the image collector coordinate system and the three-dimensional coordinate of the feature point in the pre-established model coordinate system, and a coordinate transformation matrix is obtained through solving.

In this embodiment of the present application, determining a coordinate transformation relationship according to the coordinates of the feature points in the image collector coordinate system and the coordinates of the feature points in the pre-established model coordinate system includes: matching the characteristic points on the surface of the equipment acquired by the image acquisition device with the characteristic points in the pre-established model; and determining a coordinate conversion relation according to the coordinate of the successfully matched feature point in the image collector coordinate system and the coordinate in the pre-established model coordinate system.

Illustratively, feature points on the surface of the device collected by the image collector are matched with feature points in a pre-established model, so as to determine the same feature point. And determining the coordinate conversion relation between the image collector coordinate system and the pre-established model coordinate system according to the three-dimensional coordinates of the same characteristic point in the image collector coordinate system and the three-dimensional coordinates in the pre-established model.

And S270, converting the actual three-dimensional coordinates of the fastener in the image collector coordinate system into the actual three-dimensional coordinates of the fastener in the pre-established model coordinate system according to the coordinate conversion relation.

In the embodiment of the application, the three-dimensional coordinates of the fastener are coordinates in the image collector coordinate system. Illustratively, the three-dimensional coordinates of the fastener in the image collector coordinate system can be converted into the actual three-dimensional coordinates of the fastener in the pre-established model coordinate system according to the coordinate conversion relation. The actual three-dimensional coordinates of the fastener in the pre-established model coordinate system can also be converted into the actual three-dimensional coordinates of the fastener in the image collector coordinate system.

S280, comparing the actual three-dimensional coordinates of the fastener in the pre-established model coordinate system with the theoretical three-dimensional coordinates of the fastener in the pre-established model, and determining the installation state of the fastener.

Because the actual three-dimensional coordinate and the theoretical three-dimensional coordinate of the fastener are converted into the same coordinate system, the actual three-dimensional coordinate and the theoretical three-dimensional coordinate of the fastener can be directly compared, so that whether the actual three-dimensional coordinate and the theoretical three-dimensional coordinate of the fastener are overlapped or not is determined, if the actual three-dimensional coordinate and the theoretical three-dimensional coordinate of the fastener are overlapped, the situation that the fastener is not misplaced, misassembled, neglected assembled and the like is indicated, and if the actual three-dimensional coordinate and the theoretical three-dimensional coordinate of the fastener are not overlapped, the situation that the fastener is misplaced, misassembled, neglected assembled and the like is indicated. If the actual three-dimensional coordinate and the theoretical three-dimensional coordinate of the same fastener are not coincident and have dislocation, the offset distance of the fastener can be determined according to the coordinate difference. And if the theoretical three-dimensional coordinates of the fastener in the pre-established model do not have the corresponding actual three-dimensional coordinates, indicating that the fastener is in a missing condition. If the theoretical three-dimensional coordinates of the fasteners in the pre-established model have actual three-dimensional coordinates corresponding to the theoretical three-dimensional coordinates, but the models, specifications and the like of the fasteners are inconsistent, the situation that the fasteners are mistakenly installed is explained.

According to the scheme in the embodiment of the application, the three-dimensional coordinates of the fastener are determined through multi-view vision, and the detection precision can be improved. The three-dimensional coordinate information of the fastener is accurately recovered, the actual three-dimensional coordinate of the fastener and the theoretical three-dimensional coordinate of the fastener are converted into coordinates under the same coordinate system, so that comparison is directly performed, the detection accuracy and efficiency of the installation state of the fastener are improved, the robustness is good, the operation is convenient, a Marker does not need to be pasted, the aircraft surface detection device is suitable for various aircraft surfaces, and the result is stable and reliable.

The embodiment of the application is a specific implementation mode for monitoring the installation state of a fastener. The method specifically comprises the following steps:

the monitoring flow of the triangular camera is shown in fig. 3, in the operation process of the whole system, the mechanical arm moves, the strip light source is lightened, and the triangular camera synchronously shoots the surface image of the skin of the airplane at the speed of 5 frames per second. And in the process of writing the image into the buffer area, the background performs data preprocessing in parallel, judges whether the situation of discontinuous frames occurs in the process of acquiring the image, and simultaneously generates pixel coordinates of the rivet in each photo in parallel. And the mechanical arm moves in place and pauses, the triangular camera stops shooting, the data preprocessing is continuously carried out, and the data is reserved for the defect detection module to acquire images.

And the mechanical arm completely finishes moving and returns to the Home position. After the triangular camera image shooting is completed, the data preprocessing thread arranges all rivet pixel coordinate information into a t _ points _ data. pkl file and transmits the t _ points _ data. pkl file to the visual inspection algorithm analysis module, and finally a rivet digital-analog obj file is generated.

The digital model of the aircraft skin surface rivet is compared with the standard digital model used for manufacturing, the deviation between the actual rivet position and the standard design rivet position can be analyzed, and then the situations of dislocation, missing hitting, multiple hitting and the like of the rivet position can be analyzed.

The algorithm flow is shown in fig. 4, and specifically includes: after a set of continuous images of the camera during movement is taken, the rivet position in each image is identified using a rivet detection algorithm. The detection algorithm mainly utilizes two characteristic information of rivet polishing, one is that the surface of the rivet reflects light under the illumination condition, so that the gray-scale image has higher brightness, the boundary information is more prominent, and the detection algorithm is easy to distinguish from the surface of the aircraft skin; the other is that the morphological characteristic of the rivet is round, and whether the highlighted connected region is a rivet or not can be judged through the roundness detection of the region. In addition, the data can be further filtered through the rivet area and the close rivet spacing so as to calculate accurate rivet point information.

And after the rivet position in the image is obtained, the rivet information in three pictures shot by the triangular camera at the same moment is transmitted into the triangular positioning reconstruction algorithm module. The 3D coordinates of the rivet in the world coordinate system can be reversely calculated by utilizing different position information of the same rivet point in the three images. The three images of each frame are processed, and dynamic 3D coordinate information of the rivet at different moments can be obtained.

In the point cloud splicing module, 3D coordinates of rivets at different moments are re-projected into the same world coordinate system one by one, repeated rivets are filtered out, and newly identified rivets are added frame by frame. And the camera external reference matrix is optimized by a light beam adjustment method, and the error of the reprojection is reduced to obtain the accurate rivet 3D coordinate. And finally splicing to obtain a complete machine body surface rivet 3D coordinate.

The accuracy of the camera parameters Params is the basis for the successful performance of the algorithm. In the image captured by the triangular camera, the position information of the rivet in the image needs to be obtained, so that the background information needs to be filtered. Due to the metal material of the rivet, under the appropriate illumination condition, the rivet has higher brightness value in the image as a result of specular reflection. The surface of the body is used as a background, and the optical characteristics of the body are obviously different from those of the rivet due to the fact that the body is brushed with a layer of paint.

The morphological characteristic of the rivet is circular, and whether the highlighted connected region is the rivet or not can be judged through roundness detection of the region.

Wherein, circle _ rate is roundness, S is the area of the connected domain, and C is the perimeter of the connected domain. After threshold value binarization processing, the foreground and the background of the rivet can be effectively separated. The original image is shown in fig. 5, and the image subjected to the threshold binarization processing is shown in fig. 6.

After the profile is detected, the rivets are filtered using roundness circle rate and connected field area S, and the roundness of one rivet should be close to 1. And then, the area is filtered, so that noise points in the image and tiny stains on the surface of the machine body can be effectively clear, as shown in fig. 7. The final test results are shown in fig. 8.

And binding the identified rivet pixel coordinates with the camera number and the frame serial number together to organize a t _ points data structure, and transmitting the t _ points data structure to a subsequent algorithm analysis module. And after the rivet position in the image is obtained, the rivet information in three pictures shot by the triangular camera at the same moment is transmitted into the triangular positioning reconstruction algorithm module. And constructing a two-eye matching relationship between every two triangular cameras according to position internal reference data, and sequentially performing distance constraint and angle constraint on feature points on images of the two cameras to obtain a candidate matching relationship. And matching each group of obtained characteristic points into points to establish a graph structure. And (3) obtaining a plurality of independent sets by solving the maximum independent set, recovering the matching relation of the maximum independent set, the maximum independent set and the three, and obtaining the 3D point by using a triangulation algorithm. The correct maximum independent set solution is determined by re-projecting the 3D points onto the 2D image and checking the original results.

The dynamic 3D coordinate information of the rivet at different moments can be obtained by performing the processing on the frame sequence. As shown in fig. 9. The 3D coordinates reconstructed in each frame are a unified world coordinate system with the camera coordinate system No. 1 of the frame. Only rivets that can be photographed by three cameras can be successfully positioned. The field of view of the triangular camera needs to be slightly larger than the effective area of detection.

Searching matching points to restore a camera pose matrix: and acquiring the three-dimensional coordinates of the rivet under each frame. And 3D points are converted into the same coordinate system, so that the subsequent repeated points are removed and a digital-analog is generated. Therefore, a coordinate transformation matrix Trans (rotation-translation matrix R, t) between frames needs to be established. The change in rivet position from frame to frame is a rigid body transformation, and their relative rotational translation matrices R, t are recovered, and the method of SVD decomposition can be used.

On the premise that the mechanical arm moves at a constant speed, the deviation condition of the coordinates of the rivet corresponding to the current frame and the next frame can be predicted. By finding the nearest neighbor, the relation between the rivets corresponding to the current frame and the rivets corresponding to the next frame can be determined as a corresponding relation, and the indexes ind _ src and ind _ dst are established.

And after the corresponding relation is obtained, decomposing the R and t matrixes by using SVD.

set1＝P3d1[ind_src]

set2＝P3d2[ind_dst]

mean1＝mean(set1)

mean2＝mean(set2)

Q＝(set2-mean2)·T·(set1-mean1)

U，Sigma，Vt＝SVD(Q)

R＝U·Vt

t＝mean1·T-R·(mean2·T)

Wherein, set1 and set2 are three-dimensional coordinate sets of corresponding points of two frames of images, p3d1 and p3d2 are three-dimensional coordinates of two frames of photos, ind _ src and ind _ dst are indexes of corresponding rivets, mean1 and set1 are averaged, mean2 is averaged for set2, and U, sigma and Vt are three process quantities decomposed by an SVD method. And establishing a transformation matrix from the first frame to any one frame by utilizing the continuity between the frames, so that all the three-dimensional coordinates of the rivet can be converted into the same world coordinate system.

Trans(1to n)＝Trans(1to2)·Trans(1to3)…Trans(k-1to k)

And the recovered camera pose matrix is stored and transmitted to subsequent algorithm modules for point cloud duplicate removal, visibility judgment and the like according to a standard data format.

Restoring to the 3D points of each frame in the same world coordinate system clearly sees the movement of the camera, as shown in fig. 10.

Optimizing the pose of the camera by a light beam adjustment method: since there is some error in the positioning using the triangular camera system and the calculation of the relative transformation matrix, the 3D coordinates in two adjacent frames may not be consistent for the same rivet point in the real world. However, if the matching relation is known to be accurate, for example, the point a of the frame and the point B of the next frame are the same rivet in the real world, the coordinate distances of a-3D and B-3D between two adjacent frames can be drawn by iteratively adjusting the relative transformation matrix, and finally, the positioning accuracy is improved. As shown in fig. 11.

The iterative optimization process is realized by a g2o graph optimization method, and the LM iterative method is adopted for calculation, wherein the relative pose matrix and the coordinates are used as vertexes, and the re-projected pixel coordinates are used as measuring edges. The edges are established by the previously acquired point pair matching relationship.

Point cloud de-weighting and splicing: in the three-dimensional coordinates of a real rivet, a 4 to 5 repeat rivet point with a small positional offset will appear from a real rivet position because it will be seen and located by many frames. These duplicate alternatives need to be removed and only 1 high-confidence rivet point is reserved for writing the final digifax file. As shown in fig. 12. The digital-to-analog file obtained after the deduplication is shown in fig. 13.

Digital-analog matching: and generating a detected rivet obj model, and comparing the generated rivet model with an existing digital model. Calculating and standard figures and models for each rivet point, operating a comparison program to output the deviation value of each rivet and the standard figures and counting the deviation value. The standard digifax is shown in fig. 14.

Fig. 15 is a schematic structural view of a fastener installation state detection apparatus according to an embodiment of the present invention. The device is applicable to the situation of encrypting the resources. Typically, the method can be applied to a case where the amount of resources is large and only part of the resources are encrypted. The apparatus may be implemented by software and/or hardware, and the apparatus may be integrated in an electronic device. Referring to fig. 15, the apparatus specifically includes:

the acquired image acquisition module 310 is configured to acquire images of the surface area of the device where the fastener is located through image collectors located in at least three directions to obtain at least three acquired images;

an actual three-dimensional coordinate determination module 320, configured to determine an actual three-dimensional coordinate of the fastener according to the at least three collected images based on a multi-view visual positioning algorithm;

and the installation state determining module 330 is used for determining the installation state of the fastener according to the actual three-dimensional coordinates of the fastener and the theoretical three-dimensional coordinates of the fastener in the pre-established model.

In this embodiment, the actual three-dimensional coordinate determination module 320 includes:

the two-dimensional coordinate determination unit is used for extracting and identifying the characteristics of at least three acquired images and determining the two-dimensional coordinates of the fastener in the at least three acquired images;

and the three-dimensional coordinate determination unit is used for determining the actual three-dimensional coordinate of the fastener according to the two-dimensional coordinate of the fastener in the at least three acquired images, the matching relation and the acquisition visual angles of the at least three acquired images.

In an embodiment of the present application, the apparatus further includes:

and the matching relation determining module is used for determining the matching relation of the fasteners in the at least three acquired images according to internal parameters and external parameters of the at least three image acquisition devices, epipolar geometric constraints among the at least three image acquisition devices and two-dimensional coordinates of the fasteners based on a visual analysis algorithm.

In an embodiment of the present application, the apparatus further includes:

the three-dimensional reconstruction module is used for splicing acquired images acquired by acquiring images aiming at different equipment surface areas and performing three-dimensional reconstruction on the equipment surface to acquire an actual three-dimensional structure;

accordingly, the installation state determination module 330 includes:

and the three-dimensional structure fastener detection unit is used for determining the installation state of the fastener according to the actual three-dimensional coordinates of the fastener in the actual three-dimensional structure and the theoretical three-dimensional coordinates of the fastener in the pre-established model.

In the embodiment of the application, the actual three-dimensional coordinates of the fastener are coordinates in a coordinate system of the image collector;

correspondingly, the device further comprises:

the characteristic point detection module is used for detecting the characteristic points on the surface of the equipment where the fasteners are located through the image collector and determining the coordinates of the characteristic points in the coordinate system of the image collector;

the coordinate conversion relation determining module is used for determining a coordinate conversion relation according to the coordinates of the feature points in the image collector coordinate system and the coordinates of the feature points in a pre-established model coordinate system;

and the coordinate conversion module is used for converting the actual three-dimensional coordinates of the fastener in the image collector coordinate system into the actual three-dimensional coordinates of the fastener in the pre-established model coordinate system according to the coordinate conversion relation.

In an embodiment of the present application, the coordinate transformation relation determining module includes:

the matching unit is used for matching the characteristic points on the surface of the equipment collected by the image collector with the characteristic points in the pre-established model;

and the relationship determining unit is used for determining the coordinate conversion relationship according to the coordinates of the successfully matched feature points in the image collector coordinate system and the coordinates in the pre-established model coordinate system.

In this embodiment, the installation status determining module 330 includes:

and the coordinate comparison unit is used for comparing the actual three-dimensional coordinates of the fastener in a pre-established model coordinate system with the theoretical three-dimensional coordinates of the fastener in the pre-established model to determine the installation state of the fastener.

The device for detecting the installation state of the fastener, provided by the embodiment of the application, can execute the method for detecting the installation state of the fastener, provided by any embodiment of the application, and has corresponding functional modules and beneficial effects of the execution method.

Fig. 16 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. FIG. 16 illustrates a block diagram of an exemplary electronic device 412 suitable for use in implementing embodiments of the present application. The electronic device 412 shown in fig. 16 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 16, the electronic device 412 may include: one or more processors 416; the memory 428 is configured to store one or more programs, when the one or more programs are executed by the one or more processors 416, so that the one or more processors 416 implement the method for detecting the installation state of the fastener provided in the embodiment of the present application, including:

acquiring images of the surface area of the equipment where the fastener is located by image collectors located in at least three directions to obtain at least three acquired images;

determining the actual three-dimensional coordinates of the fastener according to the at least three collected images based on a multi-view visual positioning algorithm;

and determining the installation state of the fastener according to the actual three-dimensional coordinates of the fastener and the theoretical three-dimensional coordinates of the fastener in the pre-established model.

The components of the electronic device 412 may include, but are not limited to: one or more processors or processors 416, a memory 428, and a bus 418 that couples the various device components including the memory 428 and the processors 416.

Bus 418 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, transaction ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

Electronic device 412 typically includes a variety of computer device-readable storage media. These storage media may be any available storage media that can be accessed by electronic device 412 and includes both volatile and nonvolatile storage media, removable and non-removable storage media.

Memory 428 can include computer-device readable storage media in the form of volatile memory, such as Random Access Memory (RAM)430 and/or cache memory 432. The electronic device 412 may further include other removable/non-removable, volatile/nonvolatile computer device storage media. By way of example only, storage system 434 may be used to read from and write to non-removable, nonvolatile magnetic storage media (not shown in FIG. 16, commonly referred to as "hard drives"). Although not shown in FIG. 16, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical storage medium) may be provided. In these cases, each drive may be connected to bus 418 by one or more data storage media interfaces. Memory 428 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.

A program/utility 440 having a set (at least one) of program modules 442 may be stored, for instance, in memory 428, such program modules 442 including, but not limited to, an operating device, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. The program modules 442 generally perform the functions and/or methodologies of the described embodiments of the invention.

The electronic device 412 may also communicate with one or more external devices 414 (e.g., keyboard, pointing device, display 424, etc.), with one or more devices that enable a user to interact with the electronic device 412, and/or with any devices (e.g., network card, modem, etc.) that enable the electronic device 412 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 422. Also, the electronic device 412 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) through the network adapter 420. As shown in FIG. 16, network adapter 420 communicates with the other modules of electronic device 412 over bus 418. It should be appreciated that although not shown in FIG. 16, other hardware and/or software modules may be used in conjunction with the electronic device 412, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID devices, tape drives, and data backup storage devices, among others.

The processor 416 executes various functional applications and data processing by executing at least one of other programs of the plurality of programs stored in the memory 428, for example, to implement a method of detecting a fastener installation state provided by an embodiment of the present application.

One embodiment of the present invention provides a storage medium containing computer-executable instructions which, when executed by a computer processor, perform a fastener installation status detection method, comprising:

acquiring images of the surface area of the equipment where the fastener is located by image collectors located in at least three directions to obtain at least three acquired images;

determining the actual three-dimensional coordinates of the fastener according to the at least three collected images based on a multi-view visual positioning algorithm;

and determining the installation state of the fastener according to the actual three-dimensional coordinates of the fastener and the theoretical three-dimensional coordinates of the fastener in the pre-established model.

The computer storage media of the embodiments of the present application may take any combination of one or more computer-readable storage media. The computer readable storage medium may be a computer readable signal storage medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the present application, a computer readable storage medium may be any tangible storage medium that can contain, or store a program for use by or in connection with an instruction execution apparatus, device, or apparatus.

A computer readable signal storage medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal storage medium may also be any computer readable storage medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution apparatus, device, or apparatus.

Program code embodied on a computer readable storage medium may be transmitted using any appropriate storage medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or device. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.