阅读说明：本技术 一种增强现实半实物复杂机电设备培训系统 (Augmented reality semi-physical complex electromechanical device training system ) 是由 金朋轩 于 2021-07-05 设计创作，主要内容包括：本发明提供了一种增强现实半实物复杂机电设备培训系统,包括半实物硬件构建模块、AR培训开发模块和AR融合显示模块；所述半实物硬件构建模块用于半实物硬件的设计和控制信号的生成,所述AR培训开发模块用于设备虚拟模型分类、模型点云特征计算、指导流程建立和AR数据包导出,所述AR融合显示模块用于将半实物硬件和设备虚拟模型利用增强现实三维注册技术进行虚实融合,并实现在设备实际控制信号的控制下进行运动过程模拟。本发明能够将半实物仿真和增强现实技术相结合,将真实培训设备的工作状态进行有效、直观的展示。(The invention provides an augmented reality semi-physical complex electromechanical equipment training system which comprises a semi-physical hardware construction module, an AR training development module and an AR fusion display module; the system comprises a hardware semi-physical building module, an AR training development module, an AR fusion display module and an AR fusion module, wherein the hardware semi-physical building module is used for designing semi-physical hardware and generating control signals, the AR training development module is used for classifying equipment virtual models, calculating model point cloud characteristics, establishing guide flows and exporting AR data packets, and the AR fusion display module is used for carrying out virtual-real fusion on the hardware semi-physical and the equipment virtual models by utilizing an augmented reality three-dimensional registration technology and realizing motion process simulation under the control of actual control signals of equipment. The invention can combine the semi-physical simulation and the augmented reality technology, and effectively and visually display the working state of the real training equipment.)

1. The utility model provides an augmented reality semi-physical object complicated electromechanical device training system which characterized in that: the system comprises a semi-physical hardware construction module, an AR training development module and an AR fusion display module;

the hardware construction module is used for designing the hardware and generating control signals, and comprises a signal control submodule and a part hiding control submodule; the signal control submodule is responsible for sending the state information of the equipment to the AR fusion display module, analyzing the control instruction sent by the AR fusion display module, sending the control information to the part hiding control submodule, and hiding or displaying the part by the part hiding control submodule according to the control information;

the AR training development module is used for equipment virtual model classification, model point cloud feature calculation, guidance process establishment and AR data packet export and comprises a guidance process submodule, an appearance model definition submodule, a control model definition submodule, a workpiece model definition submodule and an AR data packet publishing submodule; the guiding process submodule is used for establishing an installation process of the real object parts and guiding the installation of the hardware parts; the appearance model definition submodule is used for defining an appearance model consistent with the appearance of the real object and providing reference data for three-dimensional registration; the control model definition submodule is used for defining a control model and a model motion direction; the workpiece model definition submodule is used for defining a machined workpiece model and providing basic data for subsequent workpiece machining simulation; the AR data packet issuing submodule is used for serializing the installation process, the appearance model, the control model characteristics and the workpiece model information into an AR data packet and then sending the data packet to the AR fusion display module;

the AR fusion display module is used for carrying out virtual-real fusion on the semi-physical hardware and the equipment virtual model by utilizing an augmented reality three-dimensional registration technology and realizing motion process simulation under the control of an equipment actual control signal, and comprises an AR three-dimensional registration sub-module, an AR auxiliary guidance sub-module, a training process simulation sub-module and a workpiece state simulation sub-module; the AR three-dimensional registration sub-module is used for point cloud collection, point cloud segmentation, point cloud identification and point cloud registration of the semi-physical hardware and virtual-real fusion of the equipment virtual model and the semi-physical hardware; the AR auxiliary guidance sub-module is responsible for performing auxiliary guidance on the installation process of the installation type parts according to the installation process information in the AR data packet; the training process simulation submodule controls a control model in the AR data packet to move in real time according to the defined movement direction according to the equipment state information sent by the signal control submodule; and the workpiece state simulation submodule carries out machining phenomenon simulation on the workpiece model in the workpiece data packet according to the running track of the control model in the AR data packet.

2. The augmented reality semi-physical complex electromechanical device training system of claim 1, wherein: the establishment of the semi-physical hardware building module comprises the following steps:

a1: classifying the parts according to functions born by each part in the actual equipment, and dividing the parts into appearance parts, installation parts, operation parts, control parts and other parts;

a2: assembling appearance parts, installation parts, operation parts and control parts together to form semi-physical hardware equipment;

a3: designing a signal acquisition system for the operation type part, and accessing a control signal generated by the control type part due to state change into a signal control submodule;

a4: designing and manufacturing a mechanical structure for the installation type part, enabling the mechanical structure to be displayed or contracted and hidden in the box body, and adding a displayed or contracted and hidden state signal to the signal control submodule;

a5: and independent power supply circuits for ensuring the operation of the mechanical structure and the control circuit of the hardware equipment are designed for the hardware equipment.

3. The augmented reality semi-physical complex electromechanical device training system of claim 1, wherein: the establishment of the AR data packet comprises the following steps:

b1: importing a complete three-dimensional equipment digital model into an AR training development module;

b2: classifying the part models in the virtual model of the equipment according to the functions born by each equipment part in the actual equipment, and dividing the part models into an appearance part model, an installation part model, an operation part model, a control part model and a workpiece part model;

b3: according to the training function requirement, establishing a guidance animation of the installation type part model for the installation type part model in an AR training development module;

b4: carrying out omnibearing spatial information acquisition processing on the appearance part model and the operation part model through a virtual camera, calculating point cloud data corresponding to the appearance part model and the operation part model at each visual angle, and generating VFHF characteristics according to the point cloud data;

b5: and storing the appearance part model, the operation part model, the installation part model, the control part model, the workpiece part model, the view angle information under different view angles, the model point cloud and the model characteristic value as an AR data packet.

4. The augmented reality semi-physical complex electromechanical device training system of claim 1, wherein: the establishment of the AR fusion display module comprises the following steps:

c1: acquiring depth and texture data of semi-physical hardware equipment in real time by using a depth and texture camera of the augmented reality helmet; calculating the model characteristics of the hardware semi-physical equipment according to the acquired depth and texture data; realizing the fusion of virtual and real objects by utilizing the feature segmentation, the feature identification and the feature matching and utilizing an augmented reality three-dimensional registration algorithm;

c2: AR auxiliary guidance, which is used for performing auxiliary guidance on the installation process of the installation type parts according to the three-dimensional registration information;

c3: the real-hidden virtual-display interchange is realized, after the installation of the installation parts is finished, a power switch is turned on, and the hiding of the installation parts and the hiding of the control parts and the display of corresponding installation part models and control part models are carried out;

c4: training process simulation, namely controlling a control part model to operate by controlling an operation part according to actual operation requirements;

c5: and (3) workpiece state simulation, namely performing corresponding simulation transformation on the workpiece part model according to the instruction type in the motion process of the control part model.

5. The augmented reality semi-physical complex electromechanical device training system of claim 4, wherein: the AR three-dimensional registration comprises the following steps:

c1-1: scanning the semi-physical hardware by using an augmented reality helmet to acquire three-dimensional point cloud information P and sight line information V of the semi-physical hardware;

c1-2: performing segmentation processing on the three-dimensional point cloud information P according to a plane segmentation algorithm, wherein the segmentation result is S, if no point cloud exists in S, the segmentation fails, executing the step C1-1, otherwise, executing the step C1-3;

c1-3: intersecting the point cloud bounding boxes in the segmentation result S according to the sight line information V of the collected point cloud, wherein the intersection result is a set T, the point cloud which cannot be intersected is discarded, otherwise, the point cloud is added into the set T, and after the point cloud intersection in the S is completed, if the set T is empty, the step C1-1 is executed, otherwise, the step C1-4 is executed;

c1-4: calculating the VFHF characteristics of each point cloud block in the set T, wherein the calculation result is a set VH;

c1-5: performing feature matching on the model features of the AR data packet and the segmentation result features VH, finding the optimal point cloud block B meeting the requirements from the matching results according to the feature distance, if not, executing the step C1-1, otherwise, executing the step C1-6;

c1-6: performing SAC _ IA initial registration on the model point cloud of the AR data packet and the optimal point cloud block B, and performing fine registration by using ICP according to the SAC _ IA registration result, wherein the registration transformation matrix is M;

c1-7: transforming the virtual model to the position of the semi-physical hardware according to the registration transformation M;

c1-8: and controlling fine adjustment of the virtual model through a voice instruction, and completely overlaying the virtual model onto the semi-physical hardware.

6. The augmented reality semi-physical complex electromechanical device training system of claim 4, wherein: the AR assistance guidance comprises the following steps:

c2-1: identifying and registering the installation type parts according to the characteristics of the installation type part model;

c2-2: identifying the current state of the installation type part according to the characteristics of the installation type part, and displaying the installation method of the installation type part right in front of the installation type part;

c2-3: controlling the installation flow of the installation type parts through voice instructions;

c2-4: judging whether the installation of the installation type parts is correct or not according to the installation state of the parts, and giving a correct installation prompt if the installation of the installation type parts is incorrect;

c2-5: recording all information in the whole installation process, including installation videos, installation time, installation sequence and installation results, and evaluating the whole installation process according to the installation information.

7. The augmented reality semi-physical complex electromechanical device training system of claim 4, wherein: the exchange of the excess, the reserve and the virtual appearance comprises the following steps:

c3-1: after the installation of the installation type parts is finished, putting workpiece type parts on the equipment;

c3-2: pressing a power switch to turn on a power supply;

c3-3: retracting the installation type parts, the control type parts and the workpiece type parts into the equipment content through the control system;

c3-4: and displaying the installation type part model, the control type part model and the workpiece type part model at correct positions according to the three-dimensional registration information.

8. The augmented reality semi-physical complex electromechanical device training system of claim 4, wherein: the training process simulation comprises the following steps:

c4-1: the AR fusion display module acquires hardware state information of the semi-physical hardware module;

c4-2: analyzing hardware state information, and resetting the AR model state to make the AR model state consistent with the semi-physical hardware state;

c4-3: operating corresponding operation parts according to the training process, and transmitting operation signals to the AR fusion display module by the operation parts in real time;

c4-4: the AR fusion display module drives the corresponding control part model to move according to the operation instruction transmitted by the semi-physical platform;

c4-5: and according to the type of the operation instruction, simulating a real workpiece machining phenomenon in the motion process of the control part model.

Technical Field

The invention belongs to the technical field of electromechanical equipment training, and particularly relates to an augmented reality semi-physical complex electromechanical equipment training system.

Background

Large complex electromechanical systems are a class of systems with extremely complex structures and high technical density, and the use and maintenance process has extremely high requirements on personnel skills and tool equipment. At present, the training of complex electromechanical equipment mainly comprises two modes, namely an actual equipment based mode and a virtual simulation technology based mode. The training based on the actual equipment is limited by factors such as high equipment price, unskilled operation of trainees, limited training environment and the like, so that the training cost is high, the operation difficulty is high, the safety of the training process is poor, and operation errors and personal safety problems are easy to generate; the training based on the virtual simulation technology utilizes the computer simulation and the virtual reality technology to construct virtual equipment and a virtual workpiece model, and the virtual equipment is used for realizing the simulated machining of the virtual workpiece. Because the virtual complex electromechanical device simulation system is off-line simulation, the ideal situation of the complex electromechanical system in operation can only be reflected, and external interferences such as interference, servo error, fault, manual operation and the like in the complex device operation are not considered, only partial simulation of the actual device operation process can be realized, training personnel in the virtual training system are completely immersed in a virtual training environment generated by a computer, isolated from a real operation environment, cannot acquire visual information of a real training scene and parameters of real devices, and have poor operability and experience.

Disclosure of Invention

The invention aims to solve the technical problem of providing an augmented reality semi-physical complex electromechanical equipment training system aiming at the defects of the prior art, combining the semi-physical simulation and the augmented reality technology, and effectively and intuitively displaying the working state of real training equipment.

In order to solve the above technical problems, the present invention comprises:

an augmented reality semi-physical complex electromechanical equipment training system comprises a semi-physical hardware construction module, an AR training development module and an AR fusion display module; the hardware construction module is used for designing the hardware and generating control signals, and comprises a signal control submodule and a part hiding control submodule; the signal control submodule is responsible for sending the state information of the equipment to the AR fusion display module, analyzing the control instruction sent by the AR fusion display module, sending the control information to the part hiding control submodule, and hiding or displaying the part by the part hiding control submodule according to the control information; the AR training development module is used for equipment virtual model classification, model point cloud feature calculation, guidance process establishment and AR data packet export and comprises a guidance process submodule, an appearance model definition submodule, a control model definition submodule, a workpiece model definition submodule and an AR data packet publishing submodule; the guiding process submodule is used for establishing an installation process of the real object parts and guiding the installation of the hardware parts; the appearance model definition submodule is used for defining an appearance model consistent with the appearance of the real object and providing reference data for three-dimensional registration; the control model definition submodule is used for defining a control model and a model motion direction; the workpiece model definition submodule is used for defining a machined workpiece model and providing basic data for subsequent workpiece machining simulation; the AR data packet issuing submodule is used for serializing the installation process, the appearance model, the control model characteristics and the workpiece model information into an AR data packet and then sending the data packet to the AR fusion display module; the AR fusion display module is used for carrying out virtual-real fusion on the semi-physical hardware and the equipment virtual model by utilizing an augmented reality three-dimensional registration technology and realizing motion process simulation under the control of an equipment actual control signal, and comprises an AR three-dimensional registration sub-module, an AR auxiliary guidance sub-module, a training process simulation sub-module and a workpiece state simulation sub-module; the AR three-dimensional registration sub-module is used for point cloud collection, point cloud segmentation, point cloud identification and point cloud registration of the semi-physical hardware and virtual-real fusion of the equipment virtual model and the semi-physical hardware; the AR auxiliary guidance sub-module is responsible for performing auxiliary guidance on the installation process of the installation type parts according to the installation process information in the AR data packet; the training process simulation submodule controls a control model in the AR data packet to move in real time according to the defined movement direction according to the equipment state information sent by the signal control submodule; and the workpiece state simulation submodule carries out machining phenomenon simulation on the workpiece model in the workpiece data packet according to the running track of the control model in the AR data packet.

Further, the establishment of the hardware building module for semi-physical objects comprises the following steps:

a1: classifying the parts according to functions born by each part in the actual equipment, and dividing the parts into appearance parts, installation parts, operation parts, control parts and other parts;

a2: assembling appearance parts, installation parts, operation parts and control parts together to form semi-physical hardware equipment;

a3: designing a signal acquisition system for the operation type part, and accessing a control signal generated by the control type part due to state change into a signal control submodule;

a4: designing and manufacturing a mechanical structure for the installation type part, enabling the mechanical structure to be displayed or contracted and hidden in the box body, and adding a displayed or contracted and hidden state signal to the signal control submodule;

a5: and independent power supply circuits for ensuring the operation of the mechanical structure and the control circuit of the hardware equipment are designed for the hardware equipment.

Further, the establishing of the AR data packet includes the following steps:

b1: importing a complete three-dimensional equipment digital model into an AR training development module;

b2: classifying the part models in the virtual model of the equipment according to the functions born by each equipment part in the actual equipment, and dividing the part models into an appearance part model, an installation part model, an operation part model, a control part model and a workpiece part model;

b3: according to the training function requirement, establishing a guidance animation of the installation type part model for the installation type part model in an AR training development module;

b4: carrying out omnibearing spatial information acquisition processing on the appearance part model and the operation part model through a virtual camera, calculating point cloud data corresponding to the appearance part model and the operation part model at each visual angle, and generating VFHF characteristics according to the point cloud data;

b5: and storing the appearance part model, the operation part model, the installation part model, the control part model, the workpiece part model, the view angle information under different view angles, the model point cloud and the model characteristic value as an AR data packet.

Further, the establishing of the AR fusion display module comprises the following steps:

c1: acquiring depth and texture data of semi-physical hardware equipment in real time by using a depth and texture camera of the augmented reality helmet; calculating the model characteristics of the hardware semi-physical equipment according to the acquired depth and texture data; realizing the fusion of virtual and real objects by utilizing the feature segmentation, the feature identification and the feature matching and utilizing an augmented reality three-dimensional registration algorithm;

c2: AR auxiliary guidance, which is used for performing auxiliary guidance on the installation process of the installation type parts according to the three-dimensional registration information;

c3: the real-hidden virtual-display interchange is realized, after the installation of the installation parts is finished, a power switch is turned on, and the hiding of the installation parts and the hiding of the control parts and the display of corresponding installation part models and control part models are carried out;

c4: training process simulation, namely controlling a control part model to operate by controlling an operation part according to actual operation requirements;

c5: and (3) workpiece state simulation, namely performing corresponding simulation transformation on the workpiece part model according to the instruction type in the motion process of the control part model.

Further, the AR three-dimensional registration comprises the following steps:

c1-1: scanning the semi-physical hardware by using an augmented reality helmet to acquire three-dimensional point cloud information P and sight line information V of the semi-physical hardware;

c1-2: performing segmentation processing on the three-dimensional point cloud information P according to a plane segmentation algorithm, wherein the segmentation result is S, if no point cloud exists in S, the segmentation fails, executing the step C1-1, otherwise, executing the step C1-3;

c1-3: intersecting the point cloud bounding boxes in the segmentation result S according to the sight line information V of the collected point cloud, wherein the intersection result is a set T, the point cloud which cannot be intersected is discarded, otherwise, the point cloud is added into the set T, and after the point cloud intersection in the S is completed, if the set T is empty, the step C1-1 is executed, otherwise, the step C1-4 is executed;

c1-4: calculating the VFHF characteristics of each point cloud block in the set T, wherein the calculation result is a set VH;

c1-5: performing feature matching on the model features of the AR data packet and the segmentation result features VH, finding the optimal point cloud block B meeting the requirements from the matching results according to the feature distance, if not, executing the step C1-1, otherwise, executing the step C1-6;

c1-6: performing SAC _ IA initial registration on the model point cloud of the AR data packet and the optimal point cloud block B, and performing fine registration by using ICP according to the SAC _ IA registration result, wherein the registration transformation matrix is M;

c1-7: transforming the virtual model to the position of the semi-physical hardware according to the registration transformation M;

c1-8: and controlling fine adjustment of the virtual model through a voice instruction, and completely overlaying the virtual model onto the semi-physical hardware.

Further, the AR-assisted guidance comprises the steps of:

c2-1: identifying and registering the installation type parts according to the characteristics of the installation type part model;

c2-2: identifying the current state of the installation type part according to the characteristics of the installation type part, and displaying the installation method of the installation type part right in front of the installation type part;

c2-3: controlling the installation flow of the installation type parts through voice instructions;

c2-4: judging whether the installation of the installation type parts is correct or not according to the installation state of the parts, and giving a correct installation prompt if the installation of the installation type parts is incorrect;

c2-5: recording all information in the whole installation process, including installation videos, installation time, installation sequence and installation results, and evaluating the whole installation process according to the installation information.

Further, the exchange of the real, the hidden and the virtual appearances comprises the following steps:

c3-1: after the installation of the installation type parts is finished, putting workpiece type parts on the equipment;

c3-2: pressing a power switch to turn on a power supply;

c3-3: retracting the installation type parts, the control type parts and the workpiece type parts into the equipment content through the control system;

c3-4: and displaying the installation type part model, the control type part model and the workpiece type part model at correct positions according to the three-dimensional registration information.

Further, the training process simulation comprises the following steps:

c4-1: the AR fusion display module acquires hardware state information of the semi-physical hardware module;

c4-2: analyzing hardware state information, and resetting the AR model state to make the AR model state consistent with the semi-physical hardware state;

c4-3: operating corresponding operation parts according to the training process, and transmitting operation signals to the AR fusion display module by the operation parts in real time;

c4-4: the AR fusion display module drives the corresponding control part model to move according to the operation instruction transmitted by the semi-physical platform;

c4-5: and according to the type of the operation instruction, simulating a real workpiece machining phenomenon in the motion process of the control part model.

The invention has the beneficial effects that:

the invention can quickly establish an augmented reality semi-physical complex electromechanical equipment training system, can effectively and visually display the working state of real training equipment, avoids a plurality of limitations existing in the use of the real system, can reduce the economic cost of a simulation system and shorten the development period, and has the characteristics of good safety, good maneuverability, vivid effect and low price.

Drawings

Fig. 1 is a schematic structural view of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

An Augmented Reality (AR) system generates virtual objects that do not exist in the real environment by means of computer graphics and visualization and other technologies and accurately places them in the real environment, and integrates the virtual objects with the real environment by using a display device, so that a user is assured that the virtual objects are organic components of the real environment around the user, and a new environment with real sensory effects is presented to the user. Semi-physical simulation is also called hardware loop simulation, and is a simulation method for directly introducing part of hardware objects into a simulation system, generally comprises part of equipment hardware objects and a mathematical simulation model, is a supplement of a pure digital simulation technology and a physical simulation technology, can approach or even achieve the real effect of physical simulation, and has the characteristics of good repeatability, simplicity and convenience of pure digital simulation.

According to the invention, a semi-physical simulation technology and an augmented reality technology are combined, a semi-physical hardware platform can meet the actual operation control requirement of trainees at each stage, and an augmented reality control system ensures that the semi-physical hardware platform and an equipment virtual model can be combined in a virtual and real manner in each state of system operation, and are complementary to each other, so that the operation and control states of real equipment are presented in front of trainees all the time.

The invention provides an augmented reality semi-physical complex electromechanical equipment training system which comprises a semi-physical hardware construction module, an AR training development module and an AR fusion display module.

The hardware construction module of the semi-physical is used for the design of the hardware of the semi-physical and the generation of control signals, it includes signal control submodule and part hides the control submodule; the signal control submodule is responsible for sending the state information of the equipment to the AR fusion display module, analyzing the control instruction sent by the AR fusion display module, sending the control information to the part hiding control submodule, and hiding or displaying the part by the part hiding control submodule according to the control information.

The AR training development module is used for classifying equipment virtual models, calculating model point cloud characteristics, establishing guide processes and exporting AR data packets and comprises a guide process submodule, an appearance model definition submodule, a control model definition submodule, a workpiece model definition submodule and an AR data packet publishing submodule; the guiding process submodule is used for establishing an installation process of the real object parts and guiding the installation of the hardware parts; the appearance model definition submodule is used for defining an appearance model consistent with the appearance of the real object and providing reference data for three-dimensional registration; the control model definition submodule is used for defining a control model and a model motion direction; the workpiece model definition submodule is used for defining a machined workpiece model and providing basic data for subsequent workpiece machining simulation; the AR data packet publishing submodule is used for serializing the installation process, the appearance model, the control model characteristics and the workpiece model information into an AR data packet, and then sending the data packet to the AR fusion display module.

The AR fusion display module is used for carrying out virtual-real fusion on the semi-physical hardware and the equipment virtual model by utilizing an augmented reality three-dimensional registration technology and realizing motion process simulation under the control of an equipment actual control signal, and comprises an AR three-dimensional registration sub-module, an AR auxiliary guidance sub-module, a training process simulation sub-module and a workpiece state simulation sub-module; the AR three-dimensional registration sub-module is used for point cloud collection, point cloud segmentation, point cloud identification and point cloud registration of the semi-physical hardware and virtual-real fusion of the equipment virtual model and the semi-physical hardware; the AR auxiliary guidance sub-module is responsible for performing auxiliary guidance on the installation process of the installation type parts according to the installation process information in the AR data packet; the training process simulation submodule controls a control model in the AR data packet to move in real time according to the defined movement direction according to the equipment state information sent by the signal control submodule; and the workpiece state simulation submodule carries out machining phenomenon simulation on the workpiece model in the workpiece data packet according to the running track of the control model in the AR data packet.

The establishment of the hardware building module comprises the following steps:

a1: classifying the parts according to functions born by each part in the actual equipment, and dividing the parts into appearance parts, installation parts, operation parts, control parts and other parts;

a2: assembling appearance parts, installation parts, operation parts and control parts together to form semi-physical hardware equipment;

a3: designing a signal acquisition system for the operation type part, and accessing a control signal generated by the control type part due to state change into a signal control submodule;

a4: designing and manufacturing a mechanical structure for the installation type part, enabling the mechanical structure to be displayed or contracted and hidden in the box body, and adding a displayed or contracted and hidden state signal to the signal control submodule;

a5: and independent power supply circuits for ensuring the operation of the mechanical structure and the control circuit of the hardware equipment are designed for the hardware equipment.

The establishment of the AR data packet comprises the following steps:

b1: importing a complete three-dimensional equipment digital model into an AR training development module;

b2: classifying the part models in the virtual model of the equipment according to the functions born by each equipment part in the actual equipment, and dividing the part models into an appearance part model, an installation part model, an operation part model, a control part model and a workpiece part model;

b3: according to the training function requirement, establishing a guidance animation of the installation type part model for the installation type part model in an AR training development module;

b4: carrying out omnibearing spatial information acquisition processing on the appearance part model and the operation part model through a virtual camera, calculating point cloud data corresponding to the appearance part model and the operation part model at each visual angle, and generating VFHF characteristics according to the point cloud data;

b5: and storing the appearance part model, the operation part model, the installation part model, the control part model, the workpiece part model, the view angle information under different view angles, the model point cloud and the model characteristic value as an AR data packet.

The establishment of the AR fusion display module comprises the following steps:

c1: acquiring depth and texture data of semi-physical hardware equipment in real time by using a depth and texture camera of the augmented reality helmet; calculating the model characteristics of the hardware semi-physical equipment according to the acquired depth and texture data; realizing the fusion of virtual and real objects by utilizing the feature segmentation, the feature identification and the feature matching and utilizing an augmented reality three-dimensional registration algorithm;

c2: AR auxiliary guidance, which is used for carrying out auxiliary guidance on the installation process of the installation type parts according to the three-dimensional registration information and mainly comprises part identification, flow driving, real-time guidance, intelligent assistance and process evaluation;

c3: exchanging real hiding and virtual displaying, after the installation of the installation parts is finished, turning on a power switch, hiding the installation parts and the control parts, and displaying a corresponding installation part model and a corresponding control part model, wherein the method mainly comprises power-up of a power supply, real hiding and virtual object displaying;

c4: training process simulation, namely controlling a control part model to operate by controlling an operation part according to actual operation requirements, wherein the training process simulation mainly comprises state acquisition, program analysis, model motion and phenomenon simulation;

c5: and (3) workpiece state simulation, namely performing corresponding simulation transformation on the workpiece part model according to the instruction type in the motion process of the control part model, wherein the simulation transformation mainly comprises geometric simulation and physical simulation.

The AR three-dimensional registration includes the steps of:

c1-1: scanning the semi-physical hardware by using an augmented reality helmet to acquire three-dimensional point cloud information P and sight line information V of the semi-physical hardware;

c1-2: performing segmentation processing on the three-dimensional point cloud information P according to a plane segmentation algorithm, wherein the segmentation result is S, if no point cloud exists in S, the segmentation fails, executing the step C1-1, otherwise, executing the step C1-3;

c1-3: intersecting the point cloud bounding boxes in the segmentation result S according to the sight line information V of the collected point cloud, wherein the intersection result is a set T, the point cloud which cannot be intersected is discarded, otherwise, the point cloud is added into the set T, and after the point cloud intersection in the S is completed, if the set T is empty, the step C1-1 is executed, otherwise, the step C1-4 is executed;

c1-4: calculating the VFHF characteristics of each point cloud block in the set T, wherein the calculation result is a set VH;

c1-5: performing feature matching on the model features of the AR data packet and the segmentation result features VH, finding the optimal point cloud block B meeting the requirements from the matching results according to the feature distance, if not, executing the step C1-1, otherwise, executing the step C1-6;

c1-6: performing SAC _ IA initial registration on the model point cloud of the AR data packet and the optimal point cloud block B, and performing fine registration by using ICP according to the SAC _ IA registration result, wherein the registration transformation matrix is M;

c1-7: transforming the virtual model to the position of the semi-physical hardware according to the registration transformation M;

c1-8: and controlling fine adjustment of the virtual model through a voice instruction, and completely overlaying the virtual model onto the semi-physical hardware.

The AR-assisted guidance comprises the following steps:

c2-1: identifying and registering the installation type parts according to the characteristics of the installation type part model;

c2-2: identifying the current state of the installation type part according to the characteristics of the installation type part, and displaying the installation method of the installation type part right in front of the installation type part;

c2-3: controlling the installation flow of the installation type parts through voice instructions;

c2-4: judging whether the installation of the installation type parts is correct or not according to the installation state of the parts, and giving a correct installation prompt if the installation of the installation type parts is incorrect;

c2-5: recording all information in the whole installation process, including installation videos, installation time, installation sequence and installation results, and evaluating the whole installation process according to the installation information.

The interchange of excess and deficiency in storage and manifestation comprises the following steps:

c3-1: after the installation of the installation type parts is finished, putting workpiece type parts on the equipment;

c3-2: pressing a power switch to turn on a power supply;

c3-3: retracting the installation type parts, the control type parts and the workpiece type parts into the equipment content through the control system;

c3-4: and displaying the installation type part model, the control type part model and the workpiece type part model at correct positions according to the three-dimensional registration information.

The training process simulation comprises the following steps:

c4-1: the AR fusion display module acquires hardware state information of the semi-physical hardware module;

c4-2: analyzing hardware state information, and resetting the AR model state to make the AR model state consistent with the semi-physical hardware state;

c4-3: operating corresponding operation parts according to the training process, and transmitting operation signals to the AR fusion display module by the operation parts in real time;

c4-4: the AR fusion display module drives the corresponding control part model to move according to the operation instruction transmitted by the semi-physical platform;

c4-5: according to the operation command type, the simulation of real workpiece machining phenomena, such as machining sound, chip flying and other physical phenomena, is simulated in the motion process of the control part model.

The operation of the whole system comprises the following steps: (1) starting an AR fusion display module, wherein the AR fusion display module is connected to a signal control submodule in a semi-physical hardware construction module, and the signal control submodule sends a hardware state signal to the AR fusion display module in real time; (2) the AR fusion display module acquires the appearance characteristic D of the semi-physical hardware; (3) the AR fusion display module carries out virtual-real fusion on the appearance characteristic D and the model characteristic M defined by the AR training development module by using a three-dimensional registration technology; (4) the AR fusion display module guides the installation of hardware parts according to the installation flow defined by the AR training development and the sequence; (5) the AR fusion display module receives the power supply starting state of the semi-physical hardware building module; a hardware contraction hiding instruction is sent to the semi-physical hardware platform, and an operation model, a control model and a workpiece model are displayed at the same time; (6) after the semi-physical hardware construction module receives the contraction hiding instruction; the operation parts and the control parts can be hidden; (7) the signal control system can send a hardware state signal to the AR fusion display module in real time, the AR fusion display module correspondingly drives the virtual control model according to the hardware state information, and the change process of the workpiece is simulated in real time according to the motion information of the control model.

The invention can quickly establish the augmented reality semi-physical training system of the complex electromechanical equipment, can effectively and visually display the working state of the real training equipment, avoids a plurality of limitations existing in the use of the real system, can reduce the economic cost of the simulation system and shorten the development period, and has the characteristics of good safety, good maneuverability, vivid effect and low price.

Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.