阅读说明：本技术 基于增强现实的机械紧固部的管理方法 (Management method of mechanical fastening part based on augmented reality ) 是由 谷田雄太 冈田智仙 小林义史 菊名淳夫 中村匠 山浦美轮 山口刚二郎 中田祥吾 平井 于 2018-10-10 设计创作，主要内容包括：在以往的作业管理系统中,需要对期望管理的各个部件附加RFID标签等识别标识,且准备具有天线的紧固工具,因此运用成本变大。一种使用了对现实空间叠加假想空间而生成的增强现实空间的基于增强现实的机械紧固部的管理方法,其中,在现实紧固部(RBn)与假想紧固部(IBn)一一对应的增强现实空间中,通过相机等取得该现实紧固部被选择为紧固对象的信息,通过与该相机连接的增强现实服务器进行解析,从而能够容易地记录规定的作业按照预定计划进行的情况,能够实现运用成本低且高品质的机械紧固部的管理方法、系统。(In the conventional work management system, since it is necessary to add an identification mark such as an RFID tag to each component to be managed and prepare a fastening tool having an antenna, the operation cost increases. A management method and system for a mechanical fastening part based on augmented reality, using an augmented reality space generated by superimposing a virtual space on a real space, wherein information that the real fastening part is selected as a fastening object is acquired by a camera or the like in the augmented reality space in which the real fastening part (RBn) and the virtual fastening part (IBn) are in one-to-one correspondence, and the information is analyzed by an augmented reality server connected to the camera, so that it is possible to easily record that a predetermined operation is performed according to a predetermined plan, and it is possible to realize a management method and system for a mechanical fastening part of low operation cost and high quality.)

1. A method for managing a mechanical fastening unit based on augmented reality, the method using a mechanical fastening unit in augmented reality space created by superimposing a virtual space on a real space,

the real space comprises a real fastening part and a real tool sleeved on the real fastening part,

the imaginary space comprises an imaginary fastening part and an imaginary tool sleeved on the imaginary fastening part,

in the augmented reality space, the real fastening parts correspond to the imaginary fastening parts one to one,

an augmented reality composition system that generates the augmented reality space includes:

a camera that photographs the real space; and

an augmented reality server connected with the camera and parsing an image set by the camera,

the management method includes a fastening object determination step of fitting a sleeve head of the real tool on the real fastening portion, thereby selecting the real fastening portion as a fastening object, and transmitting information that the real fastening portion is selected as the fastening object to the virtual fastening portion.

2. The augmented reality-based mechanical fastening portion management method according to claim 1,

following the fastening object determining step, the augmented reality-based mechanical fastening portion management method further includes:

a fastening step of fastening the real fastening part with the real tool;

a tightening management information acquisition step of acquiring tightening management information in which the actual tightening section is tightened in the tightening step; and

and a fastening management information-added image storage step of storing, in the augmented real space, a fastening-end image obtained after the fastening step is ended and a fastening management information-added image obtained by adding the fastening management information to the fastening-end image.

3. The augmented reality-based mechanical fastening portion management method according to claim 2,

the real tool includes a transmission/reception unit that transmits/receives torque information of the real tightening unit, and a display unit that displays the torque information obtained by the transmission/reception unit,

and a fastening management information adding image storing step of processing the fastening management information after the augmented reality constituting system senses that the information that the real fastening part is fastened at the predetermined torque is displayed on the display part.

4. The augmented reality-based mechanical fastening portion management method according to claim 3,

the augmented reality-based management method for a mechanical fastening unit further includes an initial fastening torque storage step of recording that an initial fastening torque is generated in the real fastening unit that is attached to the real tool and selected as a fastening target, the initial fastening torque being a torque observed when the real tool starts fastening the real fastening unit.

5. The augmented reality-based mechanical fastening portion management method according to claim 4,

the implement includes the tip portion, a grip portion to be held by an operator, and a handle portion connecting the tip portion and the grip portion,

the handle portion is provided with a marking,

the mark includes direction information indicating a direction from the mark to the cuff portion, and distance information from the mark to the cuff portion.

6. The augmented reality-based mechanical fastening portion management method according to claim 5,

in the fastening object determining step, the augmented reality server performs:

acquiring real fastening part coordinates representing a position of the real fastening part analyzed in the real space captured by the camera and real tool bit coordinates representing a position of a bit of the real tool analyzed in the real space captured by the camera,

sensing a coincidence of the real tool cuff coordinates of the real tool cuff and the real fastening portion coordinates of the real fastening portion,

and transmitting information that the actual fastening part is selected as a fastening target by being fitted to the fitting head part of the actual tool to the virtual fastening parts corresponding to the actual fastening parts one by one.

7. The augmented reality-based mechanical fastening portion management method according to claim 5,

in the fastening object determining step, the augmented reality server performs:

analyzing the real space captured by the camera to obtain real fastening part coordinates representing a position of the real fastening part,

calling out the virtual tool corresponding to the real tool photographed by the camera from the virtual space to the augmented reality space,

acquiring a virtual tool bit head coordinate representing a position of a bit of the virtual tool, the virtual tool being overlapped with the real tool,

when the real tool is fitted to the real fastening part by causing the virtual tool to follow the movement of the real tool, it is determined that the virtual tool fitting head coordinates of the virtual tool fitting head part coincide with the real fastening part coordinates of the real fastening part,

and transmitting information that the actual fastening part is selected as a fastening target by being fitted to the fitting head part of the actual tool to the virtual fastening parts corresponding to the actual fastening parts one by one.

8. The augmented reality-based mechanical fastening portion management method according to claim 5,

in the fastening object determining step, the augmented reality server performs:

analyzing the real space captured by the camera to obtain real tool bit coordinates representing a position of a bit of the real tool,

generating an imaginary horizontal plane including the imaginary fastening part for reminding an operator of fastening,

generating an imaginary vertical line passing through the real tool case head coordinates of the real tool case head and intersecting the imaginary horizontal plane,

detecting a distance between an intersection point where the virtual vertical line intersects the virtual horizontal plane and a virtual fastening portion coordinate indicating a position of the virtual fastening portion in the virtual horizontal plane,

determining that the real tool bit coordinates of the bit of the real tool coincide with the imaginary fastening part coordinates of the imaginary fastening part based on the distance,

and transmitting information selected as a fastening target to the virtual fastening portions corresponding one-to-one to the actual fastening portions to which the ferrule head portion of the actual tool is fitted.

9. The augmented reality-based mechanical fastening portion management method according to claim 5,

in the fastening object determining step, the augmented reality server performs:

calling out the virtual tool corresponding to the real tool photographed by the camera from the virtual space in an augmented reality space,

the vertical axis of the cap portion of the virtual tool is overlapped with a vertical axis of a virtual fastening portion coordinate representing the position of the virtual fastening portion for reminding an operator to fasten, and the virtual tool is rotated in a horizontal plane around the vertical axes of the two,

when the actual tightening part is fitted with the sleeve head of the actual tool, information that the actual tightening part is selected as a tightening target by being fitted with the sleeve head of the actual tool is transmitted to the virtual tightening part when the moment when the virtual tool and the actual tool are fitted together is captured.

Technical Field

The present invention relates to management of a mechanical fastening section using augmented reality.

Augmented Reality (Augmented Reality) is a technology that utilizes a computer to augment a real-world environment perceived by a human. The present invention is a technique as follows: an augmented real space is generated by superimposing a virtual space constructed by a technique such as 3D-CAD (Computer Graphics) or CG (Computer Graphics) on a real space imaged by a camera or the like, and a machine fastening unit such as a device is managed using the augmented real space as described above.

Background

Conventionally, there is known a job management system as follows: information on fastening of bolts, flanges, and the like acquired by a tool and a measuring instrument, tool and measuring instrument data for work, and ID information of an operator and the like are associated with each other to ensure traceability, thereby realizing high-quality work management (for example, refer to patent document 1).

Prior art documents

Patent document

Patent document 1: japanese patent No. 5065851

Disclosure of Invention

Problems to be solved by the invention

The work management system described in patent document 1 is a system as follows: with respect to each bolt fastening operation data related to flange fastening, traceability of a fastening tool, a measuring device, a bolt used, and the like for setting and measuring the bolt fastening operation data can be ensured, and an input error of a target value of a set fastening torque or a target value of a set interference amount, and a recording error of an actually measured fastening torque value or an actually measured interference amount caused by a human error can be prevented.

However, in the work management system described in patent document 1, it is necessary to provide an RFIC tag to all bolts for fastening the flange. In addition, in order to receive and transmit information of the RFIC tag provided on the bolt, a fastening tool having an antenna extending to a head portion for gripping the bolt needs to be prepared, and thus the operation cost may increase.

Accordingly, an object of the present invention is to provide a method and a system for managing a high-quality mechanical fastening unit, which can easily record that a predetermined operation is performed according to a predetermined schedule without adding an identification mark such as an RFIC tag to the fastening unit and without transmitting and receiving data between the fastening unit and a fastening tool, and which can be operated at low cost.

Means for solving the problems

In order to solve the above problems, one of the typical augmented reality-based mechanical fastening section management methods of the present invention is a method for managing a mechanical fastening section using an augmented reality space generated by superimposing a virtual space on a real space, the real space including a real fastening section and a real tool fitted to the real fastening section, the virtual space including a virtual fastening section and a virtual tool fitted to the virtual fastening section, the augmented reality configuration system generating the augmented reality space in which the real fastening sections and the virtual fastening sections are in one-to-one correspondence in the augmented reality space, the method including: a camera that photographs the real space; and an augmented reality server connected to the camera and analyzing an image set by the camera, wherein the management method includes a fastening object determination step of fitting a sleeve head of the real tool to the real fastening portion, thereby selecting the real fastening portion as a fastening object, and transmitting information that the real fastening portion is selected as a fastening object to the virtual fastening portion.

Effects of the invention

According to the present invention, it is possible to realize a high-quality machine fastening section management method and system that can easily record that a predetermined operation is performed according to a predetermined schedule and that can be operated at low cost without adding an identification mark such as an RFIC tag to the fastening section and without transmitting and receiving data between the fastening section and a fastening tool.

The problems, configurations, and effects other than those described above will be apparent from the following description of the embodiments.

Drawings

Fig. 1 is a schematic diagram of a system configuration of a management method of a mechanical fastening unit based on augmented reality.

Fig. 2 is a schematic diagram showing a functional configuration of a helmet included in the system configuration of fig. 1.

Fig. 3 is an example of assembly work including fastening for explaining the method of managing a machine fastening portion by augmented reality according to the present invention.

Fig. 4 is a process included in the assembly work of fig. 3, and is a schematic view showing a state where the component is positioned on the base plate in real space.

Fig. 5 is a final step of the assembly work of fig. 3, and is a schematic view showing a state in which the component is fixed to the base plate by bolts in the virtual space.

Fig. 6 is a schematic view showing a process included in the assembly work of fig. 3, and showing a completed state in which the component is fixed to the base by the bolt in the augmented real space created by superimposing the virtual space of fig. 5 on the real space of fig. 4.

Fig. 7 is a schematic diagram showing a state in which bolt fastening management is started from a state in which a real bolt is inserted into a through-hole of a real component in the augmented real space of fig. 6.

Fig. 8 is a flowchart (first half) illustrating a management method of a mechanical fastening unit based on augmented reality according to the present invention.

Fig. 9 is a flowchart (second half) illustrating a method of managing a mechanical fastening unit based on augmented reality according to the present invention.

Fig. 10 is a schematic diagram showing a range of "determined as an object to be fastened" in which information that a bolt is set with a tool and selected as an object to be fastened in real space is transmitted to a virtual space.

Fig. 11 is a schematic diagram showing a relationship between a transition of fastening torque observed when fastening a bolt with a tool and a procedure of determination of a flowchart.

Fig. 12 is a schematic diagram of an image of an augmented real space displayed by a see-through display that captures a bolt changed to a display color indicating fastening completion and fastening management information displayed in the vicinity of the bolt.

Fig. 13 is an example of design drawing information for mounting a component on a base.

Fig. 14 shows an example of operation information for mounting a component on a base.

Fig. 15 shows an example of a step (S90) in which virtual bolts corresponding one-to-one to real bolts in the augmented real space are identified as objects to be fastened by real tools.

Fig. 16 shows an example of a real tool including a plurality of marks indicating positions of a head (set) portion of the tool.

Fig. 17 is another example of the step (S90) of determining the virtual bolts corresponding one-to-one to the real bolts in the augmented real space as the fastening objects to be fastened by the real tool.

Fig. 18 is another example of the step (S90) of determining the virtual bolts corresponding one-to-one to the real bolts in the augmented real space as the fastening objects to be fastened by the real tools.

Fig. 19 is another example of the step (S90) of determining the virtual bolts corresponding one-to-one to the real bolts in the augmented real space as the fastening target object fastened by the real tool.

Detailed Description

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the embodiment.

The present embodiment will be described below with reference to the drawings. The real space in the present specification refers to a space in which an image obtained by the operator's own vision or an image captured by the camera 10 is located. The virtual space in the present specification means a space constructed by digital information such as 3D-CAD and CG. Hereinafter, in order to distinguish an article in real space from an article in virtual space, a prefix of "real" is added to the article in real space, and a prefix of "virtual" is added to the article in virtual space.

< System construction >

Fig. 1 is a schematic diagram of a system configuration related to a management method of a mechanical fastening unit based on augmented reality, and fig. 2 is a schematic diagram illustrating a functional configuration of a wearable device (helmet or the like) included in the system configuration of fig. 1.

The system shown in fig. 1 is constituted by: a design information server 22 for storing design drawing information such as a part drawing, an assembly drawing, 3D-CAD, and CG (see fig. 12); a work information server 26 that stores component mounting work information and maintenance work information (see fig. 13); a torque management server 28 that manages tightening torque when the mechanical tightening part (i.e., the actual tightening part) is tightened by the actual tool RT; an analysis server 29 that can perform large-scale image analysis; a network 5 connected to these servers; the real tool RT and the wearable device 7 provided by the operator are connected to the network 5.

The wearable device 7 shown in fig. 2 is configured to be attachable to a head of an operator, and includes: a see-through type screen 14 (head mounted display device) that forms a field of view of the operator; a camera 10; and an augmented reality server 12 connected to the see-through screen 14 and the camera 10. The augmented reality server 12 forms an augmented reality space in which information of a virtual space such as 3D-CAD information obtained from a design information server 22 connected to the network 5 is superimposed on a real space (real body) imaged by the camera 10, and projects the augmented reality space on the see-through screen 14. The augmented reality server 12 is a computer having: a storage area that stores an application program for creating augmented reality and a memory area for storing various information (for example, design information and the like); and a processing unit that processes a request item indicated by the operator via the wearable device 7.

The see-through screen 14 has see-through properties so that an operator can secure a field of view, and can project an image of a real space captured by the camera 10 and a virtual real space created by the augmented reality server 12 onto the see-through screen 14. In this case, the real space projected on the see-through screen 14 is substantially the same image as the image obtained by the operator's own vision. In addition, the augmented reality server 12 may display information such as a work procedure and information such as various instructions on the see-through screen 14, thereby further improving convenience for the operator.

Since the augmented reality server 12 is connected to the network 5 and the real tool RT by wireless, the real tool RT has a transmission/reception unit that receives fastening torque information corresponding to a predetermined real bolt from the torque management server 28 via the network 5 and transmits the fastening torque information observed by the real tool RT to the augmented reality server 12 and the torque management server 28.

The augmented reality server 12 may be configured integrally with the camera 10, or the memory area and the processing unit of the augmented reality server 12 may be provided in a place separate from the work place via the network 5.

The real tool RT is a tool for tightening or loosening a mechanical fastening portion such as a bolt. The real tool RT may be a torque wrench that can replace a kit attached to a socket head corresponding to a bolt size and a bolt type, and the socket head may have various torque wrench shapes such as a wrench shape and a spanner wrench shape. The real tool RT has the following functions: fastening torque information from the initial fastening time to the end fastening time observed when fastening a real bolt RBn and residual torque information measured when the fastened bolt is loosened are collected, and these pieces of information are supplied to the torque management server 28 and the like via the augmented reality server 12 and the network 5.

The real tool RT includes a display unit (not shown) that presents fastening torque information corresponding to an arbitrary bolt size and bolt material obtained from the augmented reality server 12 or the torque management server 28, and information indicating that the real bolt RBn is fastened at a predetermined fastening torque to the operator. Since the display unit is intended to convey the above information to the operator, the information may be displayed on the see-through screen 14 or on a display unit of another wearable terminal worn by the arm of the operator, instead of the display unit being provided in the real tool RT.

The work information server 26 stores a production work instruction indicated by the production content of a product (for example, a railway vehicle, an automobile, or the like), a maintenance work instruction indicated by the content of a maintenance work, and the like. Upon request from the augmented reality server 12, the job information server 26 extracts necessary data from these job specifications and provides the extracted data.

The design information server 22, the work information server 26, and the torque management server 28 are file servers that share or store information related to design information, work information, and torque management, respectively.

The work information server 26 does not necessarily have to be independent from the design information server 22, and the work information server 26 and the design information server 22 may be operated as 1 information server in common. The analysis server 29 is used for obtaining detailed real space or virtual space coordinates (for example, coordinates indicating a three-dimensional position) of the work object by performing a large-scale (image) analysis from the image captured by the camera 10.

< content of operation >

Fig. 3 is an example of assembly work including fastening for explaining the method of managing a machine fastening portion by augmented reality according to the present invention. This assembly work is a work of fixing the real part RD2 to the real base RD1 by 4 real bolts RB1 to RB4 (hereinafter abbreviated as RBn (n is 1 to 4) or RBn) in real space. The real base RD1 includes: actual screw holes RQ1 to RQ4 each having a thread groove into which an actual bolt RBn inserted through the actual member RD2 is screwed; and a real position marker RM representing a reference position in real space. The real part RD2 includes real through-holes RP1 to RP4 through which real bolts RBn vertically penetrate at 4 corners.

< overview of tightening part management in augmented real space constructed by superimposing virtual space on real space >

Fig. 4 is a process included in the assembly work of fig. 3, and is a schematic view showing a state where the component is positioned on the base in real space. Fig. 5 is a final step of the assembly work of fig. 3, and is a schematic view showing a state in which the component is fixed to the base by the bolt in the virtual space. Fig. 6 is a schematic view showing a process included in the assembly work of fig. 3, in which a member is fixed to a base by a bolt in an augmented real space obtained by projecting the virtual space of fig. 5 onto the real space of fig. 4.

Fig. 4 is an image of the operator in real space obtained by the operator's own vision, and is a diagram in which the real part RD2 is positioned at a fixed position on the upper surface of the real base RD 1. Fig. 5 is an assembly diagram of 3D-CAD data (attitude after completion of a work of attaching the virtual component ID2 to the virtual base ID1 by virtual bolts IB1 to IB4 (hereinafter, abbreviated as IBn (n is 1 to 4) or IBn) in a virtual space (digital space). The 3D-CAD data may be stored in the design information server 22 or the augmented reality server 12 of the wearable device 7.

Fig. 6 is a schematic diagram showing a process included in the operation of fig. 3, and showing a completed state in which the component is fixed to the base by the bolt in the augmented real space created by superimposing the virtual space of fig. 5 on the real space of fig. 4.

In order to reflect an image in the augmented real space on the see-through screen 14 (corresponding to fig. 6), the augmented reality server 12 first captures (acquires) a state in which the real part RD2 is positioned on the real base RD1 (corresponding to fig. 4) with the camera 10. Next, the augmented reality server 12 acquires 3D-CAD data (assembly drawing) of the virtual space from the design information server 22. Then, the augmented reality server 12 superimposes 3D-CAD data on the captured real space from the virtual space to create an augmented reality space.

In this case, if the reference point RM in real space provided in advance at one end of the real template RD1 is overlapped with the reference point IM in virtual space at one end of the virtual template ID1, the augmented reality server 12 can easily create the augmented reality space. The augmented reality server 12 renders the created augmented reality space on a see-through screen 14. At this time point, real bolt RBn is not yet inserted into through-hole RPn of real part RD 2.

Fig. 7 is a schematic diagram illustrating a state in which bolt fastening management is started from a state in which a real bolt is inserted into a through-hole of a real component in the augmented real space of fig. 6. Fig. 7 is a diagram showing a state in which real bolt RBn is inserted into real through-hole RPn of real part RD2 constituting an augmented real space (corresponding to fig. 6) projected on see-through screen 14. Since the thread portion under the head of real bolt RBn is not yet screwed into real thread hole RQn of real base RD1, the head of real bolt RBn protrudes in the height direction from the head of virtual bolt IBn.

Since the augmented real space is created by superimposing the reference point IM in the virtual space on the reference point RM in the real space, the real bolts RBn inserted into the real through-holes RPn of the real part RD2 are in one-to-one correspondence (superimposed) with the virtual bolts IBn in the virtual space. In other words, the coordinates of real bolt RBn in real space (for example, the center coordinates of the head of the bolt) substantially coincide with the coordinates of imaginary bolt IBn in imaginary space.

However, in the case of fig. 7, real bolt RBn is not screwed into real screw hole RQn of real base RD1, and therefore, strictly speaking, the coordinates of RX-RY plane of real bolt RBn and the coordinates of IX-IY plane of virtual bolt IBn coincide, but the coordinates of real bolt RBn on RZ axis and the coordinates of virtual bolt IBn on IZ axis do not coincide.

The case where the real reference point RM provided on the real base RD1 is generated so as to overlap the virtual reference point IM of the virtual base when the virtual real space is generated as described above has been described, but the real reference point RM and the virtual reference point IM do not necessarily have to be prepared. Instead of these reference points, for example, an augmented real space may be generated by superimposing a virtual space on the real space with one of the 4 corners of the real base RD1 and the virtual base ID1 as a reference point.

< management of augmented reality-based fastening section >

Fig. 8 and 9 are flowcharts (first half) and (second half) illustrating a management method of the augmented reality-based mechanical fastening section according to the present invention. Hereinafter, a method of managing augmented reality when a worker wearing the wearable device 7 (helmet) fastens a bolt from the state of fig. 7 in an augmented reality space will be described for each step, taking the operation shown in fig. 3 as an example.

In step 10(S10), management of the augmented reality-based tightening part is started.

In step 30(S30), the augmented reality server 12 captures the real component RD2 positioned on the upper surface of the real disk RD1 by the camera 10, and acquires real space information (corresponding to fig. 4).

In step 40(S40), the augmented reality server 12 acquires virtual space information (3D-CAD information, corresponding to fig. 5) from the design information server 22 via the network 5. When the virtual space information (corresponding to fig. 5) is stored in the augmented reality server 12, the augmented reality server 12 acquires the virtual space information (corresponding to fig. 5) from the augmented reality server 12.

In step 50(S50), the augmented reality server 12 superimposes the real space information obtained in step 30 on the virtual space information obtained in step 40, thereby creating an augmented reality space. At this time, since the reference point RM in the real space coincides with the reference point IM in the virtual space, arbitrary coordinates (RXn, RYn, RZn) in the real space and arbitrary coordinates (IXn, IYn, IZn) in the virtual space coincide in a one-to-one correspondence.

Steps 60(S60) to 160(S160) are loop processing in which the number of bolts is set as the number of loops.

In step 70(S70), the augmented reality server 12 determines any one of the virtual bolts IBn in the augmented reality space in the order of fastening based on the virtual space information obtained in S50, and teaches the fastened bolt to the operator. A method of teaching the operator the virtual bolt determined in the order of fastening can be performed as follows: in the augmented real space projected on the see-through screen 14, the color of the specified virtual bolt IBn is changed, or the specified virtual bolt IBn is displayed in a blinking manner.

In step 80(S80), the operator brings the real tool RT close to the real bolt RBn on which the virtual bolt IBn taught in S70 is superimposed, and fits the sleeve portion RTH of the real tool RT on the head of the real bolt RBn.

In step 90(S90), the operator confirms that the kit portion RTH of the real tool is superimposed on the actual bolt RBn to be fastened, and completes the operation of specifying "the actual bolt RBn to be fastened" to be fastened by the real tool RT in the augmented real space projected on the see-through screen 14. Fig. 10 is a schematic view showing a range indicated by the "determination as a fastening target".

Real bolt RBn in fig. 10 substantially overlaps virtual bolt IBn in the augmented real space, but real bolt RBn in the drawing is depicted separately from virtual bolt IBn in order to facilitate understanding. Based on the instruction at S70, the operator inserts real tool RT (sleeve head RTH) over real bolt RBn to be fastened in real space at S80.

At S50, it is confirmed that the information "selected as the fastening target by overlapping the real tool RT" of the real bolt RBn is transmitted to the virtual bolt IBn because the arbitrary coordinates (RXn, RYn, RZn) in real space including the coordinates of the real bolt RBn and the arbitrary coordinates (IXn, IYn, IZn) in virtual space including the coordinates of the virtual bolt IBn overlap in one-to-one correspondence. Therefore, "determined as the fastening target" means that information on the real bolt RBn on which the real tool RT is put and which is selected as the fastening target is transmitted to the virtual bolt IBn corresponding to the real bolt RBn one by one.

If the real bolt RBn of the object to be fastened is not determined in S90, the process returns to S70, and the real bolt RBn of the object to be fastened is determined again. In S90, if the real bolt RBn of the fastening target object is determined, the process proceeds to the next step.

In step 100(S100), the augmented reality server 12 temporarily saves the coordinates lbo (IBX1, IBY1, IBZ1) in the virtual space of the virtual bolt IBn corresponding one-to-one to the real bolt RBn determined as the fastening target in the augmented reality server 12 (or any server such as the job information server 26).

In step 110(S110), augmented reality server 12 acquires fastening torque information of real bolt RBn from design information server 22 or torque management server 28 (any server), and transmits the fastening torque information to real tool RT. The fastening torque information is a fastening torque (N · m) that is specified by a bolt size, a bolt material, a use environment, and the like and generates a necessary (specified) axial force AFS. When real bolt RBn is fastened, axial force AF generated in real bolt RBn is proportional to fastening torque. Since it is not easy to directly observe the axial force AF, the fastening state of the actual bolt RBn is managed by the fastening torque T corresponding to the axial force instead of the axial force AF of the actual bolt RBn. Hereinafter, this tightening torque is referred to as a predetermined tightening torque TS.

In step 120(S120), the operator starts fastening real bolt RBn specified as the fastening target using real tool RT.

In step 125(S125), when the operator starts fastening real bolt RBn determined as the fastening target by real tool RT, real tool RT senses generation of fastening torque. At this time, the augmented reality server 12 performs image analysis of the image captured by the camera 10, and checks whether or not the coordinates RTHO (RTHX1, RTHY1, and RTHZ1) of the socket head RTH of the real tool and the coordinates RBO (RBOX1, RBOY1, and RBOZ1) of the real bolt RBn specified as the fastening target overlap with each other, or whether or not they are close to a level at which it can be determined that they overlap with each other. In S125, it can be confirmed that the real tool RT starts to tighten the real bolt RBn determined as the tightening target.

In step 128(S128), the reality tool RT transmits the sensed initial fastening torque TB to the augmented reality server 12. The real tool RT may grasp date and time information of the sensed initial tightening torque TB and transmit the date and time information together with the initial tightening torque TB to the augmented reality server 12, or the augmented reality server 12 that has received the initial tightening torque TB may store the date and time information together with the initial tightening torque TB in an arbitrary server as a part of the tightening management information 30 of the virtual bolt IBn corresponding to the real bolt RBn specified as the tightening target in the virtual space.

In step 130(S130), when the operator further fastens real bolt RBn of the fastening target object using real tool RT, the fastening torque observed in real bolt RBn reaches predetermined fastening torque TS. In real tool RT, when the tightening torque observed for real bolt RBn of the tightening target reaches predetermined tightening torque TS received from augmented reality server 12 in S110, the operator is notified that real bolt RBn is tightened at the predetermined tightening torque by blinking an LED of a display unit provided in real tool RT or by changing the color of the display unit.

The predetermined tightening torque TS has an acceptable range from a lower limit value (TSL) to an upper limit value (TSH), and the display unit of the real tool RT can represent that the tightening torque TL is equal to or less than the acceptable range (lower limit value TSL), within the acceptable range, and equal to or more than the acceptable range (upper limit value TSH) in different colors. Instead of the operation of the display unit provided in the real tool RT, the information may be displayed on the see-through screen 14 or on a display unit of another wearable terminal worn by the arm of the operator.

The real tool RT transmits the finish fastening torque TL observed when the operator finishes the fastening operation to the augmented reality server 12. The finish tightening torque TL is substantially the same as or slightly larger than the predetermined tightening torque TS. The real tool RT may grasp the date and time information at which the end fastening torque TL is sensed, and may transmit the date and time information to the augmented reality server 12 together with the end fastening torque TL, or the augmented reality server 12 that has received the end fastening torque TL may add the date and time information to the end fastening torque TL.

In step 140(S140), the augmented reality server 12 that has received the end fastening torque TL transmits the end fastening torque TL to the torque management server 28 via the augmented reality server 12 itself or the network 5, and the torque management server 28 stores the end fastening torque TL.

In step 150(S150), the augmented reality server 12 changes the display color of the virtual bolt IB1 displayed on the see-through screen 14 in one-to-one correspondence with the real bolt RB1 for which the fastening operation has been completed, to a color different from the other virtual bolts IB2 to IB4 for which the fastening operation has not been completed. The operator can distinguish the virtual bolt whose fastening is completed (completed) from the virtual bolt whose fastening operation is not completed, according to the difference in the displayed color.

In step 155(S155), the augmented reality server 12 causes the see-through screen 14 to display the fastening management information 30 such as the worker ID, the fastening completion date, the fastening completion time, and the design drawing number in the vicinity of the virtual bolt IB1 of which the display color has been changed (the fastening is completed) in S150.

The augmented reality server 12 captures an image (a still picture or a moving picture) including fastening management information 30 such as a worker ID, fastening completion date, fastening completion time, and design drawing number, which are associated with the fastening work of the real bolt RB1 displayed on the see-through screen 14 in one-to-one correspondence with these virtual bolts IB1 (hard copy of the screen of the see-through screen 14), and stores the captured image in the work information server 26 or the torque management server 28 via the augmented reality server 12 or the network 5.

Instead of the captured still picture, the moving picture of the fastening operation may be stored in a predetermined time period before the fastening end time and in a predetermined time period after the fastening end time, with reference to the fastening end time. In this animation, in addition to the fastening management information 30, a situation is recorded in which the actual bolt RBn is locked at the predetermined fastening torque TS and the actual tool RT notifies the end of fastening to the operator via the display unit.

In step 160(S160), the process returns to S50 and the cycle is repeated a number of times corresponding to the predetermined number of bolts to be fastened.

At step 170, a series of operations of fastening management using the augmented reality bolt related to the fastening operation of real bolt RBn is completed.

Fig. 11 is a schematic diagram showing a relationship between a transition of fastening torque observed when fastening a bolt with a tool and a procedure of determination of a flowchart. The horizontal axis of fig. 11 is a time axis, and the vertical axis of fig. 11 is a tightening torque observed in real bolt RBn when real bolt RBn is tightened with real tool RT.

Fig. 11 shows a transition of the fastening torque observed during the process from when the operator starts fastening the real bolt to when the fastening is completed. The horizontal axis of fig. 11 represents time, and the vertical axis of fig. 11 represents axial force AF generated in real bolt RBn. In the axial force AF, an inclined portion observed when the operator starts fastening the real bolt RBn with the real tool RT and a horizontal portion observed when the operator retracts the real tool RT are alternately observed in accordance with the number of fastening times. The horizontal portion is a step of retracting the real tool RT in preparation for the next fastening, and the fastening torque is not observed.

The initial tightening torque TB is detected in accordance with the axial force observed from the first horizontal portion, and the finish tightening torque TL is detected in accordance with the axial force observed from the third horizontal portion. Then, a predetermined axial force AFS is generated at the end portion of the third inclined portion up to the third horizontal portion, and a predetermined tightening torque TS corresponding to the predetermined axial force AFS is observed.

Next, description will be made using a flowchart shown in fig. 9. S120 of the flowchart shown in fig. 9 corresponds to point a (a: S120) of fig. 11 when the operator starts tightening real bolt RBn of the object to be fastened with real tool RT in fig. 11. S125 in fig. 9 corresponds to point B (B: S128) where the tool RT starts tightening the real bolt RBn and the initial tightening torque TB is observed from S120 in the flowchart. S130 in fig. 9 corresponds to point C (C: S130) when real tool RT fastens real bolt RBn at predetermined fastening torque TS.

Fig. 12 is a schematic diagram of an image of a strong real space displayed by the see-through display, which captures the bolt changed to a display color indicating the fastening completion and the fastening management information displayed in the vicinity of the bolt. Fig. 12 is a diagram of S155 imitating the flowchart of fig. 9, and is a final step of repeating the first cycle from S60 to S160 the number of times corresponding to the number of RBn.

At this time, only the real bolt RB1 is fastened, and therefore, only the head of the real bolt RB1 sinks downward beyond the other real bolts RB2 to 4 and overlaps the head of the virtual bolt IB 1. Since the display color of only real bolt RBn whose fastening is completed is changed to gray, the operator can easily distinguish fastening-completed bolt RB1 from unfastened bolts RB 2-4 whose fastening is not completed, and forgetting of the fastening operation can be suppressed. In the augmented real space, an image of fastening management information 30 such as a worker ID, fastening completion date, fastening completion time, fastening completion torque, and design drawing number is attached to an image in which fastening of real bolt RB1 is completed only, and the image is captured (a hard copy or screen shot of the image) and stored in an arbitrary server.

< Effect (FIG. 12) >

By creating the image shown in fig. 12, it is possible to instantaneously grasp who passes through (which operator), which design drawing is based on, and what work content is instructed, and to screw real bolt RB1 with a predetermined fastening torque.

In general, an image of a fastening target object in the augmented real space (corresponding to fig. 7) and fastening management information 30 (a worker ID, a fastening completion date, a fastening completion time, a completion fastening torque, a design drawing number, and the like) attached to the fastening target object are stored as (link or contact) information on the virtual bolt IBn in the virtual space (for example, the torque management server 28 and the like). Then, as necessary, the fastening management information 30 of the virtual bolt IBn is called from the torque management server 28 or the like as the information on the virtual bolt IBn in the virtual space, and is presented to the requester by displaying it in the augmented real space.

If the related (linked or linked) information between the virtual bolt IBn and the fastening management information 30 related to the virtual bolt IBn is damaged for some reason, the fastening management information 30 corresponding to the fastening target object cannot be retrieved, and it is difficult to grasp the state based on the fastening work performed on the fastening target object. Therefore, by acquiring and storing the image described in S155 in fig. 9, even when the information of the link (contact) is lost, the state based on the fastening work performed on the fastening target object can be instantly grasped, and thus traceability can be ensured.

Fig. 13 is an example of design drawing information for mounting a component on a base board, and fig. 14 is an example of operation information for mounting a component on a base board. The design drawing information is information included in an assembly drawing such as a drawing number, a drawing name, a machine (component) name, a bolt size, a bolt head length, a bolt material, a predetermined fastening torque, and the number of bolts.

The work information includes, for example, work content (name), work date and time, design drawing information to be referred to, predetermined worker information as a worker, vehicle information as a work target, work start time, component information, work place, work end date and time, and the like.

Fig. 15 shows an example of a step (S90) in which virtual bolts corresponding one-to-one to real bolts in the augmented real space are identified as objects to be fastened by real tools. When the machine fastening portion is managed by augmented reality, the fastening management information 30 is stored in the torque management server 28 or the like as information related to the virtual bolt IBn. Therefore, from the viewpoint of managing the fastening portion by augmented reality, it is important to accurately relate information of real bolts RBn determined as the fastening target in real space to virtual bolts IBn in a virtual space in one-to-one correspondence with the real bolts RBn.

< description of procedure for determining object to be fastened by transmitting selected real bolt information to imaginary bolt >

In S50 of fig. 8, in the augmented real space generated by superimposing the reference point RM in real space on the reference point IM in virtual space, arbitrary coordinates (RXn, RYn, RZn) in real space correspond one-to-one to arbitrary coordinates (IXn, IYn, IZn) in virtual space. Assuming that the correspondence relationship is basically established, a procedure (S90) in which information related to a real bolt RBn selected as a fastening target while the socket head RTH of a real tool is fitted in real space is transmitted (delivered) to a virtual bolt IBn corresponding to the real bolt RBn to be identified as the fastening target will be described below as a1 to a 7.

A1: the augmented reality server 12 analyzes an image including the real bolt RBn in fig. 7, which is captured by the camera 10 in real space, and calculates the coordinates RBO of the real bolt RBn (RBX1, RBY1, RBZ 1).

A2: the augmented reality server 12 calls up a 3D model (for example, the device shown in fig. 5) including the virtual bolt IBn from 3D-CAD data stored in advance in the virtual space, and acquires coordinates IBO (IBX1, IBY1, IBZ1) of an arbitrary virtual bolt IBn included in the 3D model.

A3: the augmented reality server 12 grasps, in the augmented real space, a case where the coordinates RBO (RBX1, RBY1, RBZ1) of the real bolt RBn in real space and the coordinates IBO (IBX1, IBY1, IBZ1) of the virtual bolt IBn in virtual space overlap each other or a case where the coordinates RBO (RBX1, RBY1, RBZ1) and the coordinates IBO (IBX1, IBY1, IBZ1) of the virtual bolt IBn in virtual space are close to each other to such an extent that the overlapping can be judged. The above-described processes a1 to A3 are processes for generating an augmented real space by superimposing a virtual space on a real space. With this understanding, it is confirmed again that any real bolt RBn corresponds to any virtual bolt IBn in the augmented real space.

As shown in fig. 7, in real space, when the real bolt RBn before the end of assembly is inserted into the through-hole RPn (n is 1 to 4) of the real part RD2, the head of the real bolt RBn not screwed into the real base RD1 is positioned slightly above the head of the screwed virtual bolt IBn without overlapping the head. Therefore, the X-coordinate and the Y-coordinate of the plane representing the coordinates of real bolt RBn substantially coincide with the X-coordinate and the Y-coordinate of the plane representing the coordinates of virtual bolt IBn, but the Z-coordinate of the height representing the coordinates of real bolt RBn does not coincide with the Z-coordinate of the height representing the coordinates of virtual bolt IBn. Therefore, determination is added to the determination of the overlap between real bolt RBn and virtual bolt IBn to the extent that it can be determined that the real bolt RBn and virtual bolt IBn can be determined to overlap again.

A4: the augmented reality server 12 recognizes that the real bolt RBn is selected as the fastening target in real space, based on the fact that the operator who has shot by the camera 10 has put the sleeve head RTH of the real tool on any real bolt RBn.

A5: the augmented reality server 12 geometrically calculates coordinates RTHO (RTHX1, RTHY1, and RTHZ1) of the set head RTH of the real tool from a reference point RTM1 (a marker such as an LED lamp or a two-dimensional code) included in the real tool RT imaged by the camera 10 in the real space.

A6: the augmented reality server 12 compares and verifies the coordinates RTHO (RTHX1, RTHY1, and RTHZ1) of the socket head RTH of the real tool and the coordinates RBO (RBX1, RBY1, and RBZ1) of the real bolt in the real space, and confirms that the coordinates RTHO of the socket head RTH of the real tool RT and the coordinates RBO of the real bolt coincide with each other or are within a range in which the coincidence can be determined. By this confirmation, it is confirmed that any real bolt RBn is selected as the fastening target.

A7: since the augmented reality server 12 confirms that the real bolts RBn (coordinates RBO) and the virtual bolts IBn (coordinates lbo) are in one-to-one correspondence at a3, the information of the real bolts RBn that are selected as the fastening target by being held by the sleeve head RTH of the real tool in the real space is transmitted to the virtual bolts IBn in the virtual space.

Through the above steps, it is possible to transmit information that real bolts RBn are selected as objects to be fastened to virtual bolts IBn corresponding one-to-one to real bolts RBn, and to accumulate additional information such as fastening management information 30 as information on virtual bolt IBn in a virtual space in an arbitrary server.

Further, by providing the configuration of a5, even when the camera 10 cannot capture the real bolt RBn because the real bolt RBn is blocked by a blocking object, for example, the augmented reality server 12 can obtain the coordinates of the set head RTH of the real tool RT, and can move to a7 next to a 6.

Fig. 16 shows an example of a real tool including a plurality of marks indicating positions of a head (set) portion of the tool. The real tool RT includes a sleeve head portion RTH fitted to the head of the real bolt RBn, a grip portion RTG of the real tool to be held by the operator, and a shank portion RTE connecting the sleeve head portion RTH to the grip portion RTG. On the upper surface of the shank RTE of the real tool RT arranged in the posture when the real bolt RT arranged vertically is fastened, a mark RTM1, a mark RTM3, and a mark RTM5 are provided along the axial direction of the shank RTE. On the side surface of the shank RTE of the real tool RT held in the same posture, a marker RTM2 and a marker RTM4 are provided along the axial direction of the shank RTE.

The marks RTMn (n is 1 to 5) have codes indicating a direction from each mark RTMn to the sleeve head RTH and distances L1 to L5 from each mark RTMn to the sleeve head RTH. Therefore, even when the range from the jacket head RTH to the mark RTM2 becomes the shadow of a mask and the camera 10 cannot image the jacket head RTH, the camera 10 can image the marks RTM3 to RTM 5. For example, when the camera 10 cannot clearly capture the marker RTM3, the augmented reality server 12 that has received the image of the camera 10 can recognize that the cuff RTH having the axis along the RZ direction exists at a position displaced by the distance L3 in the-RX direction from the code included in the marker RTM 3.

Therefore, the augmented reality server 12 can geometrically calculate the coordinates RTHO of the cuff head RTH (RTHX1, RTHY1, RTHZ1) based on the information of the cuff head RTH obtained from the marker RTM 3. The mark RTMn (n is 1 to 3) and the mark RTMn (n is 2, 4) are provided at positions shifted by 90 degrees around the axis of the shank RTE.

Therefore, when fastening real bolt RBn having an axis along the RZ axis, augmented reality server 12 refers to information of marker RTMn (n is 1 to 3) imaged by camera 10, and when fastening real bolt RBn having an axis along the RY axis, augmented reality server 12 mainly refers to information of marker RTMn (n is 2, 4), and can obtain coordinates of sleeve head RTH. Therefore, with the real tool shown in fig. 16, the coordinates of the box head RTH can be obtained with high quality regardless of the fastening posture of the real bolt RBn.

Fig. 17 is another example of the step (S90) of determining the virtual bolts corresponding one-to-one to the real bolts in the augmented real space as the fastening objects to be fastened by the real tool. The matters common to the flow described with reference to fig. 15 are omitted, and the specific flow shown in fig. 17 is mainly described below in B1 to B8.

B1 to B3 are substantially the same as those described in a1 to A3 described with reference to fig. 15, and therefore, the description thereof is omitted here.

B4: the augmented reality server 12 photographs a real tool RT operated by an operator in real space with the camera 10 and reads the shape thereof.

B5: the augmented reality server 12 selects a plurality of virtual tools IT (models created by 3D-CAD, etc.) stored in advance in a virtual space (arbitrary server) and calls up the virtual tools IT corresponding to the real tools RT imaged by the camera 10 in B4 one by one to the augmented reality space.

B6: the augmented reality server 12 uses the coordinates RTHO (RTHX1, RTHY1, and RTHZ1) of the set head RTH of the real tool as a reference point, and superimposes the virtual tool IT called out from the virtual space in B5 on the real tool RT in the augmented reality space.

B7: the augmented reality server 12 acquires the coordinates ITHO (ITHX1, ITHY1, and ITH 1) of the head portion ITH of the virtual tool in the virtual space when the virtual tool IT and the real tool RT are superimposed on each other in B6, and causes the virtual tool IT to follow the motion of the real tool RT after the coordinates of the head portion ITH of the virtual tool are acquired in the virtual real space.

B8: when the operator sets the real tool RT on the real bolt RBn and selects the tool to be fastened, the augmented reality server 12 compares and verifies the coordinates ITHO (ITHX1, ITHY1, ITH 1) of the socket head ITH of the virtual tool IT following the real tool RT and the coordinates IBO (IBX1, IBY1, IBZ1) of the virtual bolt Ibn corresponding to the real bolt RBn one by one, and confirms that the coordinates rtoo of the socket head ITH of the virtual tool IT and the coordinates RBO of the real bolt RBn overlap or are within a range within which the coordinates rto of the socket head ITH of the virtual tool IT can be determined to overlap. By this confirmation, an arbitrary real bolt RBn is determined as the fastening target.

Through the above-described flow, since the augmented reality server 12 recognizes that the real bolts RBn (coordinates RBO) and the virtual bolts IBn (coordinates lbo) correspond one-to-one in B3, it is possible to transmit information of the real bolts RBn selected as the fastening target by fitting the socket head RTH of the real tool in the real space to the virtual bolts IBn in the virtual space, and to accumulate additional information such as the fastening management information 30 as information related to the virtual bolts IBn in the virtual space in an arbitrary server.

In the above-described B4 and B5, the virtual tool IT corresponding to the real tool RT one-to-one is retrieved based on the shape of the real tool RT imaged by the camera 10. A marker containing virtual tool IT information corresponding to the real tool RT may be previously attached to the real tool RT, and the augmented reality server 12 may call up the virtual tool IT from the virtual space to the virtual real space based on the information of the marker photographed by the camera 10.

By providing the B7, even when the real bolt RBn is blocked by a blocking object and the camera 10 cannot capture the real bolt RBn, for example, the augmented reality server 12 can acquire the coordinates of the socket head ITH of the virtual tool IT, which correspond one-to-one to the coordinates of the socket head RTH of the real tool RT, and can move to B8 and B9 following B7.

Fig. 18 is another example of the step (S90) of determining the virtual bolts corresponding one-to-one to the real bolts in the augmented real space as the fastening objects to be fastened by the real tools. Items common to the flows described with reference to fig. 15 and 17 are omitted, and the specific flows of fig. 18 are mainly described below in C1 to C8.

C1 to C3 are substantially the same as those described in a1(B1) to A3(B3) described with reference to fig. 15 (fig. 17), and therefore, the description thereof is omitted here.

C4: the augmented reality server 12 images markers (markers) of 3 points (a real tool cuff RTH, a first marker RTM1, and a second marker RTM2) provided on a real tool RT to be imaged by the camera 10, generates a virtual line segment ILH of an arbitrary length extending downward from the real tool cuff RTH in the augmented reality space, and similarly generates a virtual line segment IL1 of the first marker RTM1 and a virtual line segment IL3 of the second marker RTM 2.

C5: the augmented reality server 12 generates a virtual plane IPL including the 3 points of the virtual line segment ILH, the virtual line segment IL1, and the virtual line segment IL2 in the augmented reality space.

C6: the augmented reality server 12 generates an imaginary vertical line IZL passing through the coordinates lbo (IBX1, IBY1, IBZ1) of the imaginary bolt IBn, which the operator is reminded to fasten, in the augmented reality space at step 70.

C7: the augmented reality server 12 calculates a distance ID1 between the contact point SILH and an intersection point SIZL on the virtual plane IPL in the augmented reality space, where the contact point SILH is a contact point with the virtual line segment IL1 on the virtual plane IPL, and the intersection point SIZL is an intersection point between the virtual plane IPL and the virtual vertical line IZL.

C8: the augmented reality server 12 determines that the virtual bolt IBn having the virtual vertical line IZL that becomes the basis of the distance ID1 calculated to be the minimum is determined as the fastening target.

Through the above-described flow, the augmented reality server 12 recognizes that the real bolts RBn (coordinates RBO) and the virtual bolts IBn (coordinates lbo) correspond one-to-one in C3, and therefore can transmit information of the real bolts RBn selected as the fastening target by being fitted with the socket head RTH of the real tool in the real space to the virtual bolts IBn in the virtual space, and can accumulate additional information such as the fastening management information 30 as information related to the virtual bolts IBn in the virtual space in any server.

C4': instead of the method for generating the virtual plane IPL described in C4, an IX-IY plane including the coordinates IBO of the virtual bolt IBn is set as the virtual plane IPL ', and an intersection point of the virtual perpendicular line ILH passing through the socket portion RTH of the real tool and the virtual plane IPL ' is set as the sil '. Then, the augmented reality server 12 calculates the distance ID1 ' on the virtual plane IPL ' between the coordinate lbo of the virtual bolt IBn on the virtual plane IPL ' and the intersection point sil ', and determines that the virtual bolt IBn having the coordinate lbo based on the calculated distance ID1 ' is to be determined as the fastening target.

In the case of C4, when the operator tilts the real tool RT, the virtual space IPL also tilts following the real tool RT, and therefore the analysis load of the augmented reality server 12 may increase. In contrast, in the case of C4 ', even if the operator tilts the real tool RT, the virtual plane IPL ' maintains its posture without being tilted, and is tilted following the tilt only by the perpendicular line ILH ' of the socket head RTH of the real tool. Therefore, the analysis load of the augmented reality server 12 is not likely to increase, and the virtual bolt IBn can be determined to be the fastening target by making the vertical line ILH' follow the inclination of the real tool RT with a small analysis load.

Fig. 19 is another example of the step (S90) of determining the virtual bolts corresponding one-to-one to the real bolts in the augmented real space as the fastening target object fastened by the real tool. Items common to the flows described with reference to fig. 15, 17, and 18 are omitted, and the specific flows of fig. 19 are mainly described below in D1 to D10.

D1 to D3 are substantially the same as those described in a1(B1, C1) to A3(B3, C3) described with reference to fig. 15 (fig. 17 and 18), and therefore, the description thereof is omitted here.

In addition, D4 to D5 are substantially the same as those described in B4 to B5 described with reference to fig. 17, and therefore, the description thereof is omitted here.

D6: the augmented reality server 12 takes the coordinates lbo (IBX1, IBY1, IBZ1) of the imaginary bolt IBn that prompted the operator to tighten the bolt next in step 70 of fig. 8.

D7: the augmented reality server 12 calls up a virtual tool IT corresponding to the real tool RT picked up by the camera 10 called up at D4 in a one-to-one manner in the virtual space, and superimposes the sleeve head ITH on the coordinates IBO (IBX1, IBY1, IBZ1) of the virtual bolt IBn acquired at D6.

D8: the augmented reality server 12 superimposes the vertical axis RZ of the tip of the virtual tool IT called up at D4 on the IZ axis of the coordinates lbo (IBX1, IBY1, IBZ1) of the virtual bolt IBn acquired at D6, that is, rotates the virtual tool IT around the IZ axis in the IX-IY plane in a state where the vertical axes of the two axes are superimposed.

D9: when the operator overlaps the sleeve head RTH of the real tool RT with the head of the real bolt RBn corresponding to the virtual bolt IBn at S70, the augmented reality server 12 captures an instant at which the virtual tool IT turned in D8 overlaps the real tool RT.

D10: when the augmented reality server 12 captures the moment when the virtual tool IT and the real tool RT that rotate overlap each other in D9, the augmented reality server 12 determines that the coordinates ith o (ITHX1, ith 1, ITHZ1) of the set head of the virtual tool IT and the coordinates RTHO (RTHX1, RTHY1, RTHZ1) of the set head of the real tool RT overlap each other.

The augmented reality server 12 compares and verifies the coordinates IBO (IBX1, IBY1, IBZ1) of the virtual bolt Ibn corresponding one-to-one to the real bolt RBn of the real tool RT to be mounted on the virtual tool IT and the coordinates RTHO (RTHX1, RTHY1, RTHZ1) of the head portion ITH of the virtual tool IT, and confirms that the coordinates RTHO of the head portion ITH of the virtual tool IT and the coordinates IBO of the virtual bolt IBn overlap or are within a range in which the overlap can be determined. By this confirmation, an arbitrary real bolt RBn is determined as the fastening target.

In the case of D8, the operator does not need to make the virtual tool IT follow the real tool RT moving in the augmented reality space, and thus the analysis load of the augmented reality server 12 can be suppressed from increasing. Therefore, the analysis load of the augmented reality server 12 is not likely to increase, and the real tool RT and the virtual tool IR can be superimposed with a small analysis load, so that the virtual bolt IBn can be specified as a fastening target as a result.

As described above, one aspect of the present invention is a method for managing an augmented reality-based mechanical fastening unit using an augmented reality space generated by superimposing a virtual space on a real space, the real space including the real fastening unit and a real tool fitted around the real fastening unit, the virtual space including a virtual fastening unit and a virtual tool fitted around the virtual fastening unit, the augmented reality constituting system generating the augmented reality space by one-to-one correspondence between the real fastening unit and the virtual fastening unit in the augmented reality space, the method comprising: a camera that photographs the real space; and an augmented reality server connected to the camera and analyzing an image set by the camera, wherein the management method includes a fastening object determination step of fitting a sleeve head of the real tool to the real fastening portion, thereby selecting the real fastening portion as a fastening object, and transmitting information that the real fastening portion is selected as a fastening object to the virtual fastening portion.

Thus, in a mechanical fastening operation such as maintenance of an automobile, the progress of the operation can be easily recorded without attaching an identification mark such as an RFIC tag to the fastening portion, and a high-quality fastening operation with low operation cost can be performed.

In another aspect of the present invention, following the fastening object specifying step, the augmented reality-based mechanical fastening unit management method further includes: a fastening step of fastening the real fastening part with the real tool; a tightening management information acquisition step of acquiring tightening management information in which the actual tightening section is tightened in the tightening step; and a fastening management information addition image storage step of storing, in the augmented real space, a fastening end image after the fastening step is ended and a fastening management information addition image obtained by adding the fastening management information to the fastening end image.

By acquiring and storing information and images indicating the situation after the fastening operation is ended in this way, it is possible to record the situation after the fastening operation is reliably ended, and to suppress forgetting of the fastening operation.

In another aspect of the present invention, the reality tool includes a transmission/reception unit that transmits/receives torque information of the reality fastening unit, and a display unit that displays the torque information obtained by the transmission/reception unit, and the augmented reality configuration system processes the fastening management information and adds an image storage step after sensing that information that the reality fastening unit is fastened at a predetermined torque is displayed on the display unit.

By acquiring and displaying the torque information in this manner, it is possible to confirm whether or not the mechanical fastening portion such as a bolt is fastened with a predetermined torque, and the quality of the fastening operation can be improved.

In another aspect of the present invention, the augmented reality-based machine fastening section management method further includes an initial fastening torque storage step of recording that an initial fastening torque, which is a torque observed when the real tool starts fastening the real fastening section, is generated in the real fastening section that is fitted with the real tool and selected as a fastening target.

In this way, by recording the occurrence of the initial torque in the actual fastening portion, forgetting of the fastening operation can be suppressed.

In another aspect of the present invention, the real tool includes the cap portion, a grip portion to be gripped by a worker, and a handle portion connecting the cap portion and the grip portion, and the handle portion includes a mark including direction information indicating a direction from the mark toward the cap portion and distance information from the mark to the cap portion.

In this way, by including the direction information indicating the direction from the mark to the cuff and the distance information from the mark to the cuff, the coordinates indicating the position of the fastening portion can be calculated.

In another aspect of the present invention, in the fastening object specifying step, the augmented reality server performs: the method includes acquiring real fastening part coordinates representing a position of the real fastening part acquired by analyzing the real space photographed by the camera and real tool bit coordinates representing a position of a bit of the real tool acquired by analyzing the real space photographed by the camera, sensing that the real tool bit coordinates of the bit of the real tool and the real fastening part coordinates of the real fastening part coincide with each other, and transmitting information that the real fastening part is inserted into the bit of the real tool and selected as a fastening object to the virtual fastening part corresponding to the real fastening part one by one.

By acquiring the actual tightening part coordinates indicating the actual tightening part position and the actual tool bit head part coordinates indicating the actual tool bit head part position, the tightening part selected as the tightening target can be specified, and the progress of the tightening operation can be recorded in real time.

In another aspect of the present invention, in the fastening object specifying step, the augmented reality server performs: analyzing the real space imaged by the camera to acquire real fastening part coordinates indicating a position of the real fastening part, calling the virtual tool corresponding to the real tool imaged by the camera from the virtual space to the augmented real space, and acquiring virtual tool set head coordinates indicating a position of a set head of the virtual tool superimposed on the real tool, when the sleeve head of the real tool is fitted to the real fastening part by making the virtual tool follow the movement of the real tool, and a step of determining that the virtual tool bit head coordinates of the virtual tool bit part overlap the real fastening part coordinates of the real fastening part, and transmitting information that the real fastening part is inserted into the real tool bit part and selected as a fastening target to the virtual fastening part corresponding to the real fastening part one by one.

In another aspect of the present invention, in the fastening object specifying step, the augmented reality server performs: analyzing the real space photographed by the camera to acquire real tool bit coordinates indicating a position of a bit of the real tool, generating a virtual horizontal plane including the virtual fastening portion that prompts an operator to fasten the real tool bit, generating a virtual vertical line that passes through the real tool bit coordinates of the bit of the real tool and intersects the virtual horizontal plane, in the virtual horizontal plane, a distance between an intersection point where the virtual vertical line intersects the virtual horizontal plane and virtual fastening part coordinates indicating a position of the virtual fastening part is detected, it is determined based on the distance that the real tool bit coordinates of the real tool bit overlap the virtual fastening part coordinates of the virtual fastening part, and information selected as a fastening target is transmitted to the virtual fastening parts one-to-one corresponding to the real fastening parts on which the real tool bit is inserted.

In another aspect of the present invention, in the fastening object specifying step, the augmented reality server performs: in the augmented real space, the virtual tool corresponding to the real tool photographed by the camera is called out from the virtual space, a vertical axis of a cap portion of the virtual tool is overlapped with a vertical axis of a virtual fastening portion coordinate representing a position of the virtual fastening portion for prompting an operator to fasten, and the virtual tool is rotated around the vertical axes of the virtual tool and the virtual fastening portion in a horizontal plane.

This makes it possible to determine whether or not the fastening portion selected as the fastening target is reliably fastened, and thus the quality of the fastening operation can be improved.

While the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

Description of reference numerals:

RBn (1 ~ 4) … real bolt (real space)

Coordinates of RBO … real bolt

IBn (n is 1 ~ 4) … imaginary bolt (imaginary space)

Coordinates of an IBO … imaginary bolt

RPn (1 to 4) … real part through hole (real space)

Rqn (1 ~ 4 for n) … real base plate screw hole (real space)

Axial force of AF … real bolt

AFS … actual bolt specified axial force

RD1 … real base plate (real space)

RD2 … real parts (real space)

Reference point of RM … real space

Reference point of IM … imaginary space

RT … reality tool

Sleeve head of RTH … reality tool

Handle of RTE … real tool

Handle part of RTG … reality tool

Coordinates of sleeve head of RTHO … real tool

Label (sign) attached to RTMn (1-5) … real tool

IT … hypothetical tool

Head of a virtual tool for ITH …

Coordinates of the sleeve head of the ITHO … hypothetical tool

IPL … imaginary plane

ILn (1 to 3) … imaginary vertical line (line segment)

Distance on the imaginary plane of ID1 …

Intersection of the imaginary vertical line of SIZL … and the imaginary plane

Contact of an imaginary line segment of the SIL1 … with an imaginary plane

TL … ending fastening torque

TS … specifies the tightening torque

Initial fastening torque of TB …

5 … network

7 … wearable device (helmet)

10 … camera

12 … Augmented Reality (AR) server 14 … see-through screen (HMD)

22 … design information server

26 … Job management Server

28 … Torque management Server

29 … resolution server

30 … fastens the coordinate systems IX, IY, and IZ … of the management information RX, RY, and RZ … in real space to the coordinate system in phantom space.