阅读说明：本技术 虚拟对象显示系统 (Virtual object display system ) 是由 大西庸士 于 2019-06-03 设计创作，主要内容包括：本发明提供一种虚拟对象显示系统,以简易的方法来显示虚拟对象。图表生成部生成将包括相机的结构要素设为节点的图表,作为控制对象的机械结构,节点选择部选择想要显示虚拟对象的节点,选择节点通知部将想要显示虚拟对象的节点通知给机械结构管理装置,转换信息计算部对包括图表中的从相机节点向显示对象的节点的路径上的控制轴节点的坐标值作为变量的转换信息进行计算,该转换信息用于根据图表计算相机节点的坐标系中的显示对象的节点的位置以及/或者姿态,转换信息通知部将转换信息通知给增强信息控制装置,坐标信息转换部将所通知的各控制轴的坐标值转换为相机节点的坐标系中的显示对象的各节点的位置以及/或者姿态。(The invention provides a virtual object display system which displays a virtual object by a simple method. The graph generating unit generates a graph in which a component including a camera is defined as a node, the graph being a machine structure to be controlled, the node selecting unit selects a node on which a virtual object is to be displayed, the selected node notifying unit notifies the machine structure management device of the node on which the virtual object is to be displayed, the conversion information calculating unit calculates conversion information including, as a variable, a coordinate value of a control axis node on a path from the camera node to the node to be displayed in the graph, the conversion information is used for calculating the position and/or posture of the display object node in the coordinate system of the camera node according to the graph, the conversion information notification unit notifies the enhanced information control device of the conversion information, and the coordinate information conversion unit converts the coordinate values of the control axes notified by the conversion information notification unit into the position and/or posture of the display object node in the coordinate system of the camera node.)

1. A virtual object display system is provided with a machine structure management device having a graph generation unit, a control point coordinate system insertion unit, a node information notification unit, a conversion information calculation unit, and a conversion information notification unit, and an augmentation information control device having a node selection unit, a selection node notification unit, and a coordinate information conversion unit,

It is characterized in that the preparation method is characterized in that,

The graph generating unit generates a graph in which a component including a camera is defined as a node, and the graph is a mechanical structure to be controlled,

The control point coordinate system insertion unit inserts a control point and a coordinate system into the graph,

The node information notifying unit notifies the enhanced information control device of node information that can be displayed,

The node selection part selects a node on which a virtual object is desired to be displayed,

The selected node notifying section notifies the machine structure management apparatus of a node on which the virtual object is to be displayed,

the conversion information calculation unit calculates conversion information for calculating a position and/or orientation of a node of a display object in a coordinate system of a camera node based on the graph, the conversion information including, as a variable, coordinate values of a control axis node on a path from the camera node to the node of the display object in the graph,

The conversion information notifying section notifies the enhancement information control device of the conversion information,

the coordinate information conversion unit converts the notified coordinate values of the respective control axes into positions and/or orientations of the respective nodes of the display object in the coordinate system of the camera node, using the conversion information.

2. The virtual object display system of claim 1,

The virtual object display system is further provided with a display device,

The enhancement information control device further includes:

An enhanced information display data calculation unit that calculates enhanced information display data for calculating a virtual object as enhanced information by the display device, based on the position and/or orientation of each node of the display object in the coordinate system of the camera node converted by the coordinate information conversion unit; and

and an enhanced information display data notification unit that notifies the display device of the enhanced information display data.

3. The virtual object display system of claim 1 or 2,

The coordinate information conversion unit converts the coordinate values acquired from the numerical controller into coordinate values of the control axes, and converts the coordinate values into positions and/or orientations of the nodes to be displayed in the coordinate system of the camera node.

4. The virtual object display system of any of claims 1-3,

The enhanced information control device further includes a storage unit having data of the graph.

5. The virtual object display system of any of claims 1-4,

The virtual object display system further includes a server that stores data of the graph.

6. the virtual object display system of any of claims 1-5,

The machine structure management device is integrated with a numerical controller of a machine tool.

7. The virtual object display system of any of claims 1-5,

The mechanical structure management device resides on the cloud.

Technical Field

The present invention relates to a virtual object display system for performing simulation using technologies of Augmented Reality (AR) and composite Reality (MR).

background

Conventionally, in the field of machine tools controlled by numerical controllers, workpieces and jigs are designed by CAD (computer-aided design) or the like. In addition, a machining program for machining a workpiece using the designed jig is created.

then, the numerical controller controls the machine tool based on the machining program, thereby machining the workpiece.

here, before the designed jig, workpiece, and machining program are actually installed in the production line, machining simulation is usually performed in order to confirm whether or not these jig, workpiece, and machining program are appropriate.

When the machining simulation is performed by checking the operation of the actual machine, it is needless to say that the machining simulation cannot be performed until the actual jig is completed. Therefore, the process is stopped before the completion of the jig.

Further, when the operation after the completion of the jig is checked, problems such as interference are found, and when the design of the jig needs to be changed, the process is further extended. In addition, a cost for changing the design of the jig is also incurred. If the design of the jig is not changed, the machining program needs to be changed, but in this case, the cycle time required for machining may be longer than the initial time.

In view of these problems, there are techniques as follows: the machining simulation is performed virtually by arithmetic processing of a personal computer or the like without performing operation confirmation by an actual machine.

For example, in the technique disclosed in patent document 1, machining simulation is performed by virtually objectifying all structures in a machine tool.

However, in the technique disclosed in patent document 1, it is necessary to create not only virtual objects of a workpiece and a jig but also virtual objects of the entire machine for each of a plurality of types of machine tools. In order to reproduce the actual operation of the machine tool, it is also necessary to perform an operation process on a virtual object on a movable portion of the machine tool. In other words, there is a problem that it is difficult to create a virtual object.

Even if the virtual object is created in this way, there is a problem that a difference from the real object occurs when the degree of reproduction of the virtual object is low.

patent document 2 discloses a technique that has been completed in view of the problems associated with the virtual object as described above. In the technique disclosed in patent document 2, an image of the inside of the machine tool is acquired by a camera, and a tool holding portion or a workpiece holding portion registered in advance is extracted as a feature point. Then, based on the positions of the feature points, a virtual object of a tool or a workpiece registered in advance is displayed in an image of the machine tool captured in real time in an overlaid manner. In this way, it is not necessary to create a virtual object for the machine tool. Hereinafter, information that overlaps with an object that actually exists in real space in this manner will be referred to as "augmented information".

However, the shooting direction of the camera needs to be fixed to a preset direction, and feature points need to be extracted.

patent document 1: japanese patent No. 4083554 specification

Patent document 2: japanese patent No. 5384178 specification

Disclosure of Invention

Therefore, an object of the present invention is to provide a virtual object display system capable of displaying a virtual object by a simple method.

(1) A virtual object display system according to the present invention (for example, a virtual object display system 10 described later) includes a machine structure management device (for example, a machine structure management device 100 described later) including a graph generation unit (for example, a graph generation unit 111 described later), a control point coordinate system insertion unit (for example, a control point coordinate system insertion unit 113 described later), a node information notification unit (for example, a node information notification unit 114 described later), a conversion information calculation unit (for example, a conversion information calculation unit 115 described later), and a conversion information notification unit (for example, a conversion information notification unit 116 described later), and an augmentation information control device (for example, an augmentation information control device 200 described later) including a node selection unit (for example, a node selection unit 211 described later), a selection node notification unit (for example, a selected node notification unit 212 described later) that generates a graph in which a component including a camera is a node, and a coordinate information conversion unit (for example, a coordinate information conversion unit 213 described later) that generates a mechanical structure to be controlled, the graph including the component including the camera as a control target, the control point coordinate system insertion unit inserting a control point and a coordinate system into the graph, the node information notification unit notifying the enhancement information control device of node information that can be displayed, the node selection unit selecting a node that is intended to display a virtual object, the selected node notification unit notifying the mechanical structure management device of the node that is intended to display the virtual object, and the conversion information calculation unit calculating conversion information for calculating a position and/or posture of the node that is the display target in a coordinate system of the camera node based on the graph, the conversion information includes, as a variable, coordinate values of control axis nodes on a path from the camera node to the node of the display object in the graph, the conversion information notification unit notifies the enhanced information control device of the conversion information, and the coordinate information conversion unit converts the notified coordinate values of the respective control axes into positions and/or postures of the respective nodes of the display object in a coordinate system of the camera node using the conversion information.

(2) The virtual object display system (for example, a virtual object display system 10 described later) according to (1) may further include a display device (for example, a display device 300 described later), and the augmentation information control device (for example, an augmentation information control device 200 described later) may further include: an enhanced information display data calculation unit (e.g., an enhanced information display data calculation unit 214 described later) that calculates enhanced information display data for calculating a virtual object as enhanced information by the display device, based on the position and/or orientation of each node of the display object in the coordinate system of the camera node converted by the coordinate information conversion unit (e.g., a coordinate information conversion unit 213 described later); and an enhanced information display data notification unit (for example, an enhanced information display data notification unit 215 described later) that notifies the display device of the enhanced information display data.

(3) In the virtual object display system (for example, the virtual object display system 10 described later) described in (1) or (2), the coordinate information conversion unit (for example, the coordinate information conversion unit 213 described later) may convert the coordinate values acquired from the numerical controller (for example, the numerical controller 150 described later) into the coordinate values of the control axes into the positions and/or orientations of the nodes of the display object in the coordinate system of the camera node.

(4) In the virtual object display systems (for example, the virtual object display system 10 described later) described in (1) to (3), the augmentation information control apparatus (for example, the augmentation information control apparatus 200 described later) may further include a storage unit (for example, the storage unit 220 described later) having data of the graph.

(5) the virtual object display system (for example, a virtual object display system 10 described later) according to (1) to (4) may further include a server that stores data of the graph.

(6) In the virtual object display systems (for example, the virtual object display system 10 described later) described in (1) to (5), the machine configuration management device (for example, the machine configuration management device 100 described later) may be integrated with a numerical controller (for example, the numerical controller 150 described later) of a machine tool (for example, the machine tool 400 described later).

(7) In the virtual object display systems (for example, the virtual object display system 10 described later) described in (1) to (5), the machine configuration management device (for example, the machine configuration management device 100 described later) may be present on the cloud.

According to the present invention, a virtual object can be displayed in a simple manner.

Drawings

Fig. 1 is a block diagram showing the basic configuration of the whole embodiment of the present invention.

Fig. 2 is a functional block diagram of the mechanical configuration management device 100 according to the embodiment of the present invention.

Fig. 3 is a functional block diagram of the numerical controller 150 according to the embodiment of the present invention.

fig. 4 is a functional block diagram of the enhanced information control apparatus 200 according to the embodiment of the present invention.

fig. 5 is an explanatory diagram of a method of generating a mechanical structure tree in the embodiment of the present invention, showing a mechanical structure.

fig. 6 is an explanatory diagram of a method of generating a mechanical structure tree in the embodiment of the present invention.

Fig. 7 is an explanatory diagram of a method of generating a mechanical structure tree in the embodiment of the present invention, showing the mechanical structure tree.

Fig. 8 is a flowchart illustrating a method of generating a mechanical structure tree in the embodiment of the present invention.

fig. 9A is an explanatory diagram of a parent-child relationship of mechanical components in the embodiment of the present invention, in which the position and orientation of 505 are affected when the coordinate value of 504 is changed, and the position and orientation of 504 are not affected even when the coordinate value of 505 is changed.

Fig. 9B is an explanatory diagram of parent-child relationships of the mechanical components in the embodiment of the present invention, in which when the coordinate value of 502 is changed, both the positions and the orientations are affected.

Fig. 10A is an explanatory diagram of a method of inserting a unit into a mechanical structure tree.

Fig. 10B is an explanatory diagram of a method of inserting a unit into a mechanical structure tree.

Fig. 10C is an explanatory diagram of a method of inserting a unit into a mechanical structure tree.

Fig. 11 is a diagram showing an example of a mechanical structure according to the embodiment of the present invention, in which 512 is a homogeneous matrix M showing a position and orientation of a connection point 520_AAnd 513 is a homogeneous matrix M representing the position and orientation of the connection point 510_B。

Fig. 12A is a diagram showing an example of a machine to be generated as a machine structure tree.

Fig. 12B is a diagram showing an example of a machine structure tree corresponding to a machine to be a target of generation of the machine structure tree.

Fig. 13 is a diagram showing an example in which a coordinate system and a control point are inserted into each node of a machine in the embodiment of the present invention.

Fig. 14 is a diagram showing an example of a mechanical structure tree into which a coordinate system and control points are inserted in the embodiment of the present invention.

fig. 15A is a diagram showing an example of a machine for inserting a shift and orientation matrix into each node in the embodiment of the present invention.

Fig. 15B is a diagram showing an example in which an offset and attitude matrix is inserted into each node of a machine in the embodiment of the present invention.

Fig. 16 is a diagram showing an operation flow of inserting a control point into a mechanical structure tree in the embodiment of the present invention.

fig. 17 is a diagram showing an example of a mechanical structure tree into which a coordinate system and control points are inserted in the embodiment of the present invention.

Fig. 18 is a diagram showing an example of information used when generating conversion information in the embodiment of the present invention, and shows a mechanical structure and a mechanical structure tree.

Fig. 19 is a diagram showing an example of information used when generating conversion information in the embodiment of the present invention.

Fig. 20 is a diagram illustrating an operation flow of the virtual object display method according to the embodiment of the present invention.

fig. 21A is a diagram showing an example of an embodiment of the present invention.

fig. 21B is a diagram illustrating an example of an embodiment of the present invention, showing a mechanical structure tree.

Fig. 22A is a diagram showing an example of an embodiment of the present invention.

fig. 22B is a diagram illustrating an example of an embodiment of the present invention, showing a mechanical structure tree.

Description of reference numerals:

10a virtual object display system;

100 a mechanical structure management device;

110 a control unit;

111 a graph generating unit;

112 node adding part;

113 a control point coordinate system insertion section;

114 a node information notification unit;

115 a conversion information calculation unit;

116 a conversion information notification unit;

120 a storage section;

150 numerical control means;

160 a control unit;

161 coordinate information notifying unit;

162 a servo motor control unit;

200 enhanced information control means;

211 a node selection unit;

212 select a node notification section;

213 coordinate information converting section;

214 enhanced information display data calculation unit;

215 enhanced information display data notification unit;

220 a storage unit;

300 a display device;

400 machine tool.

Detailed Description

< 1. Structure of virtual object display System >

Next, embodiments of the present invention will be described in detail with reference to the drawings. First, the overall configuration of the present embodiment will be described with reference to fig. 1.

The virtual object display system 10 according to the present embodiment includes a machine configuration management device 100, a numerical controller 150, an enhancement information controller 200, a display device 300, and a machine tool 400.

the machine structure management device 100 is a device specific to the present embodiment, and is configured to create a graph (hereinafter, also referred to as a machine structure tree) in which each component of the machine tool 400 is a node, and manage the machine structure using the graph, whereby the enhancement information control device 200 described later can perform control using data of the machine structure based on the machine structure tree.

more specifically, the machine structure management device 100 creates a machine structure tree representing the structure of the machine tool 400 by the method described in < 5. creation of a machine structure tree > described later, and further additionally registers a camera as a node of the machine structure tree. Since the mechanical structure tree includes the camera as a node because the mechanical structure tree has positional relationship information of each node, the enhanced information control device 200, which will be described later, knows the positional relationship and the attitude relationship with the camera for all the nodes on the mechanical structure tree.

The detailed configuration of the mechanical configuration management device 100 will be described later with reference to fig. 2.

The numerical controller 150 is a device having a function as a general numerical controller and a function of performing communication with the machine configuration management device 100. The numerical controller 150 is connected to and communicable with the machine tool 400. Then, the numerical controller 150 controls the machine tool 400 based on the movement amount of each control axis output based on the machining program incorporated in the numerical controller 150 itself, thereby machining the workpiece.

The numerical controller 150 also outputs the coordinate values of the control axes output based on the machining program to the enhanced information controller 200.

In this way, numerical controller 150 outputs the movement amount to both machine structure management apparatus 100 and machine tool 400. In this regard, the output of the movement amount from numerical controller 150 to machine structure management device 100 may be synchronized with the output of the movement amount from numerical controller 150 to machine tool 400, or may be asynchronously transferred.

The detailed configuration of the numerical controller 150 will be described later with reference to fig. 3.

the augmented information control device 200 calculates the display position and display orientation of the virtual object by the augmented reality technique, thereby performing control for appropriately displaying the virtual object.

The detailed configuration of the enhancement information control device 200 will be described later with reference to fig. 4.

The display apparatus 300 acquires the virtual object output by the augmentation information control apparatus 200 and the display position and orientation thereof. Then, based on the acquired information, the virtual object is displayed on the display device 300. The acquired information corresponds to a camera coordinate system.

the machine tool 400 is a general machine tool, and moves and rotates the control axes in accordance with the movement amounts of the control axes output from the numerical controller 150. Further, the camera fixed to machine tool 400 outputs information acquired by imaging to enhancement information control device 200.

In the present embodiment, with the above-described configuration, the user refers to the virtual object displayed in accordance with the camera coordinate system by using the display device 300, and refers to the actual structure of the machine tool 400 that is movable in accordance with the movement amount of each control axis via the display. Thus, the user can observe the appearance of the machining simulation.

The configuration shown in fig. 1 is merely an example. For example, the display device 300 may be implemented by a tablet computer type terminal. In addition, a part or all of the functions of machine configuration management apparatus 100 may be mounted on numerical controller 150. In addition, a part or all of the functions of enhancement information control device 200 may be mounted on display device 300 or numerical controller 150. The enhanced information control apparatus 200 may be implemented by a single apparatus, or may be implemented by a combination of a plurality of apparatuses. Further, the enhanced information control device 200 may be implemented by a device provided in the vicinity of the numerical controller 150 or the machine tool 400, but may also be implemented by a server device provided remotely from the numerical controller 150 or the machine tool 400 via a network, or the like. Each communication connection may be a wired connection or a wireless connection. For example, although the drawings show an example in which the communication connection between the mechanical configuration management apparatus 100, the numerical controller 150, and the enhanced information controller 200 is performed by wired connection according to Ethernet (registered trademark), the connection may be wireless connection.

< 2. Structure of mechanical Structure management device >

Fig. 2 is a functional block diagram of the mechanical structure management apparatus 100.

The machine structure management device 100 includes a control unit 110 and a storage unit 120, the control unit 110 includes a graph generation unit 111, a control point coordinate system insertion unit 113, a node information notification unit 114, a conversion information calculation unit 115, and a conversion information notification unit 116, and the graph generation unit 111 includes a node addition unit 112.

The control unit 110 is a processor that controls the entire machine configuration management device 100. The control unit 110 reads a system program and an application program stored in a ROM (not shown) via a bus, and realizes the functions of the graph generation unit 111, the node addition unit 112, the control point coordinate system insertion unit 113, the node information notification unit 114, the conversion information calculation unit 115, and the conversion information notification unit 116, which are provided in the control unit 110, according to the system program and the application program.

The graph generating unit 111 generates a mechanical structure of the machine tool 400 including a camera in a graph form. The node adding unit 112 included in the graph generating unit 111 adds a node to the generated graph. These detailed operations are described in detail in "5. generation of mechanical structure tree" below.

The control point coordinate system insertion unit 113 inserts the control points and the coordinate system into the graph of the machine structure. The detailed operation will be described in detail in "6. automatic insertion of control points and coordinate values" below.

The node information notification unit 114 notifies the enhanced information control device 200, which will be described later, of information on nodes that can be displayed.

As will be described later, after the node on which the virtual object is to be displayed is notified from the selected node notifying unit 212 of the enhanced information control device 200, the conversion information calculating unit 115 calculates, based on the graph described above, conversion information including, as a variable, coordinate values of control axis nodes on a path from the camera node to the node on which the virtual object is to be displayed, the position and/or orientation of each node on which the virtual object is to be displayed in the coordinate system of the camera node.

The conversion information may be in a matrix form, a vector form, or a yaw, tilt, and pitch form. The detailed operation is described in detail below in "calculation of conversion information" 7.

The conversion information notification unit 116 notifies the coordinate information conversion unit 213 of the enhancement information control device 200 of the conversion information calculated by the conversion information calculation unit 115.

The storage unit 120 stores information relating to the machine structure tree generated by the graph generation unit 111.

The detailed operations of the graph generation unit 111, the node information notification unit 114, the conversion information calculation unit 115, the conversion information notification unit 116, and the storage unit 120 are described in detail below in "8. method for displaying virtual objects".

< 3. Structure of numerical control apparatus

Fig. 3 is a functional block diagram of the numerical controller 150.

The numerical controller 150 includes a control unit 160, and the control unit 160 includes a coordinate information notification unit 161 and a servo motor control unit 162.

The control unit 160 is a processor that controls the numerical controller 150 as a whole. The control unit 160 reads a system program and an application program stored in a ROM (not shown) via a bus, and realizes the functions of the coordinate information notification unit 161 and the servo motor control unit 162 provided in the control unit 160 based on the system program and the application program.

The coordinate information notification unit 161 notifies the coordinate information conversion unit 213 of the enhancement information control device 200 of the coordinate information of the machine tool 400 in operation.

The servomotor control section 162 receives the movement command amount of each axis from the control section 160, and outputs the command of each axis to a servomotor (not shown).

the numerical controller 150 includes other components that are included in a general numerical controller in order to perform numerical control of the machine tool 400, but the description thereof is omitted.

< 4. Structure of enhanced information control apparatus

Fig. 4 is a functional block diagram of the enhanced information control apparatus 200.

the enhancement information control device 200 includes a control unit 210 and a storage unit 220, and the control unit 210 includes a node selection unit 211, a selected node notification unit 212, a coordinate information conversion unit 213, an enhancement information display data calculation unit 214, and an enhancement information display data notification unit 215.

The control unit 210 is a processor that controls the enhancement information control device 200 as a whole. The control unit 210 reads a system program and an application program stored in a ROM (not shown) via a bus, and realizes the functions of the node selection unit 211, the selected node notification unit 212, the coordinate information conversion unit 213, the enhanced information display data calculation unit 214, and the enhanced information display data notification unit 215, which are included in the control unit 210, according to the system program and the application program.

The node selection unit 211 selects a node on which a virtual object is to be displayed.

the selected node notification unit 212 notifies the conversion information calculation unit 115 of the mechanical configuration management device 100 of the node selected by the node selection unit 211.

The coordinate information conversion unit 213 calculates the position and/or orientation of the virtual object in the camera coordinate system based on the coordinate values of the respective control axes periodically acquired from the numerical controller 150, based on the conversion information acquired from the machine configuration management device 100.

the enhanced information display data calculation unit 214 calculates enhanced information display data for displaying enhanced information of a virtual object or the like based on a technique such as AR or MR. More specifically, the enhanced information display data calculation unit 214 converts the position and/or orientation of the virtual object in the camera coordinate system into the position and/or orientation in the screen coordinate system.

The enhanced information display data notification unit 215 transmits the enhanced information display data calculated by the enhanced information display data calculation unit 214 to the display device 300. The enhanced information display data includes the shape and the like of the virtual object, and the display position and the display posture and the like of the virtual object after being converted into the coordinate values of the screen coordinate system. The enhanced information display data may be transmitted wirelessly or by wire.

the storage unit 220 stores information relating to the chart generated by the chart generation unit 111 of the machine configuration management device 100.

The detailed operations of the node selection unit 211, the selected node notification unit 212, the coordinate information conversion unit 213, the enhanced information display data calculation unit 214, the enhanced information display data notification unit 215, and the storage unit 220 are described in detail below in "8. method for displaying virtual objects".

< 5. Generation of mechanical Structure Tree >

The machine configuration management device 100 according to the embodiment of the present invention initially generates a graph showing a machine configuration. A method of generating a mechanical structure tree as an example of a graph will be described in detail with reference to fig. 5 to 11.

A method of generating a machine structure tree representing the structure of a machine shown in fig. 5 will be described as an example. In the machine of fig. 5, the X axis is set perpendicular to the Z axis, and a tool 1 is provided on the X axis and a tool 2 is provided on the Z axis. On the other hand, a B axis is set on the Y axis, a C axis is set on the B axis, and a workpiece 1 and a workpiece 2 are set on the C axis. The method of representing the mechanical structure as a mechanical structure tree is as follows.

First, as shown in fig. 6, only the origin 201 and the nodes 202A to 202I are arranged. At this stage, the origin 201 is not connected to the node 202 and the node 202, and the names of the origin and the node are not set.

Next, the axis name (axis type) of each axis, the name of each tool, the name of each workpiece, the name of each origin, and the physical axis number (axis type) of each axis are set. Next, a parent node (axis type) of each axis, a parent node of each tool, and a parent node of each workpiece are set. Finally, the cross offsets of the respective axes (axis types), the cross offsets of the respective tools, and the cross offsets of the respective works are set. As a result, the mechanical structure tree shown in fig. 7 is generated.

Note that each node of the mechanical structure tree is not limited to the above-described information, and may or may not include, for example, an identifier (name), an identifier of its parent node, identifiers of all child nodes that are themselves parent nodes, a relative offset (cross offset) with respect to the parent node, a relative coordinate value with respect to the parent node, a relative movement direction (unit vector) with respect to the parent node, a node type (linear axis/rotation axis/cell (described later)/control point/coordinate system/origin, and the like), a physical axis number, and information relating to a conversion formula between an orthogonal coordinate system and a physical coordinate system.

by setting the values for the nodes in this manner, data having a data structure of a machine structure tree is generated in the machine structure management device 100. When another machine (or robot) is added, the origin can be added, and the node can be added.

Fig. 8 is a flowchart showing a general method of generating the mechanical structure tree, particularly, a method of setting each value to each node.

In step S11, the graph generation unit 111 acquires the values of the parameters set for the nodes.

in step S12, if the set parameter item is "parent node of itself" (yes in S12), the process proceeds to step S13. If the set parameter item is not "parent node of itself" (no at S12), the process proceeds to step S17.

In step S13, in the case where the node in which the parameter is set has already set a parent node (S13: yes), the process moves to step S14. In the case where the node having the set parameter does not set the parent node (S13: NO), the process proceeds to step S15.

In step S14, the graph generation unit 111 deletes its own identifier from the item of the "child node" that the current parent node of the node for which the parameter is set has, and updates the mechanical structure tree.

In step S15, the graph generation unit 111 sets values for the respective items of the nodes for which the parameters are set.

in step S16, the graph generation unit 111 adds its own identifier to the item of the "child node" for the parent node, updates the mechanical structure tree, and then ends the flow.

In step S17, the graph generation unit 111 sets values for the respective items of the nodes for which the parameters are set, and then ends the flow.

By using the above-described method for generating data having a data structure of a machine structure tree, the parent-child relationship between the machine components can be set.

Here, the parent-child relationship is a relationship in which, for example, when two rotation axis nodes 504 and 505 are provided as shown in fig. 9A, a change in the coordinate value of one node 504 affects unilaterally the geometric state (typically, position and orientation) of the other node 505. In this case, the nodes 504 and 505 are said to be in a parent-child relationship, with node 504 being referred to as the parent and node 505 being referred to as the child.

however, for example, as shown in fig. 9B, in a mechanical structure including two linear axis nodes 502 and 503 and four universal joints 501, there is a mechanism that affects each other such that a coordinate value (length) of one of the nodes 502 and 503 is changed to change not only a geometric state of the other but also a geometric state of the node itself. In that case, it can be seen that they are parents to each other, i.e., the parent-child relationship is bidirectional.

In this way, a mechanism in which a change in a certain node affects other nodes mutually is understood as one unit from the viewpoint of convenience, and the unit is inserted into the mechanical structure tree to generate the entire mechanical structure tree. When a cell has two connection points 510 and 520 as shown in fig. 10A and is inserted into a mechanical structure tree as shown in fig. 10B, a parent node is connected to the connection point 520 and a child node is connected to the connection point 510 as shown in fig. 10C. In addition, the cell has a transition matrix from connection point 520 to connection point 510. The transformation matrix is represented by the coordinate values of the nodes contained by the cell. In the case of a mechanical structure such as that shown in fig. 11, for example, a homogeneous matrix representing the position and posture of the connection point 520 is defined as M_AM is a homogeneous matrix representing the position and posture of the connection point 510_BThen, the coordinate value x of each linear axis node included in the conversion use means between these matrices₁、x₂As shown below.

[ number 1]

When θ and L are expressed by the following formulas,

M_B＝TM_A

Wherein T is represented as follows.

The unit representing the mechanical structure has a homogeneous conversion matrix such as T in the above-described numerical expression of [ number 1 ]. The homogeneous matrix is a 4 × 4 matrix capable of collectively representing positions and orientations as in the following expression of [ number 2 ].

[ number 2]

Attitude and position

Even when the nodes are not in a parent-child relationship with each other, a unit in which some nodes are combined into one node in advance may be defined and configured in the mechanical structure tree in order to simplify calculation processing and setting.

As described above, in the present embodiment, the mechanical structure diagram can include a unit in which a plurality of axes are integrated into one as a component.

< 6. automatic insertion of control points and coordinate values >

In order to designate various positions on the machine structure as control points and set coordinate systems of the various positions on the machine structure, the following method is implemented using the machine structure tree generated in the above-described "generation of 5. machine structure tree".

For example, in the multi-station turret machine tool 350 shown in fig. 12A, the X1 axis is set to be perpendicular to the Z1 axis, and the tool 1 is set on the X1 axis. The X2 axis is set perpendicular to the Z2 axis, and the tool 2 is set on the X2 axis. In the table, a C1 axis and a C2 axis are set in parallel on the C axis, and a workpiece 1 and a workpiece 2 are set on the C1 axis and the C2 axis, respectively. When the mechanical structure is represented by a mechanical structure tree, the mechanical structure tree shown in fig. 12B is obtained.

Taking a series of nodes connecting each workpiece to the origin of the machine as an example, as shown in fig. 13, a coordinate system and a control point are automatically inserted into the origin of the machine, the C-axis, the C1-axis, the C2-axis, the workpiece 1, and the workpiece 2, respectively. This is performed not only for the table but also for all of the X1 axis, X2 axis, Z1 axis, Z2 axis, tool 1, and tool 2, which are a series of nodes connecting the respective tools to the origin of the machine. As a result, as shown in fig. 14, the control points and the coordinate systems corresponding to the nodes are automatically inserted into all the nodes constituting the mechanical structure tree. In general, a coordinate system and a tool are designated as control points for a workpiece when machining is performed. This makes it possible to cope with various situations, such as a case where a control point is desired to be designated for a workpiece in order to move the workpiece itself to a predetermined position, and a case where a coordinate system is desired to be set for a tool itself in order to polish another tool with a certain tool.

As shown in fig. 15A, each control point and the coordinate system have an offset. Therefore, a point away from the node center can be set as the control point or the origin of the coordinate system. In addition, each control point and the coordinate system have a posture matrix. The attitude matrix represents the attitude (orientation, inclination) of the control point in the case of the attitude matrix of the control point, and represents the attitude of the coordinate system in the case of the attitude matrix of the coordinate system. In the mechanical structure tree shown in fig. 15B, the offset and attitude matrices are represented in the form of being associated with the respective corresponding nodes. Each control point and the coordinate system have information of "movement" and "cross offset" of a node existing on a path to the root of the mechanical structure tree, and can be set to each control point and coordinate system.

Fig. 16 shows a flowchart that generalizes the automatic insertion method of the control point described above. The flowchart includes a flowchart a and a flowchart B in detail, and as described later, the flowchart B is executed in the middle of the flowchart a.

First, the flowchart a will be explained.

In step S21, the graph generation unit 111 sets a machine structure tree.

In step S22, flowchart B is executed, and the flow of flowchart a ends.

next, the flowchart B will be explained.

In step S31 of the flowchart B, when the control point and the coordinate system are inserted into the node (S31: yes), the flow ends. If the control point and the coordinate system are not inserted into the node (no in S31), the process proceeds to step S32.

In step S32, the control point coordinate system insertion unit 113 inserts the control point and the coordinate system into the node, and stacks one variable n. In addition, n is 1.

In step S33, in the case where the node has the nth child node (S33: yes), the process moves to step S34. In the case where the node does not have the nth child node (S33: NO), the process moves to step S36.

in step S34, the flowchart B itself is recursively executed for the nth child node.

In step S35, n is increased by 1. That is, assuming that n is n +1, the process returns to step S33.

In step S36, a variable n is popped up, and the flow of flowchart B ends.

According to the above method, the control point coordinate system insertion unit 113 inserts the control points and the coordinate system as nodes for each node of the graph of the mechanical structure. In the above description, although the example in which the control point and the coordinate system are added as the nodes has been described, the control point coordinate system insertion unit 113 may be similarly implemented in the embodiment in which the control point and the coordinate system are included as information for each node of the graph of the mechanical structure as shown in fig. 17.

< 7. calculation of conversion information >

As described above, the conversion information calculation unit 115 calculates conversion information for converting the coordinate values of the control axis into the position and/or orientation of the virtual object in the camera coordinate system. The method of calculating the conversion information will be described in detail with reference to fig. 18 to 19.

For example, as shown in FIG. 18, at axis x₁Upper setting axis x₂Axis x₂Upper setting axis x₃N nodes are similarly connected with the end of the axis x_N. In addition, on the axis x_NThe upper control point displays a virtual object. Likewise, on the axis y₁Upper setting axis y₂On the axis y₂Upper setting axis y₃L nodes are similarly connected with the end of the axis y_L. In addition, on the axis y_LIs provided with a camera. Here, x_i，y_jThe node name indicates the coordinate value of each node.

In addition, the offset, the node type (straight line/rotation/unit/control point/coordinate system), the axis direction, the orientation matrix, and the coordinate value shown in fig. 18 are given to each node.

At this time, as shown in fig. 19, a homogeneous matrix M representing the current position and posture of the virtual object at the control point with respect to the root (mechanical origin)_objThe following equation was used.

[ number 3]

Wherein the content of the first and second substances,

Wherein the meanings of the symbols are as follows.

S_xi: a homogeneous transformation matrix of each node;

N: the number of a series of nodes connected from the root to the control point of the mechanical structure tree;

M_ctrl: the homogeneous matrix of the relative shift and posture of the control point relative to the father node is formed by the number 2 of the deviation vector and the posture matrix defined by the control point]And (4) defining.

Homogeneous transformation matrix S_xiThe number of nodes varies depending on the type of the node, and the case of a linear axis is shown below, for example.

[ number 4]

Wherein the meanings of the symbols are as follows.

x_i: node x_iSeat ofMarking a value;

ofs_xi: node x_iA relative offset vector with respect to a parent node;

v_xi: node x_iThe direction vector of movement of.

The case of the rotating shaft is shown below.

[ number 5]

wherein the meanings of the symbols are as follows.

v₁: node x_iA first component of the rotation axis direction vector of (a);

v₂: node x_ia second component of the rotation axis direction vector of (a);

v₃: node x_iA third component of the rotation axis direction vector of (1).

at this time, a homogeneous matrix X representing the current position and posture of the virtual object at the control point on the camera coordinate system_cUsing M_objThe following equation was used.

[ number 6]

Wherein the content of the first and second substances,

Wherein the meanings of the symbols are as follows.

L: the number of a series of nodes connected from the root to the camera node of the mechanical structure tree;

M_coord: homogeneous matrix of relative shift and posture of camera relative to father node according to coordinate systemthe defined offset vector and attitude matrix are composed of the above-mentioned [ number 2]]The formula of (c) is defined.

< 8. method for displaying virtual object >

Fig. 20 shows an operation flow when the virtual object is displayed. First, an outline of each step is described.

In step S41, the machine structure management device 100 adds a camera node to the machine structure tree as a new node. Thus, for example, when the structure of the machine tool 400 is as shown in fig. 21A, a camera node is added to a machine structure tree representing the structure of the machine tool 400 as shown in fig. 21B.

In step S42, the enriched information control device 200 acquires a node that can be displayed from the mechanical configuration management device 100.

In step S43, the augmentation information control apparatus 200 selects a node on which augmentation information such as a virtual object is displayed, and notifies the mechanical structure management apparatus 100 of the node.

in step S44, the machine structure management apparatus 100 derives an equation for determining the position and orientation of the node to be displayed in the camera coordinate system, and notifies the enhancement information control apparatus 200 of the equation.

In step S45, the enhanced information control device 200 acquires the coordinate values of the control axes from the numerical controller 150.

in step S46, the enhanced information control device 200 obtains the position and orientation of the node displaying the enhanced information of the virtual object or the like in the camera coordinate system using the acquired coordinate values and the equation for obtaining the position and orientation in the camera coordinate system.

In step S47, the enhancement information control device 200 obtains the position and orientation in the screen coordinate system by conversion of the coordinate system from the position and orientation in the camera coordinate system.

In step S48, the enhancement information control device 200 generates enhancement information display data, which is display data of enhancement information such as a virtual object, and outputs the data to the display device 300.

In step S49, the display device 300 displays enhancement information of the virtual object or the like. Then, the process returns to step S45.

the position and orientation of the node displaying the enhancement information such as the virtual object in the camera coordinate system may be obtained by the mechanical structure management apparatus 100, and the value indicating the position and orientation may be notified to the enhancement information control apparatus 200.

In step S47, the enhancement information control device 200 determines the position and orientation in the screen coordinate system from the position and orientation in the camera coordinate system by a known method. More specifically, a process of obtaining conversion information for converting the camera coordinate system into the screen coordinate system is performed, and the position and the posture of the virtual object are converted from the camera coordinate system into the values of the screen coordinate system based on the conversion information. The enhanced information display data includes data such as the shape of the virtual object in addition to the positions and orientations of the camera coordinate system and the screen coordinate system obtained here. The enhanced information display data is output to the display device 300, and the display device 300 can draw a virtual object at an appropriate position on the screen by using the output data.

according to the above method, a virtual object can be displayed for an arbitrary node on a display. As illustrated in fig. 22A, the present invention can also be used for a machining simulation in which a 3D model of a workpiece or a tool is placed as a virtual object. Alternatively, data related to nodes such as the rotational speeds of the spindles may be displayed. Further, by arranging an acceleration sensor and a temperature sensor and adding a sensor node to the mechanical structure tree as shown in fig. 22B, the acceleration and temperature at each location can be visually confirmed.

This enables the operator of the machine tool 400 to check the data related to the node and the numerical value of the sensor during the simulation operation.

< 9. Effect of the present embodiment

According to this embodiment, virtual objects can be arranged for all nodes registered in a graph. In particular, by adding a camera node, a virtual object can be added at an arbitrary position without separately registering a conversion equation of a movement amount or separately recognizing a unique point. In addition, identification of the initial position is not required.

< 10. modification

Although some of the graph data may be stored in the mechanical structure management apparatus 100, the graph data is not limited thereto. For example, the information may be stored in the enhanced information control apparatus 200, or may be stored in a server connected to the virtual object display system 10 via a network. The present invention can be applied to an old machine tool by storing the machine structure tree data in the enhanced information control device 200 or the server. In this case, the processing before step S43 is not executed, and step S44 is executed on the side of the enhanced information control apparatus 200.

The mechanical structure management device 100 may be incorporated into the numerical controller 150. Alternatively, the mechanical structure management apparatus 100 may exist on the cloud.

The machine configuration management device, the numerical controller, the enhancement information controller, and the machine tool described above can be realized by hardware, software, or a combination thereof. The simulation method performed by the cooperation of the machine configuration management device, the numerical controller, the enhancement information controller, and the machine tool can be realized by hardware, software, or a combination thereof. Here, the software implementation means that the computer reads and executes a program.

The program can be stored and provided to a computer using various types of non-transitory computer readable media. The non-transitory computer readable medium includes various types of tangible storage media. Examples of non-transitory computer readable media include: magnetic recording media (e.g., floppy disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only memories), CD-R, CD-R/W, semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (random access memories)). In addition, the program may also be provided to the computer through various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The computer-readable medium in the transient state can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.