阅读说明：本技术 基于增强现实观测设备内部结构的方法 (Method for observing internal structure of equipment based on augmented reality ) 是由 沈海超 于 2021-05-27 设计创作，主要内容包括：本发明实施例公开了一种基于增强现实观测设备内部结构的方法,其中,应用于终端设备的方法包括：通过所述终端设备的摄像头获取实景并进行平面检测,确定目标平面,在所述目标平面放置虚拟设备；当观测位置进入虚拟设备内部后,根据用户对当前检测画面的操作,确定当前虚拟元件节点,所述当前虚拟元件节点对应所述当前检测画面内所述虚拟设备中的虚拟元件；获取所述当前虚拟元件节点对应的虚拟元件信息,显示所述虚拟元件信息,可以对虚拟设备内部结构进行观测,获得虚拟元件信息,提升增强现实场景的互动效果。(The embodiment of the invention discloses a method for observing the internal structure of equipment based on augmented reality, wherein the method applied to terminal equipment comprises the following steps: acquiring a real scene through a camera of the terminal equipment, carrying out plane detection, determining a target plane, and placing virtual equipment on the target plane; after the observation position enters the interior of the virtual equipment, determining a current virtual element node according to the operation of a user on a current detection picture, wherein the current virtual element node corresponds to a virtual element in the virtual equipment in the current detection picture; and acquiring virtual element information corresponding to the current virtual element node, displaying the virtual element information, observing the internal structure of the virtual equipment, acquiring the virtual element information, and improving the interaction effect of the augmented reality scene.)

1. A method for observing the internal structure of equipment based on augmented reality is applied to terminal equipment and is characterized in that the method comprises the following steps:

acquiring a real scene through a camera of the terminal equipment, carrying out plane detection, determining a target plane, and placing virtual equipment on the target plane;

after the observation position enters the interior of the virtual equipment, determining a current virtual element node according to the operation of a user on a current detection picture, wherein the current virtual element node corresponds to a virtual element in the virtual equipment in the current detection picture;

and acquiring virtual element information corresponding to the current virtual element node, and displaying the virtual element information.

2. The method of claim 1, further comprising:

when the observation position is determined to enter the virtual equipment, adjusting the observation position to a target initial observation position;

the target initial observation positions are: the observation position is located on the inner surface of the outermost wall of the virtual equipment, and the observation visual angle corresponding to the observation position faces the inside of the virtual equipment.

3. The method of claim 2, wherein determining the observation location to enter the virtual device comprises:

acquiring the distance between the observation position of the camera and the center of the virtual equipment;

when the distance between the observation position where the camera is located and the center of the virtual equipment is smaller than a preset distance, the observation position is determined to enter the virtual equipment, the preset distance is larger than or equal to the maximum distance between the center of the virtual equipment and the outer surface of the virtual equipment, and the difference value between the preset distance and the maximum distance is smaller than a preset threshold value.

4. The method according to any one of claims 1 to 3, wherein the determining a current virtual element node according to the operation of the user on the current detection screen comprises:

triggering hit test processing when detecting that a preset touch operation occurs in the current detection picture, and obtaining a hit result, wherein the hit result comprises a virtual element node corresponding to the current detection picture;

if the virtual element node corresponding to the current detection picture is one, determining the virtual element node corresponding to the current detection picture as the current virtual element node;

and if the number of the virtual element nodes corresponding to the current detection picture is multiple, determining the selected virtual element node as the current virtual element node according to the selection operation of the user on the multiple virtual element nodes.

5. The method of claim 4, wherein the hit test processing comprises:

and emitting a detection ray from the camera head, and acquiring a virtual element node intersected with the detection ray.

6. The method of claim 5, further comprising:

acquiring a mark point of a virtual element node determined by the detection ray, wherein the mark point is a point of a first-time passing position on the virtual element when the detection ray passes through the virtual element;

if the number of virtual element nodes corresponding to the current detection picture is multiple, the mark points comprise multiple mark points corresponding to the multiple virtual elements, and the multiple mark points are sequenced according to the sequence that the detection ray passes through the virtual elements;

the determining, according to a selection operation of the user on the plurality of virtual element nodes, that one of the plurality of virtual element nodes is the current virtual element node includes:

when the click operation of a user on the current detection picture is detected, determining a target mark point corresponding to the click operation according to the number of the click operation and the sequence of the mark points;

and determining the virtual element node corresponding to the target marking point as the current virtual element node.

7. The method of claim 1, further comprising:

displaying a position map window in the current detection picture, wherein the position map window is used for displaying a global reference model, and the global reference model is marked with the observation position of the terminal equipment in the virtual equipment;

and under the condition that a section position acquisition instruction is detected, displaying a section view of the global reference model in the position map window, wherein the section view is determined by a cutting reference plane which is vertical to the terminal equipment screen.

8. The method of claim 7, further comprising:

when simultaneous touch operation on two positions of a screen of the terminal equipment is detected, acquiring the duration of the simultaneous touch operation;

triggering the section position acquisition instruction when the duration is not less than the preset duration;

and under the condition that the section position acquisition instruction is detected, determining the interception reference plane according to a connecting line between the two positions.

9. The method of claim 2 or 3, wherein the adjusting the observation position to a target initial observation position comprises:

acquiring an equipment adjustment reference line, wherein the equipment adjustment reference line is a connection line between an observation position where the camera is located and the center of the virtual equipment;

and adjusting the position of the virtual equipment along the equipment adjustment reference line to enable the observation position to be the target initial observation position.

10. A terminal device, comprising:

the plane detection module is used for acquiring a real scene through a camera of the terminal equipment, performing plane detection, determining a target plane, and placing virtual equipment on the target plane;

the selection module is used for determining a current virtual element node according to the operation of a user on a current detection picture after an observation position enters the interior of virtual equipment, wherein the current virtual element node corresponds to a virtual element in the virtual equipment in the current detection picture;

and the information display module is used for acquiring the virtual element information corresponding to the current virtual element node and displaying the virtual element information.

11. A terminal device comprising a processor and a memory, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the knowledge-graph construction method according to any one of claims 1 to 9.

12. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, causes the processor to carry out the steps of the method according to any one of claims 1-9.

Technical Field

The application relates to the technical field of augmented reality, in particular to a method for observing the internal structure of equipment based on augmented reality.

Background

The Augmented Reality (AR) technology is a technology that virtual information and the real world are skillfully fused, and a plurality of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, sensing and the like are widely applied, so that virtual information such as characters, images, three-dimensional models, music, videos and the like generated by a computer are applied to the real world after analog simulation, and the two kinds of information complement each other, thereby realizing the 'enhancement' of the real world.

Currently, in a scene of actually applying augmented reality, a user can operate a virtual model through simple gestures, such as zooming, rotating and the like, for example, perform interactive operations such as translation, zooming, placing and the like on a virtual home, but such operations are simple, interactive interest is low, and more experience and information about products cannot be provided for the user.

Disclosure of Invention

The invention mainly aims to provide a method for observing the internal structure of equipment based on augmented reality, which can solve the problems that the interaction of augmented reality scenes in a general scheme is simple and the internal of virtual equipment cannot be observed.

In one aspect, the application provides a method for observing an internal structure of a device based on augmented reality, which is applied to a terminal device, and the method comprises the following steps:

acquiring a real scene through a camera of the terminal equipment, carrying out plane detection, determining a target plane, and placing virtual equipment on the target plane;

after the observation position enters the interior of the virtual equipment, determining a current virtual element node according to the operation of a user on a current detection picture, wherein the current virtual element node corresponds to a virtual element in the virtual equipment in the current detection picture;

and acquiring virtual element information corresponding to the current virtual element node, and displaying the virtual element information.

Optionally, the method further includes:

when the observation position is determined to enter the virtual equipment, adjusting the observation position to a target initial observation position;

the target initial observation positions are: the observation position is located on the inner surface of the outermost wall of the virtual equipment, and the observation visual angle corresponding to the observation position faces the inside of the virtual equipment.

Optionally, the determining the observation position to enter the virtual device includes:

acquiring the distance between the camera and the center of the virtual equipment;

when the distance between the observation position where the camera is located and the center of the virtual equipment is smaller than a preset distance, the observation position is determined to enter the virtual equipment, the preset distance is larger than or equal to the maximum distance between the center of the virtual equipment and the outer surface of the virtual equipment, and the difference value between the preset distance and the maximum distance is smaller than a preset threshold value.

Optionally, the determining a current virtual element node according to the operation of the user on the current detection screen includes:

triggering hit test processing when detecting that a preset touch operation occurs in the current detection picture, and obtaining a hit result, wherein the hit result comprises a virtual element node corresponding to the current detection picture;

if the virtual element node corresponding to the current detection picture is one, determining the virtual element node corresponding to the current detection picture as the current virtual element node;

and if the number of the virtual element nodes corresponding to the current detection picture is multiple, determining the selected virtual element node as the current virtual element node according to the selection operation of the user on the multiple virtual element nodes.

Optionally, the hit test processing includes:

and emitting a detection ray from the camera head, and acquiring a virtual element node intersected with the detection ray.

Optionally, the method further includes:

acquiring a mark point of a virtual element node determined by the detection ray, wherein the mark point is a point of a first-time passing position on the virtual element when the detection ray passes through the virtual element;

if the number of virtual element nodes corresponding to the current detection picture is multiple, the mark points comprise multiple mark points corresponding to the multiple virtual elements, and the multiple mark points are sequenced according to the sequence that the detection ray passes through the virtual elements;

the determining, according to a selection operation of the user on the plurality of virtual element nodes, that one of the plurality of virtual element nodes is the current virtual element node includes:

when the click operation of a user on the current detection picture is detected, determining a target mark point corresponding to the click operation according to the number of the click operation and the sequence of the mark points;

and determining the virtual element node corresponding to the target marking point as the current virtual element node.

Optionally, the method further includes:

displaying a position map window in the current detection picture, wherein the position map window is used for displaying a global reference model, and the global reference model is marked with the observation position of the terminal equipment in the virtual equipment;

and under the condition that a section position acquisition instruction is detected, displaying a section view of the global reference model in the position map window, wherein the section view is determined by a cutting reference plane which is vertical to the terminal equipment screen.

Optionally, the method further includes:

when simultaneous touch operation on two positions of a screen of the terminal equipment is detected, acquiring the duration of the simultaneous touch operation;

triggering the section position acquisition instruction when the duration is not less than the preset duration;

and under the condition that the section position acquisition instruction is detected, determining the interception reference plane according to a connecting line between the two positions.

Optionally, the adjusting the position of the virtual device includes:

acquiring an equipment adjustment reference line, wherein the equipment adjustment reference line is a connection line between an observation position where the camera is located and the center of the virtual equipment;

and adjusting the position of the virtual equipment along the equipment adjustment reference line to enable the observation position to be the target initial observation position.

In another aspect, a terminal device is provided, including:

the plane detection module is used for acquiring a real scene through a camera of the terminal equipment, performing plane detection, determining a target plane, and placing virtual equipment on the target plane;

the selection module is used for determining a current virtual element node according to the operation of a user on a current detection picture after an observation position enters the interior of virtual equipment, wherein the current virtual element node corresponds to a virtual element in the virtual equipment in the current detection picture;

and the information display module is used for acquiring the virtual element information corresponding to the current virtual element node and displaying the virtual element information.

In another aspect, another terminal device is provided, comprising a processor and a memory, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the method of any of the above aspects and any possible implementation thereof.

In another aspect, a computer storage medium is provided, which stores one or more instructions adapted to be loaded by a processor and to perform the steps of the method of any of the above aspects and any possible implementation thereof.

The invention provides a method for observing the internal structure of equipment based on augmented reality, which comprises the steps of acquiring a real scene through a camera of terminal equipment, carrying out plane detection, determining a target plane, and placing virtual equipment on the target plane; after the observation position enters the interior of the virtual equipment, determining a current virtual element node according to the operation of a user on a current detection picture, wherein the current virtual element node corresponds to a virtual element in the virtual equipment in the current detection picture; the virtual element information corresponding to the current virtual element node is acquired, the virtual element information is displayed, the internal structure of the virtual equipment can be observed, the virtual element information is acquired, and the interaction effect of the augmented reality scene is improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Wherein:

fig. 1 is a schematic flowchart of a method for observing an internal structure of a device based on augmented reality according to an embodiment of the present application;

fig. 2 is a schematic view of a scene where an observation position enters the inside of a virtual device according to an embodiment of the present disclosure;

FIG. 3 is a schematic cross-sectional view of a screen according to an embodiment of the present disclosure;

fig. 4 is a schematic flowchart of another method for observing an internal structure of a device based on augmented reality according to an embodiment of the present application;

FIG. 5 is a schematic diagram illustrating a virtual device node hit test according to an embodiment of the present disclosure;

fig. 6 is a schematic structural diagram of a terminal device according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of another terminal device provided in the embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

Fig. 1 is a flowchart of a method for observing an internal structure of a device based on augmented reality according to an embodiment of the present application, where the method shown in fig. 1 is applicable to a terminal device, and the method includes:

101. and acquiring a real scene through a camera of the terminal equipment, carrying out plane detection, determining a target plane, and placing virtual equipment on the target plane.

The terminal device mentioned in the embodiments of the present application may be a mobile terminal device, including but not limited to a mobile terminal device such as a mobile phone, a tablet computer, a wearable device or other portable device having a touch sensitive surface (e.g., a touch screen display and/or a touch pad).

Augmented reality technology can identify a horizontal plane in the real world through plane detection, and only when the horizontal plane exists, the virtual device can be placed on a certain anchor point to be displayed. Specifically, in the embodiment of the present application, an augmented reality development library or model may be selected as needed, for example, an AR mobile terminal development library ARKit provided by an IOS system is adopted. The ARKit mentioned in the embodiment of the application is an AR development platform. Developers can use the tool set to create augmented reality applications on terminal devices.

Specifically, the AR scene implementation using the ARKit includes the following steps:

ARSCNView: loading a scene SCNScene;

SCNScene: starting a camera ARCamera to capture a scene;

ARSCNView: processing the scene data by Session;

arssessionconfiguration: tracking the camera movement and returning an ARFrame;

adding a virtual device node in an ARSCNView scene (scene);

wherein the position of the virtual device is also ensured by device tracking independent of the movements of the terminal device, device tracking in Six degrees of freedom (6 DoF) can be provided mainly by ARWorldTrackingConfiguration, including three attitude angle Yaw (Yaw), Pitch (Pitch) and Roll (Roll), and offsets along the X, Y and Z three axes in a cartesian coordinate system.

The embodiments of the present application may be implemented in combination with other augmented reality models or algorithms, which are not limited herein.

After the virtual device is placed, the virtual device is displayed on a screen of the terminal device, and a user can move the terminal device to observe the virtual device and can interact with the virtual device through various touch operations on the terminal device.

102. And after the observation position enters the inside of the virtual equipment, determining a current virtual element node according to the operation of a user on a current detection picture, wherein the current virtual element node corresponds to a virtual element in the virtual equipment in the current detection picture.

In the embodiment of the application, the terminal device can be close to the virtual device, the observation position corresponding to the terminal device changes along with the movement of the terminal device, and different observation positions can correspond to different viewing angles. When the observation position is close to a certain degree, the observation position can be considered to enter the virtual equipment, the visual angle entering the virtual equipment is displayed on the terminal equipment, and the internal structure of the virtual equipment can be observed and interacted.

In this embodiment of the application, after the observation position enters the inside of the virtual device, the user may observe the inside of the virtual device, and may interact with the virtual element inside the virtual device, and specifically, may perform various operations on the current detection picture on the terminal device to determine the current virtual element node, for example, perform a click operation, a press operation, or a slide operation on the screen to determine the virtual element in the current interaction. Instructions corresponding to different operations can be predefined according to needs to implement different interaction functions, which is not limited in the embodiments of the present application.

In an alternative embodiment, the method further comprises:

when the observation position is determined to enter the virtual equipment, adjusting the observation position to a target initial observation position;

the target initial observation positions are as follows: the observation position is located on the inner surface of the outermost wall of the virtual device, and the observation angle corresponding to the observation position faces the inside of the virtual device.

When the observation position is determined to enter the virtual equipment, the observation position can be adjusted in time, so that the observation position is located at the target initial observation position. Adjusting the position may be understood primarily as adjusting the relative position of the virtual device and the terminal device. Specifically, the observation position where the camera is located can be adjusted to be located on the inner surface of the outermost wall of the virtual device, and the observation angle faces towards the inside of the virtual device, so that the initial observation position can be fixed, tracking can be facilitated, and the size of the internal space of the virtual device can be measured.

In an optional implementation manner, the determining the observation position to enter the inside of the virtual device includes:

acquiring the distance between the observation position of the camera and the center of the virtual equipment;

when the distance between the observation position of the camera and the center of the virtual device is smaller than a preset distance, determining that the observation position enters the virtual device, wherein the preset distance is larger than or equal to the maximum distance between the center of the virtual device and the outer surface of the virtual device, and the difference between the preset distance and the maximum distance is smaller than a preset threshold.

Whether the camera enters the virtual equipment is determined by judging the distance between the observation position where the camera is located and the center of the virtual equipment.

Fig. 2 is a schematic view of a scene in which an observation position enters the virtual device, as shown in fig. 2, a preset distance is L, a terminal device can obtain a distance d between the observation position where a camera is located and the center of the virtual device, and the observation position where the camera is located can be determined by a position corresponding to the lens center of the camera. Specifically, at S1, when the distance d is greater than or equal to L, the current observation position may be considered to be outside the virtual device; at S2, if the distance d is less than L, the current observation position is considered to enter the inside of the virtual device, and optionally, the user may be prompted on the screen of the terminal device that the virtual device has entered the inside of the virtual device, and then the view angle inside the virtual device may be displayed. The preset distance L may be generally set to be equal to or slightly greater than a maximum distance m from the center of the virtual device to the outer surface thereof, that is, a difference between the preset distance L and the maximum distance m is smaller than a preset threshold a, a may be set to a smaller value as needed, and if the difference is larger, it indicates that the observation position is far away from the virtual device, and then the observation position is triggered to enter the internal scene. By the method, the scene experience can be closer to the state change of the user entering the device. In an alternative embodiment, the position adjustment method includes:

acquiring an equipment adjustment reference line, wherein the equipment adjustment reference line is a connection line between an observation position where the camera is located and the center of the virtual equipment;

and adjusting the position of the virtual device along the device adjustment reference line to enable the observation position to be the target initial observation position.

When the distance d is smaller than the preset distance L, an equipment adjustment reference line can be defined, the reference line is a connection line between the observation position where the camera is located and the center of the virtual equipment, the position of the virtual equipment is further automatically adjusted along the reference line, the observation position is located on the inner surface of the outermost wall of the virtual equipment, and the observation visual angle corresponding to the observation position faces towards the inside of the equipment.

In an alternative embodiment, when the distance d is greater than L, the observed location may be considered to be outside the virtual device; at S2, when the distance d gradually decreases to equal to L, the position relationship between the observation position and the virtual device is automatically adjusted, so that the observation position is located on the inner surface of the wall of the virtual device, which is located at the outer side of the virtual device, and the screen of the terminal device is prompted that the user has entered the virtual device.

103. And acquiring virtual element information corresponding to the current virtual element node, and displaying the virtual element information.

In the embodiment of the application, the virtual element information in the virtual equipment can be displayed. Specifically, for the selected current virtual element node, the virtual element information corresponding to the current virtual element node may be acquired from the database and displayed.

Each virtual element node established in advance may set corresponding appearance attributes and virtual element information, for example, in a SceneKit library, a SCNNode is used to manage a 3D object, and the SCNNode has a geometry attribute to manage the appearance of the object. At the selected current virtual component node, corresponding virtual component information may be obtained through the appearance attribute thereof, where the virtual component information may include a component name, a component model, a production date, a component color manufacturer, a specification, a brief introduction, and the like, which is not limited in this embodiment of the application. As shown in table 1, a virtual component information table provided in this embodiment of the present application may be displayed on a screen of a terminal device.

TABLE 1

In an optional embodiment, the method further comprises:

104. displaying a position map window in the current detection picture, wherein the position map window is used for displaying a global reference model, and the global reference model is marked with an observation position of the terminal equipment in the virtual equipment;

105. and under the condition that a section position acquisition instruction is detected, displaying a section view of the global reference model on the position map window, wherein the section view is determined by a section reference plane which is vertical to the terminal equipment screen.

Specifically, a small position map window may be displayed in the current detection picture to show a complete AR virtual reference model, which may be a 3D model, where a position of the terminal device inside the virtual object is displayed, so that the user may know the observation position of the terminal device. Optionally, the position map window may be displayed in a corner of the screen, such as an upper left corner, so as not to obscure the main viewing angle of the current detection picture.

Further, a cross-sectional view of the global reference model may be selected for display. Through the section position acquisition instruction, the current interception reference plane can be determined, so that the currently displayed global reference model is intercepted, and the current section view is obtained.

The above-mentioned section position acquiring instruction may be triggered by a preset gesture action, such as pressing the screen by two fingers at the same time.

Optionally, the method further includes:

201. when simultaneous touch operation on two positions of the screen of the terminal equipment is detected, acquiring the duration of the simultaneous touch operation;

202. triggering the section position acquisition instruction when the duration is not less than a preset duration;

203. and under the condition that the section position acquisition command is detected, determining the interception reference plane according to a connecting line between the two positions.

The preset time length may be set as required, for example, the preset time length is 3 s.

Fig. 3 is a schematic screen diagram of a cross-sectional state provided in the present application, which is taken by way of example with reference to fig. 3, where an outer frame of fig. 3 is a screen boundary, and a position map window 30 at an upper left corner of the screen displays a stereoscopic global reference model, where the global reference model includes a model virtual device 31 and a position point 32 where a virtual terminal device camera is located, where the model virtual device 31 may be in an arbitrary shape, and a rectangle in the drawing is only used as a schematic diagram. The virtual device 33 is provided outside the position map window 30, and the model virtual device 31 is a reference model of the virtual device 33. The user presses two fingers for the same time on the screen 3s or more (position 34 in the figure), the corresponding cross-sectional view can be displayed on the displayed global reference model. Specifically, the terminal device may determine a connection line between two positions pressed by the two fingers (specifically, a connection line between centers of the two positions pressed by the two fingers), determine a plane passing through the connection line and perpendicular to the screen of the terminal device as the above-mentioned clipping reference plane, and determine the cross-sectional view through the clipping reference plane. And, optionally, in conjunction with sliding the screen up, down, left, and right, the orientation of the cross-sectional view may be adjusted, such as may be facilitated by the use of an adjustment aid shown in fig. 3, including an orientation arrow, to better control the cross-sectional selection. Optionally, after the cross-section operation is completed, the global reference model may be automatically rotated to align the cross-section with the screen, i.e., to display a complete cross-section view. Alternatively, the cross-sectional state may be exited when the global reference model is clicked again. In this way, the observed position of the user can be displayed more clearly from the macro.

The method comprises the steps of obtaining a real scene through a camera of a terminal device, carrying out plane detection, determining a target plane, and placing virtual equipment on the target plane; after the observation position enters the interior of the virtual equipment, determining a current virtual element node according to the operation of a user on a current detection picture, wherein the current virtual element node corresponds to a virtual element in the virtual equipment in the current detection picture; the virtual element information corresponding to the current virtual element node is acquired, the virtual element information is displayed, the internal structure of the virtual equipment can be observed, the virtual element information is acquired, and the interaction effect of the augmented reality scene is improved. When the observation position is determined to enter the virtual equipment, the position can be adjusted, so that the observation position is fixed relative to the initial position of the virtual equipment, and observation is more convenient.

Further, fig. 4 is a schematic flowchart of another method based on an internal structure of an augmented reality observation device according to an embodiment of the present application. As shown in fig. 4, the method includes:

401. and acquiring a real scene through a camera of the terminal equipment, carrying out plane detection, determining a target plane, and placing virtual equipment on the target plane.

402. When an observation position is determined to enter the virtual device, the observation position is adjusted to a target initial observation position, and the target initial observation position is: the observation position is located on the inner surface of the outermost wall of the virtual device, and the observation angle corresponding to the observation position faces the inside of the virtual device.

The step 401 and the step 402 may refer to the detailed descriptions in the step 101 and the step 102 in the embodiment shown in fig. 1, and are not described herein again.

403. And when detecting that the preset touch operation occurs in the current detection picture, emitting a detection ray from the camera head, and acquiring a virtual element node intersected with the detection ray.

In the embodiment of the application, the preset touch operation can be set as required to trigger selection of the virtual element node, for example, the preset touch operation can be a finger clicking a screen. In the embodiment of the application, when it is detected that a preset touch operation occurs in a current detection picture, hit test (hit) processing is triggered to obtain a hit result, which includes a virtual element node corresponding to the current detection picture.

404. And acquiring a plurality of mark points corresponding to a plurality of virtual elements determined by the detection rays, wherein the mark points are sequenced according to the sequence that the detection rays pass through the virtual elements.

If a virtual component is determined by a hit test, the virtual component information can be directly displayed. In the embodiment of the application, the hit test can be triggered through one operation, and a plurality of virtual elements are detected simultaneously.

If there are multiple virtual element nodes corresponding to the current detection picture, the selected virtual element node can be determined as the current virtual element node for interaction according to the selection operation of the user on the virtual element node. Optionally, the plurality of virtual element nodes may be displayed first and then selected by the user, or the current virtual element node may be directly selected and switched through a preset touch operation triggered by the user without being displayed first.

Fig. 5 is a schematic diagram illustrating a virtual device node hit test provided in the present application. For example, a finger may be set to click on the screen, and when the screen is pressed for a preset time, the hit test is triggered to be executed once. When a hit test is triggered, a detection ray along the optical axis is emitted from the center of the origin of the camera, and the ray possibly intersects with a plurality of virtual element nodes, and a set of hit results is obtained: SCNHitTestResult, as shown in fig. 5, the detection ray emitted from the center of the origin of the camera can be displayed on the screen as emitted from the screen click point D, and when the detection ray hits 3 virtual element nodes 1, 2, and 3, the hit result is 3.

The detection ray passing through each virtual element can display a marking point at the position of the virtual element passing through for the first time. In the case where the hit includes a plurality of virtual element nodes, a corresponding plurality of marker points is obtained.

405. And when the click operation of the user on the current detection picture is detected, determining the target mark point corresponding to the click operation according to the number of the click operation and the sequence of the mark points.

The plurality of marker points may be ordered in the order in which the detection rays pass through the virtual element. For example, the detection ray sequentially passes through three virtual element nodes, and the three marking points can be respectively marked as a first marking point, a second marking point and a third marking point.

Furthermore, according to the sorting of the mark points and the number of times of the clicking operation, the user can select the virtual element to be observed through the clicking operation and view the information of the virtual element.

406. And determining the virtual element node corresponding to the target marking point as the current virtual element node.

407. And acquiring virtual element information corresponding to the current virtual element node, and displaying the virtual element information.

Specifically, the user clicks the screen once, the target mark point is a first mark point, the virtual element information corresponding to the first mark point is displayed, the screen is clicked twice, the target mark point is a second mark point, and the virtual element information corresponding to the second mark point is displayed; clicking the screen three times, wherein the target mark point is a third mark point, displaying the virtual element information corresponding to the third mark point, and so on.

For example, for one result in the obtained hit result array, that is, the current virtual component node is selected, the information showing the virtual component node may be implemented by the following steps: the hit result SCNHitTestResult includes an SCNNode attribute, and in the SceneKit library, the SCNNode is used as an appearance attribute to manage the appearance of the object, so that what element the currently selected node is can be determined according to the appearance attribute of the current virtual element node, and thus, related element information can be acquired and displayed, as described in table 1, which is not described herein again.

Optionally, when displaying the selected virtual element, the edge of the selected virtual element may be blackened or the entire virtual element may be highlighted, which is not limited in this embodiment of the application.

Further alternatively, the highlighted virtual element may be individually enlarged and the entire virtual device may be controlled to be in a transparent state, while the rotation display of the highlighted virtual element may be controlled by a preset gesture action, such as a gesture of drawing a circle clockwise on the screen. By the method, a user can observe the virtual element which is displayed at present and can observe the three-dimensional shape of the whole virtual element more conveniently.

Optionally, through preset gesture actions, such as sliding operations on a screen, the selected virtual element can be moved according to the sliding operations, so that each virtual element can be detached, the structure can be displayed more clearly, and observation is more convenient.

Based on the description of the embodiment of the method for observing the internal structure of the equipment based on the augmented reality, the embodiment of the application further discloses the terminal equipment. Referring to fig. 6, the terminal apparatus 600 includes:

a plane detection module 610, configured to obtain a real scene through a camera of the terminal device, perform plane detection, determine a target plane, and place a virtual device on the target plane;

a selecting module 620, configured to determine a current virtual element node according to an operation of a user on a current detection screen after an observation position enters a virtual device, where the current virtual element node corresponds to a virtual element in the virtual device in the current detection screen;

an information display module 630, configured to obtain the virtual component information corresponding to the current virtual component node, and display the virtual component information.

Optionally, the terminal device 600 further includes an adjusting module 640, configured to: when the observation position is determined to enter the virtual equipment, adjusting the observation position to a target initial observation position;

the target initial observation positions are as follows: the observation position is located on the inner surface of the outermost wall of the virtual device, and the observation angle corresponding to the observation position faces the inside of the virtual device.

Optionally, the adjusting module 640 is specifically configured to:

acquiring the distance between the observation position of the camera and the center of the virtual equipment;

when the distance between the observation position of the camera and the center of the virtual device is smaller than a preset distance, determining that the observation position enters the virtual device, wherein the preset distance is larger than or equal to the maximum distance between the center of the virtual device and the outer surface of the virtual device, and the difference between the preset distance and the maximum distance is smaller than a preset threshold.

Optionally, the selecting module 620 is specifically configured to:

triggering hit test processing when detecting that a preset touch operation occurs in the current detection picture to obtain a hit result, wherein the hit result comprises a virtual element node corresponding to the current detection picture;

if the virtual element node corresponding to the current detection picture is one, determining the virtual element node corresponding to the current detection picture as the current virtual element node;

and if the number of the virtual element nodes corresponding to the current detection picture is multiple, determining the selected virtual element node as the current virtual element node according to the selection operation of the user on the multiple virtual element nodes.

Optionally, the hit test processing includes:

a detection ray is emitted from the camera head, and a virtual element node intersecting the detection ray is acquired.

Optionally, the selecting module 620 is further configured to:

acquiring a mark point of a virtual element node determined by the detection ray, wherein the mark point is a point of a first passing position on the virtual element when the detection ray passes through the virtual element;

if the number of virtual element nodes corresponding to the current detection picture is multiple, the mark points comprise multiple mark points corresponding to the multiple virtual elements, and the multiple mark points are sequenced according to the sequence that the detection ray passes through the virtual elements;

when the click operation of a user on the current detection picture is detected, determining a target mark point corresponding to the click operation according to the number of the click operation and the sequence of the mark points;

and determining the virtual element node corresponding to the target marking point as the current virtual element node.

Optionally, the terminal device 600 further includes a map module 650, configured to:

displaying a position map window in the current detection picture, wherein the position map window is used for displaying a global reference model, and the global reference model is marked with an observation position of the terminal equipment in the virtual equipment;

and under the condition that a section position acquisition instruction is detected, displaying a section view of the global reference model on the position map window, wherein the section view is determined by a section reference plane which is vertical to the terminal equipment screen.

Optionally, the map module 650 is further configured to:

when simultaneous touch operation on two positions of the screen of the terminal equipment is detected, acquiring the duration of the simultaneous touch operation;

triggering the section position acquisition instruction when the duration is not less than a preset duration;

and under the condition that the section position acquisition command is detected, determining the interception reference plane according to a connecting line between the two positions.

Optionally, the adjusting module 640 is specifically configured to:

acquiring an equipment adjustment reference line, wherein the equipment adjustment reference line is a connection line between an observation position where the camera is located and the center of the virtual equipment;

and adjusting the position of the virtual device along the device adjustment reference line to make the observation position be the target initial observation position.

According to an embodiment of the present application, each step involved in the methods shown in fig. 1 and fig. 4 may be performed by each module in the terminal device 600 shown in fig. 6, and is not described herein again.

Based on the description of the embodiment of the method for observing the internal structure of the equipment based on the augmented reality, the embodiment of the application further discloses the terminal equipment. Fig. 7 is a schematic structural diagram of a terminal device according to an embodiment of the present disclosure, where the terminal device includes at least a processor 701, a memory 702, and an input/output unit 703. The processor 701 may be a Central Processing Unit (CPU), which is a final execution unit for information processing and program operation as an operation and control core of the computer system.

A computer storage medium for storing a computer program comprising program instructions may be stored in the memory 602 of the terminal device 700, and the processor 701 may execute the program instructions stored in the memory 702.

In one embodiment, the terminal device 700 described above in this embodiment of the present application may be configured to perform a series of processes, including the steps of the method in any one of the embodiments shown in fig. 1 and fig. 4, and the like.

An embodiment of the present application further provides a computer storage medium (Memory), where the computer storage medium is a Memory device in an electronic device (terminal device) and is used to store programs and data. It is understood that the computer storage medium herein may include both a built-in storage medium in the electronic device and, of course, an extended storage medium supported by the electronic device. Computer storage media provide storage space that stores an operating system for an electronic device. Also stored in this memory space are one or more instructions, which may be one or more computer programs (including program code), suitable for loading and execution by processor 1001. The computer storage medium may be a high-speed RAM memory, or may be a non-volatile memory (non-volatile memory), such as at least one disk memory; and optionally at least one computer storage medium located remotely from the processor.

In one embodiment, one or more instructions stored in a computer storage medium may be loaded and executed by a processor to perform the corresponding steps in the above embodiments; in particular implementations, one or more instructions in the computer storage medium may be loaded by the processor and executed as each step in the method shown in fig. 1 and/or fig. 4, and so on, and will not be described herein again.

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses and modules may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the division of the module is only one logical division, and other divisions may be possible in actual implementation, for example, a plurality of modules or components may be combined or integrated into another system, or some features may be omitted, or not performed. The shown or discussed mutual coupling, direct coupling or communication connection may be an indirect coupling or communication connection of devices or modules through some interfaces, and may be in an electrical, mechanical or other form.

Modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions according to the embodiments of the present application are wholly or partially generated when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on or transmitted over a computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)), or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more of the available media. The usable medium may be a read-only memory (ROM), or a Random Access Memory (RAM), or a magnetic medium, such as a floppy disk, a hard disk, a magnetic tape, a magnetic disk, or an optical medium, such as a Digital Versatile Disk (DVD), or a semiconductor medium, such as a Solid State Disk (SSD).