阅读说明：本技术 一种相对原点偏移的可视化方法、装置、电子设备及存储介质 (Visualization method and device for relative origin offset, electronic equipment and storage medium ) 是由 王迎智 齐斌 王干 于 2021-11-05 设计创作，主要内容包括：本发明是关于一种相对原点偏移的可视化方法、装置、电子设备及存储介质,包括：获取末端执行机构的柔性通道末端在第一坐标系中的第一实时位置参数；根据第一实时位置参数和柔性通道末端在第一坐标系中的初始位置区域,确定柔性通道末端与初始位置区域的实时相对位置关系,其中,初始位置区域是根据柔性通道末端的出厂前位置确定的,实时相对位置关系包括：柔性通道末端是否在初始位置区域内和柔性通道末端与初始位置区域中心的角度关系；显示实时相对位置关系。本发明通过实时计算偏移参数更新可视化图标,以最直观的效果给到操作者位置器械末端的当前相对位置,将有效改善医生的操作手感,进而提升了手术的安全系数。(The invention relates to a visualization method, a visualization device, electronic equipment and a storage medium for relative origin offset, wherein the visualization method comprises the following steps: acquiring a first real-time position parameter of the tail end of a flexible channel of a tail end executing mechanism in a first coordinate system; determining a real-time relative position relationship between the tail end of the flexible channel and an initial position area according to the first real-time position parameter and the initial position area of the tail end of the flexible channel in a first coordinate system, wherein the initial position area is determined according to a pre-factory position of the tail end of the flexible channel, and the real-time relative position relationship comprises: whether the tail end of the flexible channel is in the initial position area or not and the angle relationship between the tail end of the flexible channel and the center of the initial position area; and displaying the real-time relative position relation. The invention updates the visual icon by calculating the offset parameter in real time, gives the most intuitive effect to the current relative position of the tail end of the instrument at the position of the operator, effectively improves the operation hand feeling of a doctor, and further improves the safety factor of the operation.)

1. A method for visualizing a relative origin offset, the method comprising:

acquiring a first real-time position parameter of the tail end of a flexible channel of a tail end executing mechanism in a first coordinate system;

determining a real-time relative position relationship between the flexible channel tail end and an initial position area according to the first real-time position parameter and the initial position area of the flexible channel tail end in the first coordinate system, wherein the initial position area is determined according to a pre-factory position of the flexible channel tail end, and the real-time relative position relationship comprises: whether the flexible channel tip is within the home position region and the angular relationship of the flexible channel tip to the center of the home position region;

and displaying the real-time relative position relation.

2. The method of claim 1, wherein obtaining the first real-time position parameter of the flexible channel tip of the tip actuator in the first coordinate system comprises:

determining a first real-time position parameter of the tail end of the flexible channel of the tail end actuating mechanism in the first coordinate system according to a real-time detection value of a first magnetic sensing encoder of the base motor; the first magnetic sensing encoder is used for detecting a rotation angle of an output shaft of the base motor, the tail end executing structure comprises a transmission mechanism and a flexible channel, the flexible channel is used for a surgical instrument to pass through, and after the tail end executing mechanism is meshed with the output shaft of the base motor, the output shaft of the base motor is used for driving the transmission mechanism of the tail end executing mechanism during surgery, so that the transmission mechanism drives the flexible channel to move, and the surgical instrument is driven to move.

3. The method of claim 2, wherein the step of obtaining a first real-time position parameter of the flexible channel tip of the tip actuator in a first coordinate system is preceded by the method further comprising:

when the engagement completion of the tail end actuator and the output shaft of the base motor is detected, acquiring a first detection value of a first magnetic sensing encoder and a second detection value of a second magnetic sensing encoder, wherein the second magnetic sensing encoder is used for detecting the rotation angle of a steel wire coil in the transmission mechanism, the steel wire coil is used for tensioning or loosening a steel wire when rotating, and the steel wire is used for controlling the flexible channel to move when being tensioned or loosened;

obtaining an offset position parameter of the tail end of the flexible channel according to a difference value between the second detection value and a pre-factory detection value of the second magnetic sensing encoder, wherein the pre-factory detection value is used for representing a pre-factory position of the tail end of the flexible channel;

and obtaining a position calibration parameter of the tail end of the flexible channel according to the difference value of the first detection value and the offset position parameter.

4. The method of claim 3, wherein determining a first real-time position parameter of the flexible channel end of the end effector in the first coordinate system based on real-time sensed values of a first magnetic sensor encoder of a base motor comprises:

and calculating a difference value between a real-time detection value of a first magnetic sensing encoder of the base motor and the calibration angle parameter, and taking the calculated difference value as the first real-time position parameter of the tail end of the flexible channel in the first coordinate system.

5. The method of claim 1, wherein said determining a real-time relative positional relationship of said flexible tunnel tip to an initial positional region of said flexible tunnel tip in said first coordinate system based on said first real-time positional parameter and said initial positional region comprises:

mapping the first real-time position parameter to a second coordinate system to obtain a second real-time position parameter, wherein the first coordinate system and the second coordinate system are both planar coordinate systems, a plane where the first coordinate system is located and a plane where the second coordinate system is located are parallel to each other, and a coordinate axis of the first coordinate system and a coordinate axis of the second coordinate system form an included angle of 45 degrees;

and comparing the second real-time position parameter with the initial position area, and comparing the second real-time position parameter with the central point of the initial position area to determine the real-time relative position relationship between the tail end of the flexible channel and the initial position area.

6. The method of claim 5, wherein said comparing said second real-time location parameter to said initial location area and comparing said second real-time location parameter to a center point of said initial location area to determine a real-time relative location relationship of said flexible channel end to said initial location area comprises:

judging whether the real-time position of the tail end of the flexible channel is located in the initial position area or not according to the second real-time position parameter, if so, returning to a first mark bit, and if not, returning to a second mark bit;

and determining an angle between the real-time position of the tail end of the flexible channel and the central point of the initial position area according to the second real-time position parameter and the central point of the initial position area, wherein the angle is used as the angle relation between the tail end of the flexible channel and the center of the initial position area.

7. The method of claim 6, wherein said displaying said real-time relative positional relationship comprises:

drawing a first graph, wherein the first graph is used for representing the initial position area;

drawing a second graph according to the returned first marker bit or the second marker bit, wherein the second graph is used for representing the real-time position of the tail end of the flexible channel; if the first flag bit is returned, drawing the second graph in the first graph according to the angle relation between the tail end of the flexible channel and the center of the initial position area; if the second zone bit is returned, drawing the second graph outside the first graph according to the angle relation between the tail end of the flexible channel and the center of the initial position area;

and displaying the first graph and the second graph on a visual interface in real time.

8. The method of claim 1, further comprising:

detecting a first scale factor input by a user;

and setting the initial position area by taking the pre-factory position of the tail end of the flexible channel as a center according to the first scale factor and the first preset distance.

9. An apparatus for visualization of an offset relative to an origin, the apparatus comprising:

the acquisition module is used for acquiring a first real-time position parameter of the tail end of the flexible channel of the tail end execution mechanism in a first coordinate system;

a determining module, configured to determine a real-time relative positional relationship between the flexible channel end and an initial position region of the flexible channel end in the first coordinate system according to the first real-time position parameter and the initial position region, where the initial position region is determined according to an initial position of the flexible channel end of the end actuator, and the real-time relative positional relationship includes: whether the flexible channel tip is within the home position region and the angular relationship of the flexible channel tip to the center of the home position region;

and the display module is used for displaying the real-time relative position relation.

10. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the method of visualization of a relative origin offset as claimed in any one of claims 1 to 8.

11. A computer readable storage medium, instructions in which, when executed by a processor of a mobile terminal, enable the mobile terminal to perform the method of visualization of a relative origin offset as claimed in any one of claims 1 to 8.

Technical Field

The present disclosure relates to the field of soft tissue robots, and in particular, to a method and an apparatus for visualizing a relative origin offset, an electronic device, and a storage medium.

Background

When the remote control surgical robot is used for minimally invasive surgery, the handle operation end and the instrument execution end are scaled, and the flexible instrument structure is slender.

Although the motion relation between the handle and the tail end of the instrument is matched with high precision, compared with the traditional open surgery, a doctor is easy to lose hand feeling and the orientation feeling of the tail end of the instrument.

Disclosure of Invention

To overcome the problems in the related art, the present disclosure provides a method and an apparatus for visualizing a relative origin offset, an electronic device, and a storage medium.

According to a first aspect of the embodiments of the present disclosure, there is provided a method for visualizing a relative origin offset, including:

acquiring a first real-time position parameter of the tail end of a flexible channel of a tail end executing mechanism in a first coordinate system;

determining a real-time relative position relationship between the flexible channel tail end and an initial position area according to the first real-time position parameter and the initial position area of the flexible channel tail end in the first coordinate system, wherein the initial position area is determined according to a pre-factory position of the flexible channel tail end, and the real-time relative position relationship comprises: whether the flexible channel tip is within the home position region and the angular relationship of the flexible channel tip to the center of the home position region;

and displaying the real-time relative position relation.

Optionally, the acquiring a first real-time position parameter of the end of the flexible channel of the end actuator in the first coordinate system includes:

determining a first real-time position parameter of the tail end of the flexible channel of the tail end actuating mechanism in the first coordinate system according to a real-time detection value of a first magnetic sensing encoder of the base motor; the first magnetic sensing encoder is used for detecting a rotation angle of an output shaft of the base motor, the tail end executing structure comprises a transmission mechanism and a flexible channel, the flexible channel is used for a surgical instrument to pass through, and after the tail end executing mechanism is meshed with the output shaft of the base motor, the output shaft of the base motor is used for driving the transmission mechanism of the tail end executing mechanism during surgery, so that the transmission mechanism drives the flexible channel to move, and the surgical instrument is driven to move.

Optionally, before the step of obtaining a first real-time position parameter of the flexible channel end of the end effector in the first coordinate system, the method further includes:

when the engagement completion of the tail end actuator and the output shaft of the base motor is detected, acquiring a first detection value of a first magnetic sensing encoder and a second detection value of a second magnetic sensing encoder, wherein the second magnetic sensing encoder is used for detecting the rotation angle of a steel wire coil in the transmission mechanism, the steel wire coil is used for tensioning or loosening a steel wire when rotating, and the steel wire is used for controlling the flexible channel to move when being tensioned or loosened;

obtaining an offset position parameter of the tail end of the flexible channel according to a difference value between the second detection value and a pre-factory detection value of the second magnetic sensing encoder, wherein the pre-factory detection value is used for representing a pre-factory position of the tail end of the flexible channel;

and obtaining a position calibration parameter of the tail end of the flexible channel according to the difference value of the first detection value and the offset position parameter.

Optionally, the determining, according to a real-time detection value of a first magnetic sensor encoder of the base motor, a first real-time position parameter of the end of the flexible channel of the end actuator in the first coordinate system includes:

and calculating a difference value between a real-time detection value of a first magnetic sensing encoder of the base motor and the calibration angle parameter, and taking the calculated difference value as the first real-time position parameter of the tail end of the flexible channel in the first coordinate system.

Optionally, the determining a real-time relative position relationship between the flexible channel end and the initial position region according to the first real-time position parameter and the initial position region of the flexible channel end in the first coordinate system includes:

mapping the first real-time position parameter to a second coordinate system to obtain a second real-time position parameter, wherein the first coordinate system and the second coordinate system are both planar coordinate systems, a plane where the first coordinate system is located and a plane where the second coordinate system is located are parallel to each other, and a coordinate axis of the first coordinate system and a coordinate axis of the second coordinate system form an included angle of 45 degrees;

and comparing the second real-time position parameter with the initial position area, and comparing the second real-time position parameter with the central point of the initial position area to determine the real-time relative position relationship between the tail end of the flexible channel and the initial position area.

Optionally, the comparing the second real-time position parameter with the initial position region, and comparing the second real-time position parameter with a central point of the initial position region, and determining a real-time relative position relationship between the flexible channel end and the initial position region includes:

judging whether the real-time position of the tail end of the flexible channel is located in the initial position area or not according to the second real-time position parameter, if so, returning to a first mark bit, and if not, returning to a second mark bit;

and determining an angle between the real-time position of the tail end of the flexible channel and the central point of the initial position area according to the second real-time position parameter and the central point of the initial position area, wherein the angle is used as the angle relation between the tail end of the flexible channel and the center of the initial position area.

Optionally, the displaying the real-time relative position relationship includes:

drawing a first graph, wherein the first graph is used for representing the initial position area;

drawing a second graph according to the returned first marker bit or the second marker bit, wherein the second graph is used for representing the real-time position of the tail end of the flexible channel; if the first flag bit is returned, drawing the second graph in the first graph according to the angle relation between the tail end of the flexible channel and the center of the initial position area; if the second zone bit is returned, drawing the second graph outside the first graph according to the angle relation between the tail end of the flexible channel and the center of the initial position area;

and displaying the first graph and the second graph on a visual interface in real time.

Optionally, the method further includes:

detecting a first scale factor input by a user;

and setting the initial position area by taking the pre-factory position of the tail end of the flexible channel as a center according to the first scale factor and the first preset distance.

According to a second aspect of embodiments of the present disclosure, there is provided an apparatus for visualizing a relative origin offset, the apparatus comprising:

the acquisition module is used for acquiring a first real-time position parameter of the tail end of the flexible channel of the tail end execution mechanism in a first coordinate system;

a determining module, configured to determine a real-time relative positional relationship between the flexible channel end and an initial position region of the flexible channel end in the first coordinate system according to the first real-time position parameter and the initial position region, where the initial position region is determined according to an initial position of the flexible channel end of the end actuator, and the real-time relative positional relationship includes: whether the flexible channel tip is within the home position region and the angular relationship of the flexible channel tip to the center of the home position region;

and the display module is used for displaying the real-time relative position relation.

According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the visualization method of the relative origin offset.

According to a fourth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium having instructions which, when executed by a processor of an electronic device, enable the electronic device to perform the method of visualizing the relative origin offset.

The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:

the method can be used for acquiring a first real-time position parameter of the tail end of the flexible channel of the tail end executing mechanism in a first coordinate system; determining a real-time relative position relationship between the flexible channel tail end and an initial position area of the flexible channel tail end in the first coordinate system according to the first real-time position parameter and the initial position area of the flexible channel tail end in the first coordinate system, wherein the real-time relative position relationship comprises: whether the flexible channel tip is within the home position region and the angular relationship of the flexible channel tip to the center of the home position region; and displaying the real-time relative position relation. The visual icon is updated by calculating the offset parameter in real time, so that the effect of real-time following display is achieved, the relative position of each instrument is visually displayed in real time at a proper position on the display screen, the operation hand feeling of a doctor is effectively improved, and the safety factor of an operation is further improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.

FIG. 1 is a flow diagram illustrating a method of visualization of relative origin offset in accordance with an exemplary embodiment;

FIG. 2 is a schematic diagram illustrating the component relationships of a surgical consumable part according to an exemplary embodiment;

FIG. 3 is a block diagram illustrating a visualization device offset from an origin in accordance with an exemplary embodiment;

FIG. 4 is a block diagram illustrating an electronic device in accordance with an exemplary embodiment;

FIG. 5 is a full cross-sectional view of an end effector shown in accordance with an exemplary embodiment;

FIG. 6 is a schematic diagram illustrating the positional relationship of a solid circle to a hollow circle in accordance with an exemplary embodiment;

FIG. 7 is a schematic diagram illustrating another solid circle to hollow circle positional relationship in accordance with an exemplary embodiment;

FIG. 8 is a schematic diagram illustrating the construction of an end effector and base according to one exemplary embodiment;

FIG. 9 is a cross-sectional schematic view of a flexible channel and a steel wire shown in accordance with an exemplary embodiment;

FIG. 10 is a flow chart illustrating a calculation of positional calibration parameters for the end of a flexible channel in accordance with an exemplary embodiment;

FIG. 11 is a flow chart illustrating a method of obtaining a real-time relative positional relationship in accordance with an exemplary embodiment;

FIG. 12 is a flow chart illustrating a method of determining a real-time relative positional relationship in accordance with an exemplary embodiment;

FIG. 13 is a flow chart illustrating a method of displaying real-time relative positional relationships in accordance with an exemplary embodiment.

Description of reference numerals: 46-a multifunctional channel device; 47-an energy adapter; 47A-left energy adapter; 47B-right energy adapter; 48-a base; 49-control adapter; 49A-left control adapter; 49B-right control adapter; 463-the drive shaft; 464-a second magnetic sensing encoder; 465-a flexible channel; the section of the steel wire in the 001-X direction; the cross section of the steel wire in the 002-Y direction; 010-flexible channel cross section.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.

Fig. 1 is a flowchart illustrating a method for visualizing a relative origin offset according to an exemplary embodiment, as shown in fig. 1, including the following steps.

Step 101, acquiring a first real-time position parameter of the tail end of the flexible channel of the tail end executing mechanism in a first coordinate system.

In the present invention, the first real-time location parameter refers to: data reflecting the real-time position of the flexible tunnel end in a first coordinate system in which the component of the first real-time position in the X-axis direction is denoted as ax and the component in the Y-axis direction is denoted as ay, and thus the first real-time position parameter can be expressed in geometric components as ax and ay. In some embodiments, to obtain the first real-time position parameter, the first real-time position parameter of the end of the flexible channel of the end actuator in the first coordinate system may be determined based on real-time detection values of the first magnetic sensor encoder of the base motor. Wherein, first magnetic sensing encoder is used for detecting the turned angle of the output shaft of base motor, and end execution structure includes drive mechanism and flexible channel, and flexible channel is used for supplying surgical instruments to pass, and after end execution mechanism and the output shaft meshing of base motor, during the operation, the output shaft of base motor is used for driving end execution mechanism's drive mechanism to drive flexible channel motion through drive mechanism, and then drive surgical instruments motion.

For ease of understanding, referring to FIG. 2, FIG. 2 is a schematic diagram illustrating the components of a surgical consumable according to one exemplary embodiment. As shown in fig. 2, the a-stage is the end effector and the B-stage is the base, and when the surgical robot performs a surgical operation, the end effector and the robot base of the surgical robot are separate mechanisms, and therefore, the end effector needs to be attached to the robot base before the surgical operation.

Fig. 5 is a full cross-sectional view of an end effector shown according to an exemplary embodiment, with the specific positional relationship of the drive shaft 463, the second magnetic sensor encoder 464, and the flexible channel 465 as shown in fig. 5. The transmission shaft 463 is a part of the transmission mechanism. In the present invention, in order to distinguish the magnetic sensor encoder in the end actuator from the magnetic sensor encoder in the base, the magnetic sensor encoder in the end actuator is referred to as a second magnetic sensor encoder 464, and the magnetic sensor encoder in the base is referred to as a first magnetic sensor encoder (Gear magnetic sensor encoder).

The surgical instrument can pass through the two working channels of the tail end execution mechanism and is driven by the tail end execution mechanism to perform surgical operation. The end effector includes a drive mechanism and a flexible channel 465, the flexible channel 465 being adapted to allow passage of a surgical instrument therethrough. After the end effector is engaged with the base motor, the base motor is used to drive the actuator of the end effector during the procedure, thereby moving the flexible channel 465 through the actuator, and further moving the surgical instrument through the movement of the flexible channel 465.

Specifically, as previously described, the transmission mechanism of the end effector includes a transmission shaft 463, and when the end effector is engaged with the base, the base motor is engaged with the transmission shaft 463 to drive the transmission shaft 463 to rotate, thereby winding the wire. The other end of the steel wire is connected with the flexible channel of the tail end execution structure, and the steel wire drives the flexible channel to move when winding.

More specifically, the transmission mechanism of the end effector includes: 4 drive shafts 463, 8 wire coils and 8 wires. Wherein, set up 2 steel wire coils around every transmission shaft, two steel wire coils and transmission shaft direct or indirect meshing. The transmission shaft rotates under the drive of the output shaft of the base motor, and during the rotation of the transmission shaft, one steel wire coil around the transmission shaft is driven to rotate in the same direction with the transmission shaft, and the other steel wire coil around the transmission shaft rotates in the opposite direction with the transmission shaft. One end of each steel wire is wound on one steel wire coil, and the other end of each steel wire is connected with the tail end of the flexible channel. The wire coil is used for tightening or loosening the wire when rotating, and the wire controls the flexible channel to move when being tightened or loosened.

The relationship between the drive shaft 463 and the second magnetic sensor encoder 464 is: two second magnetic sensing encoders 464 are installed on two sides of each transmission shaft 463, and the two second magnetic sensing encoders 464 on two sides of each transmission shaft 463 are respectively used for detecting the rotation angles of the two steel wire coils. Since the total number of the drive shafts is 4, the number of the second magnetic sensor encoders is 8. Also, for convenience of understanding, in the present invention, the two second magnetic sensing encoders 464 corresponding to each transmission shaft 463 are referred to as an auxiliary a magnetic sensing encoder and an auxiliary B magnetic sensing encoder, respectively.

As shown in fig. 8, fig. 8 is a schematic diagram illustrating the construction of an end effector and base according to an exemplary embodiment. In some embodiments, the base 48 includes a base motor, wherein the base motor includes a first left motor, a second left motor, a first right motor, and a second right motor. It should be noted that, since the number of the base motors is 4, and each base motor needs to detect the rotation angle of the output shaft of the base motor through one first magnetic sensor encoder, the number of the first magnetic sensor encoders is also 4.

The end effector includes a multi-functional channel 46, an energy adapter 47, and a control adapter 49, wherein the multi-functional channel 46 includes the aforementioned drive shaft 463, a second magnetic sensor encoder 464, a flexible channel 465, and a steel wire. Wherein the flexible channel comprises a left flexible channel and a right flexible channel. The transmission shaft includes first left transmission shaft (with the first left motor meshing of base for control left side flexible channel is in the motion of X direction), second left transmission shaft (with the second left motor meshing of base, be used for controlling the motion of left side flexible channel in the Y direction), first right transmission shaft (with the first right motor meshing of base, be used for controlling the motion of right side flexible channel in the X direction), second right transmission shaft (with the second right motor meshing of base, be used for controlling the motion of right side flexible channel in the Y direction). The energy adapters 47 include a left energy adapter 47A, a right energy adapter 47B. Control adapters 49 include a left control adapter 49A and a right control adapter 49B.

Fig. 9 is a schematic cross-sectional view of a flexible channel and a wire according to an exemplary embodiment, as shown in fig. 9, wherein 010 denotes a flexible channel cross-section, 001 denotes a wire cross-section in the X-direction, i.e., a wire cross-section for pulling the flexible channel to move in the X-direction, and 002 denotes a wire cross-section in the Y-direction, i.e., a wire cross-section for pulling the flexible channel to move in the Y-direction.

In the present invention, as shown in fig. 9, the X direction and the Y direction in the first coordinate system form 45-degree angles with four directions, i.e., the top, bottom, left, and right directions in the visual sense. In other words, the X direction has a 45 degree angle with the horizontal direction and the Y direction also has a 45 degree angle with the vertical direction. Therefore, the steel wires for pulling the flexible channel to move along the X direction are respectively arranged at the upper left corner and the lower right corner of the section of the flexible channel, and the steel wires for pulling the flexible channel to move along the Y direction are respectively arranged at the upper right corner and the lower left corner of the section of the flexible channel. The advantage of arranging the wires in this way is that: when need control flexible channel when skew to the left side, need two steel wires in upper left corner and the lower left corner of simultaneous control, two steel wires compare in a steel wire for flexible channel atress is more even, helps improving flexible channel's motion stability.

In the present invention, the visual horizontal direction is set to the X 'direction, and the visual vertical direction is set to the Y' direction, thereby establishing a second coordinate system according to the X 'direction and the Y' direction.

In some embodiments, before obtaining the first real-time position parameter of the flexible channel end, the position calibration parameter of the flexible channel end may be calculated in advance, and then the first real-time position parameter of the flexible channel end may be determined according to the position calibration parameter. As shown in fig. 10, fig. 10 is a flowchart illustrating a method for calculating a position calibration parameter of a flexible channel end according to an exemplary embodiment, in order to calculate the position calibration parameter of the flexible channel end in advance, the following steps may be performed:

step 201, when it is detected that the engagement between the end effector and the output shaft of the base motor is completed, acquiring a first detection value of a first magnetic sensing encoder and a second detection value of a second magnetic sensing encoder, wherein the second magnetic sensing encoder is used for detecting a rotation angle of a steel wire coil in the transmission mechanism, the steel wire coil is used for tensioning or loosening a steel wire when rotating, and the steel wire is used for controlling the movement of the flexible channel when being tensioned or loosened.

As mentioned above, the second magnetic sensor encoder is used to detect the rotation angle of the wire coil in the transmission mechanism, the wire coil is used to tighten or loosen the wire when rotating, and the wire is used to control the flexible channel to move when being tightened or loosened.

In a specific implementation, when the engagement completion of the end actuator and the output shaft of the base motor is detected, the current detection values of the 4 first magnetic sensor encoders, namely the first detection values, can be read at the moment. At the same time, the current detection value, i.e., the second detection value, of each of the 8 second magnetic-sensing encoders is read at this time. The 8 second detection values are divided into 4 pairs, and each pair includes two second detection values belonging to the detection values of the auxiliary a magnetic sensor encoder and the auxiliary B magnetic sensor encoder corresponding to one drive shaft.

And step 202, obtaining an offset position parameter of the tail end of the flexible channel according to a difference value between the second detection value and a pre-factory detection value of the second magnetic sensing encoder, wherein the pre-factory detection value is used for representing a pre-factory position of the tail end of the flexible channel.

And the pre-factory detection value is used for representing the pre-factory position of the tail end of the flexible channel. In some embodiments, a register of the end actuator stores a pre-factory detection value of each second magnetic sensor encoder. When the end actuator is assembled to the base, the contacts on the base are electrically connected to the registers on the end actuator, and the base can access the registers of the end actuator, so as to read the pre-factory detection value of each second magnetic sensor encoder stored in the registers.

For example, before shipping, the end effector is mounted on the base of the robot system, and at this time, 4 wires of the flexible channel of the end effector are in a relaxed and flush state, and the flexible channel does not have any deviation in any direction, that is, the flexible channel can be pulled out by the same angle in any direction with the position as a central point. At this time, the current position values of all the second magnetic sensor encoders on the end actuator are written into the register of the end actuator using the writing tool.

In concrete implementation, the 8 second detection values read in step 201 are respectively different from the factory-expected detection values of the 8 second magnetic sensing encoders to calculate differences, so as to obtain 8 differences. These 8 differences are: a difference value between a second detection value of the auxiliary A magnetic sensor encoder of the first left transmission shaft and a pre-factory detection value, a difference value between a second detection value of the auxiliary B magnetic sensor encoder of the first left transmission shaft and a pre-factory detection value, a difference value between a second detection value of the auxiliary A magnetic sensor encoder of the second left transmission shaft and a pre-factory detection value, a difference value between a second detection value of the auxiliary B magnetic sensor encoder of the second left transmission shaft and a pre-factory detection value, the difference value between the second detection value of the auxiliary A magnetic sensing encoder of the first right transmission shaft and the detection value before factory shipment, the difference value between the second detection value of the auxiliary B magnetic sensing encoder of the first right transmission shaft and the detection value before factory shipment, the difference value between the second detection value of the auxiliary A magnetic sensing encoder of the second right transmission shaft and the detection value before factory shipment, and the difference value between the second detection value of the auxiliary B magnetic sensing encoder of the second right transmission shaft and the detection value before factory shipment. For simplicity of description, the difference value corresponding to the auxiliary a magnetic sensor encoder of each transmission shaft is abbreviated as Δ HomeA, and the difference value corresponding to the auxiliary B magnetic sensor encoder of each transmission shaft is abbreviated as Δ HomeB.

Then, for one transmission shaft, the Δ HomeA corresponding to the auxiliary a magnetic sensing encoder and the Δ HomeB corresponding to the auxiliary B magnetic sensing encoder are both used for feeding back the offset of the flexible channel along one direction. For example, for the first left transmission shaft, the Δ HomeA corresponding to the auxiliary a magnetic sensing encoder and the Δ HomeB corresponding to the auxiliary B magnetic sensing encoder are both used for feeding back the offset of the left flexible channel along the X direction. Therefore, in order to reduce errors, for one transmission shaft, an average value of Δ HomeA corresponding to the auxiliary a magnetic sensor encoder and Δ HomeB corresponding to the auxiliary B magnetic sensor encoder is obtained, that is ([ delta ] HomeA + [ delta ] HomeB)/2, and finally an offset position parameter is obtained. For example, for the first left transmission shaft, the Δ HomeA corresponding to the auxiliary a magnetic sensing encoder and the Δ HomeB corresponding to the auxiliary B magnetic sensing encoder are averaged, and finally the offset position parameter of the left flexible channel along the X direction is obtained.

Through the above processing, 4 offset position parameters are finally obtained, which are respectively: a shift position parameter of the left flexible channel in the X direction, a shift position parameter of the left flexible channel in the Y direction, a shift position parameter of the right flexible channel in the X direction, and a shift position parameter of the right flexible channel in the Y direction.

And step 203, obtaining a position calibration parameter of the tail end of the flexible channel according to the difference value between the first detection value and the offset position parameter.

As described above, in step 201, the first detection values of the 4 first magnetic sensor encoders are read. The 4 first magnetic sensing encoders are respectively used for detecting the rotation angles of the first left motor output shaft, the second left motor output shaft, the first right motor output shaft and the second right motor output shaft, and the four base motors are respectively used for controlling the movement of the left flexible channel along the X direction, the movement of the left flexible channel along the Y direction, the movement of the right flexible channel along the X direction and the movement of the right flexible channel along the Y direction. Therefore, the 4 first detection values read in step 201 correspond one-to-one to the 4 offset position parameters calculated in step 202. For example, a first detection value of a first magnetic sensor encoder for detecting the output shaft of the first left motor corresponds to a displacement position parameter of the left flexible channel in the X direction.

In concrete implementation, the difference between the first detection value and the offset position parameter corresponding to each other is calculated as the position calibration parameter Total Δ. Finally, 4 position calibration parameters are obtained, which are respectively: the position calibration parameters of the left flexible channel along the X direction, the position calibration parameters of the left flexible channel along the Y direction, the position calibration parameters of the right flexible channel along the X direction, and the position calibration parameters of the right flexible channel along the Y direction.

In step 101, in order to determine a first real-time position parameter of the flexible channel end in the first coordinate system according to the real-time detection value of the magnetic sensor encoder of the base motor, a difference value between the real-time detection value of the magnetic sensor encoder of the base motor and the calibration angle parameter may be calculated, and the calculated difference value may be used as the first real-time position parameter of the flexible channel end in the first coordinate system.

For understanding, for example, at the current time during the operation, a real-time detection value of the first magnetic sensing encoder corresponding to the first left motor output shaft is read, and then a difference value between the real-time detection value and a position calibration parameter of the left flexible channel along the X direction is calculated, so as to obtain a first real-time position parameter of the left flexible channel along the X direction, where the first real-time position parameter may reflect: the left flexible channel is offset in the X direction at the present time. In the same way, the remaining three first real-time location parameters may be obtained, which are respectively used to reflect: the offset of the left flexible channel in the Y direction at the current moment, the offset of the right flexible channel in the X direction at the current moment and the offset of the right flexible channel in the Y direction at the current moment.

102, determining a real-time relative position relationship between the flexible channel tail end and an initial position area according to the first real-time position parameter and the initial position area of the flexible channel tail end in the first coordinate system, wherein the initial position area is determined according to a pre-factory position of the flexible channel tail end, and the real-time relative position relationship comprises: whether the flexible channel end is within the home position area and the angular relationship of the flexible channel end to the center of the home position area.

In some embodiments, the range of the initial position region may be determined in advance by: firstly, detecting a first scale factor input by a user; and then setting an initial position area by taking the pre-factory position of the tail end of the flexible channel as a center according to a first scale factor and a first preset distance.

It should be noted that, in the present invention, the first preset distance refers to a radius or a side length corresponding to the initial position region, and specifically, the radius or the side length is selected, which is set for the initial position region in actual operation, and if the planned initial region is a circular region, the first preset distance is the radius of the initial position region, and if the planned initial region is a rectangular region, the first preset distance is the side length (length and width) of the initial position region. The first preset distance includes, but is not limited to, presetting the first preset distance before leaving a factory according to human-computer interaction experience.

In the embodiment of the present invention, the first scale factor refers to a scale value for scaling up or scaling down the first preset distance, for example, when performing a surgical operation, the target control may be input on a visual operation interface, including but not limited to a visual operation interface, according to an operation habit of a different doctor or according to a type of the surgical operation, and a corresponding scale factor is set, where the final initial position area is obtained by multiplying the first preset distance by the first scale factor. The first scale factor is, for example, 0.8, 1.2, etc., and the present application is not particularly limited.

For convenience of understanding, taking the left flexible channel as an example, an average value of the pre-factory detection value of the auxiliary a magnetic sensor encoder and the pre-factory detection value of the auxiliary B magnetic sensor encoder corresponding to the first left transmission shaft is calculated, and the average value is abbreviated as HomeX. Similarly, the average value of the pre-shipment detection value of the auxiliary a magnetic sensor encoder and the pre-shipment detection value of the auxiliary B magnetic sensor encoder corresponding to the second left propeller shaft is calculated, and the average value is abbreviated as HomeY. The coordinates (HomeX, HomeY), i.e. the pre-factory position of the left flexible channel, are then marked in a second coordinate system. And then (HomeX, HomeY) is taken as a circle center, the product of the first preset distance and the first scale factor example is taken as a radius, and a circular area is determined in the second coordinate system and taken as an initial position area.

In step 102, as shown in fig. 11, fig. 11 is a flow chart illustrating a method for obtaining a real-time relative position relationship according to an exemplary embodiment, in order to determine the real-time relative position relationship between the flexible channel end and the initial position region, the following sub-steps may be performed:

step 1021: and mapping the first real-time position parameter to a second coordinate system to obtain a second real-time position parameter, wherein the first coordinate system and the second coordinate system are both plane coordinate systems, the plane of the first coordinate system and the plane of the second coordinate system are parallel to each other, and the coordinate axis of the first coordinate system and the coordinate axis of the second coordinate system form an included angle of 45 degrees.

As mentioned above, the first coordinate system and the second coordinate system are both planar coordinate systems, the plane of the first coordinate system and the plane of the second coordinate system are parallel to each other, and the coordinate axis of the first coordinate system and the coordinate axis of the second coordinate system form an included angle of 45 degrees.

In some embodiments of the present invention, in order to perform visual display on the real-time relative position relationship, specifically, there may be a visual display interface, where a coordinate system in the visual display interface is a vertical and horizontal direction determined according to normal cognition of vision, that is, a direction of a coordinate system for calibrating vision existing in the visual display interface is a second coordinate system, a direction of a second X axis is a yaw direction of the flexible channel, and a direction of a second Y axis is a pitch direction of the flexible channel, which is equivalent to a horizontal direction (X 'direction) and a vertical direction (Y' direction) in normal cognition.

Therefore, for displaying on the visualization interface, the first real-time position parameter obtained in step 101 is mapped to the second coordinate system to obtain a second real-time position parameter, specifically, in the second coordinate system, a component of the second real-time position in the X-axis direction is denoted as Δ X ', and a component in the Y-axis direction is denoted as Δ Y', so the second real-time position parameter may be Δ X 'and Δ Y' in geometric components.

Step 1022: and comparing the second real-time position parameter with the initial position area, and comparing the second real-time position parameter with the central point of the initial position area to determine the real-time relative position relationship between the tail end of the flexible channel and the initial position area.

In step 1022, comparing the second real-time position parameter with the initial position region, and comparing the second real-time position parameter with the central point of the initial position region, as shown in fig. 12, where fig. 12 is a flowchart illustrating a method for determining a real-time relative position relationship, and determining a real-time relative position relationship between the flexible channel end and the initial position region, the following sub-steps may be specifically performed:

step 10221, according to the second real-time position parameter, determining whether the real-time position of the end of the flexible channel is within the initial position region, if the real-time position of the end of the flexible channel is within the initial position region, returning to the first flag bit, and if the real-time position of the end of the flexible channel is outside the initial position region, returning to the second flag bit.

In this embodiment of the present invention, the determining the real-time relative position relationship is to obtain the second real-time position parameter in step 1021, determine whether the real-time position of the flexible channel end is within the initial position region, return to the first flag bit if the real-time position of the flexible channel end is within the initial position region, and return to the second flag bit if the real-time position of the flexible channel end is outside the initial position region.

Further, it is illustrated in step 102 that the initial position area is determined according to the factory position of the flexible tip, and the initial area may be planned as a circular area or a rectangular area. Therefore, in some embodiments, when determining whether the real-time position of the flexible channel end is within the initial position region, two determination cases are divided.

In an embodiment of the present invention, the initial position area is set as a circular initial area, and the judgment coordinate system of the circular area should be consistent with the first coordinate system, so that the corresponding hypotenuse lengths of the X component and the Y component in the geometric space are obtained according to the triangle formula in the mathematical domain by using the second real-time position parameters (Δ X 'and Δ Y') obtained in step 1021, that is, the distance d from the real-time position of the end of the flexible channel in the first coordinate system to the origin of the initial position area, the first scale factor(s) set by the operator (doctor) is obtained, and the real-time position of the end of the flexible channel and the position relationship of the initial position area are obtained by comparing the aforementioned d and the first preset distance, that is the radius (r) of the initial position area multiplied by the first scale factor(s). Is formulated as:

d > r s (formula 1)

d < = r × s (formula 2)

In the above formula 1 and formula 2, d represents a distance from a real-time position of the flexible channel end to an origin of the initial region, r represents a first preset distance as a radius of the initial position region, and s represents a first scale factor.

If the mathematical relationship in formula 1 is satisfied, the real-time position of the flexible channel end is outside the range of the initial position area. If the mathematical relationship in formula 2 is satisfied, the real-time position of the flexible channel end is within the range of the initial position area.

In another embodiment of the present invention, the initial position area is set as a rectangular initial area, and the first preset distance is the side length (length and width) of the initial position area, so that it is determined whether the first real-time position parameters (Δ X and Δ Y) in step 101 are greater than the length multiplied by the first scale factor and the width multiplied by the first scale factor of the rectangular initial area, respectively. If both the delta X and the delta Y are smaller than the initial position area range, the real-time position of the tail end of the flexible channel is within the initial position area range, and if at least one of the delta X and the delta Y is larger than the initial position area range, the real-time position of the tail end of the flexible channel is outside the initial position area range.

The reason why two types of initial position areas are selected is that the circular initial area is set in consideration of a judgment standard using the orientation of the end actuator as a reference frame, and conforms to the judgment logic of normal consciousness of a person. The rectangular initial area is set by considering the range area which can be reached by each pair of symmetrical position steel wires along the drawing direction of the steel wires, the range area accords with the action range logic of a machine in a task space, two initial position areas can be provided for selection, and in the actual use process, a user can select a mode which is relatively accordant with the use habit and the judgment standard of the user to select the corresponding initial area, and the application is not limited specifically.

It should be noted that in the embodiment of the present invention, whichever form of initial region is selected, the initial region is displayed in a circular shape, for example, including but not limited to a hollow circle representation, rather than in a rectangular shape, on the interface displayed visually to the physician.

In summary, it is determined whether the real-time position of the end of the flexible channel is within the initial position region, if the real-time position of the end of the flexible channel is within the initial position region, the first flag bit is returned, and if the real-time position of the end of the flexible channel is outside the initial position region, the second flag bit is returned.

Step 10222, determining an angle between the real-time position of the flexible channel end and the central point of the initial position area according to the second real-time position parameter and the central point of the initial position area, as an angular relationship between the flexible channel end and the initial position area center.

And representing the X component and the Y component of the real-time position in the second coordinate system, which are far away from the central point of the initial position area, as X1 and Y1, wherein the central point of the initial position area is (HomeX, HomeY), and obtaining an included angle between the real-time position and the central point of the initial position area by performing arc tangent processing on the X component and the Y component, namely an arc tangent formula in a mathematical relationship.

Is formulated as:

angle = atan (Y1/X1) (equation 3)

In the above equation 3, angle represents an angle between the real-time position and the center point of the initial position region, atan () function represents an arctangent function, X1 represents an X component of the real-time position from the center point of the initial position region, and Y1 represents a Y component of the real-time position from the center point of the initial position region.

At this time, the obtained angle between the real-time position and the central point of the initial position area can be used as a reference for the angle relationship between the end of the flexible channel and the center of the initial position area displayed on the visual interface.

And 103, displaying the real-time relative position relationship.

In step 102, the real-time relative position relationship between the flexible channel end and the initial position region is determined, and therefore, the real-time relative position relationship is displayed on the visualization interface.

Further, as shown in fig. 13, fig. 13 is a flowchart illustrating a method for displaying real-time relative position relationship according to an exemplary embodiment, and step 103 includes the following steps:

and step 1031, drawing a first graph, wherein the first graph is used for representing the initial position area.

For convenience of understanding, taking the left flexible channel as an example, an average value of the pre-factory detection value of the auxiliary a magnetic sensor encoder and the pre-factory detection value of the auxiliary B magnetic sensor encoder corresponding to the first left transmission shaft is calculated, and the average value is abbreviated as HomeX. Similarly, the average value of the pre-shipment detection value of the auxiliary a magnetic sensor encoder and the pre-shipment detection value of the auxiliary B magnetic sensor encoder corresponding to the second left propeller shaft is calculated, and the average value is abbreviated as HomeY. In the visual display, coordinates (HomeX, HomeY), that is, the pre-factory position of the left flexible channel, are marked in the second coordinate system. And then drawing a first graph in a second coordinate system by taking (HomeX, HomeY) as a circle center and taking the product of the first preset distance and the first scale factor example as a visualization radius a, wherein the first graph is used as an initial position area and can be in a hollow circle shape.

It should be noted that the first graphic displayed on the visualization interface is only one visualization representation of the initial position area of the flexible channel end.

Step 1032, drawing a second graph according to the returned first zone bit or the second zone bit, wherein the second graph is used for representing the real-time position of the tail end of the flexible channel; if the first flag bit is returned, drawing the second graph in the first graph according to the angle relation between the tail end of the flexible channel and the center of the initial position area; and if the second zone bit is returned, drawing the second graph outside the first graph according to the angle relation between the tail end of the flexible channel and the center of the initial position area.

And determining the real-time position of the tail end of the flexible channel and the position relation of the hollow area according to the first marker bit or the second marker bit in the step 10221.

For easy understanding, taking the left flexible channel as an example, according to the requirement of the human-computer interaction experience, half of the visualization radius a of the preamble first graph is set as the radius of the second graph when the real-time position is visualized, that is, the radius is 0.5a, and the second graph may be in the form of a solid circle. Therefore, when the returned flag bit is the first flag bit, which is equivalent to that the second graph is in the first graph, that is, the solid circle is in the hollow circle, then according to the angle between the preamble real-time position and the central point of the initial position region, (HomeX, HomeY), and the visualization radius of the second graph 0.5a, the central point coordinate of the second graph, that is, (HomeX +0.5a & (angle), HomeY +0.5a & (angle), can be obtained, and according to the central point coordinate of the second graph and the visualization radius, the visualization display form of the second graph can be drawn, and at this time, the visualization display effect of the position relationship between the first graph and the second graph is shown in fig. 7.

If the returned flag bit is the second flag bit, which is equivalent to that the second graph is outside the first graph, that is, the solid circle is outside the hollow circle, the center point coordinates of the second graph, that is, (HomeX +1.5a & (angle), HomeY +1.5a & (angle)) can be obtained according to the angle between the real-time position and the center point of the initial position area, the coordinates of the center point of the initial position area, and the visualization radius 0.5a of the second graph, and the visualization display form of the second graph can be drawn according to the coordinates of the center point and the visualization radius of the second graph, at this time, the visualization display effect of the position relationship between the first graph and the second graph is shown in fig. 6.

And 1033, displaying the first graph and the second graph on a visual interface in real time.

It should be noted that in the visual display, as shown in fig. 6 and 7, the solid circles represent the real-time position of the end of the flexible channel, the hollow circles represent the initial position region, and the real-time display is performed on the visual interface to achieve the effect of real-time follow-up display, so that the operator can view the real-time relative position of the end of the flexible channel of the end executing mechanism and the initial position region with the most intuitive effect, the operation hand feeling of the doctor can be effectively improved, and the safety factor of the operation is further improved.

It is worth mentioning that the present invention further embodies the advantages of the present invention through the following description: during surgery, the initial position of the flexible channel tip corresponds to the pre-factory detection of the second magnetic sensor encoder. Since the pre-factory detection value is detected without any deflection of the flexible channel, the pre-factory detection value is referred to as an origin position, which may be referred to as an initial position of the flexible channel end.

And, the transmission shaft of the end effector has the following relation with the wire: (1) the transmission shaft rotates forwards by 180 degrees, and the steel wire traction flexible channel bends by 90 degrees along one steel wire traction direction; (2) the transmission shaft rotates 180 degrees in the opposite direction, and the steel wire pulls the flexible channel to bend 90 degrees in the opposite direction of (1). Thus, the angle of rotation of the drive shaft of the end effector may reflect the amount of offset of the flexible channel. As described above, two second magnetic sensing encoders, namely, the auxiliary a magnetic sensing encoder and the auxiliary B magnetic sensing encoder, are disposed on both sides of the transmission shaft, and can detect tension and slack in two steel wires drawn by the transmission shaft, respectively. Based on this, the rotation angle of the propeller shaft can be estimated by taking the average value of the detection value of the auxiliary a magnetic sensor encoder and the detection value of the auxiliary B magnetic sensor encoder, but there is always a one-sided feedback inaccuracy due to the passivity in the slack direction.

Therefore, in the present invention, it is considered to use the first magnetic sensing encoder (i.e., Gear magnetic sensing encoder) of the base to feed back the rotation angle of the drive shaft of the end effector because the output shaft of the motor of the base and the drive shaft of the end effector are tightly coupled in the operation mode. If the first magnetic sensing encoder of the base is required to be used for calculating the offset position, the position data of the first magnetic sensing encoder needs to be converted into position information of the transmission shaft of the end actuator. During the conversion, the following two factors need to be considered: (1) the initial position of a first magnetic sensing encoder at the output shaft of the base motor is different from the initial position of a transmission shaft of the tail end actuating mechanism; (2) before the meshing operation, the output shaft of the base motor and the transmission shaft of the tail end actuating mechanism have no transmission relation. Therefore, on the one hand, the deviation between the angle of the transmission shaft of the end effector after the engagement operation and the angle of the transmission shaft of the end effector before the engagement operation, i.e. the offset position parameter in the aforementioned step 202, needs to be considered. On the other hand, the angular value deviation between the transmission shaft of the engaged end actuator and the output shaft of the base motor, i.e. the position calibration parameter in step 203, needs to be considered.

Fig. 3 is a block diagram illustrating a visualization apparatus for relative origin offset according to an exemplary embodiment, which includes an obtaining module 301, a determining module 302, and a displaying module 303.

The acquiring module 301 is configured to acquire a first real-time position parameter of the end of the flexible channel of the end actuator in a first coordinate system.

A determining module 302, configured to determine a real-time relative position relationship between the flexible channel end and an initial position region of the flexible channel end in the first coordinate system according to the first real-time position parameter and the initial position region, where the initial position region is determined according to a pre-factory position of the flexible channel end, and the real-time relative position relationship includes: whether the flexible channel tip is within the home position region and the angular relationship of the flexible channel tip to the center of the home position region.

And a display module 303, configured to display the real-time relative position relationship.

With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Fig. 4 is a block diagram illustrating an electronic device 400 according to an example embodiment. For example, the electronic device 400 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, and the like.

Referring to fig. 4, electronic device 400 may include one or more of the following components: processing component 402, memory 404, power component 406, multimedia component 408, audio component 410, input/output interface 412, sensor component 414, and communication component 416.

The processing component 402 generally controls the overall operation of the device, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 402 can include one or more modules that facilitate interaction between the processing component 402 and other components. For example, the processing component 402 can include a multimedia module to facilitate interaction between the multimedia component 408 and the processing component 402.

The memory 404 is configured to store various types of data to support operations at the electronic device 400. Examples of such data include instructions for any application or method operating on the device, contact data, phonebook data, messages, pictures, videos, and the like. The memory 404 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

The power supply component 406 provides power to the various components of the electronic device 400. Power components 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for electronic device 400.

The multimedia component 408 comprises a screen providing an output interface between the electronic device 400 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 408 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 400 is in an operating mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 410 is configured to output and/or input audio signals. For example, the audio component 410 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 400 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 404 or transmitted via the communication component 416. In some embodiments, audio component 410 also includes a speaker for outputting audio signals.

The input/output interface 412 provides an interface between the processing component 402 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor component 414 includes one or more sensors for providing various aspects of status assessment for the electronic device 400. For example, the sensor assembly 414 may detect an open/closed state of the electronic device 400, the relative positioning of components, such as a display and keypad of the electronic device 400, the sensor assembly 414 may also detect a change in the position of the electronic device 400 or a component of the electronic device 400, the presence or absence of user contact with the electronic device 400, orientation or acceleration/deceleration of the electronic device 400, and a change in the temperature of the electronic device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 414 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 416 is configured to facilitate wired or wireless communication between the electronic device 400 and other devices. The electronic device 400 may access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 416 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the electronic device 400 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.

In an exemplary embodiment, a non-transitory computer-readable storage medium comprising instructions, such as the memory 404 comprising instructions, executable by the processor 420 of the electronic device 400 to perform the above-described method is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.

It will be understood that the invention is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.