阅读说明：本技术 虚拟键盘的显示方法、头戴显示设备及系统 (Display method of virtual keyboard, head-mounted display equipment and system ) 是由 郑光璞 舒玉龙 吴涛 周宏伟 于 2020-03-04 设计创作，主要内容包括：本发明公开了虚拟键盘的显示方法、头戴显示设备及系统,该方法包括：通过电磁接收器感应物理键盘的电磁发射器发射的电磁信号,以确定物理键盘在头戴坐标系下的第一位姿；接收物理键盘发送的通过设置在物理键盘的惯性测量单元测量得到的位姿变化信息,根据物理键盘在头戴坐标系下的初始位姿和位姿变化信息,确定物理键盘在头戴坐标系下的第二位姿；对第一位姿和第二位姿进行融合,确定物理键盘的头戴坐标系下的第三位姿；基于第三位姿向用户呈现虚拟键盘。(The invention discloses a display method of a virtual keyboard, a head-mounted display device and a system, wherein the method comprises the following steps: the method comprises the steps that an electromagnetic receiver is used for inducing an electromagnetic signal emitted by an electromagnetic emitter of a physical keyboard to determine a first pose of the physical keyboard under a head-mounted coordinate system; receiving pose change information which is sent by a physical keyboard and is obtained through measurement by an inertial measurement unit arranged on the physical keyboard, and determining a second pose of the physical keyboard in a head-mounted coordinate system according to the initial pose and the pose change information of the physical keyboard in the head-mounted coordinate system; fusing the first pose and the second pose, and determining a third pose of the physical keyboard under a head-mounted coordinate system; the virtual keyboard is presented to the user based on the third pose.)

1. A display method of a virtual keyboard in a head-mounted display device is applied to the head-mounted display device, the head-mounted display device is provided with an electromagnetic receiver, and the method comprises the following steps:

the electromagnetic receiver is used for inducing an electromagnetic signal transmitted by an electromagnetic transmitter of the physical keyboard to determine a first pose of the physical keyboard under a head-mounted coordinate system;

receiving pose change information which is sent by a physical keyboard and is obtained through measurement by an inertial measurement unit arranged on the physical keyboard, and determining a second pose of the physical keyboard in a head-mounted coordinate system according to the initial pose and the pose change information of the physical keyboard in the head-mounted coordinate system;

fusing the first pose and the second pose, and determining a third pose of the physical keyboard under a head-mounted coordinate system;

presenting a virtual keyboard to a user based on the third pose.

2. The method of claim 1, the inducing, by the electromagnetic receiver, an electromagnetic signal transmitted by an electromagnetic transmitter of a physical keyboard to determine a first position of the physical keyboard in a head-mounted coordinate system comprising:

determining a first distance and a first relative posture between the head-mounted display device and the physical keyboard according to the induction signal output by the electromagnetic receiver;

and determining a first pose of the physical keyboard under a head-mounted coordinate system according to the first distance and the first relative posture.

3. The method of claim 1, the first position comprising a first position parameter and a first pose parameter, the second position comprising a second position parameter and a second pose parameter;

the fusing the first pose and the second pose and determining a third pose of the physical keyboard under a head-mounted coordinate system comprises:

calculating a weighted average value of the first position parameter of the first pose and the second position parameter of the second pose to obtain a third position parameter;

calculating a weighted average value of the first attitude parameter of the first pose and the second attitude parameter of the second pose to obtain a third attitude parameter;

and determining a third pose of the physical keyboard under a head-mounted coordinate system according to the third position parameter and the third posture parameter.

4. The method of claim 1, further comprising:

receiving a key value sent by a physical keyboard;

determining a pressed key according to the key value based on the mapping relation between the key of the physical keyboard and the key value;

and lightening the corresponding pressed key on the virtual keyboard at preset lightening time.

5. The method of claim 1, further comprising:

receiving a vibration signal sent by a vibration motor of the physical keyboard;

and according to the vibration signal, lighting up an appointed key corresponding to the vibration signal on the virtual keyboard.

6. A display method of a virtual keyboard in a head-mounted display device is applied to a physical keyboard, the physical keyboard is provided with an electromagnetic emitter and an inertia measurement unit, and the method comprises the following steps:

transmitting an electromagnetic signal to a head-mounted display device through the electromagnetic transmitter;

and measuring the pose change information of the physical keyboard through the inertial measurement unit, and sending the pose change information to the head-mounted display equipment.

7. The method of claim 6, further comprising:

responding to the pressing operation of a user, and acquiring a key value;

and sending the key value to the head-mounted display equipment.

8. The method of claim 6, the physical keyboard further provided with a vibration motor for emitting a vibration signal if a preset designated key is pressed, the method further comprising:

responding to the operation of a user on a preset appointed key, and generating a vibration signal by the vibration motor;

and sending the vibration signal to a head-mounted display device.

9. A head-mounted display device, comprising: the device comprises an electromagnetic receiver, a first position and posture determining module, a second position and posture determining module, a third position and posture determining module and a display module;

the first pose determination module is used for inducing an electromagnetic signal transmitted by an electromagnetic transmitter of the physical keyboard through the electromagnetic receiver so as to determine a first pose of the physical keyboard under a head-mounted coordinate system;

the second pose determining module is used for receiving pose change information which is sent by the physical keyboard and is obtained by measurement through an inertial measurement unit arranged on the physical keyboard, and determining a second pose of the physical keyboard in a head-mounted coordinate system according to the initial pose and the pose change information of the physical keyboard in the head-mounted coordinate system;

the third pose determining module is used for fusing the first pose and the second pose and determining a third pose of the physical keyboard under a head-mounted display equipment coordinate system;

and the display module is used for presenting the virtual keyboard to the user based on the third pose.

10. A head-mounted display device, comprising:

a memory for storing computer instructions;

a processor for retrieving the computer instructions from the memory and executing a display method of a virtual keyboard in a head mounted display device according to any one of claims 1-5 under the control of the computer instructions.

11. A physical keyboard, comprising:

an electromagnetic signal transmitter for transmitting an electromagnetic signal to the head-mounted display device;

and the inertial measurement unit is used for measuring the pose change information of the physical keyboard and sending the pose change information to the head-mounted display equipment.

12. A physical keyboard, comprising:

a memory for storing computer instructions;

a processor for retrieving the computer instructions from the memory and executing a display method of a virtual keyboard in a head mounted display device according to any one of claims 6-8 under the control of the computer instructions.

13. A head mounted display system comprising a head mounted display device according to claim 9 or 10 and a physical keyboard according to claim 11 or 12.

Technical Field

The invention relates to the technical field of human-computer interaction, in particular to a display method of a virtual keyboard, a head-mounted display device and a system.

Background

Virtual Reality (VR) is a highly new technology that has emerged in recent years. With the explosion of the virtual reality industry, the interaction requirements of the virtual and the reality are increasingly increased in the using process of users.

At present, virtual reality device-based interactions mainly focus on speech recognition, gesture recognition, head rotation tracking, and the like. For more complicated interaction modes such as character input, attribute browsing, desktop environment and the like, a virtual keyboard needs to be displayed in the virtual reality equipment.

The virtual keyboard may be generally located based on vision, inertial sensors, ultrasound, etc. However, the method for positioning the virtual keyboard based on the vision is easily affected by the ambient light, and the positioning accuracy is poor. The virtual keyboard is positioned based on the ultrasonic waves, the positioning accuracy is high, but the frequency of the ultrasonic waves is easily affected by the Doppler effect and the temperature, and the equipment cost is high. The mode of positioning the virtual keyboard based on the inertial sensor has lower equipment cost but limited positioning precision.

Therefore, there is a need to provide a new scheme for displaying a virtual keyboard.

Disclosure of Invention

The invention aims to provide a technical scheme of a display method, a head-mounted display device and a system of a virtual keyboard.

According to a first aspect of the present invention, there is provided a display method of a virtual keyboard in a head-mounted display device, applied to the head-mounted display device, the head-mounted display device being provided with an electromagnetic receiver, the method including:

the electromagnetic receiver is used for inducing an electromagnetic signal transmitted by an electromagnetic transmitter of the physical keyboard to determine a first pose of the physical keyboard under a head-mounted coordinate system;

receiving pose change information which is sent by a physical keyboard and is obtained through measurement by an inertial measurement unit arranged on the physical keyboard, and determining a second pose of the physical keyboard in a head-mounted coordinate system according to the initial pose and the pose change information of the physical keyboard in the head-mounted coordinate system;

fusing the first pose and the second pose, and determining a third pose of the physical keyboard under a head-mounted coordinate system;

presenting a virtual keyboard to a user based on the third pose.

Optionally, the sensing, by the electromagnetic receiver, an electromagnetic signal emitted by an electromagnetic transmitter of the physical keyboard to determine a first posture of the physical keyboard in the head-mounted coordinate system includes:

determining a first distance and a first relative posture between the head-mounted display device and the physical keyboard according to the induction signal output by the electromagnetic receiver;

and determining a first pose of the physical keyboard under a head-mounted coordinate system according to the first distance and the first relative posture.

Optionally, the first position comprises a first position parameter and a first attitude parameter, and the second position comprises a second position parameter and a second attitude parameter;

the fusing the first pose and the second pose and determining a third pose of the physical keyboard under a head-mounted coordinate system comprises:

calculating a weighted average value of the first position parameter of the first pose and the second position parameter of the second pose to obtain a third position parameter;

calculating a weighted average value of the first attitude parameter of the first pose and the second attitude parameter of the second pose to obtain a third attitude parameter;

and determining a third pose of the physical keyboard under a head-mounted coordinate system according to the third position parameter and the third posture parameter.

Optionally, the method further comprises:

receiving a key value sent by a physical keyboard;

determining a pressed key according to the key value based on the mapping relation between the key of the physical keyboard and the key value;

and lightening the corresponding pressed key on the virtual keyboard at preset lightening time.

Optionally, the method further comprises:

receiving a vibration signal sent by a vibration motor of the physical keyboard;

and according to the vibration signal, lighting up an appointed key corresponding to the vibration signal on the virtual keyboard.

According to a second aspect of the present invention, there is provided a display method of a virtual keyboard in a head-mounted display device, applied to a physical keyboard provided with an electromagnetic transmitter and an inertial measurement unit, the method comprising:

transmitting an electromagnetic signal to a head-mounted display device through the electromagnetic transmitter;

and measuring the pose change information of the physical keyboard through the inertial measurement unit, and sending the pose change information to the head-mounted display equipment.

Optionally, the method further comprises:

responding to the pressing operation of a user, and acquiring a key value;

and sending the key value to the head-mounted display equipment.

Optionally, the physical keyboard is further provided with a vibration motor, and the vibration motor is configured to send out a vibration signal when a preset designated key is pressed, and the method further includes:

responding to the operation of a user on a preset appointed key, and generating a vibration signal by the vibration motor;

and sending the vibration signal to a head-mounted display device.

According to a third aspect of the present invention, there is provided a head mounted display device comprising: the device comprises an electromagnetic receiver, a first position and posture determining module, a second position and posture determining module, a third position and posture determining module and a display module;

the first pose determination module is used for inducing an electromagnetic signal transmitted by an electromagnetic transmitter of the physical keyboard through the electromagnetic receiver so as to determine a first pose of the physical keyboard under a head-mounted coordinate system;

the second pose determining module is used for receiving pose change information which is sent by the physical keyboard and is obtained by measurement through an inertial measurement unit arranged on the physical keyboard, and determining a second pose of the physical keyboard in a head-mounted coordinate system according to the initial pose and the pose change information of the physical keyboard in the head-mounted coordinate system;

the third pose determining module is used for fusing the first pose and the second pose and determining a third pose of the physical keyboard under a head-mounted display equipment coordinate system;

and the display module is used for presenting the virtual keyboard to the user based on the third pose.

According to a fourth aspect of the present invention, there is provided a head mounted display device comprising:

a memory for storing computer instructions;

a processor, configured to call the computer instruction from the memory, and execute, under control of the computer instruction, a display method of a virtual keyboard in a head-mounted display device according to a first aspect of the present invention.

According to a fifth aspect of the present invention, there is provided a physical keyboard comprising:

an electromagnetic signal transmitter for transmitting an electromagnetic signal to the head-mounted display device;

and the inertial measurement unit is used for measuring the pose change information of the physical keyboard and sending the pose change information to the head-mounted display equipment.

According to a sixth aspect of the present invention, there is provided a physical keyboard comprising:

a memory for storing computer instructions;

a processor, configured to call the computer instruction from the memory, and execute a display method of a virtual keyboard in a head-mounted display device according to the second aspect of the present invention under the control of the computer instruction.

According to a seventh aspect of the present invention, there is provided a head mounted display system comprising a head mounted display device as provided in the third or fourth aspect of the present invention, a head mounted physical keyboard as provided in the fifth or sixth aspect of the present invention.

According to one embodiment of the disclosure, the first pose of the physical keyboard is determined based on an electromagnetic positioning principle, the second pose of the physical keyboard is determined based on a positioning principle of the inertial measurement unit, and the first pose and the second pose are fused, so that the accuracy of short-time positioning of the inertial measurement unit can be utilized to make up for unsmooth pose change of the virtual keyboard obtained by electromagnetic positioning caused by intensity change of electromagnetic signals, and thus, image jitter is reduced. The embodiment combines the electromagnetic signal and the inertia measurement unit to position the physical keyboard, so that the positioning accuracy can be improved, and the positioning stability can be ensured.

Other features of the present invention and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention.

FIG. 1 provides a schematic diagram of a hardware configuration of a head mounted display system according to the present invention;

FIG. 2 is a flow chart of one embodiment of a method for displaying a virtual keyboard in a head mounted display device according to the present invention;

FIG. 3 is a flow chart of another embodiment of a method for displaying a virtual keyboard in a head mounted display device according to the present invention;

FIG. 4 is a flow chart of a method for displaying a virtual keyboard in a head mounted display device according to another embodiment of the present invention;

FIG. 5 is a block schematic diagram of one implementation structure of a head mounted display device according to the present invention;

FIG. 6 is a block schematic diagram of another implementation structure of a head mounted display device according to the present invention;

FIG. 7 is a block schematic diagram of one implementation of a physical keyboard in accordance with the present invention;

FIG. 8 is a block schematic diagram of another implementation configuration of a physical keyboard in accordance with the present invention.

Detailed Description

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.

In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

< hardware configuration >

Fig. 1 is a block diagram of a hardware configuration of a head mounted display system 100 that can be used to implement the present invention.

The head mounted display system 100 includes a head mounted display device 110 and a physical keyboard 120.

The head-mounted display device 110 may be a VR (Virtual Reality) device, an AR (Augmented Reality) device, an MR (Mixed Reality) device, and the like.

The head mounted display device 110 is provided with an electromagnetic receiver that can be used to sense electromagnetic signals emitted by an electromagnetic transmitter of a physical keyboard.

In one example, as shown in fig. 1, the head mounted display device 110 may include a processor 111, a memory 112, an interface apparatus 113, a communication apparatus 114, a display apparatus 115, an input apparatus 116, an audio apparatus 117, a camera 118, an electromagnetic receiver 119, and the like.

The processor 111 may be, for example, a central processing unit CPU, a microprocessor MCU, etc., the memory 112 may include, for example, ROM (read only memory), RAM (random access memory), non-volatile memory such as a hard disk, etc., the interface device 113 may include, for example, a USB interface, a serial interface, an infrared interface, etc., the communication device 114 may be capable of wired or wireless communication, the display device 115 may be, for example, a liquid crystal display, an L ED display, a touch display, etc., the input device 116 may include, for example, a touch screen, a handle, a body-sensory input, etc., the audio device 117 may be used to input/output voice information, the camera 118 may be used to acquire image information, may be, for example, a binocular camera, the electromagnetic receiver 119 may be used to sense electromagnetic signals emitted by an electromagnetic.

Although multiple apparatuses are shown for the head mounted display device 110 in fig. 1, the present invention may only relate to some of the apparatuses, for example, the head mounted display device 110 only relates to the processor 111, the memory 112, and the electromagnetic receiver 119.

The physical keyboard 120 is provided with an electromagnetic transmitter for transmitting an electromagnetic signal and an inertial measurement unit for acquiring pose change information of the physical keyboard.

In one example, as shown in FIG. 1, the physical keyboard 120 may include a processor 121, a memory 122, an interface device 123, a communication device 124, an electromagnetic transmitter 125, an inertial measurement unit 126, and the like.

The processor 121 may be, for example, a central processing unit CPU, a microprocessor MCU, or the like. The memory 122 includes, for example, a ROM (read only memory), a RAM (random access memory), a nonvolatile memory such as a hard disk, and the like. The interface device 123 includes, for example, a USB interface, a serial interface, an infrared interface, and the like. The communication device 124 is capable of wired or wireless communication, for example. The electromagnetic transmitter 125 may be used to transmit electromagnetic signals to a head-mounted display device. Inertial measurement unit 126 may be used to measure pose changes of physical keyboard 120.

Although multiple devices are shown for physical keyboard 120 in FIG. 1, the present invention may only relate to some of the devices, for example, physical keyboard 120 only relates to processor 111, memory 112, and electromagnetic transmitter 125 and inertial measurement unit 126.

In the above description, the skilled person can design the instructions according to the solutions provided in the present disclosure. How the instructions control the operation of the processor is well known in the art and will not be described in detail herein.

The head mounted display system shown in fig. 1 is merely illustrative and is in no way intended to limit the present disclosure, its application, or uses.

< first embodiment >

Fig. 2 is a schematic diagram of a display method of a virtual keyboard in a head-mounted display device according to an embodiment of the present specification. The display method of the virtual keyboard provided in this embodiment is implemented by computer technology, and may be implemented by the head-mounted display device 110 described in fig. 1.

The method for displaying a virtual keyboard in a head-mounted display device provided by the embodiment includes steps S2200 to S2800.

Step S2200, the electromagnetic receiver is used for inducing the electromagnetic signal emitted by the electromagnetic emitter of the physical keyboard so as to determine the first pose of the physical keyboard under the head-wearing coordinate system.

In this embodiment, the head mounted display device is provided with an electromagnetic receiver for sensing electromagnetic signals emitted by the electromagnetic transmitter of the physical keyboard. The electromagnetic receiver may be fixedly disposed on the head-mounted display device and form a fixed positional relationship with a display screen of the head-mounted display device. The arrangement position of the electromagnetic receiver induces an electromagnetic signal transmitted by an electromagnetic transmitter of the physical keyboard through the electromagnetic receiver, so that the first pose of the physical keyboard under a head-mounted coordinate system can be determined.

The head-mounted coordinate system is a three-dimensional coordinate system established by taking the center of the head-mounted display device as a coordinate origin, and takes the direction of a connecting line of the left eyepiece and the right eyepiece as an X axis, the direction perpendicular to the connecting line of the left eyepiece and the right eyepiece as a Y axis, and the direction perpendicular to a plane formed by the X axis and the Y axis as a Z axis.

In one embodiment, the step of sensing, by an electromagnetic receiver, an electromagnetic signal emitted by an electromagnetic emitter of the physical keyboard to determine the first pose of the physical keyboard in the head-mounted coordinate system may further include: steps S2210-S2220.

Step S2210, determining a first distance and a first relative posture between the head-mounted display device and the physical keyboard according to the induction signal output by the electromagnetic receiver.

In this embodiment, the electromagnetic receiver is provided with orthogonal three-axis coils, and the orthogonal three-axis coils include three sets of coils perpendicular to each other. Three groups of coils of the electromagnetic receiver induce the electromagnetic signal transmitted by the electromagnetic transmitter and output the induction signal. The output induction signal can reflect the induced electromotive force generated by the three groups of coils of the electromagnetic receiver, and the generated induced electromotive force is related to the relative position and direction between the electromagnetic transmitter and the electromagnetic receiver. The method comprises the steps of establishing an induced electromotive force matrix according to induced electromotive forces generated by three groups of coils of an electromagnetic receiver, and solving a position parameter and an attitude parameter of an electromagnetic transmitter by utilizing a six-degree-of-freedom algorithm, so that a first distance and a first relative attitude between a physical keyboard and a head-mounted display device are obtained.

Step S2220, according to the first distance and the first relative posture, a first posture of the physical keyboard under the head-mounted coordinate system is determined.

In the embodiment of the invention, the electromagnetic receiver of the head-mounted display device generates induced electromotive force by inducing the electromagnetic signal transmitted by the electromagnetic transmitter of the physical keyboard, and calculates the first pose of the physical keyboard according to the induced electromotive force, thereby realizing the positioning of the physical keyboard. The embodiment is based on electromagnetic positioning, avoids the problem that a shielding object is shielded in the positioning process, can improve the positioning precision, and realizes 360-degree tracking and positioning.

After determining the first position of the physical keyboard in the head-mounted coordinate system, entering:

and step S2400, receiving pose change information which is sent by the physical keyboard and is obtained through measurement by an inertial measurement unit arranged on the physical keyboard, and determining a second pose of the physical keyboard in a head-mounted coordinate system according to the initial pose and the pose change information of the physical keyboard in the head-mounted coordinate system.

In one embodiment, an inertial measurement unit is disposed on the physical keyboard and is configured to detect and measure acceleration, tilt, shock, vibration, rotation, and multiple degrees of freedom (Dof) motion to obtain pose change information of the physical keyboard and send the pose change information to the head-mounted display device.

In one embodiment, the initial pose of the physical keyboard in the head-mounted coordinate system refers to the pose of the physical keyboard relative to the head-mounted display device when the head-mounted display device is turned on.

The initial pose of the physical keyboard may be obtained based on electromagnetic positioning principles. When the head-mounted display device is started, an electromagnetic receiver of the head-mounted display device induces an electromagnetic signal transmitted by an electromagnetic transmitter of the physical keyboard and outputs an induction signal; according to the acquired induction signal when the head-mounted display device is started, the initial pose of the physical keyboard can be determined.

And determining a second pose of the physical keyboard in the head-mounted coordinate system according to the initial pose of the physical keyboard in the head-mounted coordinate system and the pose change information of the physical keyboard measured by the inertial measurement unit.

After determining a second position of the physical keyboard in the head-mounted coordinate system, entering:

and S2600, fusing the first pose and the second pose, and determining a third pose of the physical keyboard under a head-mounted coordinate system.

In one embodiment, the first position comprises a first position parameter and a first attitude parameter, and the second position comprises a second position parameter and a second attitude parameter.

In this embodiment, the step of fusing the first pose and the second pose and determining a third pose of the physical keyboard in the head-mounted display device coordinate system may further include: steps S2610-S2630.

Step S2610, calculate a weighted average of the first position parameter of the first pose and the second position parameter of the second pose, to obtain a third position parameter.

In this embodiment, the weights of the first location parameter and the second location parameter may be set according to engineering experience.

Step S2620, a weighted average value of the first attitude parameter of the first pose and the second attitude parameter of the second pose is obtained to obtain a third attitude parameter.

In this embodiment, the weights of the first attitude parameter and the second attitude parameter may be set according to engineering experience.

Step 2630, determining a third pose of the physical keyboard in the head-mounted coordinate system according to the third position parameter and the third posture parameter.

After determining the third posture of the physical keyboard in the head-mounted coordinate system, entering:

step S2800, present the virtual keyboard to the user based on the third pose.

In one embodiment, the third pose under the head-mounted coordinate system is converted into a world coordinate system, and a virtual keyboard is presented to the user based on the converted third pose, so that the virtual scene is matched with the real environment, and the user experience is better.

According to the embodiment of the invention, the first pose of the physical keyboard is determined based on the electromagnetic positioning principle, the second pose of the physical keyboard is determined based on the positioning principle of the inertial measurement unit, and the first pose and the second pose are fused, so that the short-time positioning accuracy of the inertial measurement unit can be utilized to make up for unsmooth pose change of the virtual keyboard obtained by electromagnetic positioning caused by the intensity change of electromagnetic signals, and the image jitter is reduced. The embodiment combines the electromagnetic signal and the inertia measurement unit to position the physical keyboard, so that the positioning accuracy can be improved, and the positioning stability can be ensured.

In an embodiment, the method for displaying a virtual keyboard in a head-mounted display device provided by this embodiment may further include: steps S3200-S3600.

Step S3200, receiving a key value sent by a physical keyboard.

In this embodiment, in response to a user pressing operation on the physical keyboard, the physical keyboard transmits a key value to the head-mounted display device. The key value may reflect a key pressed by a user.

And step S3400, determining the pressed key according to the key value based on the mapping relation between the key of the physical keyboard and the key value.

In one embodiment, a mapping relationship exists between keys of a physical keyboard and key values, when a user presses the keys, the physical keyboard sends the key values corresponding to the pressed keys to a head-mounted display device in a signal mode, and the head-mounted display device determines the pressed keys according to the obtained key values.

Step S3600, light up a corresponding pressed key on the virtual keyboard for a preset lighting time.

In one embodiment, the preset lighting time may be a time during which the pressed key is continuously in a lighting state. The lighting time is too short, so that a user cannot easily observe the pressed keys in the virtual keyboard, the lighting time is too long, and the display effect of the virtual keyboard is influenced when the user continuously presses the keys. The preset lighting time may be set according to practical experience. For example, the lighting time is 0.3 s.

For example, the key value of key a is 10000, the key value of key B is 01000, the key value of key C is 00100, the key value of key D is 00010, and the key value of key E is 00001. When the key value received by the head-mounted display is 00100, it indicates that the key C is pressed, and correspondingly lights up the key C in the virtual keyboard.

According to the embodiment of the invention, the key value sent by the physical keyboard is received, the pressed key is determined according to the key value, and the corresponding pressed key on the virtual keyboard is lightened for the preset lightening time, so that the interaction between a user and the virtual keyboard can be realized, and the user experience can be improved.

In an embodiment, the method for displaying a virtual keyboard in a head-mounted display device provided by this embodiment may further include: steps S4200-S4400.

Step S4200, receiving a vibration signal from a vibration motor of the physical keyboard.

In step S4400, according to the vibration signal, a designated key on the virtual keyboard corresponding to the vibration signal is lit.

In this embodiment, the designated key may be used to implement a particular function. The designated key may be a single key or a combination key. For example, the designated key is a combination key for version upgrade.

In this embodiment, the physical keyboard is further provided with a vibration motor, the vibration motor is used for sending a vibration signal when a preset specified key is pressed, sending the vibration signal to the head-mounted display device, and lighting the specified key corresponding to the vibration signal on the virtual keyboard according to the vibration signal to prompt a user to press a specific key.

In an embodiment, the method for displaying a virtual keyboard in a head-mounted display device provided by this embodiment may further include: steps S5200-S5400.

In step S4200, the touch position information transmitted by the physical keyboard is received.

Step S4400 presents a virtual cursor to the user according to the touch position information.

In this embodiment, the physical keyboard is further provided with a touch pad, and in response to an operation of the user on the touch pad, the physical keyboard may acquire touch position information and send the touch position information to the head-mounted display device, so that the head-mounted display device presents a virtual cursor to the user in a virtual scene.

According to the embodiment of the invention, the first pose of the physical keyboard is determined based on the electromagnetic positioning principle, the second pose of the physical keyboard is determined based on the positioning principle of the inertial measurement unit, and the first pose and the second pose are fused, so that the short-time positioning accuracy of the inertial measurement unit can be utilized to make up for unsmooth pose change of the virtual keyboard obtained by electromagnetic positioning caused by the intensity change of electromagnetic signals, and the image jitter is reduced. The embodiment combines the electromagnetic signal and the inertia measurement unit to position the physical keyboard, so that the positioning accuracy can be improved, and the positioning stability can be ensured.

< second embodiment >

Fig. 3 is a schematic diagram of a display method of a virtual keyboard in a head-mounted display device according to an embodiment of the present specification. The display method of the virtual keyboard provided in this embodiment is implemented by computer technology, and may be implemented by the physical keyboard 120 described in fig. 1.

The embodiment provides a display method of a virtual keyboard in a head-mounted display device, which comprises steps S2100-S2300.

Step S2100 transmits an electromagnetic signal to the head mounted display device through an electromagnetic transmitter.

In this embodiment, the electromagnetic transmitter is mounted on the physical keyboard, and the mounting position of the electromagnetic transmitter may be set empirically, for example, the electromagnetic transmitter may be mounted in the center of the physical keyboard, which facilitates the calculation of the first pose.

The electromagnetic transmitter is provided with orthogonal three-axis coils, and the orthogonal three-axis coils comprise three groups of coils which are vertical to each other. And controlling the three groups of coils to transmit electromagnetic signals with different frequencies so that an electromagnetic receiver of the head-mounted display device can induce the electromagnetic signals transmitted by the electromagnetic transmitter and generate induced electromotive force. The frequencies of the electromagnetic signals transmitted by the three groups of coils of the electromagnetic transmitter are different, so that the electromagnetic signals transmitted by the three groups of coils of the electromagnetic transmitter can be prevented from interfering with each other. The intensity of the electromagnetic signal transmitted by the electromagnetic transmitter can be adjusted according to the distance between the physical keyboard and the head-mounted display equipment so as to ensure that the electromagnetic receiver of the head-mounted display equipment can stably sense the electromagnetic signal.

The electromagnetic transmitter can transmit the electromagnetic signal in real time or can transmit the electromagnetic signal based on the triggering operation of a user.

In one particular example, the electromagnetic transmitter may transmit electromagnetic signals to the head-mounted display device at preset time intervals.

In one particular example, the physical keyboard may receive a request to transmit an electromagnetic signal sent by the head-mounted display device, and control the electromagnetic transmitter to operate to transmit the electromagnetic signal to the head-mounted display device in response to the request to transmit the electromagnetic signal sent by the head-mounted display device.

In one particular example, the electromagnetic transmitter may transmit an electromagnetic signal to the head mounted display device in response to a user's turn-on operation.

In one particular example, the electromagnetic transmitter operates to transmit electromagnetic signals to the head mounted display device in response to user manipulation of the physical keyboard.

In the embodiment of the invention, the electromagnetic transmitter of the physical keyboard transmits an electromagnetic signal to the head-mounted display device, so that the electromagnetic receiver of the head-mounted display device can induce the electromagnetic signal to generate induced electromotive force, and the first pose of the physical keyboard is calculated according to the induced electromotive force, thereby realizing the positioning of the physical keyboard. The embodiment is based on electromagnetic positioning, avoids the problem of shielding of a shielding object in the positioning process, and can realize 360-degree tracking and positioning.

And step S2300, measuring the pose change information of the physical keyboard through the inertial measurement unit, and sending the pose change information to the head-mounted display equipment.

In this embodiment, the inertial measurement unit is installed on the physical keyboard, and the inertial measurement unit is configured to detect and measure acceleration, tilt, shock, vibration, rotation, and multiple degrees of freedom (Dof) motion to obtain pose change information of the physical keyboard, and send the pose change information to the head-mounted display device, so that the head-mounted display device calculates a second pose of the physical keyboard according to the initial pose and the pose change information.

In an embodiment, the method for displaying a virtual keyboard in a head-mounted display device provided by this embodiment may further include: step S3100.

Step S3100, in response to a pressing operation by the user, acquiring a key value, and transmitting the key value to the head-mounted display device.

In this embodiment, a mapping relationship exists between keys of the physical keyboard and key values, when a user presses a key, the physical keyboard sends the key value corresponding to the pressed key to the head-mounted display device in a signal manner, and the head-mounted display device determines the pressed key according to the obtained key value.

For example, the key value of key a is 10000, the key value of key B is 01000, the key value of key C is 00100, the key value of key D is 00010, and the key value of key E is 00001. When the user presses key C, the physical keyboard sends key value 00100 corresponding to key C to the head-mounted display.

According to the embodiment of the invention, the key value is acquired in response to the pressing operation of the user, and the key value is sent to the head-mounted display device, so that the head-mounted display device lights the corresponding pressed key on the virtual keyboard in the preset lighting time, the interaction between the user and the virtual keyboard can be realized, and the user experience can be improved.

In one embodiment, the physical keyboard is further provided with a vibration motor for emitting a vibration signal in case that a preset designated key is pressed.

In this embodiment, the method for displaying a virtual keyboard in the head-mounted display device may further include: step S4100.

Step S4100, responding to the operation of the user on the preset appointed key, the vibration motor generates a vibration signal; the vibration signal is sent to the head mounted display device.

In this embodiment, the designated key may be used to implement a particular function. The designated key may be a single key or a combination key. For example, the designated key is a combination key for version upgrade.

In this embodiment, the physical keyboard is further provided with a vibration motor, the vibration motor is used for sending a vibration signal when a preset specified key is pressed, sending the vibration signal to the head-mounted display device, and lighting the specified key corresponding to the vibration signal on the virtual keyboard according to the vibration signal to prompt a user to press a specific key.

In an embodiment, the physical keyboard is further provided with a touch panel, and the display method of the virtual keyboard in the head-mounted display device provided by the embodiment may further include: step S5100.

Step S5100, in response to an operation of the touch panel by the user, acquires touch position information, and sends the touch position information to the head-mounted display device.

In this embodiment, the physical keyboard is further provided with a touch pad, and in response to an operation of the user on the touch pad, the physical keyboard may acquire touch position information and send the touch position information to the head-mounted display device, so that the head-mounted display device presents a virtual cursor to the user in a virtual scene.

According to the embodiment of the invention, the first pose of the physical keyboard is determined based on the electromagnetic positioning principle, the second pose of the physical keyboard is determined based on the positioning principle of the inertial measurement unit, and the first pose and the second pose are fused, so that the short-time positioning accuracy of the inertial measurement unit can be utilized to make up for unsmooth pose change of the virtual keyboard obtained by electromagnetic positioning caused by the intensity change of electromagnetic signals, and the image jitter is reduced. The embodiment combines the electromagnetic signal and the inertia measurement unit to position the physical keyboard, so that the positioning accuracy can be improved, and the positioning stability can be ensured.

< example >

As shown in fig. 4, the method for displaying a virtual keyboard in a head-mounted display device may include the following steps S401 to S409.

Step S401, transmitting an electromagnetic signal to the head-mounted display device through an electromagnetic transmitter;

step S402, an electromagnetic receiver is used for inducing an electromagnetic signal emitted by an electromagnetic emitter of the physical keyboard so as to determine a first pose of the physical keyboard under a head-mounted coordinate system;

step S403, measuring pose change information of the physical keyboard through an inertial measurement unit, and sending the pose change information to the head-mounted display device;

step S404, receiving pose change information which is sent by the physical keyboard and is obtained by measurement through an inertial measurement unit arranged on the physical keyboard, and determining a second pose of the physical keyboard under a head-mounted coordinate system according to the initial pose and the pose change information of the physical keyboard under the head-mounted coordinate system;

s405, fusing the first pose and the second pose, and determining a third pose of the physical keyboard under a head-mounted coordinate system;

step S406, presenting a virtual keyboard to the user based on the third pose;

step S407, responding to the pressing operation of the user, acquiring a key value, and sending the key value to the head-mounted display device;

step S408, receiving a key value sent by a physical keyboard;

and step S409, determining the pressed key according to the key value based on the mapping relation between the key of the physical keyboard and the key value.

According to the embodiment of the invention, the first pose of the physical keyboard is determined based on the electromagnetic positioning principle, the second pose of the physical keyboard is determined based on the positioning principle of the inertial measurement unit, and the first pose and the second pose are fused, so that the short-time positioning accuracy of the inertial measurement unit can be utilized to make up for unsmooth pose change of the virtual keyboard obtained by electromagnetic positioning caused by the intensity change of electromagnetic signals, and the image jitter is reduced. The embodiment combines the electromagnetic signal and the inertia measurement unit to position the physical keyboard, so that the positioning accuracy can be improved, and the positioning stability can be ensured.

< third embodiment >

Fig. 5 is a schematic diagram of a head-mounted display device provided by an embodiment of the invention.

The head-mounted display device can be a VR device, an AR device, an MR device, and the like.

This embodiment provides a head mounted display device 500 comprising an electromagnetic receiver 501, a first posture determination module 502, a second posture determination module 503, a third posture determination module 504 and a display module 505.

The first pose determination module 502 may be configured to sense, via the electromagnetic receiver 501, an electromagnetic signal emitted by an electromagnetic emitter of the physical keyboard to determine a first pose of the physical keyboard in the head-mounted coordinate system.

In one embodiment, the first pose determination module 502 may be further configured to determine a first distance and a first relative pose between the head mounted display device and the physical keyboard based on the sensing signal output by the electromagnetic receiver;

and determining a first pose of the physical keyboard under the head-mounted coordinate system according to the first distance and the first relative posture.

The second pose determining module 503 may be configured to receive pose change information sent by the physical keyboard and obtained through measurement by an inertial measurement unit disposed on the physical keyboard, and determine a second pose of the physical keyboard in the head-mounted coordinate system according to the initial pose and the pose change information of the physical keyboard in the head-mounted coordinate system.

The third pose determination module 504 may be configured to fuse the first pose and the second pose and determine a third pose of the physical keyboard in the head-mounted display device coordinate system.

In one embodiment, the first position comprises a first position parameter and a first attitude parameter, and the second position comprises a second position parameter and a second attitude parameter.

In this embodiment, the third posture determination module 504 may be further configured to:

calculating a weighted average value of the first position parameter of the first pose and the second position parameter of the second pose to obtain a third position parameter;

calculating a weighted average value of the first attitude parameter of the first pose and the second attitude parameter of the second pose to obtain a third attitude parameter; and the number of the first and second groups,

and determining a third pose of the physical keyboard under a head-mounted coordinate system according to the third position parameter and the third posture parameter.

The display module 505 may be used to present a virtual keyboard to the user based on the third pose.

In one embodiment, the head mounted display device 500 further comprises: a key value receiving module 506, a key determination module 507, and a first lighting module 508.

The key value receiving module 506 may be configured to receive a key value sent by a physical keyboard.

The key determining module 507 may be configured to determine a pressed key according to a key value based on a mapping relationship between keys of a physical keyboard and the key value.

The first lighting module 508 may be configured to light a corresponding pressed key on the virtual keyboard at a preset lighting time.

In one embodiment, the head mounted display device 500 further comprises: a vibration signal receiving module 509 and a second lighting module 510.

The vibration signal receiving module 509 may be used to receive a vibration signal from a vibration motor of the physical keyboard.

The second lighting module 510 may be configured to light a designated key on the virtual keyboard corresponding to the vibration signal according to the vibration signal.

Fig. 6 is a schematic diagram of a head-mounted display device provided by another embodiment of the invention.

The head-mounted display device can be a VR device, an AR device, an MR device, and the like.

The head mounted display device 600 provided by this embodiment includes a processor 601 and a memory 602, where the memory 602 stores therein computer instructions, and the computer instructions are executed by the processor 601 to execute the display method of the virtual keyboard in the head mounted display device of the foregoing first embodiment.

According to the embodiment of the invention, the physical keyboard is positioned by combining the electromagnetic signal and the inertia measurement unit, so that the positioning accuracy can be improved, and the positioning stability can be ensured. According to the embodiment, the accuracy of short-time positioning of the inertia measurement unit can be utilized to make up for unsmooth pose change of the virtual keyboard obtained by electromagnetic positioning caused by intensity change of electromagnetic signals, so that image jitter is reduced. In addition, the head-mounted display device of the embodiment is simple in structure and can be produced in large quantities.

< fourth embodiment >

FIG. 7 is a diagram of a physical keyboard provided by one embodiment of the present invention.

The physical keyboard is communicatively coupled to a head mounted display device, and this embodiment provides a physical keyboard 700 that includes an electromagnetic transmitter 701 and an inertial measurement unit 702.

The electromagnetic transmitter 701 may be used to transmit electromagnetic signals to a head mounted display device.

The inertial measurement unit 702 may be configured to measure pose change information of the physical keyboard and send the pose change information to the head-mounted display device.

In one embodiment, the physical keyboard further comprises a key value acquisition module 703.

The key value obtaining module 703 may be configured to obtain a key value in response to a pressing operation of a user, and send the key value to the head-mounted display device.

In one embodiment, the physical keyboard further comprises a vibration signal acquisition module 704.

The vibration signal acquisition module 704 may be configured to generate a vibration signal by the vibration motor in response to a user operating a preset designated key; the vibration signal is sent to the head mounted display device.

Fig. 8 is a schematic diagram of a physical keyboard provided by another embodiment of the present invention.

The physical keyboard 800 provided in this embodiment includes a processor 801 and a memory 802, and the memory 802 stores therein computer instructions, and the computer instructions are executed by the processor 801 to perform the display method of the virtual keyboard in the head mounted display device according to the foregoing second embodiment.

According to the embodiment of the invention, the physical keyboard is positioned by combining the electromagnetic signal and the inertia measurement unit, so that the positioning accuracy can be improved, and the positioning stability can be ensured. According to the embodiment, the accuracy of short-time positioning of the inertia measurement unit can be utilized to make up for unsmooth pose change of the virtual keyboard obtained by electromagnetic positioning caused by intensity change of electromagnetic signals, so that image jitter is reduced. In addition, the head-mounted display device of the embodiment is simple in structure and can be produced in large quantities.

The above embodiments mainly focus on differences from other embodiments, but it should be clear to those skilled in the art that the above embodiments can be used alone or in combination with each other as needed.

The embodiments in the present disclosure are described in a progressive manner, and the same and similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from the other embodiments, but it should be clear to those skilled in the art that the embodiments described above can be used alone or in combination with each other as needed. In addition, for the device embodiment, since it corresponds to the method embodiment, the description is relatively simple, and for relevant points, refer to the description of the corresponding parts of the method embodiment. The system embodiments described above are merely illustrative.

The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied therewith for causing a processor to implement various aspects of the present invention.

The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.

Computer program instructions for carrying out operations of the present invention may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including AN object oriented programming language such as Smalltalk, C + +, or the like, as well as conventional procedural programming languages, such as the "C" language or similar programming languages.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, by software, and by a combination of software and hardware are equivalent.

Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.