阅读说明：本技术 头戴式显示器、头戴式显示器协作系统及其方法 (Head-mounted display, head-mounted display cooperation system and method thereof ) 是由 渡边光信 桥本康宣 川前治 益冈信夫 吉泽和彦 清水宏 于 2018-06-25 设计创作，主要内容包括：头戴式显示器具备：佩戴体；显示器；通信模块,在与便携信息终端之间通信连接；视觉辨认区域检测传感器,输出用于判断经由佩戴体的使用者的视觉辨认区域中是否包括便携信息终端的状态数据；以及控制装置,与显示器、通信模块及视觉辨认区域检测传感器的每一个连接,控制装置根据状态数据判断使用者的视觉辨认区域中是否包括便携信息终端,对显示器进行显示控制以使得：在判断为视觉辨认区域中未包括便携信息终端的情况下,在显示器中显示便携信息终端的APP画面；在判断为视觉辨认区域中包括便携信息终端的情况下,在显示器中不显示APP画面。(A head-mounted display is provided with: a wearing body; a display; a communication module which is connected with the portable information terminal in a communication way; a viewing area detection sensor that outputs status data for determining whether or not the portable information terminal is included in a viewing area of a user wearing the mobile terminal; and a control device connected to each of the display, the communication module, and the viewing area detection sensor, the control device determining whether the portable information terminal is included in the viewing area of the user based on the state data, and performing display control on the display such that: displaying an APP screen of the portable information terminal on the display when the portable information terminal is determined not to be included in the visual recognition area; when it is determined that the mobile information terminal is included in the viewing area, the APP screen is not displayed on the display.)

1. A head-mounted display is characterized by comprising:

a wearing body worn on the head of a user;

a display provided in front of the eyes of the user wearing the body;

a communication module for performing communication connection with the portable information terminal;

a viewing area detection sensor that outputs status data for determining whether or not the portable information terminal is included in a viewing area of the user via the wearing body; and

a control device connected to each of the display, the communication module, and the viewing area detection sensor,

the control device determines whether the portable information terminal is included in the user's visual recognition area based on the state data, and performs display control on the display such that: displaying an APP screen on the display, the APP screen displaying an execution state of application software that implements a specific function executed by the portable information terminal, if it is determined that the portable information terminal is not included in the viewing area; and when it is determined that the portable information terminal is included in the viewing area, the APP screen is not displayed on the display.

2. The head-mounted display of claim 1,

the visual recognition area detection sensor is an outer camera provided on the front surface of the wearing body so as to include the visual recognition area of the user in the angle of view,

the control device performs image recognition processing of an object captured in an image of the outside camera, and displays the APP screen in the display if the object is not the portable information terminal, and does not display the APP screen in the display if the object is the portable information terminal.

3. The head-mounted display of claim 1,

the visual recognition area detection sensor includes:

an outer camera provided on a front surface of the wearing body so as to include a visual recognition area of the user in a viewing angle;

an inner side camera that photographs the user's eyes in the wearing body on an inner side opposite to the user's head; and

a line-of-sight detection sensor that detects a line of sight of the user based on an image from the inner camera,

the control device performs image recognition processing of an object captured in an image of the outer camera, and displays an APP screen on the display when it is determined that the object is not the portable information terminal and when it is determined that the object is the portable information terminal and it is determined from a detection result of the line-of-sight detection sensor that the portable information terminal is not in the line of sight of the user; and when the object is determined to be the portable information terminal and the portable information terminal is in the line of sight of the user, the APP screen is not displayed on the display.

4. The head-mounted display of claim 1,

the visual recognition area detection sensor includes at least one of a gyro sensor and an acceleration sensor provided on the wearing body,

the control device causes the display to transition from displaying the APP screen to not displaying the APP screen or from not displaying the APP screen to displaying the APP screen when an angular velocity included in an output from the gyro sensor and an acceleration included in an output from the acceleration sensor are equal to or greater than a predetermined threshold value.

5. The head-mounted display of claim 1,

the viewing area detection sensor is an operation detection sensor for detecting a user operation performed by the user,

the control device causes the display to transition from displaying the APP screen to not displaying, or from not displaying the APP screen to displaying, in accordance with an output from the operation detection sensor.

6. The head-mounted display of claim 1,

when the control device activates the head-mounted display after the portable information terminal is activated, the control device performs display control for displaying the APP screen without displaying the home screen displayed during normal activation processing of the head-mounted display on the display.

7. The head-mounted display of claim 2,

the display is divided into a plurality of small areas,

each small area comprising a part of the outer edge of the display,

the outer camera outputs a plurality of frames along a time series,

the control device includes small area management data that associates the small area, an outer edge included in the small area, and a position of an outer edge of an image captured by the outer camera,

the control device performs the image recognition processing on each of a 1 st frame photographed by the outside camera and a 2 nd frame photographed after the 1 st frame, determines a position where the outside edge is passed when it is determined that the portable information terminal photographed in the 1 st frame moves outside the frame through the outside edge of the 2 nd frame, specifies the small region corresponding to the passed position with reference to the small region management data, and displays the APP screen in the specified small region.

8. A head-mounted display cooperation system configured by communicatively connecting a head-mounted display and a portable information terminal,

the head-mounted display includes:

a wearing body worn on the head of a user;

a display provided in front of the eyes of the user wearing the body;

a communication module for performing communication connection with the portable information terminal;

a viewing area detection sensor that outputs status data for determining whether or not the portable information terminal is included in a viewing area of the user via the wearing body; and

a head-mounted display control device connected to each of the display, the communication module, and the viewing area detection sensor,

the portable information terminal includes:

a terminal display for displaying an APP screen that displays an execution state of application software that implements a specific function;

the terminal communication module is used for being in communication connection with the head-mounted display; and

a terminal control device connected to each of the terminal display and the terminal communication module, executing the application software,

the head-mounted display control device determines whether or not the portable information terminal is included in the user's visual recognition area based on the state data, and performs display control of the display such that: displaying an APP screen displayed in the portable information terminal on the display when it is determined that the portable information terminal is not included in the viewing area; and when it is determined that the portable information terminal is included in the viewing area, the APP screen is not displayed on the display.

9. A head-mounted display cooperation method performed between portable information terminals communicatively connected to a head-mounted display,

a step in which the head-mounted display acquires, from a viewing area detection sensor, status data for determining whether or not the portable information terminal is included in a viewing area of a user wearing the head-mounted display;

judging whether the portable information terminal is included in the user's visual recognition area or not, based on the state data;

displaying an APP screen on a display provided in the head-mounted display, the APP screen displaying an execution state of application software that realizes a specific function and is executed by the portable information terminal, when it is determined that the portable information terminal is not included in the viewing area; and

and a step of, when it is determined that the portable information terminal is included in the viewing area, performing display control on the display so that the APP screen is not displayed on the display.

Technical Field

The present invention relates to a display technology of a head mounted display (hereinafter abbreviated as "HMD").

Background

As background art in this field, patent document 1 discloses a system of "calculating positional information of a gaze point of a wearer with respect to primary information from a camera image acquired from a head-mounted display and a gaze point of a wearer of the head-mounted display with respect to the camera image, selecting and displaying secondary information associated with the positional information, and changing a display method (digest) of the displayed secondary information when the wearer observes the primary information and when the wearer does not observe the primary information".

Patent document 2 discloses "a head-mounted display device in which a user can visually recognize a virtual image and an external scene includes: a generation unit that generates a list image including a 1 st image and a 2 nd image included in a head-mounted display device, the 1 st image being a display image of an external device connected to the head-mounted display device; and an image display unit that forms a system showing a virtual image (abstract) of the generated list image.

Disclosure of Invention

In patent documents 1 and 2, after the HMD and the external display device are connected for communication, the HMD shares a display image. Therefore, when the above-described conventional technique is applied to a portable information terminal and an HMD having a communication function, the screen of the portable information terminal is displayed on the HMD when a link between the HMD and the portable information terminal is established, and therefore, even when the user does not desire or does not need it, the screen of the portable information terminal may be displayed on the HMD, which is inconvenient in use.

The present invention has been made in view of the above circumstances, and an object thereof is to provide a technique for cooperating an HMD and a portable information terminal that displays a screen of the portable information terminal on the HMD at a timing useful for a user and that is excellent in operability.

In order to solve the above problem, the present invention has the structure described in the claims. As an example, there is provided a head-mounted display including: a wearing body worn on the head of a user; a display disposed in front of the user's eyes of the wearing body; a communication module for performing communication connection with the portable information terminal; a viewing area detection sensor that outputs status data for determining whether or not the portable information terminal is included in a viewing area of the user via the wearing body; and a control device connected to each of the display, the communication module, and the viewing area detection sensor, the control device determining whether the portable information terminal is included in the viewing area of the user based on the state data, and performing display control on the display such that: displaying an APP screen on the display, the APP screen displaying an execution state of application software that implements a specific function executed by the portable information terminal, if it is determined that the portable information terminal is not included in the viewing area; and when it is determined that the portable information terminal is included in the viewing area, the APP screen is not displayed on the display.

According to the present invention, it is possible to provide a technique for cooperating an HMD and a portable information terminal that displays a screen of the portable information terminal on the HMD at a timing useful for a user and that has good operability. Further, problems, structures, and effects other than those described above will be apparent from the following description of the embodiments.

Drawings

Fig. 1 is a schematic configuration diagram of an HMD cooperation system.

Fig. 2 is a hardware configuration diagram of the HMD.

Fig. 3 is a software configuration diagram of the HMD.

Fig. 4 is a hardware configuration diagram of the portable information terminal.

Fig. 5 is a software configuration diagram of the portable information terminal.

Fig. 6 is a diagram showing a sequence example of the HMD cooperation system in embodiment 1.

Fig. 7 is a flowchart showing the flow of the APP screen display control process of embodiment 1.

Fig. 8 is a diagram showing a positional relationship between the viewing area and the portable information terminal (APP screen is not displayed).

Fig. 9 is a diagram showing a scene via the display when the APP screen is not displayed.

Fig. 10 is a diagram showing a positional relationship (APP screen display) between the viewing area and the portable information terminal.

Fig. 11 is a diagram showing a scene via the display when an APP screen is displayed.

Fig. 12 is a diagram showing a change in APP screen display start conditions according to embodiment 2.

Fig. 13 is a diagram showing the redisplay position after the mirror (frame out) is shown.

Fig. 14 is a diagram showing a sequence example of the HMD cooperation system in embodiment 4.

Fig. 15 is a diagram illustrating screen transition of the HMD.

(symbol description)

1: a wearing body; 2: a display; 3: an outer camera; 4: an inside camera; 5: an HMD control device; 10: HMD; 80: a portable information terminal; 100: an HMD collaboration system.

Detailed Description

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are denoted by the same reference numerals, and redundant description thereof is omitted.

< embodiment 1 >

Embodiment 1 is as follows: the HMD and a smart phone as a portable information terminal are connected in a communication manner, and when the portable information terminal does not enter a visual recognition area of a user wearing the HMD, an APP screen of the portable information terminal is displayed on the HMD. The "APP screen" referred to herein is a screen on which the execution state of application software executed on the portable information terminal is displayed.

Fig. 1 is a schematic configuration diagram of an HMD cooperation system 100. The HMD cooperation system 100 is configured to wirelessly communicate and connect the HMD10 and the portable information terminal 80. The portable information terminal 80 is illustrated as a smartphone, and may be a tablet terminal or a smart watch, and the type thereof is not limited as long as it is a portable image display device.

The HMD10 mainly includes a wearing body 1, a display 2, an outer camera 3, an inner camera 4, and an HMD control device 5.

The attachment 1 is a member for holding members of another HMD10 while maintaining the HMD10 worn on the head of a user M (see fig. 8 and 10) of the HMD 10. The shape is illustrated in fig. 1 as an eye lens shape, but the shape is not limited thereto.

The display 2 is a transmissive liquid crystal display, and is disposed in front of the eyes of the user M wearing the body 1.

When the side of the wearing body 1 facing the face of the user M is defined as the inside and the opposite side is defined as the outside, the outside camera 3 is a camera mounted with a CCD or CMOS and provided on the outside of the front surface of the wearing body 1. The outer camera 3 is provided on the wearing body 1 at a position and orientation including a visual field of the user M in the angle of view.

The inner camera 4 is a camera mounted with a CCD or a CMOS and provided at a position and in a direction in which the movement of the eyes of the user M can be imaged inside the wearing body 1.

The HMD control device 5 is configured using a processor and a circuit that perform communication control with the portable information terminal 80 and display control on the display 2. In the present embodiment, a configuration in which software for realizing the functions of the HMD cooperation method according to the present embodiment is executed using a general-purpose computer will be described as an example.

Fig. 2 is a hardware configuration diagram of HMD 10.

As shown in fig. 2, the HMD10 is configured to connect the HMD control device 5, the imaging unit 120, the voice unit 130, the operation unit 140, the communication module 150, the sensor unit 160, the extension I/F170, and the display unit 180 to each other via the system bus 102.

The HMD control device 5 includes: a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 103, a RAM (Random Access Memory) 104, and a storage device 110 including an HDD (Hard Disk Drive) or an SSD (Solid State Drive) having a semiconductor Memory such as a flash Memory.

The imaging unit 120 includes: an outer camera 3 and an inner camera 4; and an image signal processing unit 121 for converting the output signals of the cameras into image signals.

The voice section 130 includes: a microphone 131 for inputting voice; a voice signal processing section 132 including noise processing and amplification processing of the voice signal from the microphone 131; and a speaker 133 for playing an incoming call or voice.

The operation unit 140 includes operation buttons and operation switches, and is configured by operation members such as an ON/OFF button of a main power supply of the HMD10 and a volume dial. Each of the operation members is a member that receives an operation by a user and detects an input operation, and functions as an operation detection sensor.

The communication module 150 includes: a wireless LAN communication section 151 for connecting to a wireless LAM such as Wi-Fi (registered trademark of japan), infrared communication, or the like; and a near field wireless communication section 152 using Bluetooth (registered trademark in japan) or the like.

The sensor section 160 includes: a GPS (Global Positioning System) 161; a gyro sensor 162 for sensing a change in rotation and orientation; a geomagnetic sensor 163 for measuring azimuth; an acceleration sensor 164 for detecting data relating to tilt or movement, vibration or shock; an illuminance sensor 165 for sensing brightness of the periphery of the HMD 10; a proximity sensor 166; a distance sensor 167; and a line-of-sight detection sensor 168. The gyro sensor 162, the acceleration sensor 164, and the line-of-sight detection sensor 168 are used as visual recognition area detection sensors for detecting a visual recognition area of the user M in each embodiment.

The display unit 180 includes the display 2 and a display control unit 182, and the display control unit 182 generates display data to be displayed on the display 2 and performs output control.

Fig. 3 is a software configuration diagram of HMD 10.

The ROM103 includes a basic operation program 103a of the HMD10 and other programs 103 b. The basic operation program 103a includes a startup program of the HMD10, a display control program of a home screen image (see the home screen image 1501 in fig. 15) displayed in the middle of normal startup processing, and a communication program with the portable information terminal 80.

The RAM104 is an area where the CPU101 loads programs stored in the ROM103 and the storage device 110 during execution. During execution of each program, the RAM104 generates a basic operation execution unit 104a, an information display execution unit 104b, and a temporary storage area 104 c. The information display execution unit 104b includes: screen movement Middleware (Middleware)104b1 for displaying or not displaying the APP screen of the portable information terminal 80 on the HMD 10; and an operation execution section 104b 2.

The screen movement middleware 104b1 is a program for controlling display of an APP screen on the display 2 of the HMD 10. In the present embodiment, the middleware is configured, but may be configured by a background (Daemon) program or a driver program that operates on an OS (operating system). Instead of the screen moving middleware 104b1, the screen moving middleware may be constituted by applications that operate on the HMD10 and correspond to the applications of the personal digital assistant 80.

The storage device 110 stores various information data storage areas 110a, information display programs 110b, and other programs 110 c. The screen movement flag (see fig. 7) described later may be formed in the various information data storage area 110a or in the temporary storage area 104c of the RAM 104. In addition, small area management data, which is managed by dividing the display 2 into a plurality of small areas, which will be described later, is also stored in the various information data storage areas 110 a.

Fig. 4 is a hardware configuration diagram of the portable information terminal 80. The components having the same hardware configuration as that of the HMD10 are denoted by the same names, and are illustrated with reference numerals 201 to 282. The hardware structure differs from that of HMD10 in that: a point not provided with the line-of-sight detection sensor 168 provided in the HMD 10; and a point provided with a telephone network communication unit 253 for connecting the terminal communication module 250 to 3G and 4G lines. The inner camera 224 in the personal digital assistant 80 is: the camera that can be used for self-photographing, which is provided on the same surface (front surface) as the surface on which the terminal display 281 is provided in the portable information terminal 80, is a camera provided on the back surface of the outer camera 223.

Fig. 5 is a software configuration diagram of the portable information terminal 80. The same components as those of the software structure of HMD10 are denoted by the same names, and are shown by reference numerals 201 to 210 c. The software structure differs from that of HMD10 in that: a point at which the screen movement middleware 104b1 is not set; and a point corresponding to the APP204d as a movement of software for displaying the APP screen on the HMD10 in cooperation with the HMD 10. The mobile APP204d is an application installed in each application of the portable information terminal 80 and having a function of displaying an APP screen on the HMD10, and may be a map navigation APP, an electronic book APP, or the like, and the function of the mobile APP is not limited. In the present embodiment, the movement-compatible APP204d is used for description, but each APP may be configured to execute screen movement middleware on the portable information terminal 80, instead of corresponding to screen movement, and to cause the screen displayed on the terminal display 281 of the portable information terminal 80 to be displayed on the HMD 10.

The movement-compatible APP204d may be prepared for high-resolution display and low-resolution display according to the display capabilities of the portable information terminal 80 and HMD10 and the scenes, or may be prepared for detailed display, simplified display, and the like.

The HMD10 and the personal digital assistant 80 may be connected via near field wireless communication or wireless LAN. In addition, the 1-to-1 communication may be performed peer-to-peer, or may be performed via cloud communication.

As a scheme for moving the APP screen from the portable information terminal 80 to HMD10, the APP screen displayed on the terminal display 281 of the portable information terminal 80 may be captured and transmitted. In this case, the still image is moved. In the case of capturing a screen, the screen may be updated periodically. In addition, both the portable information terminal 80 and HMD10 may store the movement-corresponding APP204d in advance, and synchronize the movement-corresponding APP204d stored in HMD10 and the movement-corresponding APP204d stored in portable information terminal 80, without transferring the screen data in cooperation with the operation of the movement-corresponding APP204d, and display the screen having the same content as the screen on the terminal display 281 on the display 2 of HMD 10. Further, the video display may be moved from the movement-corresponding APP204d to the HMD10 by streaming media or the like.

With reference to fig. 6 to 11, an outline of processing of the HMD cooperation system 100 will be described. Fig. 6 is a flowchart showing the flow of processing of the HMD cooperation system 100. Fig. 7 is a flowchart showing the flow of the APP screen display control process according to embodiment 1. Fig. 8 is a diagram showing a positional relationship between the viewing area and the portable information terminal 80 (APP screen is not shown). Fig. 9 shows a scene via the display 2 when the APP screen is not displayed. Fig. 10 is a diagram showing a positional relationship (APP screen display) between the viewing area and the portable information terminal 80. Fig. 11 shows a scene via the display 2 at the time of APP screen display. The following description is made in accordance with the step sequence of fig. 6.

Both the personal digital assistant 80 and the HMD10 are activated (S01, S02). HMD10 and portable information terminal 80 may be activated first or simultaneously, and in the description of fig. 6, portable information terminal 80 is activated first.

The portable information terminal 80 issues a connection request to the HMD10 (S03), and a link is established (S04).

After the link is established, the mobile APP204d of the mobile information terminal 80, which is the application software stored in the mobile information terminal 80, is started (S05), and the mobile APP204d transmits an APP start notification to the HMD10 indicating that the mobile APP204d has been started (S06).

The HMD control apparatus 5 that has received the APP activation notification activates the screen movement middleware 104b1 stored in the HMD10 (S07).

The screen transition middleware 104b1 causes the outer camera 3 of the HMD10 to start shooting, and the outer camera 3 starts outputting an image to the HMD control device 5 (S08). Since the angle of view of the outer camera 3 includes the visual region of the user of the HMD10, in the present description, the imaging region of the outer camera 3 is assumed to be the visual region of the user M of the HMD 10.

The HMD control device 5 executes processing for causing the display 2 to display an APP screen (APP screen display control processing) (S09). In embodiment 1, whether or not the portable information terminal 80 is photographed in the image of the outer camera 3 is used as a display start condition (trigger) of the APP screen.

As shown in fig. 7, the screen movement APP104d1 takes in the image of the outer camera 3 (S091), and executes image recognition processing for recognizing whether or not the object (target) of the image is the portable information terminal 80 (S092). There are several specific algorithms for the image recognition processing, and for example, the front surface of the portable information terminal 80 is photographed by the outer camera 3 in advance, and the front surface image obtained by the photographing is held as training data in advance. Then, the image from the outer camera 3 may be subjected to subject extraction processing, pattern matching processing between the subject and the training data may be executed, and if the degree of coincidence is equal to or greater than a predetermined threshold value, it is determined that the portable information terminal 80 is a subject. If a front surface image is used, processing can also be performed such that: when the portable information terminal 80 is photographed upside down in the image of the outside camera 3, or the portable information terminal 80 is turned upside down and placed in a hidden place while a part of the portable information terminal 80 is shielded, it is not recognized that the object is the portable information terminal 80.

When it is determined that the subject is the portable information terminal 80 (S093/"yes"), that is, when it is determined that the portable information terminal 80 is present in the viewing area as shown in fig. 8, the HMD control apparatus 5 sets the value of the screen movement flag to "0" (S094), and does not display the APP screen on the display 2. In this case, as shown in fig. 9, the user M can visually recognize the personal digital assistant 80 through the display 2 of the HMD 10. Therefore, the user M does not visually recognize the APP screen displayed on the display 2, but directly visually recognizes the APP screen displayed on the terminal display 281 via the display 2.

When it is determined that the subject is not the portable information terminal 80 (S093/"no"), that is, when it is determined that there is no portable information terminal 80 in the viewing area as shown in fig. 10, the HMD control apparatus 5 sets the value of the screen movement flag to "1" (S095), and displays the APP screen on the display 2. In this case, as shown in fig. 11, the user M visually recognizes the APP screen displayed on the display 2.

While the display 2 of HMD10 displays the APP screen, the APP screen of terminal display 281 of portable information terminal 80 may not be displayed, or may continue to be displayed. The process of the portable information terminal 80 causing the APP screen to be displayed or not displayed, or the process of transition from display to sleep operation (operation of turning the screen dark without display) may be executed by the screen control function of the portable information terminal 80, or the screen control function of the movement-compatible APP204d may be provided.

When the end condition of the APP screen control processing is satisfied (S096/"yes"), the screen movement middleware 104b1 ends the APP screen control processing. If the end condition of the APP screen control processing is not satisfied (S096/"no"), the process returns to S091, and the image is repeatedly captured from the outer camera 3. Thus, if the termination condition is not satisfied during link establishment, the user M visually recognizes the actual APP screen of the portable information terminal 80 when viewing the portable information terminal 80 by wearing the HMD10, and displays the APP screen on the HMD10 when moving the portable information terminal 80 away from the visual region, for example, when putting down a hand holding the portable information terminal 80 or putting the portable information terminal 80 in a pocket or a bag.

When the end condition of the APP screen control process is satisfied, the link between HMD10 and portable information terminal 80 may be released, or the link may be maintained and only screen movement middleware 104b1 may be ended. Thus, when the start notification of the movement-corresponding APP204d is sent to the HMD10 again, the movement display of the APP screen can be started immediately.

The termination condition may be, for example, detection of an operation to terminate the screen movement middleware 104b1 or the movement support APP204d, or detection of an operation to turn OFF (OFF) the main power supply of the personal digital assistant 80 or HMD 10.

In fig. 6, the description has been given with the movement-compatible APP204d of the personal digital assistant 80 as "master" and the screen movement middleware 104b1 as "slave", but the screen movement middleware 104b1 may be "master" and the movement-compatible APP204d as "slave". Alternatively, a communication control method in which both are "master" may be used. That is, after the screen movement middleware 104b1 requests the portable information terminal 80 to establish a connection, the start notification of the movement-compatible APP204d may be transmitted from the portable information terminal 80 to the HMD 10.

< embodiment 2 >

The APP screen display start condition of embodiment 2 is different from that of embodiment 1. In embodiment 1, as the display start condition, it is used as a condition whether or not the personal digital assistant 80 is photographed in the image of the outer camera 3, that is, whether or not the personal digital assistant 80 is included in the visual recognition area of the user M, but the display start condition is not limited thereto. Fig. 12 is a diagram illustrating a change in APP screen display start conditions according to embodiment 2.

In embodiment 2, the following conditions are used as APP screen display start conditions.

Setting conditions: has been set to the setting of the display 2 in which the APP screen is automatically displayed on the HMD 10? (S0911). If the setting of automatic display is made, the APP screen may be displayed on the display 2 if the APP screen is not included in the viewing area.

The operating conditions are as follows: is a user operation such as a body button operation, a remote controller operation, or a gesture performed on HMD10 detected? (S0912). The APP screen may be displayed on the display 2 when the operation detection sensor detects an operation by the user.

The body movement conditions are as follows: detects the motion of the HMD10 using the outputs from the gyro sensor 162 and the acceleration sensor 164 of the HMD10 and the images output from the outer camera 3 and the inner camera 4, and virtualizes the motion as the motion of the head (head) of the user, thereby determining whether or not the motion of the head (head) of the user is detected? (S0913). The body movement condition is described as a display start condition for transition to display when the head movement is detected in the non-display state, but when the head movement is detected in the state in which the APP screen is displayed, the head may also be moved in order to observe the portable information terminal 80. When the angular velocity included in the output from the gyro sensor 162 and the acceleration included in the output from the acceleration sensor 164 are equal to or greater than a predetermined threshold value set for determining that the head is moving, the HMD control apparatus 5 causes the APP screen to transition from display to non-display or from non-display to display on the display 2. In the case of using an image, for example, when the difference between images captured at different times before and after is equal to or greater than a predetermined threshold, it may be determined that the place to be observed has changed, that is, the head has moved in order to observe another place, and the APP screen may be transitioned from display to non-display or from non-display to display on the display 2. Fig. 12 illustrates the state before the body motion detection, on the premise that the state is not displayed.

The sight line condition is as follows: is the line of sight detected to be out of sight from the portable information terminal 80? (S0914). The image recognition processing is performed on the image of the outside camera 3, and it is detected in which region of the image the portable information terminal 80 is displayed. Since the outer camera 3, the inner camera 4, and the display 2 are fixed to the wearing body 1, the relationship between the position (pixel position) in each image included in the image of the outer camera 3 and the display position of the display 2 is calibrated in advance in the display 2, and the outer camera position matching data is generated and stored.

In addition, the visual line detection sensor 168 detects the eyes of the user from the image taken by the inner camera 4 and tracks the change in the positions of the eyes, thereby detecting the movement of the visual line. Since the relationship between the position in the image corresponding to the inner camera 4 and the display position of the display 2 is also correlated, inner camera registration data that correlates these is generated and stored in advance.

Then, the image of the outer camera 3 (where the portable information terminal 80 is captured) and the image of the inner camera 4 (where the user's eyes are captured) are respectively compared with the outer camera registration data and the inner camera registration data, thereby detecting that the line of sight of the user M is out of the portable information terminal 80. Further, the APP screen may be made non-display when the portable information terminal 80 is included in the viewing area and the line of sight is on the terminal display 2, and may be made display on the display 2 when the portable information terminal 80 is not included in the viewing area and the line of sight is not on the terminal display 2. In this case, when the portable information terminal 80 is not included in the original viewing area, the APP screen may be displayed on the display 2.

As shown in fig. 12, when the screen movement middleware 104b1 determines that any one of the setting condition (S0911), the operation condition (S0912), the body movement condition (S0913), and the line of sight condition (S0914) is satisfied (yes in any one of S0911 to S0914), the value of the APP screen display flag is set to 1(S0916), and the processing is terminated. On the other hand, when it is determined that any one of the display start conditions is not satisfied (S0911 to S0914) (S0911 to S0914/all "no"), the value of the APP screen display flag is set to 0(S0915), and the process proceeds to a process of determining whether or not the end condition of the APP screen control process is satisfied (S0917). The order of determining whether or not the display start conditions are satisfied is not limited to the order of fig. 12. In fig. 12, the process proceeds to step S0916 when any one of the conditions is satisfied, but the process may proceed to step S0916 when at least one of the conditions is satisfied, that is, when any one of the conditions is satisfied or when a combination of a plurality of the conditions is satisfied.

According to the present embodiment, since the display or non-display of the APP screen can be controlled using the display start condition different from the presence or absence of the portable information terminal 80 in the viewing area described in embodiment 1, it is possible to provide an operation environment suitable for the purpose and the situation of use, and it is possible to provide an operation environment suitable for the preference of the user M by setting the condition by the user M. In embodiment 2, the presence or absence of the portable information terminal 80 in the viewing area may be combined with the display start condition.

< embodiment 3 >

Embodiment 3 is an embodiment showing a change in the display position of an APP screen. Various schemes are considered as to where the APP screen is displayed on the display 2 of the HMD 10. The following examples are given.

(1) Display to set position: the APP screen can be displayed at a predetermined position, for example, the right end of the display 2. Further, the HMD control apparatus 5 may store in advance the position at which the APP screen is displayed when the APP screen display control processing (S09) is ended the previous time, and may display the APP screen at the last end position when the APP screen display control processing (S09) is restarted next.

(2) Display to a position corresponding to the kind of APP: which position to display on the display 2 can be predetermined according to the type of the movement-corresponding APP204 d. For example, if it is a map navigation APP, in order to secure a visual field in front of the eyes, an APP screen may be displayed on the right or left end of the display 2 away from the center of the line of sight. On the other hand, if it is an electronic book APP, an APP screen or the like is displayed in the center of the display 2 for reading and writing, and the display position can be changed according to the function of the mobile APP204 d.

(3) And (3) tracking the sight line: the line of sight detection sensor 168 may continuously monitor the line of sight of the user M from the image of the inner camera 4, and display the APP screen at a position where the display 2 intersects the line of sight. For example, when the movement support APP204d is a map navigation APP, when walking with the HMD10, the map is displayed in the line of sight, and the road indicated by navigation and the actual road are easily recognized, so that the navigation information is easily understood.

(4) Regarding the redisplayed position after the portable information terminal 80 that is within the field of view of the HMD10 departs from the field of view: fig. 13 is a diagram showing the redisplay position after the mirror (frame out) is shown. When the outer camera 3 generates a plurality of frames (frames) along a time series and captures a video as shown in fig. 13, the sides forming the outer edges of the frames constituting the video, the sides a, b, c, d of the display 2, and the small regions 1, 2, 3, 4 of the display 2 are associated in advance. The small regions 1, 2, 3, and 4 are regions adjacent to the sides a, b, c, and d, respectively. The screen moving middleware 104b1 can acquire the side (outer edge) through which the portable information terminal 80 captured in the 1 st frame passes in the 2 nd frame captured thereafter and display the APP screen in the small area of the display 2 corresponding to the side.

For example, as shown in fig. 13, when the user M is looking out from the side a of the display 2 when the user M is looking out from the top of the frame captured by the outer camera 3, the APP screen is displayed in the small area 1. In addition, when the user M is looking out from the bottom side of the frame photographed by the outer camera 3, the user M looks to look out from the side c of the display 2, and therefore the APP screen is displayed in the small area 3.

In addition, it is also possible to acquire a position displayed immediately before the portable information terminal 80 moves to the outside of the display 2, and display the APP screen at the position.

(5) Position adjustment of an APP picture: a unit for adjusting the display position after moving the APP screen into HMD10 may also be provided. For example, the following may be configured: when a slide operation is performed on the terminal display 281 of the portable information terminal 80, the APP screen of the display 2 of the HMD10 changes the display position in the display 2 in conjunction with the slide operation.

According to the present embodiment, since the display position of the APP screen can be adjusted according to the usage and the preference of the user, the HMD collaboration system 100 with higher visibility can be provided.

< embodiment 4 >

Embodiment 4 relates to display control of an HMD top screen when the HMD10 is started after the movement-supporting APP204d is started. Fig. 14 is a diagram showing a sequence example of the HMD cooperation system 100 in the present embodiment. Fig. 15 is a diagram illustrating screen transition of HMD 10.

When the HMD10 is normally activated, the home screen 1501 of the HMD10 is displayed on the display 2, and a screen related to the processing of the HMD10 is displayed next.

However, in the HMD cooperation system 100, as shown in fig. 14, after the portable information terminal 80 is started (S11), the movement-corresponding APP204d is started (S12), and after a connection request is issued (S13) and a movement-corresponding APP start notification is transmitted (S14), if HMD10 is started (S15), the first screen 1501 is not displayed, and instead, the cooperation notification screen 1502 transitions to the APP-screen-in-display screen 1503 or the APP-screen-non-display standby screen 1504.

That is, in the HMD start-up (S15), the display 2 displays the top screen 1501 in the normal start-up process, but when the movement-associated APP start-up notification is received and the HMD10 is started up, the cooperation process between the HMD10 and the portable information terminal 80 is continued thereafter, so that the top screen 1501 is not displayed, a link is established (S16), and the screen movement middleware 104b1 is started (S17). The screen movement middleware 104b1 may receive the link establishment, display the cooperation notification screen 1502, start shooting with the outside camera 3 (S18), and advance the process to the APP screen display control process (S19).

In the present embodiment, the following usage scenarios are envisaged: after the portable information terminal 80 is operated to start using the movement-corresponding APP204d, the HMD10 is started by the user M who wants to perform screen cooperation in the HMD10, but in this case, the user M desires to display the APP screen earlier in the HMD 10. By displaying cooperation notification screen 1502 without displaying home screen 1501 as described above, it is possible to reduce the feeling of anxiety while waiting for display of the APP screen without causing discomfort to user M by viewing home screen 1501 that is not intended.

Further, as processing for alleviating anxiety, the cooperation notification screen 1502 may be also skipped and the APP screen display screen 1503 may be displayed. However, when the APP screen display start condition is not satisfied, the APP screen display screen 1503 is not displayed, and therefore the user M cannot confirm whether or not cooperation is possible. Therefore, if the APP screen display start condition is satisfied, the APP screen display mid-screen 1503 may be displayed immediately after the activation of the HMD10, and the cooperation notification screen 1502 may be displayed only if the APP screen display start condition is not satisfied. In this case, the screen movement middleware 104b1 may be configured to: the cooperation notification screen 1502 is not displayed in step S17, and in the case where it is determined by the processing in the APP screen display control processing (S19) that the APP screen is not displayed in the display 2, the cooperation notification screen 1502 is displayed.

According to the present embodiment, in the HMD cooperation system 100, the home screen of the HMD10 is not displayed when unnecessary, and the anxiety of the user M can be alleviated to improve convenience.

The above embodiments are not intended to limit the present invention, and various modifications may be made without departing from the scope of the present invention, and they also fall within the technical scope of the present invention. For example, the above embodiments 1 to 4 may be arbitrarily combined.