阅读说明：本技术 体验提供系统、体验提供方法及体验提供程序 (Experience providing system, experience providing method, and experience providing program ) 是由 安井裕司 德永诗园 渡边将行 西冈一树 广畑雄大 山根久和 下山田贵行 于 2018-06-15 设计创作，主要内容包括：体验提供系统具备：乘客信息检测部,其检测与车辆的乘客相关的信息；外界信息检测部,其检测所述车辆的外部的信息；输出部,其输出声音；假想影像生成部,其输出显示；以及提供部,其基于所述乘客信息检测部及所述外界信息检测部的检测结果,使由所述输出部输出的声音与由所述假想影像生成部输出的所述显示同步而动态地提供针对所述车辆的乘客的互动。(The experience providing system is provided with: a passenger information detection unit that detects information relating to a passenger of the vehicle; an external information detection unit that detects information outside the vehicle; an output unit that outputs sound; a virtual image generating unit for outputting and displaying; and a providing unit that dynamically provides interaction with a passenger of the vehicle in synchronization with the display output by the virtual image generating unit based on detection results of the passenger information detecting unit and the external world information detecting unit.)

1. An experience providing system is provided with:

a passenger information detection unit that detects information relating to a passenger of the vehicle;

an external information detection unit that detects information outside the vehicle;

an output unit that outputs sound;

a virtual image generating unit for outputting and displaying; and

and a providing unit that dynamically provides interaction with the occupant of the vehicle in synchronization with the display output by the virtual image generating unit based on detection results of the occupant information detecting unit and the external world information detecting unit.

2. The experience provision system of claim 1,

the virtual image generation unit presents a virtual image so as to be superimposed on a captured image corresponding to a seat of the vehicle.

3. The experience provision system of claim 1,

the virtual image generating unit presents a virtual image so as to correspond to a seat mirror position of the vehicle when viewed from the passenger.

4. The experience provision system of claim 1,

the providing unit adjusts the timing of speaking through the output unit based on the condition of the outside of the vehicle detected by the outside environment information detecting unit.

5. The experience provision system of claim 1,

the external information detection unit has a detection capability of information outside the vehicle that is substantially equal to a detection capability of a passenger of the vehicle of information outside the vehicle.

6. The experience provision system of claim 1,

the interaction provided by the providing section includes a sound corresponding to a musical composition heard in the vehicle detected by the passenger information detecting section.

7. The experience provision system of claim 1,

the providing unit synchronizes the music outputted from the output unit with the display outputted from the virtual image generating unit.

8. The experience provision system of claim 7,

the passenger information detection unit includes a storage unit that detects and stores a state of a passenger of the vehicle with respect to the interaction provided by the providing unit, and the passenger information detection unit changes the interaction available to the passenger of the vehicle based on the state of the storage unit.

9. The experience provision system of claim 1,

the vehicle is an autonomous vehicle and the vehicle is,

the providing unit is configured to determine whether the passenger information detecting unit is in an automatic driving period of the vehicle, and to limit at least a part of operation patterns of the interaction during a manual driving period.

10. The experience provision system of claim 1,

the providing portion performs the interaction using a determining portion provided outside the vehicle.

11. The experience provision system of claim 1,

the interaction is provided by an output of an external processing unit that processes signals detected by the passenger information detecting unit and the external information detecting unit outside the vehicle.

12. The experience provision system of claim 1,

the passenger information detection portion is configured to be able to detect the interaction provided by the providing portion,

the providing section dynamically provides an interaction with respect to an occupant of the vehicle based on detection results of the occupant information detecting section and the external world information detecting section including the provided interaction.

13. The experience provision system of claim 1,

when there are a plurality of the interactions that the providing part should provide,

the providing unit determines the execution timing or the execution availability of the plurality of interactions based on the detection results of the passenger information detecting unit and the external information detecting unit.

14. The experience provision system of claim 1,

the experience providing system includes an event determining unit that detects occurrence of an event based on detection results of the passenger information detecting unit and the external information detecting unit,

the event determination unit includes:

a local determination unit that determines an event on the vehicle; and

and a remote determination unit that determines an event by an external server connected to the vehicle in communication.

15. The experience provision system of claim 1,

the experience providing system is provided with:

an event determination unit that detects occurrence of an event based on detection results of the passenger information detection unit and the external information detection unit; and

an interaction control section that controls the interaction provided by the providing section based on detection results of the passenger information detecting section and the external information detecting section,

the event determination unit continues to detect the occurrence of the event in parallel with the execution period of the interactive control unit, and,

the providing portion dynamically provides the interaction based on an occurrence of the event detected during provision of the interaction.

16. An experience provision method, wherein,

the vehicle mount computer performs the following operations:

detecting passenger information;

detecting external information;

outputting sound;

outputting and displaying; and

based on the passenger information and the external information, the output sound and the output virtual image are synchronized to dynamically provide interaction for the passengers of the vehicle.

17. An experience providing program, wherein,

causing the vehicle mount computer to perform the following:

detecting passenger information;

detecting external information;

outputting sound;

outputting and displaying; and

based on the passenger information and the external information, the output sound and the output virtual image are synchronized to dynamically provide interaction for the passengers of the vehicle.

Technical Field

The present invention relates to an experience providing system, an experience providing method, and an experience providing program.

The present application claims priority based on japanese patent application No. 2017-118702, filed in japan on 16/6/2017, the contents of which are incorporated herein by reference.

Background

Conventionally, studies have been made on automatic driving in which at least one of speed control and steering control is automatically performed. In connection with this, a technique of displaying an image having a depth so that an observer can feel a sense of depth is disclosed (for example, see patent document 1).

Prior art documents

Patent document

Patent document 1: japanese patent laid-open publication No. 2017-30737

Disclosure of Invention

Problems to be solved by the invention

In the conventional art, contents concerning pleasure of the occupant of the vehicle are not considered.

The present invention has been made in view of such circumstances, and an object thereof is to provide an experience providing system, an experience providing method, and an experience providing program that can provide a vehicle occupant with a conventionally-unavailable enjoyment.

Means for solving the problems

The experience providing system, the experience providing method, and the experience providing program according to the present invention adopt the following configurations.

(1): one aspect of the present invention relates to an experience providing system including: a passenger information detection unit that detects information relating to a passenger of the vehicle; an external information detection unit that detects information outside the vehicle; an output unit that outputs sound; a virtual image generating unit for outputting and displaying; and a providing unit that dynamically provides interaction with a passenger of the vehicle in synchronization with the display output by the virtual image generating unit based on detection results of the passenger information detecting unit and the external world information detecting unit.

(2): in the aspect (1), the virtual image generation unit may present the virtual image so as to overlap the captured image corresponding to the seat of the vehicle.

(3): in the aspect of (1), the virtual image generation unit may present a virtual image so as to correspond to a seat mirror position of the vehicle when viewed from the passenger.

(4): in the aspect of (1), the providing unit adjusts the timing of speaking through the output unit based on the condition of the outside of the vehicle detected by the outside environment information detecting unit.

(5): in the aspect of (4), the external information detection unit may have a capability of detecting information outside the vehicle that is substantially equal to a capability of detecting information outside the vehicle that is detected by a passenger of the vehicle.

(6): in the aspect of (1), the interaction provided by the providing section includes a sound corresponding to a musical composition heard in the vehicle detected by the passenger information detecting section.

(7): in the aspect (1), the providing unit synchronizes the music output by the output unit with the display output by the virtual image generating unit.

(8): in the aspect of (7), the passenger information detection unit includes a storage unit that detects and stores a state of a passenger of the vehicle with respect to the interaction provided by the providing unit, and the passenger information detection unit changes the interaction available to the passenger of the vehicle based on the state of the storage unit.

(9): in the aspect of (1), the vehicle is an automatically driven vehicle, and the providing unit is configured to determine whether or not the passenger information detecting unit is in an automatic driving period of the vehicle, and to limit at least a part of the operation pattern of the interaction during manual driving.

(10): in the aspect of (1), the providing section performs the interaction using a judging section provided outside the vehicle.

(11): in the aspect (1), the interaction is provided by an output of an external processing unit that processes signals detected by the passenger information detecting unit and the external information detecting unit outside the vehicle.

(12): in the aspect of (1), the passenger information detection unit is configured to be able to detect an interaction provided by the providing unit, and the providing unit dynamically provides the interaction with the passenger of the vehicle based on detection results of the passenger information detection unit and the external world information detection unit including the provided interaction.

(13): in the aspect of (1), when there are a plurality of interactions to be provided by the providing unit, the providing unit determines the execution timing or the execution availability of the plurality of interactions based on the detection results of the passenger information detecting unit and the external world information detecting unit.

(14): in the aspect of (1), the experience providing system may further include an event determination unit that detects occurrence of an event based on detection results of the passenger information detection unit and the external world information detection unit, and the event determination unit may include: a local determination unit that determines an event on the vehicle; and a remote determination unit that determines an event by an external server connected to the vehicle in communication.

(15): in the aspect of (1), the experience providing system includes: an event determination unit that detects occurrence of an event based on detection results of the passenger information detection unit and the external information detection unit; and an interaction control unit that controls the interaction provided by the providing unit based on detection results of the passenger information detection unit and the external world information detection unit, wherein the event determination unit continues to detect the occurrence of the event in parallel with an execution period of the interaction control unit, and the providing unit dynamically provides the interaction based on the occurrence of the event detected during the provision period of the interaction.

(16): one aspect of the present invention relates to an experience providing method, wherein an on-board computer performs the following operations: detecting passenger information; detecting external information; outputting sound; outputting and displaying; and synchronizing the outputted sound with the outputted virtual image based on the passenger information and the external information to dynamically provide interaction to a passenger of the vehicle.

(17): one aspect of the present invention relates to an experience providing program in which an in-vehicle computer is caused to perform the following operations: detecting passenger information; detecting external information; outputting sound; outputting and displaying; and synchronizing the outputted sound with the outputted virtual image based on the passenger information and the external information to dynamically provide interaction to a passenger of the vehicle.

Effects of the invention

According to (1), (5), (14), (16) or (17), it is possible to provide the passengers of the vehicle with fun which has not been available in the past.

According to (2), the passenger of the vehicle can be given the illusion that the virtual image is riding on the vehicle.

According to (3), the passenger of the vehicle can be caused to make a visual error in which the virtual image is seated on the seat of the vehicle.

According to (4), the speech and the motion of the virtual image can be provided to the passenger with the fun of dynamically changing according to the conditions of the passenger of the vehicle and the conditions outside the vehicle.

According to (6), the virtual image can be made to make a speech that can obtain resonance of the occupant of the vehicle.

According to (7) or (8), it is possible to provide the passengers of the vehicle with interactive fun which has not been available before.

According to (9), safety during manual driving of the vehicle can be improved.

According to (10), by performing the interaction using the determination section provided outside the vehicle, it is possible to achieve cost reduction of the vehicle as compared with the case where the determination section is provided inside the vehicle.

According to (11), the interaction is provided by the output of the external processing portion that processes the signals detected by the passenger information detecting portion and the external information detecting portion outside the vehicle, so that the cost reduction of the vehicle can be achieved.

According to (12) or (13), the interaction more suitable for the state of the passenger or the environment around the passenger can be provided to the passenger.

According to (15), the providing section can appropriately provide the interaction to the passenger even during the provision of the interaction.

Drawings

Fig. 1 is a block diagram of an experience providing system.

Fig. 2 is a diagram showing an example of an in-vehicle device including a passenger information detection unit.

Fig. 3 is a diagram showing an example in which the projection device projects a virtual image and a virtual seat on the front window.

Fig. 4 is a diagram showing an example in which the projection device projects a virtual image and a virtual seat on the left window.

Fig. 5 is a diagram showing an example in which the projection device projects a virtual image and a virtual seat on the rear view mirror.

Fig. 6 is a diagram showing an example in which the hologram projection device projects a virtual image on an empty seat of a vehicle.

Fig. 7 is a diagram showing an example of a face of a virtual image displayed on the touch panel display.

Fig. 8 is a diagram showing an example of a face on which a virtual image is displayed by the transmissive plate type display.

Fig. 9 is a flowchart showing an example of a flow of processing and the like in which the speech generating unit generates original data of a speech regarding a virtual image of a sightseeing spot.

Fig. 10 is a flowchart showing an example of a flow of processing and the like of the speech generating unit to generate the original data of the speech of the virtual image prompting the passenger of the vehicle to sing.

Fig. 11 is a sequence diagram for explaining an example of processing in the experience providing server and the vehicle.

Fig. 12 is a structural diagram of a vehicle having an automatic driving function.

Fig. 13 is a diagram for explaining a processing procedure of the automatic driving.

Fig. 14 is a diagram showing an example of a functional configuration included in the experience providing system 1A.

Fig. 15 is a sequence diagram showing the flow of processing in the case where interaction is performed in the vehicle.

Detailed Description

Embodiments of an experience providing system, an experience providing method, and an experience providing program according to the present invention will be described below with reference to the drawings. An experience providing system is a system that provides an interaction (e.g., entertainment content) to a passenger of a vehicle for the passenger of the vehicle to enjoy. The interaction includes a sound corresponding to a musical composition heard in the vehicle.

[ integral Structure ]

Fig. 1 is a block diagram of an experience providing system 1. The experience providing system 1 includes, for example, a content providing server 100 that provides content, and a vehicle 200 that receives the provision of content from the content providing server 100 and provides the content to passengers. These components can communicate with each other via a network NW. The Network NW includes the internet, a WAN (Wide Area Network), a LAN (Local Area Network), a public line, a provider device, a private line, a wireless base station, and the like.

[ vehicle ]

First, the vehicle 200 will be explained. The vehicle 200 includes, for example, a communication unit 210, a passenger information detection unit 220, an external world information detection unit 230, a providing unit 240, an audio input unit 250, a virtual image selection unit 260, a content selection unit 270, and a storage unit 280.

The communication unit 210 is, for example, a wireless communication module for connecting to the network NW. The communication unit 210 performs wireless communication based on Wi-Fi, DSRC (Dedicated Short Range Communications), Bluetooth (registered trademark), or other communication standards. As the communication unit 210, a plurality of communication units corresponding to the applications may be prepared. The communication unit 210 communicates with the content providing server 100 via the network NW.

The passenger information detection unit 220 detects information related to a passenger of the vehicle 200. The information related to the passenger refers to, for example, the expression of the passenger, the presence or absence of speech of the passenger, whether or not the passenger is seated in a seat in the vehicle, and the like. The passenger information detection unit 220 includes, for example, a part or all of an in-vehicle camera, an in-vehicle microphone, a seating sensor, and the like. The passenger information detection unit 220 detects information related to the passengers of the vehicle 200 at a predetermined cycle, for example, and transmits the detected information to the content providing server 100 via the communication unit 210 and the network NW.

The external information detection unit 230 detects information outside the vehicle 200. The external information is, for example, a scene outside the vehicle, a sound outside the vehicle, and the like. The external information detection unit 230 includes, for example, a part or all of an external camera, an external microphone, and the like. External information detection unit 230 detects information outside vehicle 200 at a predetermined cycle, for example, and transmits the detected information to content providing server 100 via communication unit 210 and network NW.

The providing unit 240 includes, for example, a virtual image generating unit 242, an audio output unit 244, and a content playback unit 246. The virtual image generation unit 242 generates a virtual image of a person that makes the passenger of the vehicle 200 enjoy, for example. The virtual image generation unit 242 presents (displays) a virtual image of a person visually recognizable by the passenger of the vehicle 200 based on the raw data received from the content providing server 100. The virtual image generator 242 includes a part or all of a projector, a window, a mirror, a display, a hologram projector, and the like, for example.

The sound output unit 244 outputs, for example, a sound for making the occupant of the vehicle 200 pleasant. The audio output unit 244 outputs audio based on the raw data received from the content providing server 100. The sound output unit 244 includes, for example, a speaker.

The content playback unit 246 provides the contents for the passenger of the vehicle 200 in which the sound output by the sound output unit 244 and the operation of the virtual image presented by the virtual image generation unit 242 are synchronized (or coordinated). That is, the content playback unit 246 synchronizes the audio (e.g., music) output from the audio output unit 244 with the display (e.g., the operation of the virtual image) output from the virtual image generation unit 242. The content playback unit 246 coordinates (i.e., interlocks) the operation of the sound with the operation of the virtual video by using, for example, the mmd (mikumikudance) described in japanese patent application laid-open No. 2016-.

The voice input unit 250 receives an input such as a singing voice of a passenger of the vehicle 200. The voice input unit 250 includes, for example, a microphone for singing. The voice input unit 250 transmits a signal (data) indicating the singing voice of the passenger of the vehicle 200 to, for example, the singing practice scoring unit 134 of the content providing server 100 via the communication unit 210 and the network NW.

The virtual image selecting unit 260 receives selection of a virtual image of a person presented by the virtual image generating unit 242 by a passenger of the vehicle 200. The virtual image selecting unit 260 transmits data indicating the received selected content to the content providing server 100.

Content selection unit 270 receives a selection of a content provided by content providing server 100 by a passenger of vehicle 200. Content selector 270 transmits data indicating the received selected content to content providing server 100.

The storage unit 280 is implemented by an HDD (Hard Disk Drive), a flash Memory, a RAM (Random access Memory), a ROM (Read Only Memory), or the like. The storage unit 280 stores information received from the content providing server 100, information related to the passenger detected by the passenger information detection unit 220, information outside the vehicle 200 detected by the external information detection unit 230, and data such as the singing voice of the passenger of the vehicle 200 received by the voice input unit 250.

Fig. 2 is a diagram showing an example of the in-vehicle device including the passenger information detection unit 220. In the example of fig. 2, the passenger information detection unit 220 includes an in-vehicle camera 220A, an in-vehicle microphone 220B, and a seating sensor 220C. The in-vehicle camera 220A captures, for example, the face of the passenger and outputs an image. The in-vehicle camera 220A transmits the data of the captured image to the content providing server 100.

The in-vehicle microphone 220B acquires sound in the vehicle. The in-vehicle microphone 220B transmits the acquired data of the in-vehicle sound to the content providing server 100.

The seating sensor 220C detects whether or not a passenger is seated on the seat on which the seating sensor 220C is provided, based on the pressure applied to the seating sensor 220C. The seating sensor 220C transmits a signal (data) indicating whether or not a passenger is seated to the content providing server 100.

In the example of fig. 2, the external world information detection unit 230 includes vehicle exterior cameras 230A to 230C and a vehicle exterior microphone 230D. The external information detection unit 230 may include an illuminance sensor, a raindrop sensor, a humidity sensor, a navigation device including a GNSS receiver and map information, a gyro sensor, a vehicle speed sensor, and the like.

The exterior camera 230A captures an image of a scene in front of the vehicle 200. The off-vehicle camera 230A transmits data of the captured image of the scene to the content providing server 100.

The vehicle exterior camera 230B captures an image of the scene on the left side of the vehicle 200. The exterior camera 230B transmits the data of the captured image of the scene to the content providing server 100.

The exterior camera 230C captures an image of the scene on the right side of the vehicle 200. The off-vehicle camera 230C transmits the data of the captured image of the scene to the content providing server 100.

The exterior microphone 230D acquires sound outside the vehicle. The external microphone 230D transmits the acquired external sound data to the content providing server 100.

In the example of fig. 2, the virtual image generating unit 242 includes a projection device 242A, a front window 242B, a left side window 242C, a rear view mirror 242E, a hologram projection device 242G, a touch panel display device 242H, and a transmissive plate type display 242I.

The projection device 242A projects a virtual image on the front window 242B, the left window 242C, or the rear mirror 242E, and presents a virtual image visually recognizable to the passenger of the vehicle 200. The hologram projection device 242G projects a virtual image (hologram) on an empty seat, and presents a virtual image that can be visually confirmed by the passenger of the vehicle 200. The projection device 242A or the hologram projection device 242G presents a virtual image at a scale of at least 70% or more. The projection device 242A or the hologram projection device 242G presents a virtual image that creates an illusion that the occupant of the vehicle 200 is seated on the seat of the vehicle 200.

Fig. 3 is a diagram showing an example in which the projector 242A projects a virtual image VM and a virtual seat VS on the front window 242B. In the example of fig. 3, the projection device 242A presents a virtual image VM seated on a virtual seat VS on the front window 242B. That is, the virtual image generating unit 242 presents the virtual image so that the photographed images corresponding to the seat are superimposed.

Fig. 4 is a diagram showing an example in which the projector 242A projects the virtual image VM and the virtual seat VS on the left window 242C. In the example of fig. 4, the projection device 242A, which is rotatable in the horizontal direction as indicated by the double-headed arrow in fig. 2 and is capable of changing the projection direction, presents a virtual image VM seated on the virtual seat VS in the left window 242C.

Fig. 5 is a diagram showing an example in which the projector 242A projects the virtual image VM and the virtual seat VS on the rear mirror 242E. In the example of fig. 5, the projector 242A presents a virtual image VM on the rear view mirror 242E while being seated on the virtual seat VS.

Fig. 6 is a diagram showing an example in which the hologram projector 242G projects a virtual image VM on an empty seat S of the vehicle 200. In the example of fig. 6, the hologram projection device 242G presents a virtual image VM that can be visually confirmed by the passenger of the vehicle 200 on the vacant seat S of the vehicle 200.

Returning to fig. 2, touch panel display device 242H displays a virtual image on the screen, and presents a virtual image that can be visually confirmed by the occupant of vehicle 200. The touch panel display device 242H displays a virtual image on a scale of at least 70% or more, for example.

Fig. 7 is a diagram illustrating an example in which the touch panel display device 242H displays the face of the virtual image VM. In the example of fig. 7, the touch panel display device 242H presents the face of the virtual image VM on the screen of the touch panel display device 242H.

Returning to fig. 2, the transmissive plate type display 242I is disposed, for example, between the driver seat and the passenger seat. The transmissive plate type display 242I displays a virtual image on the screen, and presents the virtual image visually recognizable to the passenger of the vehicle 200. The transmissive plate type display 242I displays a virtual image on a scale of at least 70% or more, for example. The transmissive panel type display 242I is, for example, a transparent LED display in which LEDs are arranged on a transparent substrate, an OLED (Organic Light Emitting Diode) having a structure in which an Organic EL (Electroluminescence) element having a laminated film in which a functional Organic material is sandwiched between 2 electrodes is arranged on a transparent substrate, or the like. The transmissive panel display 242I may be a head-up display type display having a projection device embedded in the headrest of the driver's seat and a polarizing plate, for example.

Fig. 8 is a diagram showing an example in which the transmissive plate type display 242I displays the face of the virtual image VM. In the example of fig. 8, the transmissive plate display 2421 presents the face of the virtual image VM on the screen of the transmissive plate display 242I. That is, the virtual image generating unit 242 presents the virtual image so as to correspond to the seat mirror image position of the vehicle when viewed from the passenger.

In the example of fig. 2, the audio output portion 244 includes a left front speaker 244A, a right front speaker 244B, a left rear speaker 244C, and a right rear speaker 244D.

The voice input unit 250 includes, for example, a microphone 250A for singing.

The virtual image selection unit 260 includes, for example, a touch panel display device 242H. The passenger of the vehicle 200 selects the virtual image presented by the virtual image generating unit 242 from among the virtual images of the plurality of persons by performing a touch input to the touch panel display device 242H, for example.

The content selector 270 includes, for example, a touch panel display device 242H. The passenger of the vehicle 200 selects a desired content from a plurality of contents that can be provided by the content providing server 100 by, for example, performing a touch input to the touch panel display device 242H.

[ content providing Server ]

The content providing server 100 includes, for example, a communication unit 110, a storage unit 120, and a control unit 130. Some or all of the functional configurations included in content providing server 100 may be mounted on vehicle 200. Some or all of the functional configurations included in vehicle 200 may be mounted on content providing server 100.

The communication unit 110 is, for example, a communication module for connecting to the network NW. The communication unit 110 includes, for example, a NIC (Network Interface Card). Communication unit 110 communicates with vehicle 200 via network NW.

The storage unit 120 is implemented by an HDD, a flash memory, a RAM, a ROM, and the like. The storage unit 120 stores information received from the vehicle 200, information transmitted to the vehicle 200, raw data generated in the content providing server 100, and the like. In a case where the content providing server 100 provides the karaoke content to the passenger of the vehicle 200, the karaoke content data 122 (for example, data of accompaniment music, lyrics, and the like) is stored in the storage unit 120.

The control Unit 130 is realized by a processor such as a CPU (Central Processing Unit) executing a program stored in the storage Unit 120. The control unit 130 includes, for example, a raw data generation unit 132 and a singing performance evaluation unit 134. Some or all of these functions may be realized by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), GPU (Graphics Processing Unit), or the like, or may be realized by cooperation of software and hardware. The program may be stored in advance in a storage device such as an HDD (Hard Disk Drive) or a flash memory, may be stored in a removable storage medium such as a DVD or a CD-ROM, or may be installed in the storage device by mounting the storage medium in the Drive device.

The raw data generating unit 132 generates raw data of the contents of the passenger of the vehicle 200 in accordance with the voice (e.g., singing, speaking, etc. of the virtual image) output from the voice output unit 244 of the vehicle 200 and the movement of the virtual image generated by the virtual image generating unit 242 of the vehicle 200. Therefore, the raw data generating unit 132 includes, for example, a speech generating unit 132A, a singing voice generating unit 132B, and an operation generating unit 132C.

The speech generating unit 132A generates raw data of the speech of the virtual image (i.e., the sound of the virtual image speech) presented by the virtual image generating unit 242 of the vehicle 200. The speech generating unit 132A has an automatic Voice Response (Interactive Voice Response) function with AI (artificial intelligence) mounted thereon.

AI refers to a technique for manually realizing human intelligent work such as learning, inference, and judgment using a computer. AI also includes mechanical learning and deep learning (deep learning). Mechanical learning refers to a method using the following algorithm: data is analyzed for specific matters in the society, and the results are learned, judged, or predicted. Deep learning refers to a method for installing machine learning. In the deep learning, tasks may be classified according to all kinds of support methods that can be performed (or expected) by the machine. The machine learning and the deep learning are techniques for learning of the AI. Machine learning has a feature in which a developer analyzes data by the AI itself to find regularity and rules, instead of programming all operations in advance. That is, machine learning is learning in AI that is capable of performing a specific task through training. The deep learning is obtained by further development of mechanical learning. In deep learning, a framework used when analyzing information and data is different from that of machine learning. In deep learning, a multilayer neural network is used which is created by simulating human nerves, thereby improving the ability of a computer to analyze and learn data.

The automatic sound response function refers to a function of responding to the speech of the occupant of the vehicle 200 using a pre-recorded sound or a dynamically generated sound.

The speech generating unit 132A generates the raw data of the speech of the virtual image based on the speech of the occupant of the vehicle 200 detected by the occupant information detecting unit 220 of the vehicle 200, and establishes a conversation between the occupant of the vehicle 200 and the virtual image.

The speech generating unit 132A may generate the original data of the speech of the virtual image based on the information outside the vehicle 200 detected by the external information detecting unit 230 of the vehicle 200. For example, the speech generation unit 132A generates original data of the speech of the virtual image with respect to the detection result of the external information detection unit 230 (for example, when it starts raining or when the road starts to be congested). In this case, the detection capability of the external information detection unit 230 is set to the same degree as the detection capability of the information outside the vehicle 200 by the passenger. As a result, the speech generating unit 132A can generate the raw data of the speech that obtains the virtual image of the resonance of the passenger. For example, the speech generating unit 132A generates original data of the speech of the virtual image based only on the information on the direction in which the virtual image is directed. For example, when the range that can be viewed from the passenger becomes narrow as the vehicle 200 moves or the visibility becomes poor as the weather deteriorates, the detection capability of the external world information detection unit 230 decreases in accordance with the decrease in the detection capability of the passenger for the information outside the vehicle 200.

The external information detection unit 230 detects the start of rain using, for example, the technique described in international publication No. 2014/007175. In this example, when a portion having a luminance higher than the surrounding for a certain time or more is present at substantially the same position in the image captured by the vehicle exterior camera 230A while the vehicle 200 is traveling, although the background is flowing far away, the external world information detection unit 230 determines that the portion is a water droplet attached to the lens of the vehicle exterior camera 230A, and detects that it is beginning to rain. The external information detection unit 230 transmits data indicating the start of rainfall to the speech generation unit 132A of the content providing server 100 via the network NW.

The external world information detection unit 230 detects that a road has started to be congested by using the technique described in fig. 7 of japanese patent application laid-open No. 11-064031, for example. In this example, the external world information detection unit 230 detects the current position of the vehicle 200 by the GNSS receiver, the gyro sensor, and the vehicle speed sensor. The vehicle speed sensor detects the vehicle speed of the vehicle 200. The gyro sensor detects the traveling direction of the vehicle 200. When the vehicle speed of the vehicle 200 is equal to or lower than the predetermined value and continues for a predetermined time or longer, the external information detection unit 230 determines that the road has started to be congested. The external information detection unit 230 transmits data indicating that the road starts to be congested to the speech generation unit 132A of the content providing server 100 via the network NW.

The speech generating unit 132A may generate original data of the speech of the virtual image of the sightseeing spot outside the vehicle, for example, which is detected by the external information detecting unit 230. In this example, the external information detection unit 230 includes a navigation device including map information including sightseeing spots and a GNSS receiver. The external information detection unit 230 detects that the vehicle 200 is located in a sightseeing spot by using the technique described in japanese patent No. 5921162, for example. The external information detection unit 230 transmits data indicating that the vehicle 200 is located at a sightseeing spot to the speech generation unit 132A of the content providing server 100 via the network. The storage unit 120 of the content providing server 100 stores therein the sightseeing information of each of the plurality of sightseeing spots and a script of a virtual image speech of each sightseeing spot. The speech generating unit 132A generates original data of speech regarding the virtual image of the sightseeing spot based on the script stored in the storage unit 120.

Fig. 9 is a flowchart showing an example of a flow of processing and the like in which the speech generating unit 132A generates original data of speech regarding a virtual image of a sightseeing spot. First, the control unit 130 of the content providing server 100 acquires the route of the vehicle 200 determined by the navigation device of the external information detection unit 230 of the vehicle 200 and the map information provided in the navigation device via the network NW (step S100). The control unit 130 determines whether or not there is a sightseeing spot on the acquired route of the vehicle 200 (step S101). When the sightseeing spot does not exist on the path of the vehicle 200, the processing of the routine of the present flowchart is ended. On the other hand, when the sightseeing spot is located on the route of the vehicle 200, the control unit 130 obtains the current position of the vehicle 200 detected by the GNSS receiver of the navigation device via the network NW (step S102). The control unit 130 calculates the distance between the current position of the vehicle 200 and the sightseeing spot (step S103). The control unit 130 obtains the vehicle speed of the vehicle 200 detected by the vehicle speed sensor of the external information detection unit 230 via the network NW (step S104). The control unit 130 calculates the time required for the vehicle 200 to reach the sightseeing spot based on the distance and the vehicle speed (step S105). The speech generating unit 132A of the content providing server 100 generates original data of speech about a virtual image of a sightseeing spot, and transmits the data to the audio output unit 244 of the vehicle 200 via the network NW (step S106). The control unit 130 determines whether the vehicle 200 has reached the sightseeing spot (step S107). When the vehicle 200 does not reach the sightseeing spot, the control unit 130 executes step S107 again after a predetermined time has elapsed. On the other hand, when the vehicle 200 reaches the sightseeing spot, the control unit 130 transmits a control signal for causing the audio output unit 244 of the vehicle 200 to output a speech related to the virtual image of the sightseeing spot to the audio output unit 244 via the network NW (step S108). In this way, control unit 130 adjusts the timing of causing sound output unit 244 to output speech based on the external condition of the vehicle.

The speech generating unit 132A may generate raw data of speech on a virtual image of an advertisement outside the vehicle, for example, detected by the external information detecting unit 230. In this example, the storage unit 280 of the vehicle 200 stores dictionary data. The external information detection unit 230 recognizes character strings included in images captured by the vehicle exterior cameras 230A to 230C by using the technique described in japanese patent application laid-open No. 2015-153032, for example. Specifically, the external information detection unit 230 recognizes a character string included in the image by OCR (optical character Recognition). In the technique described in japanese patent application laid-open No. 2015-153032, although the recognized character string is translated, the external information detecting unit 230 determines whether or not the recognized character string is an advertisement based on dictionary data stored in the storage unit 280. When the recognized character string is an advertisement, the external information detection unit 230 transmits the data indicating the meaning of the advertisement to the speech generation unit 132A of the content providing server 100 via the network NW. The storage unit 120 of the content providing server 100 stores a script of the speech of the virtual image of each of the plurality of advertisements. The speech generating unit 132A generates original data of the speech of the virtual image of the advertisement based on the script stored in the storage unit 120.

Consider a case where although the content providing server 100 provides the karaoke content to the passenger of the vehicle 200, the passenger of the vehicle 200 sings without matching with the accompaniment music of the karaoke content. In this case, the speech generator 132A may generate the original data of the speech of the virtual image prompting the passenger of the vehicle 200 to sing, based on the detection result of the voice input unit 250 of the vehicle 200 indicating the intention of the passenger of the vehicle 200 not to sing in accordance with the accompaniment music of the karaoke content. In this example, the singing voice of the passenger of the vehicle 200 detected by the singing microphone of the voice input unit 250 is transmitted to the control unit 130 of the content providing server 100 via the network NW.

The control portion 130 determines whether the passenger of the vehicle 200 sings in accordance with the accompaniment music based on the received singing voice (in detail, no voice because no singing is made) of the passenger of the vehicle 200 and the accompaniment music of the karaoke content data 122. The storage unit 120 of the content providing server 100 stores a script of a virtual image that urges the passenger of the vehicle 200 to sing. When the passenger of the vehicle 200 sings without matching with the accompaniment music, the speech generating unit 132A generates the original data of the speech of the virtual image prompting the passenger of the vehicle 200 to sing based on the script stored in the storage unit 120. In the case where the passenger of the vehicle 200 sings without matching with the accompaniment music, the speech generating section 132A may generate the original data of the virtual image for cheering and cheering (cheering and beating beat sounds) matching with the accompaniment music. When the speech generating unit 132A generates the raw data of the beat sound of the virtual image, the motion generating unit 132C may generate the raw data of the beat motion of the virtual image matching the beat sound.

Fig. 10 is a flowchart illustrating an example of a flow of processing and the like of the speech generating unit 132A to generate the original data of the speech of the virtual image prompting the passenger of the vehicle 200 to sing. First, the control unit 130 of the content providing server 100 determines whether or not the content providing server 100 is in a period in which the karaoke content is provided to the passenger of the vehicle 200 (step S200). When the content providing server 100 is not in the period of providing the karaoke content, the processing of the routine of the flowchart is ended. On the other hand, when the content providing server 100 is in a period of providing the karaoke content, the control unit 130 acquires the singing voice of the passenger in the vehicle 200 detected by the singing microphone of the voice input unit 250 in the vehicle 200 (step S201). The control section 130 determines whether or not the passenger of the vehicle 200 is singing (step S202). In the case where the passenger of the vehicle 200 is singing, the processing of the routine of the present flowchart is ended. On the other hand, when the passenger of the vehicle 200 does not sing, the speech generating unit 132A of the content providing server 100 generates original data of the speech of the virtual image prompting the passenger of the vehicle 200 to sing, and transmits the original data to the audio output unit 244 of the vehicle 200 via the network NW (step S203). The control unit 130 acquires the singing voice of the passenger of the vehicle 200 (step S204). The control section 130 determines whether or not the passenger of the vehicle 200 starts singing (step S205). When the passenger of the vehicle 200 starts singing, the processing of the routine of the present flowchart is ended. On the other hand, when the passenger of the vehicle 200 does not start singing, the speech generating unit 132A generates original data of the applause sound of the virtual image prompting the passenger of the vehicle 200 to sing, and outputs the generated data to the sound output unit 244 of the vehicle 200 via the network NW (step S206). The control unit 130 acquires the singing voice of the passenger of the vehicle 200 (step S207). The control section 130 determines whether or not the passenger of the vehicle 200 starts singing (step S208). When the passenger of the vehicle 200 starts singing, the processing of the routine of the present flowchart is ended. On the other hand, when the passenger of the vehicle 200 does not start singing, the motion generating unit 132C generates raw data of the beating beat motion of the virtual image prompting the passenger of the vehicle 200 to sing, and the speech generating unit 132A generates raw data of the beating beat sound of the virtual image and transmits the raw data to the virtual image generating unit 242 and the audio output unit 244 of the vehicle 200 via the network NW (step S209).

The singing voice generating unit 132B generates raw data of the singing voice of the virtual image output by the voice output unit 244 of the vehicle 200. When the content providing server 100 provides the karaoke content to the passenger of the vehicle 200, the singing voice generating unit 132B may generate the original data of the singing voice of the virtual image which is singed in accordance with the accompaniment music of the karaoke content output by the voice output unit 244 of the vehicle 200. In this example, the original data of the singing voice of the virtual video is generated using the technique described in japanese patent application laid-open No. 2008-58379, for example. The singing voice generating unit 132B is, for example, a conventional (japanese: コンキヤテイ イテイブ) type or a corpus type TTS (text-to-speech) system that records voices of persons who actually exist, decomposes the voices into voice elements, creates a database of the voice elements, and synthesizes the voice elements of the database to generate desired voices. The singing voice generating section 132B may be a parametric TTS system.

Even when the singing voice generating unit 132B of the content providing server 100 generates the original data of the singing voice of the virtual image and the voice output unit 244 of the vehicle 200 outputs the singing voice of the virtual image, the control unit 130 of the content providing server 100 may stop transmitting the original data of the singing voice of the virtual image to the voice output unit 244 of the vehicle 200 when the passenger of the vehicle 200 does not start singing. In this case, instead of generating the raw data of the singing voice of the virtual image by the singing voice generating unit 132B, the speech generating unit 132A may generate raw data of the speech of the virtual image for the passenger of the vehicle 200, and the sound output unit 244 of the vehicle 200 may output the speech of the virtual image for the passenger of the vehicle 200.

The motion generating unit 132C generates raw data of the motion of the virtual image presented by the virtual image generating unit 242 of the vehicle 200. As described above, the motion of the virtual image presented by the virtual image generating unit 242 is coordinated with the speech and/or singing voice of the virtual image by using the MMD, for example.

When the content providing server 100 provides the karaoke content to the passenger of the vehicle 200, the operation of the virtual image presented by the virtual image generating unit 242 and the accompaniment music of the karaoke content are coordinated based on the raw data generated by the operation generating unit 132C.

For example, when the expression or the like of the passenger detected by the passenger information detecting unit 220 of the vehicle 200 and/or the scene or the like outside the vehicle detected by the external world information detecting unit 230 change, the speech generating unit 132A may generate the raw data so that the speech of the virtual image output by the sound output unit 244 dynamically changes according to the change, and the motion generating unit 132C may generate the raw data so that the motion of the virtual image generated by the virtual image generating unit 242 dynamically changes. That is, in this example, the speech generator 132A and the motion generator 132C generate the raw data based on the latest information without accumulating the information detected by the passenger information detector 220 and the external information detector 230 in the storage unit 120. For example, the passenger information processing unit 410 may include a storage unit (storage device or information storage area). The storage unit stores the interaction state provided by the occupant of the vehicle with respect to the providing unit 240, which is detected by the occupant information processing unit 410. The providing unit 240 may change the interaction available to the occupant of the vehicle based on the state of the storage unit. For example, the providing unit 240 may change the interaction based on information that associates the state of the occupant of the vehicle with the interaction, among the information stored in the storage unit 280. The change of the interaction may be performed by the raw data generation unit 132. In this case, the raw data generation unit 132 changes the interaction based on the information in which the state of the occupant of the vehicle and the interaction are associated with each other, among the information stored in the storage unit 120.

The sound output unit 244 of the vehicle 200 outputs the speech and the singing voice of the virtual image based on the raw data generated by the speech generating unit 132A and the singing voice generating unit 132B. The virtual image generating unit 242 of the vehicle 200 presents a virtual image of the movement based on the raw data generated by the movement generating unit 132C. By presenting a virtual image in which speech and motion can be dynamically changed, the passenger of the vehicle 200 can have the illusion that the virtual image is also present in the vehicle 200.

The singing strength scoring unit 134 scores the singing strength of the passenger of the vehicle 200 singing in accordance with the accompaniment music of the karaoke contents when the content providing server 100 provides the karaoke contents to the passenger of the vehicle 200. The higher the scoring result of the singing strength scoring unit 134 is, the more the virtual image generating unit 242 of the vehicle 200 raises the level of the virtual image that the passenger of the vehicle 200 can select by the virtual image selecting unit 260.

Fig. 11 is a sequence diagram for explaining an example of processing in the content providing server 100 and the vehicle 200.

(step S1) the content selection unit 270 of the vehicle 200 accepts selection of a content by the occupant of the vehicle 200.

(step S2) the vehicle 200 transmits data representing the selected content (in this example, the karaoke content) to the content providing server 100.

(step S3) the virtual image selection unit 260 of the vehicle 200 receives selection of a virtual image by the passenger of the vehicle 200.

(step S4) the vehicle 200 transmits data indicating the selected virtual image to the content providing server 100.

(step S5) the content providing server 100 transmits data of accompaniment music, lyrics, and the like of the karaoke content to the vehicle 200.

(step S6) the occupant information detection unit 220 of the vehicle 200 detects information relating to the occupant of the vehicle 200.

(step S7) the vehicle 200 transmits the detected information relating to the occupant of the vehicle 200 to the content providing server 100.

(step S8) the external information detection unit 230 of the vehicle 200 detects information outside the vehicle 200.

(step S9) the vehicle 200 transmits the detected information outside the vehicle 200 to the content providing server 100.

(step S10) based on the information on the occupant of the vehicle 200 and the information outside the vehicle 200, the speech generator 132A of the content providing server 100 generates the raw data for outputting the speech of the virtual image at the audio output unit 244 of the vehicle 200. The singing voice generating unit 132B generates raw data for outputting the singing voice of the virtual image in the voice output unit 244 of the vehicle 200. The motion generating unit 132C generates raw data used for generating the motion of the virtual image in the virtual image generating unit 242 of the vehicle 200.

(step S11) the content providing server 100 transmits the generated raw data to the vehicle 200.

(step S12) the providing unit 240 of the vehicle 200 provides the passenger of the vehicle 200 with the contents of the virtual image in accordance with the movement and the sound (the speech, the singing voice, etc. of the virtual image).

(step S13) the occupant information detection unit 220 of the vehicle 200 detects information relating to the occupant of the vehicle 200 during the provision of the content. This information represents the reaction of the passenger to the vehicle 200 for the provided content.

(step S14) the vehicle 200 transmits information relating to the occupant of the vehicle 200 detected during the provision of the content to the content providing server 100.

(step S15) the external information detection unit 230 of the vehicle 200 detects information outside the vehicle 200 during the provision of the content.

(step S16) the vehicle 200 transmits information of the outside of the vehicle 200 detected during the provision of the content to the content providing server 100.

(step S17) based on the information on the occupant of the vehicle 200 and the information outside the vehicle 200 during the provision of the content, the speech generating unit 132A generates raw data for outputting speech of the virtual image, the singing voice generating unit 132B generates raw data for outputting singing voice of the virtual image, and the operation generating unit 132C generates raw data for generating the operation of the virtual image.

(step S18) the content providing server 100 transmits the generated raw data to the vehicle 200.

(step S19) the providing unit 240 of the vehicle 200 provides the passenger of the vehicle 200 with the contents in which the speech, singing voice, and movement of the virtual image dynamically change, based on the raw data generated during the provision of the contents.

The vehicle 200 on which the passenger who receives the provision of the content rides is, for example, an autonomous vehicle that requires substantially no driving operation. Hereinafter, a case will be described in which the vehicle 200 on which the passenger who receives the provision of the content is an autonomous vehicle. The vehicle 200 on which the passenger who accepts the provision of the content rides may be a manually driven vehicle.

[ automatic Driving vehicle ]

Fig. 12 is a structural diagram of a vehicle 200 having an automatic driving function. The vehicle 200 includes, for example, an outside monitoring unit 310, a navigation device 330, a recommended lane determining device 340, an automatic driving control unit 350, a driving force output device 360, a brake device 362, and a steering device 364, in addition to the components (the communication unit 21O, the passenger information detection unit 220, the outside information detection unit 230, the providing unit 240, the sound input unit 250, the virtual image selection unit 260, and the storage unit 280) shown in fig. 1.

The outside monitoring unit 310 includes, for example, a camera, a radar, a LIDAR (Light Detection and ranging), an object recognition device that performs a sensor fusion process based on the output thereof, and the like. External monitoring section 310 estimates the type of an object (particularly, a vehicle, a pedestrian, and a bicycle) present in the periphery of vehicle 200, and outputs the estimated type of the object together with information on the position and speed of the object to automatic driving control section 350. Some of the components may be shared between the external monitor unit 310 and the external information detection unit 230.

The navigation device 330 includes, for example, an HMI (Human machine Interface) 332, a GNSS receiver 334, and a navigation control device 336. The HMI332 includes, for example, a touch panel display device, a speaker, a microphone, and the like. The GNSS receiver 334 measures the position of the vehicle (the position of the vehicle 200) based on radio waves transmitted from GNSS satellites (for example, GPS satellites). The navigation control device 336 includes, for example, a cpu (central Processing unit) and various storage devices, and controls the entire navigation device 330. The storage device stores map information (navigation map). The navigation map is a map in which roads are represented by nodes and lines. The navigation control device 336 refers to the navigation map and determines a route to the destination specified by the HMI332, based on the position of the vehicle 200 detected by the GNSS receiver 334. The navigation control device 336 may transmit the position and destination of the vehicle 200 to a navigation server (not shown) using the communication unit 210, and acquire a route returned from the navigation server. Some of the components may be shared between the navigation device 330 and the external information detection unit 230. That is, the navigation device and the navigation device 330 provided in the external information detection unit 230 may be a common navigation device.

The recommended lane determining device 340 includes, for example, an MPU (Map Positioning Unit) and various storage devices. The storage device stores map information with higher precision than the navigation map. The high-accuracy map information includes, for example, information on the road width, gradient, curvature, signal position, and the like for each lane. The recommended lane determining device 340 determines a recommended lane that is preferable for traveling along the route input from the navigation device 330, and outputs the determined recommended lane to the automatic driving control unit 350.

The automatic driving control Unit 350 includes one or more processors such as a CPU and an MPU (Micro Processing Unit) and various storage devices. The automatic driving control unit 350 automatically drives the vehicle 200 so as to avoid contact with an object whose position and speed are input from the outside monitoring unit 310, in principle of driving on the recommended lane determined by the recommended lane determining device 340. The automatic driving control unit 350 executes various events in sequence, for example. The events include: a constant speed driving event for driving in the same driving lane at a constant speed, a follow-up driving event for following a preceding vehicle, a lane change event, a junction event, a branch event, an emergency stop event, a toll station event for passing a toll station, a hand-over event for ending automatic driving and switching to manual driving, and the like. During the execution of these events, there are also cases where actions for avoidance are planned based on the surrounding conditions of the vehicle 200 (presence of a surrounding vehicle or pedestrian, lane narrowing due to road construction, etc.).

The automatic driving control unit 350 generates a target track on which the vehicle 200 will travel in the future. The target trajectory includes, for example, a velocity element. For example, the target track is displayed as a track in which points (track points) to be reached by the vehicle are sequentially arranged. The track point is a point to which the vehicle should arrive at every predetermined travel distance, and in contrast, a target speed and a target acceleration at every predetermined sampling time (for example, several fractions of sec or so) are generated as a part of the target track. The track point may be a position to which the vehicle at the sampling time should arrive at every predetermined sampling time. In this case, information of the target velocity and the target acceleration is displayed at intervals of the track points.

Fig. 13 is a diagram for explaining a processing procedure of the automatic driving. First, as shown in the above figure, the navigation device 330 determines a route. This route is, for example, a rough route in which no distinction is made between lanes. Next, as shown in the figure, the recommended lane determining device 340 determines a recommended lane that is easy to travel along the route. As shown in the following figure, the automatic driving control means 350 generates a trajectory point for traveling along the recommended lane as much as possible while avoiding an obstacle or the like, and controls a part or all of the driving force output device 360, the brake device 362, and the steering device 364 so as to travel along the trajectory point (and the accompanying speed curve). Such action assignment is merely an example, and the automatic driving control unit 350 may perform processing in one dimension, for example.

The driving force output device 360 outputs a running driving force (torque) for running of the vehicle to the driving wheels. The driving force output device 360 includes, for example, a combination of an internal combustion engine, a motor, a transmission, and the like, and a power ECU that controls them. The power ECU controls the above configuration based on information input from the automatic drive control unit 350 or information input from a drive operator not shown.

The brake device 362 includes, for example, a caliper, a hydraulic cylinder that transmits hydraulic pressure to the caliper, an electric motor that generates hydraulic pressure in the hydraulic cylinder, and a brake ECU. The brake ECU controls the electric motor so as to output a braking torque corresponding to a braking operation to each wheel, based on information input from the automated driving control unit 350 or information input from the driving operator. The brake device 362 may be provided with a mechanism for transmitting the hydraulic pressure generated by the operation of the brake pedal included in the driving operation unit to the hydraulic cylinder via the master cylinder as a backup. The brake device 362 is not limited to the above-described configuration, and may be an electronically controlled hydraulic brake device that controls an actuator based on information input from the automatic steering control unit 350 and transmits the hydraulic pressure of the master cylinder to the hydraulic cylinder.

The steering device 364 includes, for example, a steering ECU and an electric motor. The electric motor changes the direction of the steered wheels by applying a force to a rack-and-pinion mechanism, for example. The steering ECU drives the electric motor to change the direction of the steered wheels based on information input from the automatic steering control unit 350 or information input from the steering operator.

The providing unit 240 of the vehicle 200 provides the occupant of the vehicle 200 with the content of synchronizing (or coordinating) the operation of the sound output by the sound output unit 244 and the virtual image presented by the virtual image generating unit 242 during the automatic driving of the vehicle 200. During manual driving of the vehicle 200, the providing portion 240 does not provide the content to the passenger of the vehicle 200. For example, the providing unit 240 of the vehicle 200 acquires information indicating whether or not the automated driving is being performed from the automated driving control unit 350 or the control unit of the vehicle, and restricts at least a part of the operation pattern of the interaction during the manual driving of the vehicle 200. The restricted action profile is an action profile that may possibly obstruct the attention of the driver.

In the above-described embodiment, the providing unit 240, the virtual image generating unit 242, the audio output unit 244, and the content playback unit 246 are provided in the vehicle 200, and the raw data generating unit 132, the speech generating unit 132A, the singing voice generating unit 132B, and the motion generating unit 132C are provided in the content providing server 100 (i.e., outside the vehicle), but they may be provided only in the vehicle 200 or only outside the vehicle.

< modification example >

In the modification, the functional configuration and the processing of the above embodiment will be described in detail. Fig. 14 is a diagram showing an example of a functional configuration included in the experience providing system 1A. The experience providing system 1 includes the functional configuration shown in fig. 14 in addition to the functional configurations of fig. 2 and 12. The passenger information detection unit 220, the external environment information detection unit 230, and the providing unit 240 in fig. 13 have the same functional configurations as the passenger information detection unit 220, the external environment information detection unit 230, and the providing unit 240 described in fig. 2.

Vehicle 200 further includes, for example, signal generation unit 400, local determination unit 510, local UI processing unit 610, and local storage unit 700. The signal generation unit 400 has the following functional configuration in addition to the functional configuration of the audio input unit 250. The signal generating unit 400 includes, for example, a passenger information processing unit 410, an external information processing unit 420, and an environmental information processing unit 430.

The passenger information processing unit 410 extracts information used by the local determination unit 510 or the remote determination unit 550 from the detection result of the passenger information detection unit 220, and transmits the extracted information to the local determination unit 510 or the remote determination unit 550. For example, the passenger information processing unit 410 transmits an image captured by a camera to the local determination unit 510 or the remote determination unit 550. The external information processing unit 420 extracts information used by the local judgment unit 510 or the remote judgment unit 550 from the detection result of the external information detection unit 230, and transmits the extracted information to the local judgment unit 510 or the remote judgment unit 550. The external information processing unit 420 transmits, for example, a detection result of a sensor for observing the state of the external environment to the local determination unit 510 or the remote determination unit 550.

The environment information processing unit 430 acquires the detection results of various sensors and the like that acquire information on the environment and the information provided by the providing unit 240, extracts information used by the local determining unit 510 or the remote determining unit 550 from the acquired information, and transmits the extracted information to the local determining unit 510 or the remote determining unit 550. For example, when the index indicated by the sound pressure signal acquired by each microphone deviates from the reference value by a predetermined degree, the signal generation unit 400 extracts the sound pressure signal and transmits the extraction result to the local determination unit 510 or the remote determination unit 550 while detecting a change in the sound pressure signal at a certain level.

The local judgment section 510 makes a judgment about the environmental information based on the detection result of the signal generation section 400. The judgment about the environment information is, for example, evaluation of an event and an environment attribute not related to the context of input/output. That is, the local determining unit 510 determines whether or not the information acquired from the signal generating unit 400 satisfies the occurrence condition of the event. The local determination unit 510 includes, for example, a local event determination unit 520, a local attribute determination unit 530, and a transmission signal generation unit 540.

The local event determination unit 520 determines whether or not the detection result (environmental information) of the signal generation unit 400 satisfies a predetermined condition. When determining that the predetermined condition is satisfied, the local event determination unit 520 transmits information indicating the determination result to the local UI processing unit 610. The prescribed condition refers to detection of a detection result associated with an input-output context.

The local attribute determining unit 530 determines whether or not a predetermined environmental attribute is included in the detection result of the signal generating unit 400. The environment attribute is, for example, information indicating an attribute indicating whether or not the vehicle is in a traveling state, information indicating an attribute indicating whether or not a building exists around the vehicle, information indicating an attribute indicating whether or not a sightseeing spot exists around the vehicle, and information indicating an attribute indicating predetermined sound data (e.g., music). The transmission signal generating unit 540 transmits the processing result of the local event determining unit 520 or the local attribute determining unit 530 to the local UI processing unit 610.

The local UI processing section 610 performs context processing. The local UI processing unit 610 performs evaluation of input/output context and generation of response. The local UI processing unit 610 includes, for example, a local intention estimating unit 620, a local response generating unit 630, and a transmission signal generating unit 640.

The local intention inference section 620 determines whether or not the sound data is an input with intention based on the sound data. The local response generator 630 causes the virtual image generator 242 to generate a virtual image according to the state of the passenger. The transmission signal generation unit 640 transmits the processing result of the local intention estimation unit 620 or the local response generation unit 630 to the providing unit 240. The information stored in the local storage unit 700 will be described later.

The experience providing system 1 further includes, for example, a remote determination unit 550, a remote UI processing unit 650, a remote storage unit 710, and external reference information 800. The remote determination unit 550, the remote processing unit 650, the remote storage unit 710, and the external reference information 800 are provided in a place different from the vehicle.

The remote determination unit 550 includes, for example, a remote event determination unit 560, a remote attribute determination unit 570, and a transmission signal generation unit 580. The remote event determination unit 560 determines whether or not the detection result (environmental information) of the signal generation unit 400 satisfies a predetermined condition. The remote attribute determination unit 570 determines whether or not the detection result of the signal generation unit 400 includes a predetermined environmental attribute. The transmission signal generation section 580 transmits the processing result of the remote event determination section 560 or the remote attribute determination section 570 to the remote UI processing section 650. The providing unit 240 performs interaction in the vehicle 200 by transmitting the processing result of the remote event determining unit 560 or the remote property determining unit 570 to the remote UI processing unit 650.

The remote UI processing section 650 performs context processing. The remote UI processing unit 650 performs evaluation of input/output context and generation of response. The remote UI processing unit 650 includes, for example, a remote intention estimating unit 660, a remote response generating unit 670, and a transmission signal generating unit 680. The remote intention estimating section 660 determines whether or not the sound data is an input with intention based on the sound data. The remote response generator 670 causes the virtual image generator 242 to generate a virtual image according to the state of the passenger. The transmission signal generation unit 680 transmits the processing result of the remote intention estimation unit 660 or the remote response generation unit 670 to the providing unit 240.

The remote storage unit 710 stores, for example, the same information as the information stored in the local storage unit 700. The external reference information 800 includes information such as sound data of music or the like and a pattern of the sound data.

[ specific examples ]

Hereinafter, a process of executing an interaction in a vehicle will be described with reference to fig. 15. The signal generation unit 400 acquires, for example, an ambient sound in the vehicle cabin, and transmits the extracted sound data to the local determination unit 510, the remote determination unit 550, and the UI processing units (the local UI processing unit 610 and the remote UI processing unit 650) every predetermined second (for example, 5 seconds) based on the acquired ambient sound (step S300).

When determining that the pattern of the audio data transmitted from the signal generation unit 400 is similar to the pattern of the audio data representing music, the local event determination unit 520 determines that music is to be played (step S302). The local attribute determining section 530 determines an attribute with respect to the sound data. In the case where the attribute exists in the feature, the local attribute judgment section 530 transmits the judgment result to the local UI processing section 610 (step S304).

The remote event determination unit 560 determines whether or not an event such as the playback of music has occurred based on the audio data and the external reference information 800 acquired from the signal generation unit 400 (step S306), determines attributes such as "singer a" and "music B", and transmits the determination result to the remote UI processing unit 650 (step S308).

Next, the local intention estimating unit 620 of the local UI processing unit 610 determines whether or not the voice data is an input with intention based on the voice data (step S310). The local UI processing unit 610 determines whether or not the audio data is input with intention, for example, using an AI, a prestored algorithm, or the like. In step S310, it is determined that the input is not an input with intent. After that, music is played back through the providing section 249 (step S312). That is, when there are a plurality of interactions to be provided, the providing unit 240 determines the execution timing or execution permission of the plurality of interactions based on the detection results of the passenger information processing unit 410 and the external information processing unit 420, and provides the interaction to the passenger based on the determination result.

In the next cycle, the signal generation unit 400 acquires the sound in the vehicle cabin, and separates the acquired sound into music sound data and speech data (step S314). Next, the local event determination unit 520 determines that an event "interest in music" has occurred when the music sound data and the speech data are acquired and it is determined from the data that humming is underway or that the passenger is performing an action with a tempo (step S316).

The local event determination unit 520 specifies the playback position and the content information corresponding to the singer a based on the music information (information such as music data, lyrics, and musical interval) specified based on the external reference information 800 as "the event to be the playback position detection target has occurred", and feeds back the specification result to the signal generation unit 400 or the local UI processing unit 610 (step S318).

The local UI processing unit 610 determines whether or not a predetermined operation is performed in a state where the playback position is specified and additional information such as lyrics is acquired (step S320). For example, the local storage unit 700 stores information that a predetermined event and an action executed at the time of the event are associated with each other. For example, the local UI processing unit 610 determines that "intervention timing has occurred" when a specific state such as the following occurs based on the information acquired from the signal generating unit 400 (step S320). The specific state is a state in which the passenger stops humming the song, a state in which the passenger's motion disappears, or a state in which the music is interlude.

When the intervention timing is acquired from the local determination unit 510, the local UI processing unit 610 executes an operation based on the available content information stored in the local storage unit 700 (step S322). The available content information is information in which the event and the action have a correspondence relationship. The action means "do this song like? "etc. output of confirmation interest, generation of singing matching with the subsequent music, etc. The processes in steps S320 and S322 may be executed by the remote UI processing unit 650.

As described above, the providing part 240 dynamically provides the interaction for the occupant of the vehicle based on the detection result of the occupant information processing part 410 including the provided interaction. In this case, the providing unit 240 may further add the detection result of the external information processing unit 420 to dynamically provide the interaction with the occupant of the vehicle.

[ processing in the case where a plurality of events are in parallel, in addition to the above-described processing ]

For example, the local judging unit 510 (or the remote judging unit 550) continues the occurrence detection of the event in parallel with the execution period of the local UI processing unit 610 (the remote UI processing unit 650), and the providing unit 240 dynamically provides the interaction based on the occurrence of the event detected during the provision of the interaction.

For example, when the passenger moves the body in accordance with the music (or the interlude) to form a rhythm, the local response generation unit 630 causes the virtual image generation unit 242 to generate the virtual image so as to mirror the movement of the passenger in accordance with the music. For example, when the passenger moves his/her body and a rhythm is formed as described above, the local response generation unit 630 causes the virtual image generation unit 242 to generate a virtual image such that the hand is swung or clapped in accordance with the intervention timing.

When the local event determination unit 520 detects that there is a sight point such as a sightseeing point outside based on the sensor data extracted by the external world information processing unit 420 (for example, data of the feature amount extracted from the captured image by the camera) and determines that the occurrence of the "point of interest finding" event has occurred, the local response generation unit 630 outputs a comment to the outside world such as "that is the point of interest C" to the provision unit 240. Here, since there are the above-described parallel events, there is a case where another event such as "generation of intervention timing" and a response thereto are repeated. In this case, the local UI processing unit 610 determines the priority of each of the repeated events and responses thereto, and adjusts the information provided or output by the response according to the priority.

According to the embodiment described above, the experience providing system 1 is provided with the sound output unit 244 that outputs sound, the virtual image generating unit 242 that presents the virtual image of the person visually recognizable by the passenger of the vehicle 200, and the providing unit 240 that provides the content for the passenger of the vehicle 200 in which the sound output by the sound output unit 244 and the motion of the virtual image presented by the virtual image generating unit 242 are coordinated, thereby providing the passenger of the vehicle 200 with a conventionally-unavailable enjoyment.

While the present invention has been described with reference to the embodiments, the present invention is not limited to the embodiments, and various modifications and substitutions can be made without departing from the scope of the present invention.

Description of reference numerals:

1 … experience provision system, 100 … content provision server, 110 … communication section, 120 … storage section, 122 … karaoke content data, 130 … control section, 132 … raw data generation section, 132a … speech generation section, 132B … singing sound generation section, 132C … action generation section, 134 … singing practice scoring section, 200 … vehicle, 210 … communication section, 220 … passenger information detection section, 220a … in-vehicle camera, 220B … in-vehicle microphone, 220C … seating sensor, 230 63 230 … outside information detection section, 230a … out-vehicle camera, 230B … out-vehicle camera, 230C … out-vehicle camera, 230D … out-vehicle microphone, 240 … provision section, 242a … projection device, 242B … front window, 242C … left side window 242, E … rear view mirror 242, G … projection …, holography 242 … virtual image generation section, 242a … projection device … display panel … display device, 244 … audio output unit, 244a … speaker, 244B … speaker, 244C … speaker, 244D … speaker, 246 … content playback unit, 250 … audio input unit, 250a … microphone for singing, 260 … virtual image selection unit, 270 … content selection unit, and 280 … storage unit.