Method and system for risk control in switching driving modes
阅读说明:本技术 用于切换驾驶模式中的风险控制的方法和系统 (Method and system for risk control in switching driving modes ) 是由 H·郑 T·P·小戴利 D·W·刘 于 2017-12-19 设计创作,主要内容包括:本示教涉及用于切换车辆的模式的方法、系统和介质。接收与车辆以及车辆中的驾驶者的状态有关的实时数据,且其用于根据任务模型确定将要执行以便将车辆从当前模式切换到该不同模式的一组任务。基于与当前模式相关联的第一风险,确定切换要求的第一持续时间。推定完成各个任务所需要的任务持续时间。基于所要求的第一持续时间以及基于完成该组任务需要的任务持续时间确定的总持续时间,确定与切换相关联的第二风险。如果第二风险满足一判据,切换车辆模式。(The present teachings relate to methods, systems, and media for switching modes of a vehicle. Real-time data relating to the status of the vehicle and the driver in the vehicle is received and used to determine a set of tasks to be performed in order to switch the vehicle from the current mode to the different mode, in accordance with a task model. A first duration of a handover requirement is determined based on a first risk associated with a current mode. The task duration required to complete each task is estimated. A second risk associated with the handover is determined based on the required first duration and a total duration determined based on a task duration required to complete the set of tasks. If the second risk satisfies a criterion, the vehicle mode is switched.)
1. A method implemented on a machine having at least one processor, memory, and a communication platform connectable to a network, for switching a vehicle from a current mode to a different mode, the method comprising:
receiving real-time data relating to a vehicle and a status of a driver in the vehicle;
determining, based on the real-time data, a set of tasks to be performed to switch the vehicle from the current mode to the different mode, according to a task model;
determining a first duration of time required to switch from the current mode to the different mode based on a first risk associated with the current mode;
for each of the set of tasks, estimating a task duration required to complete the task;
inferring a second risk of switching from the current mode to the different mode based on the required first duration and a total duration determined based on a task duration required to complete the set of tasks; and
switching from the current mode of the vehicle to the different mode in response to the second risk satisfying a criterion.
2. The method of claim 1, wherein the real-time data includes intrinsic performance parameters and extrinsic performance parameters, the intrinsic performance parameters specifying conditions inside the vehicle that limit vehicle operating performance, the extrinsic performance parameters specifying conditions outside the vehicle that limit vehicle operating performance.
3. The method of claim 1, wherein the task duration for each of the set of tasks is estimated based on a state of the driver and/or a driver profile.
4. The method of claim 3, wherein the task duration for each of the set of tasks is further inferred based on a response time profile, the response time profile including information corresponding to a time required for a human driver to complete the task, and the driver profile indicating a task duration required for the driver to complete the task.
5. The method of claim 1, wherein the criterion corresponds to a total duration of time required to complete the set of tasks being less than or equal to the first duration of time.
6. The method of claim 1, further comprising:
generating a switch alert instruction in response to the second risk meeting a criterion prior to switching based on the set of tasks to be performed and a task duration required to complete each of the set of tasks.
7. The method of claim 1, wherein the status of the driver:
presume based on information obtained from a plurality of sensors disposed within the vehicle; and is
Including at least one of the posture of the driver, the functional state of the driver, the health state of the driver, and the mental state of the driver.
8. A machine-readable medium having information stored thereon for switching a vehicle from a current mode to a different mode, wherein the information, when read by a machine, causes the machine to perform:
receiving real-time data relating to a vehicle and a status of a driver in the vehicle;
determining, based on the real-time data, a set of tasks to be performed to switch the vehicle from the current mode to the different mode, according to a task model;
determining a first duration of time required to switch from the current mode to the different mode based on a first risk associated with the current mode;
for each of the set of tasks, estimating a task duration required to complete the task;
inferring a second risk of switching from the current mode to the different mode based on the required first duration and a total duration determined based on a task duration required to complete the set of tasks; and
switching from the current mode of the vehicle to the different mode in response to the second risk satisfying a criterion.
9. The medium of claim 8, wherein the real-time data includes intrinsic performance parameters that specify conditions inside the vehicle that limit vehicle operating performance and extrinsic performance parameters that specify conditions outside the vehicle that limit vehicle operating performance.
10. The medium of claim 8, wherein the task duration for each of the set of tasks is estimated based on a state of the driver and/or a driver profile.
11. The medium of claim 10, wherein the task duration for each of the set of tasks is further inferred based on a response time profile, the response time profile including information corresponding to a time required for a human driver to complete the task, and the driver profile indicating a task duration required for the driver to complete the task.
12. The medium of claim 8, wherein the criterion corresponds to a total duration of time required to complete the set of tasks being less than or equal to the first duration of time.
13. The medium of claim 8, wherein the information, when read by the machine, further causes the machine to perform:
generating a switch alert instruction in response to the second risk meeting a criterion prior to switching based on the set of tasks to be performed and a task duration required to complete each of the set of tasks.
14. The medium of claim 8, wherein the driver's status:
presume based on information obtained from a plurality of sensors disposed within the vehicle; and is
Including at least one of the posture of the driver, the functional state of the driver, the health state of the driver, and the mental state of the driver.
15. A system for switching a vehicle from a current mode to a different mode, the system comprising:
a handover risk estimator configured to:
receiving real-time data relating to a vehicle and a status of a driver in the vehicle; and
determining, based on the real-time data, a set of tasks to be performed to switch the vehicle from the current mode to the different mode, according to a task model;
a response time estimator configured to:
determining a first duration of time required to switch from the current mode to the different mode based on a first risk associated with the current mode; and
for each of the set of tasks, estimating a task duration required to complete the task;
a handover risk estimator configured to:
inferring a second risk of switching from the current mode to the different mode based on the required first duration and a total duration determined based on a task duration required to complete the set of tasks; and
switching from the current mode of the vehicle to the different mode in response to the second risk satisfying a criterion.
16. The system of claim 15, wherein the real-time data includes intrinsic performance parameters and extrinsic performance parameters, the intrinsic performance parameters specifying conditions inside the vehicle that limit vehicle operating performance, the extrinsic performance parameters specifying conditions outside the vehicle that limit vehicle operating performance.
17. The system of claim 15, wherein the task duration for each of the set of tasks is estimated based on a state of the driver and/or a driver profile.
18. The system of claim 17, wherein the task duration for each of the set of tasks is further inferred based on a response time profile, the response time profile including information corresponding to a time required for a human driver to complete the task, and the driver profile indicating a task duration required for the driver to complete the task.
19. The system of claim 15, wherein the criterion corresponds to a total duration of time required to complete the set of tasks being less than or equal to the first duration of time.
20. The system of claim 15, further comprising an alert instruction generator configured to:
generating a switch alert instruction in response to the second risk meeting a criterion prior to switching based on the set of tasks to be performed and a task duration required to complete each of the set of tasks.
Technical Field
The present teachings relate generally to the field of automotive vehicles and hybrid vehicles. In particular, the present teachings relate to a mechanism for operating a vehicle and its augmented behavior planning interface (augmented behavor planning interface).
Background
Automated vehicles employ a variety of computational approaches to assist in automated vehicle operation. Recently, in the automotive industry, much focus has been placed on operating a vehicle in an automatic mode in a safe manner. Autonomous vehicles allow an occupant (driver) to manually switch from a human driving mode (i.e., a mode in which the operator fully performs vehicle control) to an automatic mode (i.e., a mode in which the vehicle is essentially self-driving). Vehicles with two operating mode (automatic and manual) capabilities are hybrid drive vehicles.
In the past, switching between automatic and manual driving modes was done manually. In operation, in some cases, the handover itself may present a hazard if it is done in a particular situation. In addition, even when the autonomous vehicle detects a potential hazard and decides to switch control to the human driver, the manner in which the human driver takes over control in the manual mode may depend on the state of the occupant. For example, in an autonomous driving mode, the occupant may have given less attention, or may even fall into the dream. These real situations in autonomous driving have not been addressed in the past.
Another aspect relates to automatic switching of a hybrid driving mode vehicle in the opposite direction, i.e. from a human driving mode to an automatic driving mode. Conventional methods do not address this direction switch, no mention how to do it in a safe manner.
Therefore, there is a need for a solution to these problems.
Disclosure of Invention
The teachings disclosed herein relate to methods, systems, and programming for augmented behavior planning in automated and hybrid vehicle driving environments. In particular, the present teachings relate to methods, systems, and programming for a mechanism to perform handoff (handover) operations associated with different operating modes of a vehicle.
By one aspect of the present disclosure, a method is disclosed that is implemented on a machine having at least one processor, memory, and a communication platform connectable to a network for switching a vehicle from a current mode to a different mode. The method comprises the steps of: real-time data relating to the vehicle and the status of a driver in the vehicle is received, and a set of tasks to be performed to switch the vehicle from a current mode to the different mode is determined based on the real-time data according to a task model. The method further comprises the following steps: a first duration required to switch from the current mode to the different mode is determined based on a first risk associated with the current mode, and a task duration required to complete the task is estimated for each of the set of tasks. The method infers a second risk of switching from the current mode to the different mode based on the requested first duration and a total duration determined based on a duration of the tasks required to complete the set of tasks, and switches the vehicle from the current mode to the different mode in response to the second risk satisfying a criterion.
By one aspect of the present disclosure, a system for switching a vehicle from a current mode to a different mode is disclosed. The system includes a switching risk estimator configured to receive real-time data relating to a state of the vehicle and a driver in the vehicle, determine a set of tasks to be performed to switch the vehicle from a current mode to the different mode based on the real-time data according to a task model. The system also includes a response time estimator configured to determine a first duration required to switch from the current mode to the different mode based on a first risk associated with the current mode, and estimate, for each of the set of tasks, a task duration required to complete the task. The system also includes a switch risk estimator configured to estimate a second risk of switching from the current mode to the different mode based on the required first duration and a total duration determined based on a duration of tasks required to complete the set of tasks. The switching risk estimator switches from the current mode of the vehicle to the different mode in response to the second risk satisfying a criterion.
Other concepts relate to software for implementing the present teachings with respect to developing vehicle systems. According to this concept, a software product includes at least one machine-readable non-transitory medium and information carried by the medium. The information carried by the medium may be executable program code data, parameters associated with the executable program code and/or information related to the user, requests, content or information related to social groups, etc.
In one example, a machine-readable medium having information stored thereon for switching a vehicle from a current mode to a different mode is disclosed, wherein the information, when read by a machine, causes the machine to perform the steps of: receiving real-time data relating to a vehicle and a status of a driver in the vehicle; a set of tasks to be performed to switch the vehicle from the current mode to the different mode is determined based on the real-time data according to the task model. The steps further include determining a first duration required to switch from the current mode to the different mode based on a first risk associated with the current mode, and for each of the set of tasks, inferring a task duration required to complete the task. Further, the steps include inferring a second risk of switching from the current mode to the different mode based on the requested first duration and a total duration determined based on a duration of tasks required to complete the set of tasks, and switching the vehicle from the current mode to the different mode in response to the second risk satisfying a criterion.
Additional advantages and novel features will be set forth in part in the description which follows and in part will become apparent to those skilled in the art upon examination of the following description and drawings or may be learned by manufacture or operation of the examples. The advantages of the present teachings may be realized and attained by practice and application of the various aspects of the methods, apparatus, and combinations particularly pointed out in the detailed examples discussed below.
Drawings
The methods, systems, and/or programming described herein are further described in terms of exemplary embodiments. These exemplary embodiments are described in detail with reference to the accompanying drawings. These embodiments are non-limiting exemplary embodiments, in which like reference numerals represent like structures throughout the several views of the drawings, and in which:
FIG. 1 illustrates an exemplary diagram showing switching operating modes among vehicles, in accordance with one embodiment of the present teachings;
FIG. 2 illustrates an exemplary block diagram of a vehicle driving mode switching unit according to one embodiment of the present teachings;
FIG. 3 shows an illustrative flow chart of an exemplary process performed by a vehicle driving mode switching unit according to one embodiment of the present teachings;
FIG. 4 illustrates a chart that presents information associated with real-time intrinsic and extrinsic data in accordance with various embodiments of the present teachings;
FIG. 5 illustrates a chart presenting information associated with real-time vehicle data in accordance with various embodiments of the present teachings;
FIG. 6 illustrates a chart presenting information associated with a real-time human state in accordance with various embodiments of the present teachings;
FIG. 7 illustrates an exemplary block diagram of a risk evaluator included in a vehicle driving mode switching unit, according to an embodiment of the present teachings;
FIG. 8 illustrates an exemplary chart that sets forth the types of risk assessments according to one embodiment of the present teachings;
FIG. 9 shows an illustrative flow chart summarizing an exemplary process performed by a risk evaluator contained in a driving mode switching unit of a vehicle in accordance with an embodiment of the present teachings;
FIG. 10 illustrates an exemplary block diagram of a driver status analyzer included in a risk evaluator in accordance with an embodiment of the present teachings;
FIG. 11 shows an illustrative flow chart summarizing an exemplary process performed by a driver status analyzer contained in a risk evaluator unit of a vehicle in accordance with an embodiment of the present teachings;
FIG. 12 illustrates an exemplary block diagram of a switching risk determiner included in a driving mode switching unit, in accordance with an embodiment of the present teachings;
FIG. 13 depicts an illustrative flow chart summarizing an exemplary process performed by a handover risk determiner according to an embodiment of the present teachings;
FIG. 14 illustrates an exemplary block diagram of an automatic mode-to-human driver (A-H) mode switching risk determiner, according to an embodiment of the present teachings;
FIG. 15 shows an illustrative flow chart summarizing an exemplary process performed by an (A-H) mode switch risk determiner according to an embodiment of the present teachings;
FIG. 16 illustrates a chart that presents information associated with response times in accordance with various embodiments of the present teachings;
FIG. 17A illustrates an exemplary block diagram of a human driver mode to automatic (H-A) mode switching risk determiner, according to an embodiment of the present teachings;
FIG. 17B illustrates an exemplary table summarizing a plurality of vehicle operating modes according to an embodiment of the present teachings;
FIG. 18 depicts an exemplary flowchart outlining an exemplary process performed by an (H-A) mode switch risk determiner according to one embodiment of the present teachings;
FIG. 19 illustrates an exemplary block diagram of a handover alert control unit according to one embodiment of the present teachings;
FIG. 20A shows an illustrative flow chart summarizing an exemplary process performed by a handover alert control unit according to one embodiment of the present teachings;
FIG. 20B illustrates an exemplary mission plan presenting different mediums for alerting a driver of a vehicle in accordance with an embodiment of the present teachings;
FIG. 21 illustrates an exemplary block diagram of a multi-modal switching alert unit in accordance with an embodiment of the present teachings;
FIG. 22 shows an illustrative flow chart summarizing an exemplary process performed by a multimodal switch alert unit in accordance with an embodiment of the present teachings;
FIG. 23 illustrates the architecture of a mobile device that can be used to implement a dedicated system incorporating the present teachings; and is
FIG. 24 illustrates the architecture of a computer, which can be used to implement a specific purpose system incorporating the present teachings.
Detailed Description
In the following detailed description, by way of example, numerous specific details are set forth in order to provide a thorough understanding of the relevant teachings. However, it will be apparent to one skilled in the art that the present teachings may be practiced without these specific details. In other instances, well-known methods, procedures, components, and/or circuits have been described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.
An autonomous vehicle is a vehicle that is capable of detecting its environment and navigating through the environment without human input. Automotive vehicles detect their surroundings using a variety of techniques, such as radar, laser, GPS, odometers, and computer vision. The advanced control system interprets the sensor information to identify the appropriate flight path as well as the obstacles and associated signs. Automated vehicles include a control system that is capable of analyzing sensor data in order to distinguish between different vehicles on a roadway. Potential benefits of automotive vehicles include reduced mobility and infrastructure costs, increased safety, increased mobility, increased user satisfaction, reduced crime rates. Specifically, significant reductions in traffic collisions and associated costs due to autonomous vehicles, including less need for insurance. Automated vehicles are expected to increase traffic flow, free travelers from driving and navigation chores, and reduce fuel consumption.
However, there are certain obstacles to the widespread adoption of automotive vehicles. For example, disputes about liability, the period of time required to replace an existing vehicle inventory, individual conflict with out-of-control, consumer safety concerns, implementation of a viable legal framework, establishment of government regulations, loss of privacy risks, and safety concerns are certain factors that prevent adoption of automated vehicles. Thus, the following describes a framework of vehicle systems that use an automatic mode of operation as well as a human driver mode of operation. Note that in this disclosure, the terms "automatic mode" and "automatic driving mode" are used interchangeably to correspond to automatic operation of the vehicle, while the terms "human driver mode" and "manual mode" are used interchangeably to correspond to operation of the vehicle by a human driver.
FIG. 1 shows an illustrative view showing operating mode switching among vehicles, according to an embodiment of the present teachings. As shown in fig. 1, the vehicle includes a driving
By one embodiment, the driving
Upon receiving an instruction indicating an intention to perform the vehicle operation mode switching, the multimodal switching
FIG. 2 illustrates an exemplary block diagram of the driving
The current risk evaluator 230 is configured to determine a risk of operating the vehicle in the current operating mode. By one embodiment, to determine the risk, the current risk assessor 230 receives as input real-time vehicle data, real-time intrinsic and extrinsic data, sensor data, the driver profile 210, and map/road configuration data 220. The map/road configuration data 220 provides information about the geographic location where the vehicle is currently located. Additionally, the map/road configuration data 220 may include information corresponding to traffic (traffic) within the vehicle's geographic location. The driver profile 210 data includes information about the driver's driving history. Such information may include, for example, the number of times the driver has been involved in a violation within a predetermined period of time, the model of the vehicle the driver is operating, characteristics of the driver (e.g., whether the driver is disabled, whether the driver is myopic, whether the law requires the driver to wear a device that assists the driver in operating the vehicle (e.g., prescription glasses), weather conditions under which the driver is preferably recommended not to operate the vehicle, etc.).
Turning to FIG. 5, exemplary real-time vehicle data input to current risk evaluator 230 is shown. In particular, FIG. 5 shows an illustrative chart that presents information associated with real-time vehicle data in accordance with various embodiments of the present teachings. For example, the real-time vehicle data may include information about the current location of the vehicle (e.g., latitude and longitude locations), information about the surroundings of the vehicle, the current visibility experienced by the driver of the vehicle (e.g., amount of sun glare based on the current time), controls specific to the weight or quality of the vehicle (e.g., the current weight of occupants in the vehicle and other control parameters related to vehicle operation), and information about vehicle maintenance (e.g., current fuel volume, tire conditions, brakes, wipers), and other important parameters that determine whether the vehicle should be operated in a safe manner.
The present risk assessor 230 also receives real-time data (extrinsic and intrinsic vehicle performance) that is used in determining the risk of the vehicle's current operating mode. Turning to FIG. 4, an illustrative chart is shown that presents information associated with real-time intrinsic and extrinsic data in accordance with various embodiments of the present teachings. As shown in fig. 4, the data related to the intrinsic performance of the vehicle may include information about vehicle operating parameters, such as a supported speed of the vehicle, a supported safety level, a defect in the vehicle (e.g., low tire pressure, defective brake pads, defective headlights/taillights, etc.), and a reliability factor of the vehicle. The data relating to the vehicle's external performance may include information external to the vehicle, such as the type and condition of the road the vehicle is traversing (e.g., the steepness/slope of the road, the speed limit permitted for the road, the current vehicle and human traffic on the road, and road surface conditions such as whether the road is slippery), weather information corresponding to the vehicle environment (e.g., the amount of ice, snow, or precipitation on the road the vehicle is traversing), visibility into the vehicle environment, external events such as whether there is an accident on the vehicle's path, whether the vehicle is traversing a restricted zone (e.g., a hospital or school), or whether there is construction on the vehicle's path that results in a detour from the vehicle's originally planned path, and so forth.
A further input to the present risk assessor 230 is real-time sensor data that is used to determine the current status of the vehicle driver. Turning to FIG. 6, an illustrative chart is shown that presents information associated with a vehicle driver state in accordance with various embodiments of the present teachings. Such information may include data corresponding to the attitude of the driver and/or occupant (e.g., detected position, sitting position, whether the driver is wearing a seat belt, etc.) and the position of the driver and occupant within the vehicle. Additional information associated with the driver status may include information about the health of the driver, the functional status of the driver (i.e., whether the driver is drowsy, asleep, or drowsy, and whether the driver is intoxicated), the mental status of the driver (e.g., the level of alertness of the driver), and information about the nature of the journey made by the driver (e.g., whether the driver and/or occupants are severely injured, and whether the vehicle is traveling on the road to a hospital, etc.). The sensor data used by the current risk assessor 230 to detect the driver state described above is described below with reference to FIG. 10.
Upon receiving the above-described input, risk assessor 230 determines the risk involved in the current operating mode of the vehicle. Specifically, the risk assessor 230 determines whether it is safe to operate the vehicle in the current operating mode. Details regarding the calculation of this risk are described below with reference to fig. 7. In calculating the risk, the current risk assessor 230 determines whether the calculated risk satisfies a particular criterion (e.g., whether the calculated risk is above or below a predetermined first threshold risk level). The predetermined first threshold risk level associated with a particular operating mode of the vehicle may correspond to a safety level for operating the vehicle in the particular operating mode of the vehicle. In particular, if the calculated risk is above a first predetermined threshold risk level, it may be deemed unsafe to operate the vehicle in the current mode, and if the calculated risk is below the first predetermined threshold risk level, it may be deemed that the vehicle may continue to operate in the current mode. Based on the calculated risk satisfying (or not satisfying) the criterion, the risk evaluator 230 may determine whether it is safe to operate the vehicle in the current operating mode or whether a switch from the current mode to a different mode of operating the vehicle is to be performed.
By one embodiment, if the calculated risk associated with the current operating mode is within the allowable limit (e.g., the risk is below a first predetermined threshold risk level), the vehicle remains in the current vehicle operating mode (i.e., no transition to the vehicle operating mode is performed). However, if the calculated risk does not satisfy the criterion (e.g., the calculated risk is above a first predetermined threshold risk level), the current risk evaluator 230 actuates the
The
By one embodiment, the
Instead, the current risk assessor 230 may perform exception handling procedures, such as stopping the vehicle in a safe manner. For example, consider that the current vehicle operating mode is a human driving mode. If the risk determiner 230 determines that, for example, the driver of the vehicle is experiencing a disease emergency or life threatening event (detected via a driver status analyzer included in the risk evaluator 230 and described below with reference to FIG. 7), then the risk determiner 230 may not initialize the
By way of an embodiment, if current risk evaluator 230 determines that the current risk associated with the current vehicle operating mode is above a first predetermined threshold associated with the current mode (the safety level of the current mode) and below a second predetermined threshold level associated with the current mode, the current risk evaluator actuates switch
Conversely, if the risk associated with switching the vehicle to a different mode is determined to be high by the switching
FIG. 3 shows an illustrative flow chart of an exemplary process performed by the driving
Further, in step 320, based on the information received in step 310, the process determines the current driving mode of the vehicle and the current status of the driver operating the vehicle. Details regarding the detection of the current state of the driver operating the vehicle are described below with reference to fig. 10, while details regarding the determination of the current operating mode of the vehicle are described below with reference to fig. 7.
Further, in step 330, the process assesses a risk associated with the current operating mode of the vehicle. Thereafter, in step 340, the process executes a query to determine if the risk of operating the vehicle in the current mode, if assessed, is abnormally high. For example, as previously described, the process determines whether the assessed risk is above a second predetermined threshold associated with the current operating mode of the vehicle. If the response to the query is positive, the process moves to step 395, where the exception handling process is performed as previously described. However, if the response to the query is not qualitative, the process moves to step 350.
In step 350, the process determines a risk of a switch, i.e., a risk associated with transitioning the vehicle to a different vehicle operating mode. By way of an embodiment, and as described below with reference to fig. 12, the risk of switching may be calculated based on a driver of the vehicle performing at least one task within an estimated time of performing the at least one task, wherein the estimated time is determined based on a state of the driver. In addition, it must be understood that upon switching from the current mode to a different operating mode of the vehicle, the risk of the different operating mode of the vehicle does not exceed the risk threshold level associated with the different mode.
The process then moves to step 360 where a query is performed to determine if the risk of handover assessed in step 350 is high. By way of an example, the process may determine whether there are other switching risks associated with transitioning the vehicle to the different mode. If the response to the query in step 360 is positive, the process moves to step 395 and an exception handling process is performed, otherwise the process moves to step 370.
In step 370, the process selects at least one alert medium from a plurality of alert medium sources to alert a driver of the vehicle of an emergency switch in operating modes of the vehicle. By one embodiment, the alert medium is selected based on the risk of handover assessed in step 350. In addition, with one aspect of the present disclosure, a warning medium may be selected based on the driver's status, wherein the warning medium warns the driver to perform a specific task (described below with reference to fig. 21) in order to efficiently switch/switch the operating mode of the vehicle.
In step 380, the process performs warning based on the selected medium, and thereafter in step 390, the driving
Turning now to fig. 7, an exemplary block diagram of a risk evaluator 230 included in a driving
The risk assessor 230 as shown in FIG. 7 includes a
By one embodiment, the
The current mode detector 760 is configured to determine a current operating state of the vehicle. By way of one embodiment, the real-time vehicle data previously described with reference to FIG. 5 can be used to determine a vehicle operating mode. For example, sensors disposed within the vehicle may be configured to determine whether the driver is operating the vehicle by detecting, for example, whether a steering wheel is being operated by the driver, whether an accelerator and/or brake pedal is being controlled by the driver, and the like. It must be understood that a variety of mechanisms can be used to determine whether the vehicle is in the human driver operating mode or the automatic operating mode.
Based on the determination of the current vehicle operating mode, current mode detector 760 activates either automatic mode risk detector 730 or manual mode risk detector 770 to determine the risk associated with the respective vehicle operating mode. As inputs, the automatic mode risk detector 730 receives the real-time data and the real-time vehicle data described above with reference to fig. 4 and 5. As inputs, the manual mode risk detector 770 receives real-time data and real-time vehicle data, driver status, and information contained in the driver profile 780.
By way of an embodiment, the automatic mode risk detector 730 may determine a risk event based on environmental conditions. For example, in foggy weather, the automatic mode risk detector 730 may determine that the vehicle is not safe to operate automatically, e.g., a collision detector of the vehicle may have difficulty accurately detecting a nearby vehicle. In a similar manner, when the vehicle is traversing a steep icy road, it may be feasible to operate the vehicle under human control, i.e., the automatic mode may not be suitable for controlling the vehicle under severe (icy) road conditions. In addition, there may also be certain geographic areas where autonomous driving is prohibited or deemed unsafe, such as acceleration lanes, drive-off lanes, toll booths, known construction zones, school zones, and portions of roads near these areas. The risk associated with these events is determined by the automatic mode risk detector 730.
Similar to the scenario of the dangerous autopilot scenario described above, by one embodiment, the manual mode risk detector 770 may determine the situation: where it may be dangerous to run the vehicle under manual control. For example, the manual mode risk detector 770 may determine that the driver is likely to encounter an accident in the imminent future based on the detected driver state and the driver profile 780. Such a determination may be based on the traffic density in the vicinity of the vehicle under consideration. As another example, the risk associated with a manual mode of operation of the vehicle may correspond to detecting that the vehicle has deviated from its predetermined path several times (within a predetermined time window). For example, an in-vehicle sensor may detect that the vehicle is being operated by the driver in such a manner: the vehicle is off center in the lane and enters the adjacent lane. The manual mode risk detector 770 may treat the occurrence of such an event as a risky event based on the amount of traffic in the vicinity of the vehicle. Similarly, the driver state detector may determine that the driver is in a drowsy, and/or yawned state, and running the vehicle in the human driving mode may be at risk.
According to the driving risk model 720, the automatic mode risk detector 730 and the manual mode risk detector 770 detect respective risks associated with operating the vehicle in the automatic mode or the manual mode, respectively. Specifically, based on the respective input information, the risk patterns 720 are used by the automatic pattern risk detector 730 and the manual pattern risk detector 770 to assess the risk of operating the vehicle in the respective modes.
By one embodiment, the automatic mode risk detector 730 and the manual mode risk detector 770 input information about the driver and/or the detected scenarios about the geographic location through which the vehicle is traversing to the risk report generator 740. The risk report generator 740 uses the driving risk model 720 to generate a risk report associated with an automatic mode or a manual mode of vehicle operation, respectively, and outputs a current risk associated with the operating mode of the vehicle.
Additionally, with one embodiment of the present disclosure, the automatic mode risk detector 730 and the manual mode risk detector 770 provide inputs (indicative of the separately detected risks) to the anomaly risk processor 750. As described below with reference to FIG. 12, with one embodiment, upon detecting a risk in a current operating mode of the vehicle, the vehicle system may determine a risk associated with switching the vehicle to a different operating mode. If the risk associated with the different operating modes is within the allowable limit, the vehicle system may perform an operation to switch the operating mode of the vehicle.
However, if the risk associated with the different vehicle operating mode is not acceptable, the current risk evaluator 230 may activate an abnormal risk processor 750, which may be configured to perform safety-related functions as previously described based on the driver profile. For example, the abnormal risk processor 750 may perform processing such as bringing the vehicle to an immediate stop, initiating a call for government regulatory assistance (e.g., auto-dialing 911 or calling roadside assistance via a mobile phone associated with the vehicle), sending the vehicle GPS address to a server that monitors the operation of a particular vehicle and other similar vehicles. It must be understood that in order to perform the function of bringing the vehicle into a complete stop described above, it may comprise the processing steps of: for example, the current speed of the vehicle, the current lane in which the vehicle is traveling, nearby traffic, identifying the shoulder area of the road (e.g., the side parking area), and gradually bringing the vehicle to a complete stop in a safe manner.
FIG. 9 shows an illustrative flow chart outlining an exemplary process performed by risk evaluator 230 according to one embodiment of the present teachings. The process begins at step 905, where the activity of the driver and/or occupant of the vehicle under consideration (described below with reference to FIG. 10) is detected via various sensors.
Based on the information detected by the sensors and a driver detection model (described below with reference to FIG. 10), the process in step 910 analyzes the driver/occupant status. In step 915, the current operating state of the vehicle is determined. For example, the current mode detector 760 shown in fig. 7 may detect a current operating mode of the vehicle.
The process further proceeds to step 920 where a query is performed to determine if the vehicle is operating in the automatic mode. If the response to the query is positive, the process moves to step 925, otherwise, if the vehicle is operating in manual mode, the process moves to step 930.
In step 930, the process detects a risk associated with a vehicle operating in a manual mode. As introduced previously, the determination of such risk may be based at least on the detected state of the driver, information contained in the driver profile, real-time information about the vehicle the driver is operating. It must be understood that the information contained in the driver profile is static driver information corresponding to the characteristics of the driver, such as whether the driver is disabled, myopic, medical information of the driver, etc., and the information related to the detected driver's state is dynamic information based on the current behavior of the driver.
Returning to step 920, if it is determined that the vehicle is operating in the automatic mode, the process moves to step 925 and obtains the operating state of the vehicle in the automatic mode. For example, the process may determine the location of the vehicle, the current speed at which the vehicle is traveling, and similar information. By way of example, the process may be configured to determine a level of an automatic mode (level) at which the vehicle is operating (described below with reference to FIG. 17B).
Upon determining the respective risk of operating the vehicle in the automatic mode (step 935) or in the human driving mode (step 930), the process moves to step 940. In step 940, a query is performed to determine whether exception handling is required, i.e., the risk of the determined automatic mode or human driving mode of the vehicle is abnormally high. If the response to the query is positive, the process moves to step 950, where the vehicle is operated in an exception handling mode, performing at least one of the exception handling functions previously described.
However, if the response to the query in step 940 is negative, the process moves to step 955, where a risk assessment report is generated. Additionally, the generated risk assessment report may be displayed on a display panel included within the vehicle and/or transmitted to a remote server for monitoring purposes, as shown in step 960.
FIG. 10 illustrates an exemplary block diagram of a driving
By one aspect of the present disclosure, the
By one embodiment, the
Information detected by the
By one embodiment, the detected driver behavior is input to a
According to an embodiment of the present disclosure, information about the behavior of the driver may be input to the
Based on the functional state of the driver estimated by the
FIG. 11 shows an illustrative flow chart summarizing an exemplary process performed by
The process then proceeds to step 1140, where the driver's behavioral characteristics are detected. Specifically, in step 1140, based on the information detected in step 1130, the behavioral characteristic model may be used to detect behavioral characteristics, such as whether the driver is asleep, drowsy, nausea, etc. The behavioral characteristics detected in step 1140 may be used in steps 1150, 1160, and 1170 to estimate the functional status of the driver, the health of the driver, and the mental status of the driver, respectively, as described above with reference to fig. 10.
The detected functional state, health and mental state of the driver may be used in step 1180 to generate an overall state of the driver. For example, the current driver state generator (1090 in FIG. 10) may be used to determine an overall state of the driver based on detected mental, functional, and health information of the driver.
The process then proceeds to step 1190, where the generated driver state from step 1180 may be output for inclusion in the current mode risk report generator (740 in FIG. 7) and/or for use by the switching risk determiner and switching risk evaluator to determine, for example, a risk associated with the current mode in which the vehicle is operating.
FIG. 12 illustrates an exemplary block diagram of a
As an input, the
By way of an embodiment, a-H
In addition, a-H
However, if either the a-H
Fig. 13 shows an illustrative flow chart outlining an exemplary process performed by the
The process further proceeds to step 1320, where a query is performed to determine whether the vehicle intends to switch from an automatic mode of operating the vehicle to a human drive (A-H) mode, or whether the vehicle intends to switch from a human drive of operating the vehicle to an automatic (H-A) mode. If the response to the query is to switch from automatic mode to human driving mode (i.e., A-H switch), the process moves to step 1325. If the response to the query is to switch from human driving to automatic mode (i.e., H-A), the process moves to step 1350.
In step 1325, the switch risk determiner determines a risk associated with performing a switch from an automatic mode to a human driving mode of operating the vehicle. As will be described in detail later with reference to fig. 14 and 17B, when switching from the automatic mode to the human driver mode, the driver is presented with a set of presumed tasks to be performed in order to switch the operating mode of the vehicle. Based on whether the driver performs the set of presumed tasks, the a-H
Thereafter, the process in step 1330 performs a query to determine whether a high risk is associated with transitioning to human driving mode. If the response to this query is positive, the process moves to step 1345, where an exception handling process is performed. However, if the response to the query in step 1330 is negative, the process moves to step 1335.
In step 1335, the process generates an a-H switch command alert that is associated with a set of tasks to be performed by the vehicle driver before a successful switch to the human drive mode of vehicle operation can occur. Thereafter, the process in step 1340 generates (and outputs) an A-H vehicle control signal to instruct the control system of the vehicle to perform a switching operation while the driver is performing the set of presumed tasks.
Similar to steps 1325 through 1345 described above, the switch risk determiner in steps 1350 through 1370 performs functions related to switching from a human driving mode to an automatic mode of operating the vehicle. Specifically, in step 1350, the switch task determiner determines a risk associated with performing a switch from a human driving mode to an automatic mode of operating the vehicle.
The process then moves to step 1355, where a query is made to determine if the high risk is associated with a transition to an automatic mode of operating the vehicle. If the response to the query is positive, the process moves to step 1370, where an exception handling process is performed. However, if the response to the query in step 1335 is not qualitative, the process moves to step 1360.
In step 1360, the process generates an H-a switch order alert that is associated with a set of tasks to be performed by the vehicle driver when a successful switch to the automatic mode of operating the vehicle can occur. Thereafter, the process generates (and outputs) an H-A vehicle control signal in step 1365 to instruct the vehicle control system to perform the switch to the auto-run mode.
FIG. 14 illustrates an exemplary block diagram of an automatic mode-to-human driver/manual (A-H) mode switching
With an embodiment of the present disclosure, the a-H
Additionally, as inputs, switching
Before switching to the manual operating mode of the vehicle, a judgment may be required to verify whether the driver is able to operate the vehicle. Such a determination may include verifying the posture of the driver, determining the driver's level of vigilance, and the like. By way of an embodiment, the driver's level of vigilance may be determined by applying a series of tasks to the driver and determining whether the driver is performing the indicated tasks in a satisfactory manner (where, for example, the tasks may be displayed in the form of instructions on a display panel contained within the vehicle). These tasks may include instructing the driver to perform lane changes, displaying objects (e.g., flashing icons) on the vehicle display panel, and tracking the movement of the driver's eyes to determine the degree of vigilance, among other things. Additionally, a certain amount of time may be required to determine the health of the driver (e.g., via medical sensors such as blood pressure, heart rate monitors, etc. that measure key indicators of the driver, which are processed by sensors (described below with reference to FIG. 24) included in the vehicle system).
Additionally, based on the real-time information received by switching
As shown in fig. 14, the output of the switching
The
By way of an example, as shown in fig. 16, the response time required to perform the set of inferred tasks may be determined based on driver physiological information (e.g., the driver's current posture or attitude), user/driver information (e.g., the time required to previously switch the manual operating mode, the driver's status, etc.), extrinsic factors (e.g., road conditions, road visibility, traffic volume on the road), and intrinsic factors of the vehicle (e.g., the vehicle's allowable speed, the vehicle's allowable safety, the vehicle's condition (e.g., the number of vehicle defects)), and so forth.
The estimated set of tasks to be performed and the estimated corresponding response times to perform the tasks are input to the
Turning to fig. 15, an illustrative flow chart outlining an exemplary process performed by the (a-H) mode switch risk determiner is shown in accordance with an embodiment of the present disclosure. The process begins in step 1510, where (a-H) the switch mode risk determiner receives real-time information including intrinsic and extrinsic properties of the vehicle and real-time vehicle data.
The process then moves to step 1520 where (a-H) the switch mode risk determiner presumes a set of tasks to be performed by the human driver to successfully switch the vehicle operating mode to the human driving mode. Note that with one embodiment, the set of tasks may be inferred based on the received real-time information according to a switching task model.
In step 1530, the response time, i.e., the amount of time required for the vehicle operator to complete each of the set of tasks, is estimated. It must be appreciated that the response time may be calculated based on at least one of the driver profile, the current driver state, and the universal response time profile as previously described with reference to FIG. 14.
The process further proceeds to step 1540, where the risk associated with switching the vehicle operation mode to the human driving mode is determined. As previously described, the risk of switching may be determined based on a comparison of the total estimated time to perform the set of tasks and the time at which the vehicle should transition to the different mode (e.g., human driver mode), which is determined based on a risk level associated with operating the vehicle in a current mode (e.g., automatic mode).
Next, the process in step 1550 executes a query to determine (in step 1540) whether the determined risk is high. For example, a determination may be made to verify whether the risk determined in step 1540 violates a particular criterion, e.g., TEWhether greater than T. If the response to the query in step 1550 is positive, the process proceeds to step 1570, otherwise the process proceeds to step 1560.
In step 1560, the criterion (i.e., T) is not violated in response to the determined risk in step 1540EBelow T), the process generates an a-H switching alert command signal that is further used in step 1580 to output (e.g., display on a panel) the set of presumed tasks to the driver of the vehicle. Details regarding the presentation of the set of tasks to the driver are described in more detail below with reference to fig. 19 and 21. However, if the risk in step 1550 is determined to be high, the process performs an exception handling step as previously described in step 1570.
FIG. 17A illustrates an exemplary block diagram of a human driver to automatic (H-A) mode switching risk determiner according to an embodiment of the present teachings. As previously described, the H-a
As shown in fig. 17A, the H-a
Based on the received input, the
With one embodiment of the present disclosure, the H-A
By an embodiment, the determined sequence of tasks, driver status, and current vehicle data are input to the
By one embodiment, the
By one embodiment, the risk associated with switching to the automatic mode may be determined based on a level/degree of risk associated with the current operating mode of the vehicle. For example, similar to the description provided above with reference to the a-H handoff risk determiner, the H-a handoff risk determiner determines a risk level associated with operating the vehicle in a human driver mode. Based on the risk level, the H-A switch risk determiner obtains an amount of time that the vehicle should switch to the automatic operating mode for safety considerations. Additionally, the automatic
Based on the assessed risk being within an acceptable range, the automatic
As described above, it must be understood that, in the present embodiment, the respective functions/tasks to be performed by the vehicle driver correspond to the tasks to be performed in the automatic operation mode. Additionally, if one of the tasks determined by the auto-
Additionally, with one embodiment, when the risk associated with transitioning to the automatic mode is unacceptable, the
FIG. 17B provides a classification of various vehicle operating modes based on six different levels (ranging from fully manual to fully automatic systems) in accordance with an embodiment of the present teachings. The classification may be based on the amount of driver intervention and attention required. As shown in fig. 17B, there are six vehicle operating modes/levels:
mode 0, where the automated system issues an alert, but there is no vehicle control.
Mode 1 (also referred to as "hands-on" mode below) is a mode in which the driver of the vehicle and the automatic system share vehicle control. For example, Adaptive Cruise Control (ACC), where the driver controls the steering function and the automated system controls the vehicle speed, is an example of mode 1 operation. In addition, features such as parking assist (where steering is automatic and speed is manual) and Lane Keeping Assist (LKA) are examples of mode 1 operation of the vehicle. In mode 1 operation, the driver must be ready to regain full control of the vehicle at any time.
Mode 2 (also referred to herein as "hands-off" mode) is an operating mode in which the automatic system assumes full control of the vehicle (acceleration, braking, and steering). The driver monitors the driving and expects to be ready to intervene immediately in the event that the automated system fails to respond properly.
Mode 3 (also referred to as "out of eye" mode) is a mode in which the driver can safely divert his/her attention away from the driving task, e.g., the driver can text or watch a movie. The vehicle handles situations that require immediate response, such as emergency braking. In this mode, the driver must still be ready to intervene within some limited time frame specified by the manufacturer when the vehicle requires to do so. When actuated by a human driver, the vehicle achieves full control of all aspects of the driving of slow moving traffic at speeds up to 60 kilometers per hour. In some instances, the vehicle is programmed to operate in this mode only on highways with physical barriers separating oncoming traffic.
Mode 4 (also referred to herein as "centrifugal" mode) is similar to mode 3, but does not even require driver attention for safety purposes, i.e., the driver can safely fall asleep or leave the driver's seat. With an embodiment, the autopilot performance is supported only in limited areas or in certain situations, such as traffic jams. Outside of these areas or situations, the vehicle must be able to safely suspend the journey, i.e. stop the vehicle if the driver does not regain control.
In mode 5 (also referred to herein as "steering wheel selectable" mode), no human intervention is required. Thus, as previously described, with one embodiment, the driving
Thus, by one embodiment, the H-A switch risk determiner may determine the risk associated with switching from a human driver mode to all levels of the automatic mode shown in FIG. 17B. It must be appreciated that switching to a lower level of vehicle automation may require the vehicle automation system to perform a smaller number of tasks than switching to a fully automated level. However, a lower level of automatic control may require a greater level of driver vigilance.
Turning to fig. 18, an illustrative flow chart outlining an exemplary process performed by an (H-a) switching pattern risk determiner is shown in accordance with an embodiment of the present disclosure. The process begins at
The process then moves to step 1820, where (H-a) the switch mode risk determiner infers a set of tasks to be performed by the vehicle when switching the vehicle operation mode to the automatic mode, based on the automatic takeover task model. Additionally, in
In
Based on the determined risk associated with switching the vehicle to the autonomous mode (in step 1840) that the criteria are not violated, the process generates an H-a switch alert command signal and a control signal (in step 1860) in
Additionally, a switch alert command signal may be output (e.g., on a display panel) to display the set of presumed tasks to be performed by the vehicle driver. Details regarding the presentation of the set of tasks to the driver are described in more detail below with reference to fig. 19 and 21.
FIG. 19 illustrates an exemplary block diagram of a switching alert control unit 250 included in a driving mode switching unit (block 130 in FIG. 1) of a vehicle system, according to one embodiment of the present teachings. The switch alert control unit 250 includes an alert instruction analyzer 1910, a vehicle model 1920, a user profile 1930, an alert time determiner 1950, an alert content determiner 1940, an alert medium determiner 1970, a task/alert configuration model 1960, a plurality of multi-modal media models 1980, and a multi-modal alert instruction generator 1990.
By one embodiment, the alert directive analyzer 1910 receives an alert directive signal corresponding to a switch from automatic to human drive (A-H) mode or from human drive to automatic (H-A) mode of vehicle operation mode. Specifically, the alert instruction analyzer 1910 receives one of an A-H handoff alert instruction signal from an A-H handoff instruction generator (block 1450 of FIG. 14) and an H-A handoff alert instruction signal from an H-A handoff instruction generator (block 1760 of FIG. 17). It must be understood that the a-H switch alert command and the H-a switch alert command signal each correspond to a set of tasks to be performed in order to switch the operating mode of the vehicle. In addition, the alert instruction analyzer 1910 obtains an estimate of the time required to complete each task (referred to herein as the task duration).
The warning instruction analyzer 1910 inputs each analyzed task to a warning time determiner 1950. The alert time determiner 1950 determines the amount of time needed to perform an alert operation (referred to herein as an alert duration). In particular, the alert duration corresponds to a time period: a warning prompt is presented to the vehicle driver to ensure that the driver performs the corresponding task in due course. By one embodiment, the duration of the alert for each task is determined based on the current state of the driver. In addition, the alert duration for each task may be adjusted based on the current state of the driver. Details regarding adjusting the alert duration for each task are described below with reference to FIG. 20B.
For example, consider a task having a 30 second task duration, i.e., the vehicle driver expects to complete the task within 30 seconds. In this case, the warning time determiner 1950 may determine that a warning duration of 10 seconds is to be used to provide a warning prompt to the driver, thereby notifying the driver of the corresponding task to be performed. By one embodiment, the alert duration is a fraction of the task duration. Specifically, referring to the example above, a warning time of 10 seconds (within a time window of 30 seconds) is presented to the driver (e.g., at the beginning of the time window) to inform the driver to perform a task within the estimated window of 30 seconds. It must be understood that the warning time having an initial duration of 10 seconds may extend to, for example, 15 seconds when it is determined that the driver has not performed the assigned task.
Accordingly, as described above, the alert time determiner 1950 may determine alert durations for various tasks based on the current state of the driver. For example, consider a case where a driver is expected to perform a particular task, and the driver state analyzer (fig. 7) determines that the driver is not fully awake (e.g., yawned, drowsy, etc.). In such a case, the alert time determiner 1950 may determine that there is a longer alert duration than the driver state analyzer determines that the driver is fully alerted. It must be noted that the task duration for completing a specific task is estimated by the switching time estimator (e.g., 1720 in fig. 17) based on the driver state.
For each task, the alert time determiner 1950 inputs each task and an alert duration associated with the task to the alert medium determiner 1970 and the alert content determiner 1940. Additionally, as inputs, the alert medium determiner 1970 receives information about the vehicle model 1920, a user profile 1930 corresponding to the driver operating the vehicle, and the current status of the vehicle driver. By an embodiment, the alert medium determiner 1970 is configured to select (from a pool of the multi-medium models 1980) at least one medium to be used as a platform for providing alerts to the driver. Specifically, the warning medium determiner 1970 associates each task with the at least one medium 1980 for warning the driver to perform the corresponding task.
By one embodiment, the selected medium may be determined based on the driver's preferences. The driver's preferences may be predetermined and stored a priori within the driver profile 1930. Additionally or alternatively, the type of media selected may be determined based on the driver's condition. For example, if the driver is in a yawned state (as determined by the driver
By one embodiment, the alert content determiner 1940 determines alert content associated with each task. For example, for a display type medium, the content may correspond to a particular message to be displayed on the vehicle display panel. The displayed message serves as a warning to the driver of the vehicle. In a similar manner, the audio content may correspond to content to be played via an audio system included in the vehicle. The content related to the audio or video mode of the alert may also include decibel level (i.e., how loud the audio message should be), the frequency at which the alert is presented, the brightness of the displayed message on the panel, whether the message is displayed with an effect such as blinking, and the like.
With regard to the vibration-type media alert, the alert content determiner 1940 may determine a magnitude of the vibration. For example, if the vibration pattern alerting the driver includes providing a vibration to the steering wheel or the driver's seat, the alert content determiner may determine a magnitude of the corresponding vibration to be applied to the driver. As shown in fig. 19, alert content determiner 1940 may also use task/alert configuration model 1960 to associate various tasks with at least one media source and present alerts to the driver.
In addition, the type of media source used for alerting purposes (as determined by alert media determiner 1970) and alert content (as determined by alert content determiner 1940) are input to multimodal alert instructions generator 1990. Upon receiving input from the alert content determiner 1940 and the alert medium determiner 1970, the multimodal alert instruction generator 1990 generates a sequence of tasks to be performed by the driver.
By way of an embodiment, the multimodal alert instruction generator 1990 is also configured to generate an alert plan for the task for prompting the driver. In particular, the multimodal alert instruction generator 1990 determines a sequence in which tasks will be presented to the driver. For example, the multimodal alert instruction generator 1990 may generate a plan based on task durations of various tasks, wherein the task with the greatest task duration is presented to the driver first. Alternatively, the multimodal alert instruction generator 1990 may first present the driver with the task having the minimum task duration. In addition, the multimodal alert instructions generator 1990 may generate a task plan based on criticality scores (i.e., priorities) associated with the various tasks. The criticality score of a task corresponds to a level of urgency for performing the task. For example, when switching to an automatic mode of operating the vehicle, the task of, for example, removing the driver's foot from the accelerator pedal may be considered more important than, for example, changing lanes before switching the vehicle operating mode. In such a scenario, the multimodal alert instruction generator 1990 may generate a plan in which the task of removing the driver's foot from the pedal (i.e., the task with higher priority) is first presented to the driver. A detailed description of the execution of the multimedia alerting instruction is given below with reference to fig. 21.
FIG. 20A shows an illustrative flow chart outlining an exemplary process performed by the handover alert control unit 250 in accordance with one embodiment of the present teachings. The process begins at
In
The process then moves to step 2050, where the switching alert control unit determines the alert content to be associated with each task. For example, as previously described with reference to fig. 19, the alert content determiner 1940 determines alert content to be associated with each task. In addition, in
Upon determining the contents and media to be associated with the respective tasks (
Turning now to FIG. 20B, an
Additionally, FIG. 20B shows at least one alert medium (from a set of four
It must be understood that
In addition, as shown with reference to task 2, a variety of media may be used to alert the driver to the corresponding task to be performed. Note that the various media used to alert the driver to perform the necessary tasks may be presented simultaneously, as shown in FIG. 20B, and/or may be presented in a sequential manner (i.e., one after the other). In addition, as previously described, the alert duration for each task may be configured based on the state of the driver. As shown in fig. 20B,
For example, if it is observed that the driver has not performed a task at the time the alert is presented, the duration of the alert may be adjusted. In a similar manner, the intensity of the type of media used to alert the driver to perform the corresponding task may also be adjusted as indicated by arrows 2089 a-2089 c. It must be appreciated that when using multiple media sources to alert the driver to perform a task (e.g., as shown in fig. 20B for task 2), the alert duration and intensity of each media type may be controlled in an independent manner. Additionally, with one embodiment, the alert duration for each task is preferably a fraction of the corresponding task duration. It must be noted that the alert duration for a particular task may have a maximum duration that is slightly less than the corresponding task duration (assuming that the vehicle driver takes a short period of time to perform the task). In addition, as previously described, if the driver of the vehicle does not perform a particular task within the estimated task duration, the vehicle control can terminate the operation of switching the vehicle mode and, instead, initiate the exception handling process.
FIG. 21 illustrates an exemplary block diagram of a multi-modal
The plurality of multimodal media models 2130 includes at least selectable sound media 2130-1, vibration media 2130-2, lights 2130-3, visual media 2130-4, enhanced visual media 2130-5, and audio media 2130-6. By one embodiment, each media contained in multimodal media model 2130 includes a corresponding media generator configured to select and actuate the corresponding media. For example, as shown in FIG. 21, the plurality of
As an input, the multi-model
By one embodiment, the multimodal alert instructions analyzer 2110 uses the
Accordingly, for each task, the selected media is input to the
By one embodiment, the multimodal
By one embodiment, information captured by the plurality of sensors may be input to alert
Turning to fig. 22, an illustrative flow chart is shown that outlines an exemplary process performed by the multimodal
The process further proceeds to step 2230, where, based on the media selected for the respective task (i.e., selected by the alert media determiner 1970 in fig. 19), the multimodal
In addition, the process coordinates actuation of all media associated with the particular task in step 2240 using the alert coordinator of the multimodal
The process then moves to step 2260, where the multimodal switching alert unit observes the effect of the alert. As previously described, the multimodal switch alert unit collects information from multiple sensors to observe whether alerts presented to a user are being followed. Additionally, the process determines in step 2270 whether the planned task is complete based on the observations of step 2260.
In response to the driver not completing the planned task, the multimodal
FIG. 23 illustrates an architecture of a mobile device 2300 that may be used to implement particular systems embodying the present teachings. Such a mobile device 2300 includes, but is not limited to, a smart phone, a tablet, a music player, a handheld game, a Global Positioning System (GPS) receiver, and a wearable computing device (e.g., glasses, a wristwatch, etc.), or in any other modality. The mobile device 2300 in this example includes: one or more Central Processing Units (CPUs) 2340; one or more Graphics Processing Units (GPUs) 2330; a memory 2360; a communication platform 2310, such as a wireless communication module; a memory 2390; one or more input/output (I/O) devices 2350; a display or projector 2320-a for vision-based presentation; and one or more multimodal interface channels 3020-b. The multi-modal channels may include auditory channels or other media channels for signaling or communication. Any other suitable components, including but not limited to a system bus or a controller (not shown), may also be included in mobile device 2300. As shown in fig. 23, a mobile operating system 2370 (e.g., iOS, Android, windows phone, etc.) and one or more applications 2380 may be loaded from memory 2390 into memory 2360 for execution by CPU 2340.
To implement the various modules, units, and functions thereof described in this disclosure, a computer hardware platform may be used as a hardware platform for one or more of the elements described herein. The hardware elements, operating system, and programming languages of such computers are conventional in nature, and it is assumed that those skilled in the art are sufficiently familiar with them to adapt these techniques to the present teachings presented herein. A computer with user interface elements may be used to implement a Personal Computer (PC) or other type of workstation or terminal device, but the computer may also operate as a server if suitably programmed. It is believed that one skilled in the art is familiar with the structure, programming, and general operation of such computer devices, and thus the drawings may be self-explanatory.
FIG. 24 illustrates an architecture of a computing device that can be used to implement particular systems that implement the present teachings. This particular system implementing the present teachings has a functional block diagram of a hardware platform that includes user interface elements. The computer may be a general purpose computer or a special purpose computer. Both of which can be used to implement a particular system for use with the present teachings. Such a computer 2400 may be used to implement any of the components or aspects of the present teachings as described herein. Although only one such computer is shown for convenience, the computer functions associated with the present teachings described herein may be implemented in a distributed fashion across several similar platforms to spread the processing load.
For example, the computer 2400 includes a COM port 2450 connected to a network connected thereto to facilitate data communications. The computer 2400 also includes a Central Processing Unit (CPU)2420 in the form of one or more processors for executing program instructions. An exemplary computer platform includes: an internal communication bus 2410; various forms of program storage and data memory, such as a disk 24170, Read Only Memory (ROM)2430 or Random Access Memory (RAM)2440, for various data files to be processed and/or communicated by a computer and possibly program instructions to be executed by a CPU. The computer 2400 also includes an I/O component 2460 that supports input/output flows in the form of different media between the computer and other components herein (e.g., interface elements 2480). An exemplary type of interface element may correspond to different types of sensors 2480-a configured on an autonomous vehicle. Another type of interface element may correspond to a display or projector 2480-b for vision-based communication. There may also be additional components for other multimodal interface channels, such as: an auditory device 2480c for audio-based communication; and/or a component 2480-d for signaling, e.g., a signal to cause a vehicle component (e.g., a vehicle seat) to vibrate, based on the communication. The computer 2400 may also receive programming and data via network communications.
Thus, embodiments of the methods of the present teachings as outlined above may be implemented in a program. Program aspects of the present technology may be viewed as an "article of manufacture" or "article of manufacture" typically in the form of executable code and/or associated data carried on or implemented in a machine-readable medium. Tangible, non-transitory "memory" type media include any or all of memory or other memory for a computer, processor, etc., or associated modules thereof, such as various semiconductor memories, tape drives, disk drives, etc., that may provide storage for software programming at any time.
All or a portion of the software may sometimes be transmitted over a network, such as the internet or various other telecommunications networks. For example, such a transfer may enable loading of software from one computer or processor into another (e.g., from a management server or search engine operator's host or other enhanced advertisement server onto a hardware platform of a computing environment or other system implementing the computing environment or similar functionality associated with the present teachings). Thus, another type of medium that can carry software elements includes optical, electrical, and electromagnetic waves, for example, used through physical interfaces between local devices, through wired and optical fixed networks, through various air links. The physical elements carrying such waves (e.g., wired or wireless links, optical links, etc.) are also considered to be media carrying software. As used herein, unless limited to a tangible "storage" medium, terms such as a computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.
Thus, a machine-readable medium may take many forms, including but not limited to, tangible storage media, carrier wave media, or physical transmission media. Non-volatile storage media include any storage device, such as optical or magnetic disks, such as any computer, etc., which may be used to implement the system shown in the figures or any component thereof. Volatile storage media includes dynamic memory, such as the main memory of such computer platforms. Tangible transmission media include: coaxial cables, copper wire and fiber optics, including the wires that comprise a bus within a computer system. Carrier-wave transmission media can take the form of electrical or electromagnetic signals, or acoustic or light waves such as those generated during Radio Frequency (RF) and Infrared (IR) data communications. Common forms of computer-readable media therefore include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any other optical medium, punch cards, paper tape, any other physical storage medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave transporting data or instructions, a link or cable carrying such a carrier wave, or any other medium from which a computer can read programming code and/or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a physical processor for execution.
It will be apparent to those skilled in the art that the present teachings are applicable to numerous modifications and/or enhancements. For example, although the implementation of the various components described above may be implemented in a hardware device, it may also be implemented as a software-only solution, for example installed on an existing server. Additionally, the teachings disclosed herein may be implemented as firmware, a firmware/software combination, a firmware/hardware combination, or a hardware/firmware/software combination.
While the present teachings and/or other examples have been described above, it will be appreciated that various modifications may be made thereto, and that the subject matter disclosed herein may be implemented in various forms and examples, and that the present teachings may be applied in numerous applications, only some of which have been described herein. The appended claims are intended to claim any and all such applications, modifications and variations that fall within the true scope of the present teachings.