阅读说明：本技术 信息处理方法、信息处理装置以及记录介质 (Information processing method, information processing apparatus, and recording medium ) 是由 小川贵生 小出雅士 于 2019-07-25 设计创作，主要内容包括：本发明提供一种服务器装置的信息处理方法、信息处理装置以及记录介质,该信息处理方法,获取从设置在空间内的一个以上的传感器中的至少其中之一获取的第1信息；基于所述第1信息,检测存在于所述空间内的物品的破损；基于所述第1信息,推测发生了所述物品的破损的状况；根据所述推测的状况,输出使自走式装置在所述空间内执行规定的动作的第2信息。(The invention provides an information processing method, an information processing device and a recording medium of a server device, the information processing method comprises the steps of obtaining 1 st information obtained from at least one of more than one sensors arranged in a space; detecting a breakage of an article present in the space based on the 1 st information; estimating a state of occurrence of breakage of the article based on the 1 st information; and outputting 2 nd information for causing the self-propelled device to execute a predetermined operation in the space, based on the estimated state.)

1. An information processing method of an information processing apparatus, comprising:

acquiring 1 st information acquired from at least one of one or more sensors disposed in a space;

detecting a breakage of an article present in the space based on the 1 st information;

estimating a state of occurrence of breakage of the article based on the 1 st information;

and outputting 2 nd information for causing the self-propelled device to execute a predetermined operation in the space, based on the estimated state.

2. The information processing method according to claim 1,

the 2 nd information is information for causing the self-propelled apparatus to output a predetermined sound in the space based on the estimated state.

3. The information processing method according to claim 1,

further, the 3 rd information is output to cause the presentation means to present information for changing the estimated condition.

4. The information processing method according to claim 1,

the self-propelled device is a self-propelled cleaning machine,

the 2 nd information is information for causing the self-propelled cleaning machine to clean the damaged article in the space based on the estimated state.

5. The information processing method according to claim 1,

the presumed condition is a condition of intrusion of a suspicious person into the space,

the 2 nd message is a message for causing the self-propelled device to perform an action in the space that interferes with the suspicious person.

6. The information processing method according to claim 5,

further acquiring image data in which the suspicious person is photographed from a photographing apparatus disposed in the space,

the acquired image data and notification information for notifying the existence of the suspicious person are further transmitted.

7. The information processing method according to claim 5,

when the damage information indicating that the self-propelled device is damaged is acquired, the 4 th information requesting repair of the self-propelled device is further output.

8. The information processing method according to claim 1,

further, 5 th information is output for the presentation device to present information relating to the article for suppressing the occurrence of the estimated situation.

9. The information processing method according to claim 1,

the one or more sensors include at least one of a microphone device and a camera device disposed within the space,

the 1 st information contains at least one of sound data acquired by the microphone device and image data acquired by the photographing device,

the estimation of the situation estimates the situation in which the damage of the article has occurred, based on at least one of the sound data and the image data.

10. The information processing method according to claim 1,

acquisition of the 1 st information, acquiring the 1 st information at prescribed time intervals,

the estimation of the situation estimates the situation in which the damage of the article has occurred based on a plurality of 1 st information acquired within a predetermined period of time with reference to a time point at which the damage of the article has occurred.

11. An information processing apparatus characterized by comprising:

an acquisition unit that acquires 1 st information acquired from at least one of one or more sensors provided in a space;

a detection unit that detects a breakage of an article present in the space based on the 1 st information;

an estimation unit configured to estimate a state in which the article is damaged based on the 1 st information; and the number of the first and second groups,

and an output unit that outputs 2 nd information for causing the self-propelled device to perform a predetermined operation in the space, based on the estimated state.

12. A recording medium that is readable by a computer and stores an information processing program, the information processing program causing the computer to function as:

acquiring 1 st information acquired from at least one of one or more sensors disposed in a space;

detecting a breakage of an article present in the space based on the 1 st information;

estimating a state of occurrence of breakage of the article based on the 1 st information; and the number of the first and second groups,

and outputting 2 nd information for causing the self-propelled device to execute a predetermined operation in the space, based on the estimated state.

Technical Field

The present invention relates to an information processing method, an information processing apparatus, and a computer-readable recording medium storing an information processing program for causing a device to execute a predetermined operation.

Background

Conventionally, there is known an electric cleaning device that recognizes a registered foreign object stored in a storage unit by comparing a captured image captured by an imaging unit with an image of the registered foreign object stored in the storage unit during a cleaning operation and performing image recognition (see, for example, japanese patent application laid-open No. 5771885). When the foreign object is recognized, the electric cleaning device controls the suction driving unit based on the control pattern stored in the storage unit corresponding to the recognized foreign object, and displays a display for identifying the recognized foreign object on the display screen.

However, in the above-described conventional technology, the occurrence of damage to the article is not estimated, and further improvement is required.

Disclosure of Invention

The present invention has been made to solve the above-described problems, and an object of the present invention is to provide an information processing method, an information processing apparatus, and a computer-readable recording medium having an information processing program recorded thereon, which are capable of causing a self-propelled apparatus to execute a predetermined operation in response to a situation in which an article is damaged.

An information processing method according to an aspect of the present invention is an information processing method of an information processing apparatus, including the steps of: acquiring 1 st information acquired from at least one of one or more sensors disposed in a space; detecting a breakage of an article present in the space based on the 1 st information; estimating a state of occurrence of breakage of the article based on the 1 st information; and outputting 2 nd information for causing the self-propelled device to execute a predetermined operation in the space, based on the estimated state.

Drawings

Fig. 1 is a schematic diagram showing the configuration of an equipment control system according to embodiment 1 of the present invention.

Fig. 2 is a schematic diagram showing a configuration of a server device according to embodiment 1 of the present invention.

Fig. 3 is a schematic diagram showing an example of the device operation information stored in the device operation information storage unit according to embodiment 1.

Fig. 4 is a schematic diagram showing an example of the article information stored in the article information storage unit in embodiment 1.

Fig. 5 is a schematic diagram showing an example of movie information stored in the service information storage unit according to embodiment 1.

Fig. 6 is a schematic diagram showing an example of restaurant information stored in the service information storage unit according to embodiment 1.

Fig. 7 is a flowchart 1 for explaining the operation of the server device according to embodiment 1 of the present invention.

Fig. 8 is a flow chart of fig. 2 for explaining the operation of the server device according to embodiment 1 of the present invention.

Fig. 9 is a schematic diagram for explaining the operation of the apparatus in situation 1 in which an article slips off a human hand during a daily operation in embodiment 1.

Fig. 10 is a schematic diagram for explaining the operation of the device in the 2 nd situation where a plurality of people are quarreling in embodiment 1.

Fig. 11 is a schematic diagram for explaining the operation of the device in situation 3 in which a suspicious person enters in embodiment 1.

Fig. 12 is a schematic diagram showing the configuration of the 1 st server device according to embodiment 2 of the present invention.

Fig. 13 is a schematic diagram showing the configuration of the 2 nd server device according to embodiment 2 of the present invention.

Detailed Description

(basic knowledge of the invention)

In the above-described conventional technique, an image of a foreign object other than the cleaning target is registered in advance in a storage unit, a captured image captured during the cleaning operation is compared with the image of the foreign object, the registered foreign object is recognized, and a display for specifying what the recognized foreign object is displayed on a display screen. In addition, as the foreign matter, from the viewpoint of avoiding a trouble or a breakage of the electric cleaner, "plastic bags", "books", "wires", "screws", and the like are listed, and from the viewpoint of ensuring the cleaning performance of the sucked foreign matter and avoiding stains and breakage, "books", "micro SD cards", "paper money", "jewels", and the like are listed. That is, in the conventional art, although foreign matter other than the cleaning target can be recognized, the cleaning target such as a broken cup cannot be recognized.

In the conventional art, when an article is damaged, it is not assumed that the article is damaged. Therefore, the conventional art does not disclose or suggest that the electric cleaner should perform a predetermined operation in response to the occurrence of damage to the article.

In order to solve the above problem, an information processing method according to an aspect of the present invention is an information processing method of an information processing apparatus, including: acquiring 1 st information acquired from at least one of one or more sensors disposed in a space; detecting a breakage of an article present in the space based on the 1 st information; estimating a state of occurrence of breakage of the article based on the 1 st information; and outputting 2 nd information for causing the self-propelled device to execute a predetermined operation in the space, based on the estimated state.

With this configuration, the state of damage to the article is estimated based on the 1 st information acquired from at least one of the one or more sensors provided in the space, and the 2 nd information for causing the self-propelled device to perform a predetermined operation in the space is output based on the estimated state. Therefore, when the article present in the space is damaged, the self-propelled device can be caused to perform a predetermined operation in accordance with the situation in which the damage of the article has occurred.

In the information processing method, the 2 nd information may be information for causing the self-propelled device to output a predetermined sound in the space according to the estimated state.

With this configuration, the self-propelled device can output a predetermined sound in the space according to the state of damage to the article. For example, when the damaged article is a situation in which a plurality of people are quarreling, the self-propelled apparatus can be caused to perform an operation of moving while outputting a voice of the quarrelining plurality of people.

In the information processing method, the 3 rd information for presenting the information for changing the estimated state by the presentation device may be further output.

With this configuration, since the 3 rd information for changing the information of the estimated situation is presented by the presentation device, the situation in which the article damage has occurred can be changed. For example, when a situation in which a plurality of people have quarreling the object is occurring, information suitable for the conciseness of the plurality of people who are quarreling the object may be presented.

In the information processing method, the self-propelled device may be a self-propelled cleaning machine, and the 2 nd information may be information for causing the self-propelled cleaning machine to clean the damaged article in the space based on the estimated state.

According to this configuration, since the self-propelled device is a self-propelled cleaning machine, the self-propelled cleaning machine can clean a damaged article in the space in accordance with the occurrence of the article damage. For example, when the damaged article occurs in a situation where the article slips off a hand of a person during a daily operation, the self-propelled cleaning machine may be caused to perform an operation of cleaning the damaged article.

In the information processing method, the estimated situation may be a situation in which a suspicious person enters the space, and the 2 nd information may be information for causing the self-propelled device to execute an operation in the space that interferes with the suspicious person.

According to this configuration, when the situation in which the article is damaged is a situation in which a suspicious person enters the space, the self-propelled cleaning machine can be caused to perform an operation to disturb the suspicious person in the space.

In the information processing method, the image data of the suspicious person may be acquired from an imaging device disposed in the space, and the acquired image data and notification information for notifying the presence of the suspicious person may be transmitted.

According to this configuration, since the image data in which the suspicious person is photographed is acquired from the photographing device disposed in the space, and the acquired image data and the notification information for notifying the existence of the suspicious person are transmitted, it is possible to notify the existence of the suspicious person to other people.

In the information processing method, when the damage information indicating that the self-propelled device is damaged is acquired, the 4 th information requesting repair of the self-propelled device may be further output.

With this configuration, when the self-propelled device is damaged, the self-propelled device can be automatically requested to be repaired.

In the information processing method, the 5 th information for causing the presentation device to present the information about the article for suppressing the occurrence of the estimated situation may be further output.

According to this configuration, the information on the article for suppressing the occurrence of the article damage is presented by the presentation device, so that the occurrence of the article damage can be suppressed. For example, when the situation in which the article is broken is a situation in which a suspicious person enters the space, information on the antitheft article may be presented as an article for suppressing the situation in which the suspicious person enters the space.

In the information processing method, the one or more sensors may include at least one of a microphone device and an imaging device provided in the space, the 1 st information may include at least one of sound data acquired by the microphone device and image data acquired by the imaging device, and the estimation of the situation may be performed based on at least one of the sound data and the image data to estimate the situation in which the damage of the article has occurred.

According to this configuration, it is possible to estimate with high accuracy the occurrence of damage to the article based on at least one of the sound data acquired by the microphone device provided in the space and the image data acquired by the imaging device provided in the space.

In the information processing method, the 1 st information may be acquired at predetermined time intervals, and the situation may be estimated based on a plurality of 1 st information acquired within a predetermined period of time with reference to a time point at which the breakage of the article has occurred.

According to this configuration, since the 1 st information is acquired at predetermined time intervals, and the state of occurrence of breakage of the article is estimated based on the plurality of 1 st information acquired within a predetermined period with reference to the time point at which breakage of the article has occurred, for example, the state of occurrence of breakage of the article can be estimated with higher accuracy using past image data for a certain period from the time point at which breakage of the article has occurred.

An information processing apparatus according to another aspect of the present invention includes: an acquisition unit that acquires 1 st information acquired from at least one of one or more sensors provided in a space; a detection unit that detects a breakage of an article present in the space based on the 1 st information; an estimating unit that estimates a state in which the article is damaged based on the 1 st information; and an output unit for outputting 2 nd information for causing the self-propelled device to perform a predetermined operation in the space, based on the estimated state.

With this configuration, the state of damage to the article is estimated based on the 1 st information acquired from at least one of the one or more sensors provided in the space, and the 2 nd information for causing the self-propelled device to perform a predetermined operation in the space is output based on the estimated state. Therefore, when the article present in the space is damaged, the self-propelled device can be caused to perform a predetermined operation in accordance with the situation in which the damage of the article has occurred.

A recording medium according to another aspect of the present invention is a computer-readable recording medium storing an information processing program for causing a computer to function as: acquiring 1 st information acquired from at least one of one or more sensors disposed in a space; detecting a breakage of an article present in the space based on the 1 st information; estimating a state of occurrence of breakage of the article based on the 1 st information; and outputting 2 nd information for causing the self-propelled device to execute a predetermined operation in the space, based on the estimated state.

With this configuration, the state of damage to the article is estimated based on the 1 st information acquired from at least one of the one or more sensors provided in the space, and the 2 nd information for causing the self-propelled device to perform a predetermined operation in the space is output based on the estimated state. Therefore, when the article present in the space is damaged, the self-propelled device can be caused to perform a predetermined operation in accordance with the situation in which the damage of the article has occurred.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments are merely examples embodying the present invention, and are not intended to limit the technical scope of the present invention.

(embodiment 1)

Fig. 1 is a schematic diagram showing the configuration of an equipment control system according to embodiment 1 of the present invention. As shown in fig. 1, the equipment control system includes a server device 3, a Gateway (GW)5, a 1 st sensor 11, a 2 nd sensor 12, a self-propelled cleaner 21, and a display device 22.

The gateway 5, the 1 st sensor 11, the 2 nd sensor 12, the self-propelled cleaner 21, and the display device 22 are disposed in the house 10. The gateway 5 is communicably connected to the 1 st sensor 11, the 2 nd sensor 12, the self-propelled cleaner 21, and the display device 22 by wireless communication. The gateway 5 is communicably connected to the server apparatus 3 via the network 4. The network 4 is, for example, the internet.

The 1 st sensor 11, the 2 nd sensor 12, the self-propelled cleaner 21, and the display device 22 are communicably connected to the server device 3 via the gateway 5. The 1 st sensor 11, the 2 nd sensor 12, the self-propelled cleaner 21, and the display device 22 may be directly communicably connected to the server device 3 without the gateway 5.

Fig. 2 is a schematic diagram showing a configuration of a server device according to embodiment 1 of the present invention. The server apparatus 3 is communicably connected to a sensor group 1 including a plurality of sensors disposed in a house 10 and a device group 2 including a plurality of devices disposed in the house 10. The sensor group 1 includes various sensors such as the 1 st sensor 11 and the 2 nd sensor 12. The equipment group 2 includes various kinds of equipment such as a self-propelled cleaner 21, a display device 22, and an information device 23. In fig. 2, the gateway 5 is omitted.

The 1 st sensor 11 is, for example, a microphone device, collects sound in the house 10, and transmits sound data to the server device 3. The 2 nd sensor 12 is, for example, an imaging device, and images the inside of the house 10 and transmits image data to the server device 3. In addition, the sensor group 1 may also include a thermal image sensor and a vibration sensor. The sensors constituting the sensor group 1 may be installed on the wall, floor, and furniture of the house 10, or may be mounted on any device in the device group 2.

The self-propelled cleaning machine 21 is an example of a self-propelled device, and performs dust collection and cleaning while autonomously moving. The self-propelled cleaning machine 21 cleans the floor while autonomously moving on the floor in the house 10. In general, the self-propelled cleaner 21 is connected to a charging device (not shown) provided at a predetermined location in the house 10, and when a user presses a cleaning start button provided on the main body of the self-propelled cleaner 21 or receives cleaning prompt information from the server device 3, the user leaves the charging device and starts cleaning. The self-propelled cleaning machine 21 includes a control unit, a camera, a speaker, a drive unit, a cleaning unit, and a communication unit, which are not shown.

The control unit controls the cleaning operation of the self-propelled cleaner 21. The driving unit moves the self-propelled cleaner 21. The driving unit includes a driving wheel for moving the self-propelled cleaning machine 21 and a motor for driving the driving wheel. The drive wheels are provided at the bottom of the self-propelled cleaner 21. The cleaning unit is provided at the bottom of the self-propelled cleaner 21 to suck the dust-suction object.

The camera captures the traveling direction of the self-propelled cleaner 21. The communication unit transmits image data captured by the camera to the server device 3. Then, the communication unit receives cleaning instruction information for starting cleaning from the server device 3. And a control unit which starts cleaning when the cleaning prompt information is received by the communication unit. The cleaning prompt information includes a damage position where the article 6 is damaged in the house 10. The damaged position is a position where the article 6 such as a cup or a plate is damaged. After moving to the damaged position, the self-propelled cleaner 21 captures an image of the dust collection object present at the damaged position, and transmits the captured image data to the server device 3. Then, the self-propelled cleaning machine 21 cleans the damaged position, and returns to the charging device.

And a speaker for outputting a predetermined sound in response to the occurrence of the damage to the article. For example, when a plurality of people are quarreling in a situation where damage to an article has occurred, the speaker outputs a sound for soothing the quarreling plurality of people.

In embodiment 1, the equipment control system includes the self-propelled cleaning machine 21 as an example of the self-propelled device, but the present invention is not particularly limited thereto, and a self-propelled robot such as a pet robot may be included as an example of the self-propelled device. The self-propelled robot has a function other than the cleaning function of the self-propelled cleaner 21.

The display device 22 is disposed on a wall of a predetermined room in the house 10. The facility control system according to embodiment 1 may further include a plurality of display devices 22. The plurality of display devices 22 may be disposed on the walls of rooms such as a living room, a kitchen, a bedroom, a bathroom, a toilet, and an entrance. The display device 22 may be an information terminal such as a smartphone or a tablet computer. The display device 22 includes a communication unit, a display unit, and an input unit, which are not shown.

The communication unit receives information indicating the state of the device from the device disposed in the house 10. Then, the communication unit receives the presentation information from the server apparatus 3.

The display unit is, for example, a liquid crystal display device, and displays various kinds of information. The display unit displays information on devices disposed in house 10. The display unit may display, for example, a current state of the washing machine, and a current state of the air conditioner. The display unit displays the presentation information received by the communication unit.

The input unit is, for example, a touch panel, and receives an input operation by a user. The input unit receives an input of an operation instruction to a device disposed in house 10. The input unit receives an input of an operation instruction to the air conditioner, and receives an input of an operation instruction to the lighting device, for example. The communication unit transmits the operation instruction input through the input unit to the device.

The information device 23, which is, for example, a smartphone, a tablet computer, a personal computer, or a mobile phone, has a function of communicating with the outside. The information device 23 includes a communication unit not shown. The communication unit receives image data for capturing suspicious persons intruding into the house 10 and notification information for notifying the presence of suspicious persons from the server device 3, and transmits the received image data and notification information to a server device for police management. The server apparatus 3 transmits the image data and the notification information to a server apparatus managed by the police through the information device 23 in the house 10, whereby the police can specify the transmission sources of the image data and the notification information.

In embodiment 1, the server device 8 transmits the image data and the notification information to the server device managed by the police via the information device 23, but the present invention is not particularly limited to this, and the server device 3 may directly transmit the image data and the notification information to the server device managed by the police without via the information device 23.

The equipment group 2 includes a washing machine, a lighting device, an air conditioner, an electric door, an electronic lock, an air cleaner, and the like, in addition to the self-propelled cleaner 21, the display device 22, and the information equipment 23. The devices constituting the device group 2 include, for example, home appliances, information devices, and home equipment machines.

The server device 3 includes a communication unit 31, a processor 32, and a memory 33.

The communication unit 31 includes a sensing data receiving unit 311, a control information transmitting unit 312, a presentation information transmitting unit 313, and a notification information transmitting unit 314. The processor 32 includes a damage detection unit 321, a situation estimation unit 322, a damage position determination unit 323, a device operation determination unit 324, a damaged article determination unit 325, a substitute article determination unit 326, a presentation information generation unit 327, a service determination unit 328, and a notification information generation unit 329. The memory 33 includes a device operation information storage unit 331, an article information storage unit 332, and a service information storage unit 333.

The sensing data receiving unit 311 acquires sensing data (1 st information) acquired by at least one of one or more sensors provided in the house 10 (space). The sensing data receiving section 311 receives sensing data from each sensor of the sensor group 1. The sensing data (1 st information) includes sound data acquired by the 1 st sensor 11 (microphone device) and image data acquired by the 2 nd sensor 12 (photographing device). The sensing data receiving section 311 receives sound data as sensing data from the 1 st sensor 11 and image data as sensing data from the 2 nd sensor 12.

Also, the sensing data receiving section 311 receives sensing data from each device of the device group 2. The device group 2 also includes devices provided with sensors. The device provided with the sensor transmits the sensing data to the server apparatus 3. As described above, the self-propelled cleaner 21 includes the camera. For this reason, the sensor data receiving unit 311 receives image data as sensor data from the self-propelled cleaner 21. The display device 22 may include a microphone and a camera, and the sensing data receiving unit 311 may receive audio data and image data as sensing data from the display device 22.

Breakage detection unit 321 detects breakage of an article present in house 10 based on the sensing data (1 st information) received by sensing data reception unit 311. The breakage detection unit 321 detects breakage of an article when the sound data received from the 1 st sensor 11 includes characteristics of a sound generated when the article is broken. The memory 33 may store, for example, frequency components of a plurality of damage sounds such as sounds of broken ceramics or glass in advance. The breakage detection unit 321 compares the frequency components of the sound data received from the 1 st sensor 11 with the frequency components of the plurality of breakage sounds stored in the memory 33, and detects breakage of the article when the two frequency components match.

The damage detector 321 may estimate the damage of the article based on the sound data received from the 1 st sensor 11, using the sound data when the damage of the article has occurred and a prediction model for machine learning using the damage of the article as teacher data. In this case, the prediction model is stored in advance in the memory 33.

The breakage detection unit 321 may detect breakage of the article based on image data captured by the 2 nd sensor 12. For example, the sensor data receiving unit 311 may acquire temporally continuous image data from the 2 nd sensor 12. The damage detection unit 321 analyzes the acquired image data, and may detect that the article is damaged even when the image data includes a situation in which the article slips off a hand of a person in the house 10 and the article is broken on the floor.

The breakage detection unit 321 may detect breakage of the article by using sensing data from another sensor such as a vibration sensor. The breakage detection unit 321 may detect breakage of an article by using sensor data from a plurality of sensors in the sensor group 1.

The situation estimation unit 322 estimates the situation where the article damage has occurred based on the sensing data (1 st information). The situation estimation unit 322 estimates a situation where the article is damaged based on at least one of the audio data and the image data. That is, the sensor data receiving unit 311 acquires image data at predetermined time intervals. The situation estimating unit 322 estimates a situation where the article damage has occurred based on a plurality of image data acquired within a predetermined time with reference to a time point at which the article damage has occurred. The breakage detection unit 321 can identify the time when the breakage of the article occurs by recognizing the characteristic component of the sound of the breakage of the article from the sound data. The situation estimation unit 322 estimates the situation where the damage to the article has occurred based on a plurality of image data acquired within a predetermined period of time from the identified time point where the damage to the article has occurred.

For example, when an article slips off a hand of a person during a usual movement of the person, the article is damaged. Further, for example, when a plurality of people are quarreling, one of the plurality of people throws an article, and the article is broken. Further, for example, when a suspicious person intrudes into the house 10, the suspicious person destroys the article, thereby damaging the article. Therefore, the situations in which the damage of the article occurs include, for example, the 1 st situation in which the article slips off a hand of a person during a daily operation, the 2 nd situation in which a plurality of persons quarrel, and the 3 rd situation in which a suspicious person intrudes. The situation estimation unit 322 estimates which situation of occurrence of the damage of the article is a 1 st situation in which the article slips off a hand of a person, a 2 nd situation in which a plurality of persons have quarreling, or a 3 rd situation in which a suspicious person intrudes in a daily operation.

The situation estimation unit 322 analyzes the plurality of image data before the time when the article damage occurs, and identifies the hand of the person and the article included in the plurality of image data. Further, the situation estimation unit 322 estimates that the situation in which the article is damaged is the 1 st situation in which the article slips off the hand of the person in the daily operation, when the article slips off the hand of the person. The situation estimation unit 322 may acquire sound data before the time when the article damage occurs, and estimate that the situation in which the article damage occurs is the 1 st situation in which the article slips off from the hand of the person during daily operation when the sound data includes a sound that is scaring the person.

The situation estimation unit 322 analyzes the plurality of image data before the time when the article damage occurs, and identifies the hand of the plurality of persons and the article included in the plurality of image data. Further, the situation estimation unit 322 estimates that the situation in which the article damage has occurred is the 2 nd situation in which a plurality of people are quarreling when one of the plurality of people throws the article. The situation estimation unit 322 may acquire the sound data before the time when the article damage occurs, and estimate that the situation when the article damage occurs is the 2 nd situation when a plurality of people make a quarrel when the sound data includes a sound of a quarrel.

The situation estimation unit 322 may acquire the sound data before the time when the article damage has occurred, and estimate that the situation in which the article damage has occurred is the 2 nd situation in which a plurality of people struggle when the sound volume of the sound of the plurality of people included in the sound data is equal to or greater than the threshold value. The situation estimation unit 322 may acquire a plurality of image data before the time when the article damage occurs, and recognize an operation in which a plurality of people included in the plurality of image data are quarreling. The situation estimation unit 322 may detect vibration generated when a plurality of people are quarreling by using a vibration sensor.

Further, the situation estimation unit 322 analyzes the plurality of image data before the time when the article damage occurs, and recognizes a person included in the plurality of image data. When recognizing that the person is not a resident of the house 10 registered in advance, the situation estimation unit 322 estimates that the situation in which the damage of the article has occurred is the 3 rd situation in which the suspicious person enters. Further, the situation estimation unit 322 may estimate that the situation in which the article damage has occurred is the 3 rd situation in which the suspicious person enters, when the person who is not the resident of the house 10 registered in advance is recognized and the resident of the house 10 registered in advance is not recognized.

The situation estimation unit 322 may estimate the situation in which the article damage has occurred from the image data before the time at which the article damage has occurred, using the image data before the time at which the article damage has occurred and a prediction model for performing machine learning using the situation in which the article damage has occurred as teacher data. In addition, the prediction model is stored in the memory 33 in advance.

The situation in which the article is damaged in embodiment 1 is not limited to the above-described situations 1 to 3.

Damaged position specifying unit 323 specifies the damaged position of the article in house 10. The memory 33 may store a layout diagram of the house 10 in a two-dimensional coordinate space, for example. Further, the self-propelled cleaning machine 21 may create a map by moving in the house 10 and transmit the created map to the server device 3. The damage position specifying unit 323 specifies the coordinates of the sound generating source of the damage to the article on the layout as the damage position.

Further, the damage position specifying unit 323 can more accurately specify the source of the sound of damage to the article by collecting the sound of damage to the article with a plurality of microphones. The damaged position specifying unit 323 may specify the position of damage to the article based on the image data captured by the 2 nd sensor 12.

The equipment operation determination unit 324 determines a predetermined operation to be executed by the self-propelled cleaner 21 (self-propelled device) in the house 10 (space) based on the situation estimated by the situation estimation unit 322. The device operation determination unit 324 may determine a predetermined operation to be executed by a device other than the self-propelled cleaner 21, based on the situation estimated by the situation estimation unit 322. Further, the device operation determination unit 324 may determine not only the devices constituting the device group 2 but also predetermined operations to be executed by the sensors constituting the sensor group 1, based on the situation estimated by the situation estimation unit 322.

The device operation information storage unit 331 stores device operation information in which a situation in which the article is damaged and an operation to be executed by the device are associated with each other in advance.

Fig. 3 is a schematic diagram showing an example of the device operation information stored in the device operation information storage unit according to embodiment 1.

As shown in fig. 3, the occurrence of the article damage corresponds to the operation to be executed by the apparatus. In the 1 st situation where the article slips off the hand of the person during the daily operation, the operation of causing the self-propelled cleaning machine 21 to suck the damaged article and the operation of presenting the substitute article for the damaged article on the display device 22 correspond to each other. In the situation 2 where a plurality of people are quarreling, the operation of moving the self-propelled cleaner 21 while outputting the sound of soothing the quarreling plurality of people and the operation of presenting a restaurant or movie suitable for the settlement on the display device 22 correspond to each other. In the 3 rd situation in which a suspicious person enters, the operation of moving the self-propelled cleaning machine 21 while interfering with the steps of the suspicious person, the operation of imaging the suspicious person by the imaging device (the 2 nd sensor 12), and the operation of transmitting the image data of the suspicious person imaged and the notification information for notifying the presence of the suspicious person to the police by the information device 23 are associated with each other.

The operation of the equipment according to the situation in which the article is damaged is not limited to the above.

The device operation determination unit 324 refers to the device operation information storage unit 331, and determines to cause the device to execute a predetermined operation corresponding to the situation estimated by the situation estimation unit 322.

The device operation determination unit 324 determines the operation of the self-propelled cleaner 21 for suctioning the damaged article and the operation of the display device 22 for presenting a substitute for the damaged article, when it is estimated that the article has slipped off from the hand of the person during the daily operation, as the 1 st situation. When it is estimated that the situation 2 is a situation in which a plurality of people are quarreling, the device operation determination unit 324 determines the operation of the self-propelled cleaner 21 that moves while outputting the sound of the quarreling plurality of people and the operation of the display device 22 that presents restaurants or movies suitable for the settlement. When it is estimated that the 3 rd situation is a suspicious person entering, the device operation determination unit 324 determines the operation of the self-propelled cleaning machine 21 that moves while interfering with the steps of the suspicious person, the operation of the imaging device (the 2 nd sensor 12) that images the suspicious person, and the operation of the information device 23 that transmits the image data that images the suspicious person and the notification information for notifying the presence of the suspicious person to the police.

The device operation determination unit 324 controls the operation of the self-propelled cleaner 21. The device operation determination unit 324 generates control information (2 nd information) for causing the self-propelled cleaner 21 (self-propelled device) to perform a predetermined operation in the space, based on the estimated state. Here, the control information is cleaning instruction information for causing the self-propelled cleaner 21 to clean a damaged article in the house 10 (space) based on the estimated situation. When the operation of the self-propelled cleaner 21 for suctioning the damaged article is determined, the device operation determination unit 324 moves the self-propelled cleaner 21 to the damaged position determined by the damaged position determination unit 323, and generates cleaning instruction information for causing the self-propelled cleaner 21 to clean the damaged article at the damaged position.

The control information transmitting unit 312 outputs control information (2 nd information) for causing the self-propelled cleaner 21 (self-propelled device) to perform a predetermined operation in the house 10 (space) based on the estimated situation. The control information transmitting unit 312 transmits the cleaning instruction information generated by the device operation determining unit 324 to the self-propelled cleaning machine 21. Upon receiving the cleaning instruction information, the self-propelled cleaning machine 21 moves to the damaged position, images an object to be cleaned as a damaged article at the damaged position, transmits the imaged image data to the server device 3, and sucks the object to be cleaned. The sensor data receiving unit 311 acquires information on the cleaning target object by the self-propelled cleaner 21. For example, the information on the dust collection target is information on the appearance of the dust collection target. The information on the external appearance includes an image captured by a camera provided in the self-propelled cleaner 21. The image includes an object to be cleaned. The sensor data receiving unit 311 receives the image data of the object to be cleaned transmitted from the self-propelled cleaner 21.

When the device operation determination unit 324 determines an operation to present a substitute for the damaged article on the display device 22, the damaged article identification unit 325 identifies the damaged article made of the dust collection target object based on the image data including the dust collection target object received from the self-propelled cleaner 21. The object to be cleaned is a broken article. The damaged article identification unit 325 identifies a damaged article based on the appearance of the dust collection target. The damaged article identification unit 325 identifies an image including the dust collection object, and identifies a damaged article based on the identified image. The memory 33 may store a table in which images of a plurality of articles and names (product names) of the articles are associated with each other in advance. The damaged article identification unit 325 compares the image data obtained by imaging the dust collection object with the images of the plurality of articles stored in the memory 33, and identifies the name of the article corresponding to the image of the article partially matching the image of the dust collection object as the name of the damaged article.

The article information storage unit 332 stores article information related to an article.

Fig. 4 is a schematic diagram showing an example of the article information stored in the article information storage unit in embodiment 1.

As shown in fig. 4, the article information includes an article number for identifying the article, a product name of the article, a type of the article, a color of the article, a size of the article, a weight of the article, a material of the article, a price of the article, a manufacturer of the article, and a sales shop of the article. The article information storage unit 332 stores article information in which an article number, a product name of an article, a type of an article, a color of an article, a size of an article, a weight of an article, a material of an article, a price of an article, a manufacturer of an article, and a sales shop of an article are associated with each other.

The article information shown in fig. 4 is merely an example, and may include other information such as an image of an article. Further, all the article information may be managed by one table, or may be managed by being dispersed into a plurality of tables.

The substitute item identification unit 326 identifies a substitute item associated with the damaged item based on the item information relating to the damaged item. Substitute article identification unit 326 acquires article information of the damaged article from article information storage unit 332.

For example, the substitute article may be the same article as the broken article. In this case, the substitute item identification unit 326 identifies the same item as the damaged item as a substitute item.

For example, the substitute article may have the same properties as the damaged article. In this case, the substitute item identification unit 326 identifies an item having the same attribute as the damaged item as a substitute item. The attribute is, for example, the color, size, weight, or material of the article. The substitute item identification unit 326 identifies an item having the same color, size, weight, and material as the damaged item as a substitute item. The substitute item identification unit 326 may identify an item having the same color, size, weight, or material as the damaged item as a substitute item.

Further, for example, the substitute article may be an article having an attribute similar to that of the damaged article. In this case, the substitute item identification unit 326 identifies an item having an attribute similar to that of the damaged item as a substitute item. The attribute is, for example, the color, size or weight of the article. The substitute item determination unit 326 determines an item similar to at least one of the color, size, and weight of the damaged item as a substitute item. For example, the color similar to blue is bluish purple or the like, and the similar color for each color may be stored in advance. Also, the damaged article is of a similar size, e.g., a width, length, and height of the damaged article in the range of-1 cm to +1 cm. Further, the article having a size similar to that of the damaged article is not limited to the article having a size within the range of the predetermined value as described above, and may be an article having a size within the range of a predetermined ratio, for example, an article having a size within the range of-10% to + 10% of the width, length, and height of the damaged article. Further, an article of similar weight to the weight of the broken article is, for example, an article of weight in the range of-10 grams to ten 10 grams of the weight of the broken article. In addition, the articles having a weight similar to the weight of the damaged articles are not limited to the articles having a weight within the range of the predetermined value as described above, but may be articles having a weight within the range of a predetermined ratio, for example, articles having a weight within the range of-10% to + 10% of the weight of the damaged articles.

For example, the substitute article may be a material having the same properties as the damaged article and having a higher strength than the damaged article. In this case, the substitute article identification unit 326 identifies an article having the same attributes as the damaged article and a material having a higher strength than the damaged article as a substitute article. The attribute is, for example, the color of the article. When the broken article is made of ceramic, the substitute article identification unit 326 identifies, as a substitute article, a metal article that is the same color as the broken article and has a higher intensity than the broken article.

The memory 33 may include a user information storage unit that stores user information related to a user. The user information includes a user ID for identifying the user, the name of the user, the address of the user, the date of birth of the user, the blood type of the user, the home structure of the user, and the belongings of the user. The substitute article identifying unit 326 may identify the user who owns the damaged article identified by the damaged article identifying unit 325, and acquire the user information of the identified user from the user information storage unit. Then, the substitute item identification unit 326 may identify a substitute item associated with the damaged item based on the item information on the damaged item and the user information on the owner of the damaged item.

For example, the user information includes owned article information indicating a plurality of owned articles owned by the owner. The substitute item identification unit 326 may identify an item of a type different from the damaged item from among the plurality of owned items indicated by the owned item information, and identify, as the substitute item, an item of the same type as the damaged item and having at least one attribute as the identified item among the plurality of sold items being sold. The user information may include living information indicating a living location of the owner. The substitute item identification unit 326 may identify a substitute item that can be purchased in a store within a predetermined range from the position of residence indicated by the residence information.

The presentation information generating unit 327 generates presentation information on the substitute item identified by the substitute item identifying unit 326. The reminder information includes an image representing the appearance of the substitute item. The prompt message may include an image showing the appearance of the substitute item and a target image for ordering the substitute item.

The presentation information transmitting unit 313 outputs presentation information on the substitute item specified by the substitute item specifying unit 326. That is, the presentation information transmitting unit 313 transmits the presentation information generated by the presentation information generating unit 327 to the display device 22. The display device 22 displays the received presentation information in a predetermined manner.

In addition, when it is estimated that the situation 2 is a situation in which a plurality of people are quarreling, the control information generated by the device operation determination unit 324 is sound output instruction information for causing the self-propelled cleaner 21 (self-propelled device) to output a predetermined sound in the house 10 (space) according to the estimated situation. When the device operation determination unit 324 determines an operation of moving the self-propelled cleaner 21 while outputting the voices of the loud multiple persons, it moves the self-propelled cleaner 21 to the damage position determined by the damage position determination unit 323 and generates voice output instruction information for moving the self-propelled cleaner 21 while outputting the voices of the loud multiple persons at the damage position.

The control information transmitting unit 312 transmits the audio output instruction information generated by the device operation determining unit 324 to the self-propelled cleaner 21. When receiving the sound output instruction information, the self-propelled cleaning machine 21 moves from the charging position to the damage position and outputs a loud sound for a plurality of people at the damage position. At this time, the self-propelled cleaner 21 may output the sound of a large number of people who have performed a quarantining work and may suck the dust collection object (damaged article). After the suction of the cleaning object is completed, or after a predetermined time has elapsed from the start of the sound output, the self-propelled cleaner 21 returns to the charging position.

When the device operation determination unit 324 determines to cause the display device 22 to display an operation of a restaurant or a movie suitable for the settlement, the service identification unit 328 refers to the service information storage unit 333 and identifies a restaurant or a movie to be presented to a plurality of people who have made a quarrel.

The service information storage 333 stores in advance service information including movie information on a movie being shown and restaurant information on restaurants.

Fig. 5 is a schematic diagram showing an example of movie information stored in the service information storage unit according to embodiment 1.

As shown in fig. 5, the movie information includes the title of the movie, the subject of the movie, the showing location, the showing schedule, and the free seat information. The service information storage unit 333 stores movie information in which the title, the subject, the showing location, the showing schedule, and the free seat information of the movie are associated with each other.

The movie information shown in fig. 5 is an example, and may include other information such as actors performing in the movie. Further, all movie information may be managed in one table, or may be managed in a plurality of tables.

The service determination section 328 determines a movie to be prompted to a plurality of people in a quarrel. The service determination section 328, for example, determines a movie suitable for the material to be resolved and available for viewing. In addition, it is preferable that the movie information contains a material for which the movie is suitable for the resolution. The movie available for viewing is a movie whose showing start time is after the current time and which has an empty seat. Further, in the case where the memory 33 stores in advance user information on each of a plurality of people in a quarrel and the user information includes information on favorite movie titles, the service identification unit 328 may identify a movie corresponding to a movie title that is commonly liked by a plurality of people in a quarrel, with reference to the user information.

Fig. 6 is a schematic diagram showing an example of restaurant information stored in the service information storage unit according to embodiment 1.

As shown in fig. 6, the restaurant information includes the name of the restaurant, the type of food, the location of the restaurant, the business hours of the restaurant, and the vacancy information. The service information storage unit 333 stores restaurant information in which the name of the restaurant, the type of food, the location of the restaurant, the business hours of the restaurant, and the vacancy information are associated with each other.

The restaurant information shown in fig. 6 is an example, and may include other information such as a menu and an image in the restaurant. Further, all restaurant information may be managed in one table, or may be managed in a plurality of tables.

The service determination 328 determines the restaurants to be suggested to the multiple persons in the quarreling. The service determination section 328, for example, determines a restaurant type suitable for the settlement and a restaurant that can eat. In addition, whether the restaurant is of a type suitable for settlement or not is preferably included in the restaurant information in advance. Also, a restaurant that can have a meal refers to a restaurant that is currently in business hours and has an empty seat. Further, in the case where the memory 33 stores in advance user information on each of the plurality of people in the quarrel and the user information includes information on a type of favorite dish, the service identification unit 328 may identify a restaurant corresponding to a type of dish that the plurality of people in the quarrel commonly enjoy, with reference to the user information.

In embodiment 1, the service identification unit 328 may identify either one of a restaurant and a movie for presenting to a plurality of people in a quarrel, or both a restaurant and a movie for presenting to a plurality of people in a quarrel.

Further, in embodiment 1, the device operation determination unit 324 determines the operation of presenting the restaurant or movie suitable for the settlement on the display device 22 when it is estimated that a plurality of people are in the 2 nd situation, but the present invention is not particularly limited to this, and the device operation determination unit 324 may determine the operation of presenting the service suitable for the settlement on the display device 22 when it is estimated that a plurality of people are in the 2 nd situation. When the device operation determination unit 324 determines that the display device 22 is to present the service suitable for the disaggregation, the service specification unit 328 refers to the service information storage unit 333 and specifies the service to present to a plurality of people who are quarreling.

The presentation information generating unit 327 generates presentation information (3 rd information) for causing the display device 22 (presentation device) to present information for changing the estimated situation. The prompt information generation unit 327 generates prompt information on restaurants or movies determined by the service determination unit 328 as being suitable for settlement.

The presentation information transmitting unit 313 outputs presentation information for causing the display device 22 (presentation device) to present information for changing the estimated situation. Here, the prompt information transmitting unit 313 transmits the prompt information about the restaurant or movie specified by the service specifying unit 328 to the display device 22. That is, the presentation information transmitting unit 313 transmits the presentation information generated by the presentation information generating unit 327 to the display device 22. The display device 22 displays the received presentation information in a predetermined manner.

When it is estimated that the 3 rd situation is a suspicious person entering, the control information generated by the device operation determination unit 324 is disturbance operation instruction information for causing the self-propelled cleaning machine 21 (self-propelled device) to execute an operation that disturbs the suspicious person in the house 10 (space). When determining the operation of moving the self-propelled cleaner 21 while interfering with the steps of the suspicious person, the device operation determination unit 324 moves the self-propelled cleaner 21 to the damaged position determined by the damaged position determination unit 323, and generates interference operation instruction information for moving the self-propelled cleaner 21 while interfering with the steps of the suspicious person at the damaged position.

The control information transmitting unit 312 transmits the disturbance operation instruction information generated by the device operation determining unit 324 to the self-propelled cleaner 21. The self-propelled cleaning machine 21 moves from the charging position to the damaged position upon receiving the disturbance operation instruction information, and moves while disturbing the steps of the suspicious person at the damaged position. At this time, the self-propelled cleaning machine 21 measures the distance to the suspicious individual by the distance sensor, and moves while keeping a constant distance from the suspicious individual. After the suspicious person disappears from the house 10 (space), the self-propelled cleaner 21 sucks the dust collection object (damaged article) and returns to the charging position. Further, the self-propelled cleaning machine 21 may be moved so as to close the entrance of the house 10, and the suspicious person may be confined in the house 10 until the police arrives.

When the motion of imaging the suspicious person by the imaging device (the 2 nd sensor 12) is determined, the sensing data receiving unit 311 acquires image data of the suspicious person from the 2 nd sensor 12 disposed in the house 10 (space). The sensing data receiving unit 311 outputs the image data to the notification information generating unit 329.

The notification information generation unit 329 generates notification information for notifying the presence of suspicious persons. The notification information includes, for example, information indicating the presence of a suspicious person and the address of the house 10. The notification information generation unit 329 outputs the image data and the notification information acquired by the sensing data reception unit 311 to the notification information transmission unit 314.

The notification information transmitting unit 314 transmits the image data acquired by the sensing data receiving unit 311 and the notification information notifying the existence of the suspicious person to the information device 23 when determining the operation of the information device 23 that transmits the image data in which the suspicious person is captured and the notification information notifying the existence of the suspicious person to the police. The information device 23 receives the image data and the notification information from the server apparatus 3, and transmits the received image data and notification information to a server apparatus managed by the police.

When the self-propelled cleaner 21 detects a damage of itself, damage information indicating that the self-propelled cleaner 21 is damaged may be transmitted to the server device 3. The sensor data receiving unit 311 may receive damage information transmitted from the self-propelled cleaning machine 21. The notification information generation unit 329 may generate notification information (4 th information) for requesting the manufacturer to repair the self-propelled cleaner 21 when the damage information is received. The notification information transmitting unit 314 may transmit notification information for requesting the manufacturer to repair the self-propelled cleaner 21 to the information device 23. The information device 23 may receive notification information requesting the manufacturer to repair the self-propelled cleaner 21 from the server apparatus 3, and transmit the received notification information to a server apparatus managed by the manufacturer.

Fig. 7 is a 1 st flowchart for explaining an operation of the server device according to embodiment 1 of the present invention, and fig. 8 is a 2 nd flowchart for explaining an operation of the server device according to embodiment 1 of the present invention. Although fig. 7 illustrates an example of detecting a damage based on audio data, the damage may be detected based on other sensor data such as the image data described above.

First, the sensing data receiving section 311 receives sound data as sensing data from the 1 st sensor 11 (step S1).

Next, damage detector 321 determines whether or not damage to the article in house 10 is detected, using the sound data received by sensor data receiver 311 (step S2). At this time, the breakage detection unit 321 detects breakage of the article when the frequency component of the sound data received from the sensing data reception unit 311 matches the frequency component of the sound data of breakage of the article stored in advance. Here, if it is determined that breakage of the article has not been detected (no at step S2), the processing returns to step S1.

On the other hand, when it is determined that the breakage of the article is detected (yes in step S2), the sensing data receiving unit 311 receives, as sensing data, a plurality of image data acquired within a predetermined period of time from the time when the breakage of the article is detected from the 2 nd sensor 12 (step S3). The sensor data receiving unit 311 may request the 2 nd sensor 12 to transmit a plurality of pieces of image data acquired within a predetermined period of time from the time when the breakage of the article is detected, and may receive a plurality of pieces of image data transmitted from the 2 nd sensor 12 in response to the request. The sensor data receiving unit 311 may receive image data from the 2 nd sensor 12 periodically and store the received image data in the memory 33. In addition, when the occurrence of the damage to the article is detected, the situation estimation unit 322 may read out, from the memory 33, a plurality of pieces of image data acquired within a predetermined period of time in the past from the time when the occurrence of the damage to the article is detected.

Next, the situation estimating unit 322 estimates the situation where the damage of the article has occurred, based on the plurality of image data acquired within the past predetermined period from the time when the damage of the article has been detected (step S4). In embodiment 1, the situation estimation unit 322 estimates that the situation in which the article damage has occurred is one of the 1 st situation in which the article has slipped off from a person's hand during a daily operation, the 2 nd situation in which a plurality of persons have quarreling, and the 3 rd situation in which a suspicious person has entered.

Next, the situation estimation unit 322 determines whether the estimation result is the 1 st situation in which the article has slipped off from the hand of the person during the daily operation (step S5).

Here, if it is determined that the situation is situation 1 in which the article has slipped off from the hand of the person (yes at step S5), damaged position identifying unit 323 identifies the damaged position of the article in house 10 (step S6).

Next, the device operation determination unit 324 refers to the device operation information storage unit 331 and determines the operation of the device corresponding to the 1 st situation estimated by the situation estimation unit 322 (step S7). Here, if the estimation result is the 1 st situation in which the article has slipped off from the hand of the person, the device operation determination unit 324 determines the operation of causing the self-propelled cleaning machine 21 to suck the damaged article and the operation of causing the display device 22 to present a substitute article for the damaged article.

Next, the device operation determination unit 324 moves the self-propelled cleaner 21 to the damage position determined by the damage position determination unit 323, and generates cleaning instruction information for causing the self-propelled cleaner 21 to clean the damaged article at the damage position (step S8).

Next, the control information transmitting unit 312 transmits the cleaning instruction information generated by the device operation determining unit 324 to the self-propelled cleaning machine 21 (step S9). The self-propelled cleaning machine 21 receives the cleaning instruction information from the server device 3, and moves to the damage position included in the cleaning instruction information. When the self-propelled cleaner 21 reaches the damage position, the camera captures an image of the object to be cleaned, and image data obtained by the capturing is transmitted to the server device 3. The self-propelled cleaner 21 sucks the dust-suction object if transmitting image data including the dust-suction object to the server apparatus 3.

Next, the sensor data receiving unit 311 receives image data including the dust collection object from the self-propelled cleaner 21 as sensor data (step S10).

Next, the damaged article identification unit 325 identifies a damaged article made of the dust collection object based on the image data including the dust collection object received from the self-propelled cleaner 21 (step S11). The damaged article identification unit 325 compares the images of the plurality of articles stored in advance with the image of the dust collection object included in the image data, and identifies a damaged article including the dust collection object. For example, when the object to be cleaned is a fragment of a ceramic cup, the damaged article identification unit 325 identifies an image of an article partially matching the image of the fragment included in the image data, and identifies an article corresponding to the identified image of the article as a damaged article.

Next, the substitute item identification unit 326 acquires the item information of the damaged item from the item information storage unit 332 (step S12).

Next, the substitute item identification unit 326 identifies a substitute item associated with the damaged item based on the item information on the damaged item (step S13). For example, the substitute item identification unit 326 identifies an item identical to the damaged item as a substitute item.

Next, the presentation information generator 327 generates presentation information on the substitute item identified by the substitute item identifier 326 (step S14).

Next, the presentation information transmitting unit 313 transmits the presentation information generated by the presentation information generating unit 327 to the display device 22 (step S15). The display device 22 receives the presentation information transmitted from the server device 3 and displays the received presentation information. The display device 22 displays presentation information while the self-propelled cleaner 21 sucks the dust collection object. The display device 22 may display a presentation message when the self-propelled cleaner 21 starts suctioning the cleaning target object. The display device 22 may continue to display the presentation information after the self-propelled cleaner 21 finishes suctioning the object to be cleaned.

Fig. 9 is a schematic diagram for explaining the operation of the device in the 1 st situation in which the article slips off the hand of the person in the daily operation in embodiment 1.

When the article 6 is damaged as a result of the article 6 slipping off the hand of the person 61 during the daily operation, the self-propelled cleaning machine 21 moves to the damaged position to suck the damaged article 6. Then, the display device 22 installed indoors displays the presentation information 221 including the image 222 for confirming whether or not the same substitute article as the damaged article 6 is purchased.

As shown in fig. 9, the prompt message 221 contains, for example, "do you want to buy a new cup? "such a character, an image showing the appearance of the substitute item, and an image 222 of a button for switching to an order screen for ordering the substitute item.

Further, the device operation determination unit 324 may generate a presentation message for notifying the user of the start of cleaning when the cleaning instruction information is generated. In this case, the control information transmitting unit 312 may transmit the cleaning instruction information generated by the device operation determining unit 324 to the self-propelled cleaning machine 21 and transmit the presentation information generated by the device operation determining unit 324 to the display device 22. The display device 22 may also display a prompt message for notifying the user of the start of cleaning. In this case, the prompt message contains, for example, "do you? Is there no bar broken? I are ready to begin sweeping such words.

On the other hand, in step S5 of fig. 7, if it is determined that the estimation result is not the 1 st situation in which the article has fallen off the hand of a person (no in step S5), the situation estimation unit 322 determines whether the estimation result is the 2 nd situation in which a plurality of persons have quarreling (step S16).

Here, if it is determined that the situation is the 2 nd situation in which a plurality of people are quarreling (yes at step S16), damaged position identifying unit 323 identifies the damaged position of the article in house 10 (step S17).

Next, the device operation determination unit 324 refers to the device operation information storage unit 331 and determines the operation of the device corresponding to the 2 nd situation estimated by the situation estimation unit 322 (step S18). Here, when it is estimated that a plurality of people are quarreling in situation 2, the device operation determination unit 324 determines an operation of moving the self-propelled cleaner 21 while outputting a voice of the plurality of people in quarreling and an operation of presenting a restaurant or movie suitable for the settlement on the display device 22.

Next, the device operation determination unit 324 moves the self-propelled cleaner 21 to the damage position determined by the damage position determination unit 323, and generates sound output instruction information for moving the self-propelled cleaner 21 at the damage position while outputting sounds for soothing a plurality of people in a quarreling (step S19).

Next, the control information transmitting unit 312 transmits the audio output instruction information generated by the device operation determining unit 324 to the self-propelled cleaning machine 21 (step S20). When receiving the sound output instruction information, the self-propelled cleaning machine 21 moves from the charging position to the damage position, and outputs a loud sound for a plurality of people at the damage position. Then, the self-propelled cleaner 21 sucks the dust collection object (damaged article). After the suction of the dust-suction object is completed or after a predetermined time has elapsed from the start of the sound output, the self-propelled cleaner 21 returns to the charging position.

Next, the service identification unit 328 refers to the service information storage unit 333 and identifies restaurants that are to be presented to a plurality of people who have made a quarrel (step S21). The service identification unit 328 identifies the type of cooking commonly liked by a plurality of people in a quarrel, and identifies a restaurant that can have a meal and corresponds to the identified restaurant type, by referring to the user information stored in advance.

In the processing of fig. 8, the service identification unit 328 identifies restaurants to be presented to a plurality of people in a quarreling, but the present invention is not limited to this, and may identify movies to be presented to a plurality of people in a quarreling.

Next, the presentation information generator 327 generates presentation information on restaurants appropriate for the settlement determined by the service determiner 328 (step S22).

Next, the presentation information transmitting unit 313 transmits the presentation information generated by the presentation information generating unit 327 to the display device 22 (step S23). The display device 22 receives the presentation information transmitted from the server device 3 and displays the received presentation information. The display device 22 displays presentation information while the self-propelled cleaner 21 is outputting sound. The display device 22 may display the presentation information when the self-propelled cleaner 21 starts outputting the sound. The display device 22 may continue to display the presentation information after the self-propelled cleaner 21 finishes outputting the sound.

Fig. 10 is a schematic diagram for explaining the operation of the device in the 2 nd situation where a plurality of persons are quarreling in embodiment 1.

When a plurality of people 62, 63 make a quarrel, and the article 6 is damaged, the self-propelled cleaner 21 moves to the damaged position and outputs a sound for soothing the plurality of people 62, 63 making a quarrel. In fig. 10, for example, the self-propelled cleaner 21 outputs a sound of "good and cool down". The display device 22 installed indoors displays presentation information 223 for presenting restaurants suitable for the plurality of people 62 and 63 in quarreling.

As shown in fig. 10, the prompt message 223 includes, for example, "do not want to eat italian cuisine in M restaurant? "such text and a reservation button for reserving a restaurant. If the reservation button is pressed, the screen shifts to a reservation screen for reserving a restaurant.

On the other hand, in step S16 of fig. 8, when determining that the estimation result is not the 2 nd situation in which a plurality of people are quarreling (no in step S16), the situation estimation unit 322 determines whether the estimation result is the 3 rd situation in which a suspicious person enters (step S24).

Here, if it is determined that the estimation result is not the 3 rd situation in which the suspicious person enters (no in step S24), that is, if the situation in which the damage of the article cannot be estimated, the process is terminated. In addition, when it is determined that the estimation result is not the 3 rd situation in which the suspicious person enters, the device operation determination unit 324 may determine the operation of causing the self-propelled cleaning machine 21 to suck the damaged article.

On the other hand, if it is determined that the situation is the 3 rd situation in which a suspicious person enters (yes at step S24), damaged position identifying unit 323 identifies the damaged position of the item in house 10 (step S25).

Next, the device operation determination unit 324 refers to the device operation information storage unit 331 and determines the operation of the device corresponding to the 3 rd situation estimated by the situation estimation unit 322 (step S26). Here, when it is estimated that the 3 rd situation is a suspicious person intrusion, the device operation determination unit 324 determines an operation of moving the self-propelled cleaning machine 21 while interfering with the steps of the suspicious person, an operation of imaging the suspicious person by the imaging device (the 2 nd sensor 12), and an operation of transmitting image data of imaging the suspicious person and notification information for notifying the presence of the suspicious person to the police by the information device 23.

Next, the device operation determination unit 324 moves the self-propelled cleaning machine 21 to the damage position determined by the damage position determination unit 323, and generates disturbance operation instruction information for moving the self-propelled cleaning machine 21 at the damage position while disturbing the steps of the suspicious person (step S27).

Next, the control information transmitting unit 312 transmits the disturbance operation instruction information generated by the device operation determining unit 324 to the self-propelled cleaning machine 21 (step S28). The self-propelled cleaning machine 21 moves from the charging position to the damaged position upon receiving the disturbance operation instruction information, and moves while disturbing the steps of the suspicious person at the damaged position. After the suspicious person disappears from the house 10 (during the air break), the self-propelled cleaning machine 21 sucks the dust collection object (damaged article) and returns the dust collection object to the charging position.

Next, the sensor data receiving unit 311 receives image data for capturing a suspicious person from the 2 nd sensor 12 disposed in the house 10 (space) (step S29).

Next, the notification information generation unit 329 generates notification information for notifying the presence of suspicious persons (step S30).

Next, the notification information transmitting unit 314 transmits the image data acquired by the sensing data receiving unit 311 and the notification information generated by the notification information generating unit 329 to the information device 23 (step S31). The information device 23 receives the image data and the notification information from the server apparatus 3, and transmits the received image data and notification information to a server apparatus managed by the police.

Fig. 11 is a schematic diagram for explaining the operation of the device in situation 3 in which a suspicious person enters according to embodiment 1.

When suspicious person 64 enters and article 6 is damaged, self-propelled cleaner 21 moves to the damaged position and moves while interfering with the steps of suspicious person 64. In fig. 11, for example, the self-propelled cleaning machine 21 moves around the suspicious person 64 while keeping a predetermined distance from the suspicious person 64. Then, the 2 nd sensor 12 transmits image data of the suspicious individual 64 to the server apparatus 3. The information device 23 receives image data obtained by imaging the suspicious individual 64 and notification information for notifying the presence of the suspicious individual 64 from the server apparatus 3, and transmits the received image data and notification information to a server apparatus managed by the police.

Further, after the information device 23 transmits the image data and the notification information to the server apparatus for police management, the display apparatus 22 installed indoors may display the presentation information 224 for presenting that the police have been alerted. As shown in fig. 11, the prompt message 224 contains, for example, a word "the police has been warned".

In this way, the occurrence of damage to the article is estimated based on the 1 st information acquired from at least one of the one or more sensors provided in the space, and the 2 nd information for causing the self-propelled cleaner 21 to perform the predetermined operation in the space is output based on the estimated occurrence. Therefore, when an article present in the space is damaged, the self-propelled cleaner 21 can be caused to perform a predetermined operation in accordance with the occurrence of the damage of the article.

The presentation information generating unit 327 may generate presentation information (5 th information) for causing the display device 22 (presentation device) to present information on the article for suppressing the occurrence of the estimated situation. The presentation information transmitting unit 313 may transmit (output) presentation information (5 th information) for causing the display device 22 (presentation device) to present information on the article for suppressing the occurrence of the estimated situation. For example, when it is estimated that a 3 rd situation is entered by a suspicious person, the presentation information generation unit 327 may generate presentation information related to an antitheft article and transmit the presentation information to the display device 22.

(embodiment 2)

The equipment control system according to embodiment 1 includes one server device, but the equipment control system according to embodiment 2 includes two server devices.

Fig. 12 is a schematic diagram showing the configuration of the 1 st server device according to embodiment 2 of the present invention, and fig. 13 is a schematic diagram showing the configuration of the 2 nd server device according to embodiment 2 of the present invention.

The equipment control system according to embodiment 2 includes the 1 st server device 3A, the 2 nd server device 3B, the gateway 5 (not shown), the 1 st sensor 11, the 2 nd sensor 12, the self-propelled cleaner 21, the display device 22, and the information equipment 23. In embodiment 2, the same components as those in embodiment 1 are denoted by the same reference numerals, and descriptions thereof are omitted. The sensor group 1 includes various sensors such as a 1 st sensor 11 and a 2 nd sensor 12. The equipment group 2 includes various kinds of equipment such as a self-propelled cleaner 21, a display device 22, and an information device 23. In addition, in fig. 12 and 13, the gateway 5 is omitted.

The 1 st server apparatus 3A is communicably connected to the sensor group 1, the device group 2, and the 2 nd server apparatus 3B via a network. The 2 nd server apparatus 3B is communicably connected to the device group 2 and the 1 st server apparatus 3A via a network.

The 1 st server apparatus 3A is operated by, for example, a platform (platform). The 2 nd server apparatus 3B is operated by, for example, a 3 rd party platform (third party).

The 1 st server device 3A includes a communication unit 31A, a processor 32A, and a memory 33A.

The communication unit 31A includes a sensing data reception unit 311, a control information transmission unit 312, a notification information transmission unit 314, a damaged article information transmission unit 315, and a device operation information transmission unit 316. The processor 32A includes a damage detection unit 321, a situation estimation unit 322, a damage position determination unit 323, a device operation determination unit 324, a damaged article determination unit 325, and a notification information generation unit 329. The memory 33A includes a device operation information storage unit 331.

The damaged article information transmitting unit 315 transmits damaged article information indicating the damaged article identified by the damaged article identifying unit 325 to the 2 nd server device 3B.

The device operation information transmitting unit 316 transmits device operation information indicating an operation of the restaurant or movie which is suitable for the settlement determined by the device operation determining unit 324 and is presented to the 2 nd server apparatus 3B by the display device 22.

The 2 nd server device 3B includes a communication unit 31B, a processor 32B, and a memory 33B.

The communication unit 31B includes a presentation information transmitting unit 313, a damaged article information receiving unit 317, and a device operation information receiving unit 318. The processor 32B includes a substitute item identification unit 326 and a presentation information generation unit 327. The memory 33B includes an article information storage unit 332.

Damaged article information receiving unit 317 receives damaged article information transmitted from server apparatus 1 a. The substitute article identification unit 326 identifies a substitute article associated with the damaged article based on the damaged article information received by the damaged article information receiving unit 317.

The device operation information receiving unit 318 receives the device operation information transmitted from the 1 st server apparatus 3A. When the device operation information receiving unit 318 receives the device operation information indicating the operation of presenting the restaurant or movie suitable for the settlement on the display device 22, the service specifying unit 328 refers to the service information storage unit 333 and specifies the restaurant or movie to be presented to a plurality of people in a quarreling.

In embodiment 2, 1 st server device 3A transmits damaged article information and device operation information to 2 nd server device 3B, but the present invention is not particularly limited to this, and 2 nd server device 3B may transmit a request for damaged article information and device operation information to 1 st server device 3A, and 1 st server device 3A may transmit damaged article information and device operation information to 2 nd server device 3B in response to the request.

In embodiment 2, the device control system may include a plurality of 2 nd server apparatuses 3B.

In the above embodiments, each component may be configured by dedicated hardware, or may be realized by executing a software program applied to each component. Each component can be realized by causing a program execution unit such as a CPU or a processor to read and execute a software program stored in a recording medium such as a hard disk or a semiconductor memory.

A part or all of the functions of the apparatus according to the embodiment of the present invention are typically realized by an integrated circuit lsi (large Scale integration). That is, the chips may be individually formed, or a part or all of the chips may be included in one chip. The integrated circuit is not limited to an LSI, and may be realized by a dedicated circuit or a general-purpose processor. An fpga (field Programmable Gate array) which can be programmed after LSI manufacturing or a reconfigurable processor which can reconfigure connection and setting of circuits inside LSI can be used.

Further, a part or all of the functions of the device according to the embodiment of the present invention may be realized by causing a processor such as a CPU to execute a program.

All the numbers used above are for specifically explaining the present invention, and the present invention is not limited to the exemplified numbers.

The order in which the steps shown in the flowcharts are executed is for specifically explaining the example of the present invention, and the steps may be performed in an order other than the above order as long as the same effects can be obtained. Moreover, a part of the above steps may be performed simultaneously (in parallel) with other steps.

The information processing method, the information processing apparatus, and the computer-readable recording medium storing the information processing program according to the present invention are capable of causing the self-propelled apparatus to execute a predetermined operation in accordance with a situation where damage to an article has occurred, and therefore, are useful as an information processing method, an information processing apparatus, and a computer-readable recording medium storing an information processing program for causing a device to execute a predetermined operation.