阅读说明：本技术 构建地图及定位方法、客户端、移动机器人及存储介质 (Map construction and positioning method, client, mobile robot and storage medium ) 是由 不公告发明人 于 2020-04-16 设计创作，主要内容包括：本申请公开一种构建地图及定位方法、客户端、移动机器人及存储介质。其中,环构建地图的方法包括：从多个地图片段中选择利用第一构建地图方式而构建的第一地图片段和第二地图片段；其中,多个地图片段是利用至少一种构建地图方式而构建的,第一地图片段和第二地图片段是基于移动机器人所检测到的中断情况而被单独构建的；依据第一地图片段和第二地图片段中描述同一物理空间的两个地图位置的位姿偏差,将第一地图片段和第二地图片段融合成一个地图片段。本申请有效减少因中断情况的影响而使得移动机器人所构建的地图误差较大的问题；以及将单独构建的多个地图片段进行融合的方式实现了移动机器人将多个地图片段进行整合的目的。(The application discloses a map construction and positioning method, a client, a mobile robot and a storage medium. The method for constructing the map by the ring comprises the following steps: selecting a first map segment and a second map segment which are constructed by utilizing a first map construction mode from a plurality of map segments; wherein the plurality of map segments are constructed using at least one map construction approach, the first map segment and the second map segment being constructed separately based on the detected outage situation of the mobile robot; and fusing the first map segment and the second map segment into one map segment according to the pose deviations of the two map positions describing the same physical space in the first map segment and the second map segment. The method and the device effectively reduce the problem that the map constructed by the mobile robot has larger error due to the influence of the interruption condition; and the aim of integrating a plurality of map segments by the mobile robot is fulfilled by fusing a plurality of map segments which are constructed independently.)

1. A method of constructing a map for processing a plurality of map segments constructed by a mobile robot, the method comprising:

selecting a first map segment and a second map segment which are constructed by utilizing a first map construction mode from a plurality of map segments; wherein the plurality of map segments are constructed using at least one map construction approach, the first and second map segments being constructed separately based on a detected outage of the mobile robot;

and fusing the first map segment and the second map segment into one map segment according to the pose deviations of two map positions describing the same physical space in the first map segment and the second map segment.

2. The method of constructing a map of claim 1, wherein the plurality of map segments further comprises a third map segment constructed using a second construction mapping approach; wherein the third map segment begins at the end position of the first map segment and the third map segment ends at the start position of the second map segment.

3. The method of constructing a map of claim 1, wherein the outage condition comprises at least one of: the map segment construction may be interrupted by an intervention of an external force, by a loss of data required for the first map construction mode, or by an intervention of a generated control command.

4. The method of constructing a map according to claim 1, wherein the step of selecting a first map segment and a second map segment constructed using a first map construction method from among a plurality of map segments comprises:

selecting a previously constructed first map segment from the plurality of map segments before construction of the second map segment is completed; or

Selecting a first map segment and a second map segment that have been constructed from the plurality of map segments, wherein the second map segment is constructed after the first map segment.

5. The method of constructing a map according to any one of claims 1 to 4, wherein the step of selecting a first map segment and a second map segment constructed using a first map construction method from a plurality of map segments comprises:

and selecting a first map segment closest to the second map segment from the plurality of map segments according to the adjacency relation of the map segments.

6. The method of constructing a map according to claim 5, wherein the adjacency is determined by detecting an overlap of at least two map segments in accordance with a construction order; and/or the adjacency is determined by detecting an image match in at least two map segments.

7. The method of claim 1, wherein the step of fusing the first map segment and the second map segment into one map segment comprises: and splicing the first map segment and the second map segment according to the pose deviation to obtain a new map segment.

8. The method for constructing a map according to claim 7, wherein the step of stitching the first map segment and the second map segment according to the pose deviation to obtain a new map segment comprises:

aligning the second map segment with the first map segment according to the pose deviation of two map positions describing the same physical space in the first map segment and the second map segment;

and updating the map data in the first map segment according to the aligned first map segment and the aligned second map segment so as to splice the new map segment.

9. The method of constructing a map of claim 8, wherein the step of aligning a first map segment with a second map segment according to a pose deviation of two map locations in the first and second map segments that describe the same physical space comprises:

determining the pose deviation of each map position in each map segment according to the matched image characteristics in the first map segment and the second map segment;

and according to the pose deviation, performing at least one of rotation transformation, translation transformation or scaling transformation on the second map segment to align the second map segment and the first map segment.

10. The method of constructing a map according to claim 7, wherein the step of fusing the first map segment and the second map segment into one map segment comprises at least one of:

removing redundant information in the new map segment;

and updating the information related to the positioning in the new map segment.

11. The method of claim 1, wherein the step of fusing the first map segment and the second map segment into one map segment comprises: and fusing the second map segment constructed later into the first map segment constructed earlier to obtain a fused map segment.

12. The method of constructing a map according to claim 2, wherein the second map construction manner is a manner of constructing a map segment using movement data of the mobile robot.

13. The method of constructing a map of claim 2, further comprising: and correcting the third map segment according to the fused map segment.

14. The method of constructing a map of claim 1, further comprising:

and the fused map segments are classified into the map segments so as to select a new first map segment and a new second map segment from the map segments and execute the operation of fusing the first map segment and the second map segment into one map segment.

15. The method of constructing a map of claim 1, further comprising: and sending the map segments and/or the fused map segments to a client for display by the client.

16. A positioning method for a mobile robot, comprising:

acquiring an image shot by a camera device of the mobile robot;

determining the geographical position information of the mobile robot for shooting the image according to the information related to positioning in at least one pre-stored map segment;

wherein the at least one map segment comprises: map segments constructed with a method of constructing a map according to any one of claims 1 to 15, and/or map segments constructed separately based on detected interruptions of the mobile robot.

17. A control system of a mobile robot, comprising:

the interface device is used for being in communication connection with the client;

a storage device storing at least one program and storing the constructed plurality of map segments; wherein the plurality of map segments includes a first map segment and a second map segment;

processing means, connected to said storage means and to said interface means, for executing said at least one program for coordinating said storage means and said interface means to perform a method of constructing a map as claimed in any one of claims 1 to 15 and/or to perform a positioning method as claimed in claim 16.

18. A mobile robot, comprising:

the interface device is used for being in communication connection with the client;

a storage device storing at least one program and storing the constructed plurality of map segments; wherein the plurality of map segments includes a first map segment and a second map segment;

the mobile device is used for controlled execution of mobile operation;

processing means connected to said storage means, interface means, and mobile means for executing said at least one program to coordinate said storage means, interface means, and mobile means to perform a method of constructing a map as claimed in any one of claims 1 to 15 and/or to perform a positioning method as claimed in claim 16.

19. A client, comprising:

the interface module is used for being in communication connection with the mobile robot to acquire a plurality of map segments; wherein the map segments are constructed by at least one map construction method, and the map segments comprise map segments which are constructed independently based on the interruption condition detected by the mobile robot;

and the display module is used for displaying the map segments in a distinguishing manner according to segment display rules.

20. The client of claim 19, wherein the segment display rules comprise at least one of: and gradually filling and displaying each region picture segment to obtain a rule, and displaying the plurality of map segments by using different display colors.

21. The client of claim 19, wherein the displayed map segment is a map segment being built by the mobile robot, a map segment that has not been synthesized, or a synthesized map segment.

22. The client of claim 19, wherein the display module is further configured to refresh the displayed plurality of map segments according to a fusion change in the obtained plurality of map segments.

23. The client of claim 19, further comprising: the map construction method comprises a construction module, a map selection module and a map display module, wherein the construction module is used for selecting a first map segment and a second map segment which are constructed by utilizing a first map construction mode from a plurality of map segments; fusing the first map segment and the second map segment into one map segment according to the pose deviations of two map positions describing the same physical space in the first map segment and the second map segment;

correspondingly, the display module displays each regional picture segment containing the fused map segment according to the segment display rule.

24. The client of claim 19, wherein the display module is further configured to display the positioning location information of the mobile robot in one of the map segments.

25. A computer-readable storage medium, characterized by storing at least one program which, when invoked, executes and implements a method of constructing a map as claimed in any one of claims 1 to 15 and/or performs a positioning method as claimed in claim 16.

Technical Field

The application relates to the field of computer data processing, in particular to a map construction and positioning method, a client, a mobile robot and a storage medium.

Background

A mobile robot is a machine device that automatically performs work. It can accept human command, run the program programmed in advance, and also can operate according to the principle outline action made by artificial intelligence technology. The mobile robot can be used indoors or outdoors, can be used for industry or families, can be used for replacing security patrol, replacing people to clean the ground, and can also be used for family companions, auxiliary office work and the like. Depending on the application fields of different mobile robots, the mobile robots used in the respective fields may be moved in different ways, for example, the mobile robots may be wheeled, walking, chain, or the like. With the updating iteration of the mobile technology of the mobile robot, the mobile information provided by the sensor is utilized to carry out instant positioning and map construction (SLAM for short) so as to provide more accurate navigation capability for the mobile robot, and the mobile robot can move autonomously more effectively. However, in the case of a robot cleaner, the distance traveled by the roller wheel on the ground made of different materials is different, which makes it possible for the map constructed by the SLAM technology in this field to be greatly different from the map of the actual physical space.

Disclosure of Invention

In view of the shortcomings of the related art, the present application is directed to a map building and positioning method, a client, a mobile robot and a storage medium, which overcome the technical problem of the related art that a map built by a mobile robot through movement measurement has a large error with an actual physical space.

To achieve the above and other related objects, a first aspect of the disclosure provides a method for constructing a map, for processing a plurality of map segments constructed by a mobile robot, the method including: selecting a first map segment and a second map segment which are constructed by utilizing a first map construction mode from a plurality of map segments; wherein the plurality of map segments are constructed using at least one map construction approach, the first and second map segments being constructed separately based on a detected outage of the mobile robot; and fusing the first map segment and the second map segment into one map segment according to the pose deviations of two map positions describing the same physical space in the first map segment and the second map segment.

In certain embodiments of the first aspect of the present application, the plurality of map segments further includes a third map segment constructed using the second construction mapping approach; wherein the third map segment begins at the end position of the first map segment and the third map segment ends at the start position of the second map segment.

In certain embodiments of the first aspect of the present application, the interruption condition comprises at least one of: the map segment construction may be interrupted by an intervention of an external force, by a loss of data required for the first map construction mode, or by an intervention of a generated control command.

In certain embodiments of the first aspect of the present application, the step of selecting a first map segment and a second map segment constructed using a first construction map approach from a plurality of map segments comprises: selecting a previously constructed first map segment from the plurality of map segments before construction of the second map segment is completed; or selecting a first map segment and a second map segment that have been constructed from the plurality of map segments, wherein the second map segment is constructed after the first map segment.

In certain embodiments of the first aspect of the present application, the step of selecting a first map segment and a second map segment constructed using a first construction map approach from a plurality of map segments comprises: and selecting a first map segment closest to the second map segment from the plurality of map segments according to the adjacency relation of the map segments.

In certain embodiments of the first aspect of the present application, the adjacency is determined by detecting an overlap of at least two map segments according to a construction order; and/or the adjacency is determined by detecting an image match in at least two map segments.

In certain embodiments of the first aspect of the present application, the step of merging the first map segment and the second map segment into one map segment includes: and splicing the first map segment and the second map segment according to the pose deviation to obtain a new map segment.

In certain embodiments of the first aspect of the present application, the step of stitching the first map segment and the second map segment according to the pose deviation to obtain a new map segment includes: aligning the second map segment with the first map segment according to the pose deviation of two map positions describing the same physical space in the first map segment and the second map segment; and updating the map data in the first map segment according to the aligned first map segment and the aligned second map segment so as to splice the new map segment.

In certain embodiments of the first aspect of the present application, the step of aligning the second map segment and the first map segment according to a pose deviation of two map locations describing the same physical space in the first map segment and the second map segment comprises: determining the pose deviation of each map position in each map segment according to the matched image characteristics in the first map segment and the second map segment; and according to the pose deviation, performing at least one of rotation transformation, translation transformation or scaling transformation on the second map segment to align the second map segment and the first map segment.

In certain embodiments of the first aspect of the present application, the step of merging the first map segment and the second map segment into one map segment includes at least one of: removing redundant information in the new map segment; and updating the information related to the positioning in the new map segment.

In certain embodiments of the first aspect of the present application, the step of merging the first map segment and the second map segment into one map segment includes: and fusing the second map segment constructed later into the first map segment constructed earlier to obtain a fused map segment.

In certain embodiments of the first aspect of the present application, the second map construction means is means for constructing a map segment using movement data of the mobile robot.

In certain embodiments of the first aspect of the present application, the method of constructing a map further comprises: and correcting the third map segment according to the fused map segment.

In certain embodiments of the first aspect of the present application, the method of constructing a map further comprises: and the fused map segments are classified into the map segments so as to select a new first map segment and a new second map segment from the map segments and execute the operation of fusing the first map segment and the second map segment into one map segment.

In certain embodiments of the first aspect of the present application, the method of constructing a map further comprises: and sending the map segments and/or the fused map segments to a client for display by the client.

A second aspect of the present application provides a positioning method for a mobile robot, including: acquiring an image shot by a camera device of the mobile robot; determining geographic position information of image features in the map segments according to at least one pre-stored map segment and positioning-related information; determining the geographical position information of the image shot by the mobile robot according to the geographical position information of the determined image characteristics; wherein the at least one map segment comprises: map segments constructed using a method of constructing a map as described in any one of the first aspects, and/or map segments constructed separately based on detected interruptions by the mobile robot.

A third aspect of the present application provides a control system of a mobile robot, including: the interface device is used for being in communication connection with the client; a storage device storing at least one program and storing the constructed plurality of map segments; wherein the plurality of map segments includes a first map segment and a second map segment; processing means, coupled to the storage means and the interface means, for executing the at least one program to coordinate the storage means and the interface means to perform the method of constructing a map according to any one of the first aspect and/or to perform the positioning method according to the second aspect.

A fourth aspect of the present application provides a mobile robot comprising: the interface device is used for being in communication connection with the client; a storage device storing at least one program and storing the constructed plurality of map segments; wherein the plurality of map segments includes a first map segment and a second map segment; the mobile device is used for controlled execution of mobile operation; processing means, connected to said storage means, interface means, and mobile means, for executing said at least one program to coordinate said storage means, interface means, and mobile means to perform a method of constructing a map as described in any one of the first aspects and/or to perform a positioning method as described in the second aspect.

A fifth aspect of the present application provides a client, comprising: the interface module is used for being in communication connection with the mobile robot to acquire a plurality of map segments; wherein the map segments are constructed by at least one map construction method, and the map segments comprise map segments which are constructed independently based on the interruption condition detected by the mobile robot; and the display module is used for displaying the map segments in a distinguishing manner according to segment display rules.

In certain embodiments of the fifth aspect of the present application, the segment display rules comprise at least one of: and gradually filling and displaying each region picture segment to obtain a rule, and displaying the plurality of map segments by using different display colors.

In certain embodiments of the fifth aspect of the present application, the displayed map segment is a map segment that the mobile robot is building, a map segment that has not yet been synthesized, or a synthesized map segment.

In certain embodiments of the fifth aspect of the present application, the display module is further configured to refresh the displayed plurality of map segments according to a fusion change in the acquired plurality of map segments.

In certain embodiments of the fifth aspect of the present application, the client further comprises: the map construction method comprises a construction module, a map selection module and a map display module, wherein the construction module is used for selecting a first map segment and a second map segment which are constructed by utilizing a first map construction mode from a plurality of map segments; fusing the first map segment and the second map segment into one map segment according to the pose deviations of two map positions describing the same physical space in the first map segment and the second map segment; correspondingly, the display module displays each regional picture segment containing the fused map segment according to the segment display rule.

A sixth aspect of the present application provides a computer-readable storage medium, characterized by storing at least one program which, when invoked, executes and implements the method of constructing a map according to any one of the first aspects, and/or executes the positioning method according to the second aspect.

In summary, the method for constructing the map, the client, the mobile robot and the storage medium of the application adopt a mode of independently constructing a plurality of map segments according to the interruption condition, so that the problem of large map error constructed by the mobile robot due to the influence of the interruption condition is effectively reduced; the purpose that the mobile robot integrates the map segments is achieved by fusing the map segments which are constructed independently, and the mobile robot is facilitated to carry out positioning operation by using the obtained map segments so as to quickly determine the position of the mobile robot in a physical space; in addition, the map segments constructed by the mobile robot and the fused map segments are sent to the client side for being displayed to the user, and the user can view the constructed and fused image segments of all places conveniently.

Other aspects and advantages of the present application will be readily apparent to those skilled in the art from the following detailed description. Only exemplary embodiments of the present application have been shown and described in the following detailed description. As those skilled in the art will recognize, the disclosure of the present application enables those skilled in the art to make changes to the specific embodiments disclosed without departing from the spirit and scope of the invention as it is directed to the present application. Accordingly, the descriptions in the drawings and the specification of the present application are illustrative only and not limiting.

Drawings

The specific features of the invention to which this application relates are set forth in the appended claims. The features and advantages of the invention to which this application relates will be better understood by reference to the exemplary embodiments described in detail below and the accompanying drawings. The brief description of the drawings is as follows:

fig. 1 is a flow chart of a hardware structure diagram of a mobile robot according to an embodiment of the present invention.

FIG. 2 shows a schematic diagram of three map segments constructed for the present control system with two breaks without stopping movement.

FIG. 3 shows two map segments constructed for one interruption during movement by the control system of the subject application.

FIG. 4 shows an example of a first map segment and a second map segment having a neighboring relationship according to the present application.

FIG. 5 shows a flow chart of a method of one embodiment of the present application for constructing a map.

Fig. 6 is a flowchart illustrating an embodiment of a positioning method according to the present application.

Fig. 7 is a schematic diagram illustrating an architecture of a client according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present application is provided for illustrative purposes, and other advantages and capabilities of the present application will become apparent to those skilled in the art from the present disclosure.

In the following description, reference is made to the accompanying drawings that describe several embodiments of the application. It is to be understood that other embodiments may be utilized and that changes in the module or unit composition, electrical, and operation may be made without departing from the spirit and scope of the present disclosure. The following detailed description is not to be taken in a limiting sense, and the scope of embodiments of the present application is defined only by the claims of the issued patent. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

Although the terms first, second, etc. may be used herein to describe various elements, information, or parameters in some instances, these elements or parameters should not be limited by these terms. These terms are only used to distinguish one element or parameter from another element or parameter. For example, a first map segment may be referred to as a second map segment, and similarly, a second map segment may be referred to as a first map segment, without departing from the scope of the various described embodiments. The first map segment and the second map segment are both describing one map segment, but they are not the same map segment unless the context clearly dictates otherwise. Depending on context, for example, the word "if" as used herein may be interpreted as "at … …" or "at … …".

Also, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes" and/or "including," when used in this specification, specify the presence of stated features, steps, operations, elements, components, items, species, and/or groups, but do not preclude the presence, or addition of one or more other features, steps, operations, elements, components, species, and/or groups thereof. The terms "or" and/or "as used herein are to be construed as inclusive or meaning any one or any combination. Thus, "A, B or C" or "A, B and/or C" means "any of the following: a; b; c; a and B; a and C; b and C; A. b and C ". An exception to this definition will occur only when a combination of elements, functions, steps or operations are inherently mutually exclusive in some way.

For the mobile robot, the SLAM technology is adopted to construct a map, and the map is influenced by the ground material, so that the measurement error of a mobile sensor of the mobile robot is caused. In some embodiments, the mobile robot is configured with a camera device, during the movement of the mobile robot in the physical space based on the movement task, the mobile robot performs visual positioning analysis on the images of different positions to realize continuous positioning, the mobile robot is used for constructing at least a map segment of the physical space corresponding to the movement track, and when the mobile robot continues to move and completes the movement task, the map segment is completed into a map of the physical space corresponding to the movement track. The method for performing visual positioning by using a camera is also called VSLAM, and uses a mapping relationship between a pixel position of at least one image feature (usually not less than 4) in a corresponding image in at least one image captured by a mobile robot during movement and a position point of the mobile robot and the corresponding image feature in a physical space to calculate position information of the mobile robot in the physical space during movement. The map of the physical space corresponding to the movement track can be constructed by accumulating the position relations by utilizing the VSLAM technology. Although the method effectively improves the error problem caused by friction between the mobile robot and the ground in the SLAM technology, the mobile robot still needs the SLAM method to construct a map under the influence of ambient light when the mobile robot is not enough to extract enough image features from the shot image. This makes map errors that mobile robots have constructed historically large. When the mobile robot reuses a map with a large error, the mobile robot cannot accurately perform moving operations such as navigation, relocation, and the like.

In order to improve the accuracy of a map constructed by a mobile robot in a complex environment, the application provides a map construction method. The complex environment is, for example, the ambient light in the physical space does not have stability at different time periods/different positions, for example, the light is brighter in the place where people walk in the living room and the light is darker under the tea table/sofa. Complex environments are also exemplified by mobile robots that do not have unity in the material of the moving surface (e.g., floor) that they contact during movement, e.g., floor materials for living rooms including tiles and carpets.

It should be noted that the complex environment may be not only a private environment such as a home environment, but also any environment in which the mobile robot is movable, such as a storage environment, a park environment, and a public environment such as an airport/train station. Mobile robots in different environments may perform different tasks, and for this purpose, the mobile robots are exemplified by cleaning robots, logistics robots, distribution robots, and the like.

Please refer to fig. 1, which is a schematic diagram of a hardware structure of a mobile robot. The mobile robot 1 includes at least: storage means 11, moving means 12, camera means 13, and processing means 14, the mobile robot may further comprise interface means 15.

Wherein the storage means 11 stores at least one program. The storage means 11 comprise at least one memory. In embodiments, the memory may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state storage devices. In certain embodiments, the memory may also include memory that is remote from the one or more processors, such as network attached memory that is accessed via RF circuitry or external ports and a communications network, which may be the internet, one or more intranets, local area networks, wide area networks, storage area networks, and the like, or suitable combinations thereof. The memory device also includes a memory controller. The memory controller may control access to the memory by other components in the mobile robot, such as the CPU and peripheral interfaces.

The processing device 14 is configured to execute the at least one program to coordinate the operations of the storage device 11 and the mobile device 12 for building the map while moving.

The processing device 14 includes at least one processor, and the at least one processor is connected to the at least one memory for executing and implementing at least one embodiment described in the method of constructing a map when the at least one program is executed. In an embodiment, the processor is operatively coupled with a memory and/or a non-volatile storage device. More specifically, the processor may execute instructions stored in the memory and/or the non-volatile storage device to perform operations in the computing device, such as generating an image and/or transmitting an image to an electronic display. As such, the processor may include one or more general purpose microprocessors, one or more special purpose processors, one or more field programmable logic arrays, or any combination thereof.

Wherein, the moving device 12 is used for driving the moving robot to move integrally, and the moving device executes moving operation under the control of the processing device. In one embodiment, the moving device 12 includes a driving control device and at least two roller sets. Wherein at least one of the at least two roller groups is a controlled roller group. The driving control device is connected with the processing device and drives the controlled roller group to roll based on the control instruction output by the processing device.

The drive control device comprises a drive motor. The driving motor is connected with the roller group and is used for directly driving the roller group to roll. The drive control device may also include one or more processors (such as a CPU or a micro-processing unit (MCU)) dedicated to controlling the drive motor. For example, the micro-processing unit is used for converting a control instruction output by the processing device into an electric signal for controlling a driving motor, and controlling the rotating speed, the steering direction and the like of the driving motor according to the electric signal so as to drive the cleaning robot to move. The processor in the drive control means may be shared with the processor in the processing means or may be provided independently. For example, the processor in the drive control device serves as a slave processing device, the processor in the processing device serves as a master device, and the drive control device performs movement control based on control of the processing device. Or a processor in the drive control means is common to a processor in the processing means. The drive control device receives the control command output by the processing device through the program interface. The driving control device drives the controlled roller group to roll based on the control instruction output by the processing device.

The camera device 13 is used to take at least one image during the movement of the mobile robot, the taken image being used at least for constructing a map.

In some embodiments, the camera device 13 includes, but is not limited to: cameras, video cameras, camera modules integrated with optical systems or CCD chips, and camera modules integrated with optical systems and CMOS chips, and the like. According to the requirement of actual imaging, the lens that can be adopted by the camera or video camera includes but is not limited to: standard lenses, telephoto lenses, fisheye lenses, wide-angle lenses, and the like.

The interface device 15 is used to provide a function of data communication with an external apparatus. Wherein the external device includes, but is not limited to, at least one of: the system comprises a cloud server system, an intelligent terminal for personal use, a monitoring terminal for security linkage, other mobile robots and the like; also included, but not limited to: and the external equipment is used for expanding the operation capability, the reading authority and the like of the mobile robot, such as a U disk, a U shield and the like. Examples of such interface devices include, but are not limited to, at least one of: wireless network interfaces, serial interfaces, etc. Examples of the wireless network interface include: a network interface for accessing a wireless local area network such as WiFi, a network interface for interfacing a mobile network, and the like. The serial interface is exemplified by an interface conforming to serial standards such as USB, RS232, and the like. For example, the interface means communicates with a mobile phone of a user to send the constructed map segment to the mobile phone side, so that the user can view the constructed at least one map segment.

In addition to the hardware devices described in the above examples, the mobile robot further includes various sensing devices (not shown), such as: a collision sensor for detecting a collision, a distance measuring sensor/depth sensor for detecting a peripheral obstacle, a displacement sensor/rotation pulse sensor/speed sensor for detecting the movement of the mobile device, an angle sensor for detecting the rotation angle (or attitude) of the movement of the mobile device, a cliff sensor (also called ground detection sensor) for detecting the suspension of the ground in front of the travel of the mobile robot, a collision sensor (such as a pressure sensor) for detecting a collision, and the like. Each sensing device and the camera device form a data acquisition system for sensing the surrounding environment and acquiring corresponding perception data of the mobile robot. Correspondingly, the perception data includes but is not limited to: visual data, sensory data, etc. Examples of visual data include images, depth maps, and the like. Examples of the sensing data include: movement data indicating movement of the mobile device, measurement data indicating a relative positional relationship between the mobile robot and an obstacle, warning data indicating a hazard warning of a traveling direction of the mobile robot, and the like. The movement data is, for example, displacement data and angle data of a mobile robot (or a mobile device called a mobile robot) moving between two positions. The measurement data include, for example, distance data and direction data between the mobile robot and the obstacle. Examples of the warning data include collision data and floating data.

The processing device 14 of the mobile robot coordinates the mobile device 12, the storage device 11, the data acquisition system (not shown), the interface device 14, and the like to execute the map building scheme provided in the present application by calling at least one program. Wherein. The processing device uses at least one of the sensed data and the visual data provided by the data acquisition system to control the movement of the mobile device 12, and the software and hardware of the mobile device form a control system of the mobile robot, and the control system achieves the purpose of acquiring a map of a physical space moved by the mobile robot by executing the method for constructing the map according to the application. For convenience of description, the processing device coordinates each hardware device of the mobile robot to perform each step of the map building method, also called the control system to perform the process of the map building method.

In examples provided herein, the control system maps based on at least one of sensed data and visual data provided by the data acquisition system in one or more of a variety of ways to map.

In some examples, if the control system is capable of providing a way of constructing a map, the control system performs the map construction in a way of constructing the map based on the acquired data satisfying the need of the way of constructing the map during movement of the mobile robot, and does not perform the map construction in a way of constructing the map when the acquired data not satisfying the need of the way of constructing the map, at which time the control system stores the map constructed when the acquired data satisfies the need of the way of constructing the map as a map segment.

In still other examples, if the control system is capable of providing two map building manners, during the movement of the mobile robot, the control system performs map building in accordance with one of the map building manners when the acquired data satisfies a need of the other map building manner, and performs map building in accordance with the other map building manner when the acquired data satisfies a need of the other map building manner, at which time, the control system divides the maps built in the first map building manner and the second map building manner into different map segments and stores the map segments.

According to the above descriptions of the examples, one way to construct a map referred to in this application is by way of computer processing to construct a map segment.

The map construction method used for constructing a map segment is as follows: for example, during a movement, the control system can stably acquire visual data and sensing data required for constructing a map, and construct a map segment using the same data processing manner using the visual data and the sensing data.

Taking a scheme of constructing a map based on the VSLAM technology as an example, in some embodiments, the control system corresponds an initial position where the mobile robot is located when a map construction method is adopted for map construction to an initial coordinate position of a constructed map segment, performs positioning operation by using visual data and sensing data acquired at different positions of the mobile robot during current movement, and constructs the map segment during the movement by using positioning operation which is continuously iterated.

In some embodiments, the positioning operation is performed by using a positioning scheme with a publication number of CN107907131A or CN109643127A, and is incorporated herein in its entirety.

In other embodiments, the positioning operation may be performed by the following method: during the movement, the data which can be processed by the control system at least comprises a current image, the control system determines a first image feature pair set of the first image matched with the current image and determines at least one pair of second image feature pair sets of the current image matched with the second image; the first image, the second image and the current image are obtained by shooting at three shooting positions which are not collinear; and determining the coordinate information of the current position of the mobile robot according to the position relationship of the first image feature pair set in the first image and the current image, the position relationship of the second image feature pair set in the second image and the current image, and the position relationship between the coordinate information of the shooting positions corresponding to the first image and the second image in the space coordinate system of the physical space. The first image and the second image may be images captured at different positions during the current movement of the imaging device of the mobile robot, or may be images captured during a certain movement in the past. Correspondingly, the position relationship between the coordinate information of the shooting positions corresponding to the first image and the second image in the spatial coordinate system of the physical space (referred to as the position relationship between the two shooting positions for short) may be measured by the movement data during the current movement or measured by the movement data during a certain movement in the past times.

In still other embodiments, the positioning operation may be performed, for example, as follows: during the movement of the mobile robot, the data which can be processed by the control system comprises an image and measurement data, wherein the measurement data is depth information of at least one pixel unit in the indoor space corresponding to the image. For example, the depth information is distance information and orientation information that measure a mobile robot to a top area of the indoor space. The control system acquires a first image and a second image which are respectively shot by the camera device at different positions; the pixel unit of the second image is provided with a second image feature matched with the first image feature in the first image, and an image feature pair is formed between the first image feature and the second image feature; and determining position change information of the mobile equipment between different positions according to the current spatial scale parameter and the pixel position offset of the image feature pair in the first image and the second image.

Taking the map construction scheme based on the SLAM technology as an example, in some embodiments, the control system corresponds a start position where the mobile robot is located when the map construction is performed in a map construction manner to a start coordinate position of a map segment to be constructed, data that the control system can process during movement of the mobile robot includes movement data, measurement data, and the like, measures a positional relationship between positions traveled during the movement and the start position using the movement data, measures a positional relationship between obstacles during the movement and the positions traveled during the movement using the measurement data, and constructs a map segment according to the positional relationship measured during the movement.

It should be noted that map segments obtained by using a map construction method may be obtained by using any of the above examples; or according to at least two positioning examples and carrying out positioning correction; or other such techniques based on SLAM or VSLAM.

It should be noted that, in order to be compatible with the data structure describing the map segment, the manner of constructing the map may be differentiated according to the type of the data structure describing the map segment. The data structure is a data rule used for describing map segments and determined according to data required by the map building mode and data which can be output. For example, the data structure includes: vector data set in a unit grid, and an image captured by the mobile robot during movement and its capture position, image features and their pixel positions in different images, and the like. As another example, the data structure includes: vector data set in a unit grid, and position information between the mobile robot and an obstacle detected during movement. The technician determines, based on the different data structures provided in the examples above, that the control system employs different types of maps.

In some implementations, when the control system switches from a VSLAM technology-based map building approach a to a VSLAM technology-based map building approach B based on data provided by the data acquisition system, and both approaches a and B of the map building describe map segments in the same data structure, one approach that the control system provides to build a map may be considered by the skilled person to include approach a and approach B. For example, if the data structure of a map segment generated by using a map construction method including a positioning solution such as CN107907131A is the same as the data structure of a map segment generated by using a map construction method including a positioning solution such as CN109643127A, a map construction method used by the control system for the same generated map segment includes: the way of constructing maps with positioning schemes like CN107907131A and the way of constructing maps with positioning schemes like CN109643127A are continued.

In other specific examples, the first map building manner and the second map building manner are different schemes designed based on the VSLAM technology/SLAM technology, for example, the first map building manner is a manner of building a map by using a positioning scheme including CN 107907131A; a second way of constructing maps is to construct maps using a positioning solution comprising, for example, CN 109643127A.

In other specific examples, the first map building manner and the second map building manner may be different schemes designed based on the VSLAM technology and the SLAM technology, respectively, for example, the first map building manner is a map building manner designed based on the VSLAM technology; the second map construction method is a map construction method designed based on the SLAM technology.

In some specific examples, the first map construction mode represents a mode of generating map segments by using a first data structure, and the second map construction mode represents a mode of generating map segments by using a second data structure, wherein the first data structure and the second data structure are incompatible.

For convenience of description, the first map building mode and the second map building mode mentioned in the following steps are distinguished according to the map building modes of the map segments built according to the same data structure, and the distinguishing mode has the advantage of improving the capacity of the control system for fusing a plurality of map segments.

The map segment described in each of the above examples describes not only the path traveled by the mobile robot during the movement and the geographical position relationship of the obstacle obtained along the way, but also the positioning information for identifying the geographical position and the geographical position information (such as coordinate data) corresponding thereto are marked on the map segment. The positioning information is, for example, an image feature in an image, or a depth data feature. The positioning information includes, for example, contour features, angle features, point features, line features, and the like, which reflect obstacles, and are obtained from the image/sensing data.

When the control system detects an interruption, the map segments constructed using a map construction method are saved as individual map segments.

The interruption condition is an event (event)/instruction (instruction)/interruption (interrupt) or the like that disables the control system to perform a mapping operation using a currently used mapping method during one movement operation of the mobile robot. Examples of the interrupt condition include any one of: the map segment construction may be interrupted by an external force intervention, by a loss of data required for the first map construction mode, or by an intervention of a generated control command.

The condition that the movement is interrupted due to the intervention of an external force mainly refers to the condition that a control system confirms the moving path of the mobile robot during the movement or the moved physical space is not changed under the control of an internal control mechanism according to the data provided by the data acquisition system. For example, when the user (or pet) forces the mobile robot to change the moving direction, the control system detects that the acquired posture information of the mobile robot does not accord with the issued moving instruction, and then determines that the interrupt condition is generated. For another example, when the user holds the mobile robot from room a to room B, the control system detects that the time length for generating the suspension data exceeds a preset warning time length threshold value, and then determines that the interrupt condition is generated. When an interruption condition occurs, the control system treats the previously constructed map segment as a separate map segment.

In this case, the map segment construction is interrupted due to the absence of data required for the first map construction method. The data required by the first map building mode refers to data which is required by the data acquisition system for the control system to execute the first map building mode. In order to ensure that the execution of the subsequent steps is beneficial to improving the accuracy of the constructed map, the first map construction mode is a map construction scheme designed based on the VSLAM technology. When the required data is missing, the control system generates an interrupt condition and treats the previously constructed map segment as a separate map segment.

In some examples, the data required by the first mapping approach is data that is necessary to be provided by the data acquisition system when the first mapping approach is run. Taking the first MAP building method as the MAP building method a as an example, if the data required for the first MAP building method includes the image and movement data provided by the data acquisition system during movement, the control system determines that the MAP segment MAP0 is interrupted in the building when the control system fails to acquire the movement data, thereby obtaining the individual MAP segment MAP 0.

In other examples, the data required for the first mapping mode is data selected from sufficient data provided by the data acquisition system when the first mapping mode is running. Taking the first map building mode as an example of a mode of building map segments by using the map building mode A or B according to the same data structure, wherein the data required for building the map mode A comprises an image and movement data provided by the data acquisition system during movement, and the data required for building the map mode B comprises a plurality of images at different positions provided by the data acquisition system during movement; when the control system takes the image and the mobile data as input data, a MAP construction is executed by adopting a MAP construction mode A in a first MAP construction mode to generate a MAP segment MAP1 and the MAP segment MAP is gradually expanded along with the moving path of the mobile robot; when the control system cannot acquire the mobile data, the control system adopts a plurality of images shot at different positions in the first MAP building mode as input data to execute a MAP building mode B and continues to expand the MAP segments MAP 1; when the control system is unable to acquire an image, the control system is unable to execute the construction mode a or B of the first MAP construction mode, i.e., discontinue constructing the MAP segment MAP1, based on other data provided by the data acquisition system, thereby resulting in the individual MAP segment MAP 1.

The case where the movement is interrupted due to the intervention of the generated control command mainly means that the control system changes the current movement path or movement mode of the mobile robot according to the received control command. In other words, the control instruction is used to cause the mobile robot to interrupt the currently executed movement operation and then execute the movement operation related to the control instruction. When generating the control command, the control system treats the previously constructed map segment as a separate map segment. For example, the control system changes the moving operation of constructing the map into the moving operation determined by executing a navigation route based on the control instruction, and then the control system takes the map segment constructed before as an independent map segment, wherein the navigation route is planned by utilizing at least one map segment constructed.

Wherein, the control instruction is generated in any one of the following situations: the power management data may be obtained by performing identification analysis on each data provided by the data acquisition system, by performing analysis on power management data provided by a power management system of the mobile robot, or by performing acquisition based on a control operation of an external device. Examples of the control instructions include but are not limited to: including control commands for the direction/distance of movement, control commands relating to functions that can be provided by the mobile robot, etc. The control instruction including the moving direction/moving distance includes, for example: the control system recognizes the gesture/posture of the person in the image to obtain a control instruction including adjusting the moving direction, or the control system listens to the interface device to obtain a control instruction related to changing the moving route, or the like. Examples of control instructions related to functions that can be provided by the mobile robot are as follows: for example, the mobile robot is a cleaning robot, and the control instructions related to the functions provided by the mobile robot comprise a control instruction for dumping garbage to a specified position according to the detected dust collection amount of a dust collection box and a control instruction for replenishing water to the specified position according to the detected water level of a water storage tank which is too low; for another example, the mobile robot is a warehousing and transportation robot, and the control command related to the function provided by the mobile robot includes a control command for changing the placement position generated according to the acquired package address change information. The control instructions related to the functions that the mobile robot can provide may also be related to power management data provided by the power management system of the mobile robot; for example, the control system generates a control command for the recharge movement according to the remaining battery data provided by the power management system.

If the mobile robot can only provide the first map construction method, the mobile robot obtains a plurality of map segments which are individually constructed by the first map construction method according to the interruption condition detected by the control system. If the mobile robot is capable of providing at least a first map construction mode and a second map construction mode, it obtains a plurality of map segments that are individually constructed using the first map construction mode and the second map construction mode, respectively, according to the interruption detected by the control system. In some practical applications, the accuracy of the map segment constructed by using the second map construction method is lower than that of the map segment constructed by using the first map construction method, for example, the second map construction method is a map construction method designed based on the SLAM technology; the first map construction mode is a map construction mode designed based on the VSLAM technology.

It should be noted that the mobile robot can provide more ways of constructing maps to accommodate more complex physical spaces. As described in the examples above, if multiple interruptions are detected during movement, the control system may switch between different types of ways to construct a map without stopping the movement to generate multiple separate map segments.

And the generated individual map segments at least have an incidence relation for constructing the sequence. For example, the control system constructs a plurality of individual map segments in sequence in accordance with the order in which the interruption situations are generated, starting from when the moving operation is started to be performed.

In some examples, the separate map segments generated also have an incidence of start/end positions end-to-end. For example, the control system starts when the moving operation is started to be executed, and takes the end position of the map segment before the interrupt condition is generated as the start position of the map segment after the interrupt condition is generated. In some specific examples, when different kinds of map building manners are adopted according to the interruption situation, the picture segments are connected end to end. For example, referring to fig. 2, which shows a schematic diagram of three MAP segments constructed by the control system under two interruption situations without stopping movement, wherein a solid line is a path of the mobile robot described in each simple MAP segment, and a dashed box is used to represent an individual MAP segment, as shown in the figure, the control system constructs a MAP segment MAP _ a by using a first MAP construction method from a starting point position P0 of movement, constructs a MAP segment MAP _ b by using a second MAP construction method from an end point position P1 of the MAP segment MAP _ a when the first interruption situation occurs, and constructs a MAP segment MAP _ c by using the first MAP construction method from an end point position P2 of the MAP segment MAP _ b until a movement end point position P3 when the second interruption situation occurs. The first interruption condition is, for example, that the control system cannot obtain the image required by the first map building manner, and the second interruption condition is, for example, that the control system obtains the image required by the first map building manner again. In other embodiments, the map segments are connected end-to-end when the same type of map construction is used depending on the interruption. For example, referring to fig. 3, two MAP segments constructed by the control system being interrupted once during the movement are shown, wherein a solid line is a path of the mobile robot described in each simple MAP segment, and a dashed box is used to represent an individual MAP segment, as shown in the figure, the control system starts to construct the first MAP segment MAP _ a 'by using the first MAP construction manner from the start point position P0' of the movement, determines that an interruption occurs when the control system detects that the duration of the suspended data abnormality exceeds a preset duration threshold, and ends when the suspended data abnormality is detected, the control system continues to construct the second MAP segment MAP _ b 'by using the first MAP construction manner from the end point position P1' of the first MAP segment MAP _ a 'to the movement end point position P3'.

In other examples, there is also an adjacency between the two generated separate map segments. In some specific examples, please refer to fig. 4, which shows an example that the first map segment and the second map segment have an adjacent relationship, as shown in the figure, when the area B 'in the second map segment a' constructed by the mobile robot has an overlap of the positioning information with the area B in the previous first map segment a, there is an adjacent relationship between the first map segment and the second map segment.

When the number of the map segments which are separated by using the first map construction mode is more than or equal to two, the control system executes the following steps, the map segments are effectively integrated by using a mode of carrying out fusion operation on the map segments which are separated by using the first map construction mode under the condition that the mobile robot cannot continuously construct a map for a physical space by using a first map construction room, so that the accuracy of the integrated map segments is improved, and the accuracy of the integrated map segments is not reduced due to the influences of areas where the control system does not construct the map/a second map construction mode with lower accuracy adopted by the control system and the like.

Please refer to fig. 5, which is a flowchart illustrating a method for constructing a map according to an embodiment of the present application. The method is mainly executed by a control system of the mobile robot. The method can be executed in an online state or an offline state. Wherein the online state represents a state in which the current map segment is being constructed, e.g., the online state represents a state in which the mobile robot is performing the map construction in the first/second map construction manner. The offline state indicates a state in which a map segment is not constructed, for example, a state in which system resources are sufficient during charging of the mobile robot or during standby.

In step S100, a first map segment and a second map segment constructed by a first map construction method are selected from a plurality of map segments; wherein the plurality of map segments are constructed using at least one of a first map segment and a second map segment that are constructed separately based on a detected outage of the mobile robot.

For convenience of description, it should be noted that the first map segment may also be referred to as a second map segment, and is used only for the description scheme herein.

When the map segment is in the online state, the control system selects a first map segment constructed before from the plurality of map segments while constructing the second map segment by using the first map construction mode. Wherein the frequency of selecting the first map segment is lower than the update frequency of the second map segment when the second map segment is constructed. When in the offline state, the control system selects a first map segment and a second map segment that have been constructed from the plurality of map segments.

Here, the first map segment is randomly selected from the map segments and tried to be performed with the second map segment in the subsequent step S110, or the map segments are sequentially selected from front to back (or from back to front) according to the construction order as the first map segment and tried to be performed with the second map segment in the subsequent step S110, so as to fuse the two map segments. The control system can also select a first map segment closest to the second map segment from the plurality of map segments according to the adjacency relation of the map segments.

An example of the method for determining that the second map segment has an adjacent relationship with at least one map segment constructed by using the first map construction method is to match the positioning information of each map segment constructed by using the first map construction method with the positioning information in the second map segment, and when the matched positioning information pair is obtained, select the first map segment, which has an adjacent relationship with the second map segment. Taking the positioning information in the first map segment and the second map segment as the features in the image as an example, the control system selects a plurality of image features in one image of one generated map segment to match with a plurality of image features in one image of the second map segment, and when at least a preset number of image features are successfully matched, selects the map segment as the first map segment, and executes step S110.

The method for determining that the second map segment has an adjacent relation with at least one map segment constructed by the first map construction method further includes detecting an overlapping condition of at least two map segments according to the construction sequence. The control system detects at least one map segment having an overlap with the second map segment in sequence in accordance with the construction sequence, and selects the first detected map segment as the first map segment. Taking the three MAP segments shown in fig. 2 as an example, the control system detects whether the MAP segment MAP _ a and the MAP segment MAP _ b have an overlap condition with the MAP segment MAP _ c in order from front to back in accordance with the construction order, and takes the MAP segment MAP _ a as the first MAP segment and the MAP segment MAP _ c as the second MAP segment when it is determined that the MAP segment MAP _ a and the MAP segment MAP _ c have an overlap condition.

In this case, since the control system builds a plurality of individual map sections on the basis of the interruption situation, it is possible/established that the same physical space is described in terms of the respective coordinate systems between the selected first map section and the second map section. For the case that it is not determined that the first map segment and the second map segment describe the same physical space, the control system performs the recognition operation by the aforementioned image matching manner to determine whether to continue to perform step S110, i.e., when it is determined that the selected first map segment and the selected second map segment describe the same physical space, the control system continues to perform step S110, otherwise, re-performs step S100 to re-select at least one of the first map segment and the second map segment and perform step S110.

In step S110, the first map segment and the second map segment are fused into one map segment according to the pose deviations of two map positions describing the same physical space in the first map segment and the second map segment.

And for the condition that the first map segment and the second map segment are determined to describe the same physical space, the control system also determines the pose deviation between the two map positions, and fuses the first map segment and the second map segment by using the determined pose deviation to obtain a new map segment.

The control system matches the positioning information in the first map segment and the second map segment to obtain a positioning information pair by using the positioning information in the data structures of the first map segment and the second map segment and the geographic position information (also called coordinate information) of the positioning information in the respective map segments, and fuses the first map segment and the second map segment by using the positioning information to obtain a new map segment by using the geographic position and attitude deviation of the local part of the corresponding actual object in the first map segment and the second map segment respectively. Wherein the positioning information can be described by image features, depth features, and/or the like. The geographic pose deviation comprises a displacement deviation, and/or an angular deviation, of a local portion of the actual object between the geographic locations in the first map segment and the second map segment, respectively.

In some embodiments, the step S110 includes: step S111, aligning the second map segment with the first map segment according to the pose deviation of two map positions describing the same physical space in the first map segment and the second map segment; and step S112, updating the map data in the first map segment according to the aligned first map segment and second map segment to splice the new map segment.

Taking the positioning information as an image feature as an example, the control system determines the pose deviation for each map position in each map segment according to the matched image feature in the first map segment and the second map segment; and performing at least one of a rotation transformation, a translation transformation, or a zoom transformation on the second map segment to align the second map segment with the first map segment according to the pose deviation. Wherein the pose deviation comprises a displacement deviation and/or an angle deviation of the matched image feature pairs between map positions in the map segments.

In a specific embodiment, the control system selects a first image in the first map segment and a second image in the second map segment, performs matching operation on a plurality of first image features in the first image and a plurality of second image features in the second image, and further obtains each piece of first map location information of each matched first image feature in the first map segment and each piece of second map location information of each matched second image feature in the second map segment when obtaining the matched image feature pairs; according to the pixel position deviation of the matched multiple image feature pairs in the first image and the second image, the first map position information POS1 of each first image feature in the multiple image feature pairs in the first map fragment and the mapping relation between the image coordinate system of the first image and the position coordinate system of the first map fragment, and obtaining the map position information POS 2' of each second image feature in the matched multiple image feature pairs in the first map fragment; according to the condition that each map position information POS 2' is approximately overlapped with each second position information POS2 of the plurality of second image features in the second map segment, the control system determines the map pose deviation of the physical space corresponding to the image feature pairs in the two map segments. Wherein the pixel pose deviation comprises: a displacement offset, and/or an angular offset between pixel locations in the first image and the second image for the matched pairs of image features. The approximately overlapping condition may be set according to a systematic error or a preset fixed error value of the first map-building method used, or according to a position error of each map position information POS 2' and each second position information POS2 in the second map segment. And the physical space corresponding to the image feature pairs is a part or all of the same physical space described in the first map segment and the second map segment.

The map pose deviations of the physical space corresponding to the image feature pairs in the two map segments and the pose deviations of the two map positions describing the same physical space in the first map segment and the second map segment can be regarded as equivalent or within a preset error range. The preset error range is set according to at least one of a system error of the first map building mode, a preset fixed error value or an area of the map segment.

And the control system determines the pose deviations of two map positions describing the same physical space in the first map segment and the second map segment according to the obtained pose deviations, and aligns the second map segment with the first map segment according to the determined pose deviations.

And the control system integrally converts each piece of geographic position information in the second map segment according to the pose deviation so as to enable the first map segment and the second map segment to finish alignment operation. And performing matrix transformation on each piece of geographic position information in one of the map segments by the alignment operation according to the rotation angle, the translation displacement and the scaling described by the pose deviation. For example, the control system performs at least one of a secondary rotation transformation, a translation transformation, or a scaling transformation on the second segment. For another example, the control system performs the transformation on the second image segment in blocks to reduce the calculation amount of a single calculation.

And after alignment, obtaining a new map segment which is formed by splicing the first map segment and the second map segment together. In some embodiments, the step S110 further includes: performing data fusion on the positioning information in the two map segments, for example, removing redundant information in a new map segment; and updating the information related to positioning in the new map segment. The redundant information is generated after map areas describing the same physical area in the first map segment and the second map segment are spliced. For example, the control system deletes the map position information and the positioning information thereof overlapped in the second map segment generated after the splicing. Examples of the information related to positioning updated by the control system include: and geographic position information of the shot image originally in the first map section or the second map section and geographic position information corresponding to the image features in the image are respectively geographic position information and the like in the new map section. For example, the control system updates the geographical position information of the shot image marked in the non-overlapping map region in the original second map segment and the position information corresponding to the image feature to the corresponding geographical position information in the new map segment.

In some computer processing approaches, the control system fuses the second map segment that is constructed later into the first map segment that is constructed earlier to obtain a fused map segment. For example, subsequently constructed map segments are continuously fused to previously fused map segments, thereby reducing the amount of computation in fusing subsequent map segments.

The mobile robot selects various map building modes to switch during moving according to the interrupt condition, so that a first map segment, a third map segment and a second map segment are sequentially built in a plurality of built map segments according to the building sequence by way of example; the first map segment and the second map segment are map segments constructed by adopting a first map construction mode, and the third map segment is a map segment constructed by adopting a second map construction mode. After the first map segment and the second map segment are fused into a new map segment, a third map segment connected between the original two map segments can be corrected according to the fused map segment. Thus, by modifying the third map segment, the control system may present a more accurate map to the user.

The control system determines the pose deviation of the third map segment and the new map segment according to the first joint position of the first map segment and the third map segment before fusion, the second joint position of the second map segment and the third map segment before fusion and the positions corresponding to the positions in the map segment after fusion, and corrects the third map segment by using the obtained pose deviation. The first joining position and the second joining position may be position coordinate points or position areas.

The pose deviation can be determined based on the geographic pose deviation between the positioning information of the position area of the first joining position and the positioning information of the position area of the second joining position, or the pose deviation when the first map segment and the second map segment are spliced.

When the selected first map segment and the second map segment in the plurality of map segments are fused into a new map segment, the new map segment is called one of the plurality of map segments. On the basis, the control system continues to perform the fusion operation of the map segments by re-performing the above steps S100 and S110, that is, the fused map segments are classified into the map segments, so as to select a new first map segment and a new second map segment from the map segments and perform the operation of fusing the first map segment and the second map segment into one map segment. Therefore, the map segments which are independently constructed due to the influence of the interrupt condition are merged into the map segments which can describe the geographic environment in the whole physical space moved by the mobile robot.

In order to reduce the system resources occupied by the control system for executing the method, the control system can start executing the method according to the resource occupation condition. Or, some steps in the method are executed according to a preset time interval, for example, two map segments are selected and whether the two map segments include an overlapping area is detected, if so, other steps in the method are executed, for example, a step of merging the first map segment and the second map segment into one map segment, and the like.

With at least one map segment constructed/fused, the control system is also capable of performing positioning operations on the mobile robot, including repositioning when the interrupt condition disappears, and positioning operations during the mobile robot's execution of the navigation task. For example, the control system initiates a repositioning operation using the obtained at least one map segment when it detects that the lifted interruption condition has disappeared. As another example, the control system performs a positioning operation using the obtained at least one map section during the navigation movement to prevent the mobile robot from deviating from the set navigation route.

Please refer to fig. 6, which is a flowchart illustrating an embodiment of a positioning method. In step S200, an image captured by the imaging device of the mobile robot is acquired. And the image records the entity environment in the view field range of the current position of the mobile robot.

In step S210, determining the geographical location information of the mobile robot capturing the image according to the information related to positioning in the at least one pre-stored map segment.

The control system extracts image features in the image and matches the extracted image features in the information relating to the position in one of the map sections to obtain the geographical position of the extracted image features in the respective map section. For example, the control system matches the image features in the image related to positioning in a certain map segment with the extracted image features, and when the matched image feature pairs and the corresponding image pairs are determined, according to the pixel pose deviation between the image feature pairs, the map position information of the image features in the corresponding map segment in the corresponding image feature pairs in the map segment and the mapping relation between the image coordinate system of the image and the position coordinate system of the map segment, the map position information of the shooting position of the shooting image in the map segment is obtained.

In order to facilitate the user to view each map segment constructed by the mobile robot, the control system further performs step S120 (not shown), and sends the map segments and/or the merged map segment to the client for the client to display.

Here, the client is provided in an electronic device held by a user, and the client provides software communicable with the mobile robot to the user. With the hardware of the electronic device scheduled by the client, a user can view a plurality of map segments currently stored in the mobile robot through a display screen of the electronic device, which may be a plurality of original map segments constructed in sequence; or at least one map segment containing the fused map segments.

To this end, the electronic device includes: the device comprises an interface unit, a processing unit, a storage unit and a display unit. The interface unit includes a network interface, and may further include a data interface such as a USB interface for connecting to an external device. Wherein the network interface example comprises at least one of the following: a wireless local area network access module, a mobile network access module, a wired network access module, etc.

The storage unit stores at least one program. The storage unit includes at least one memory. In embodiments, the memory may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state storage devices. In certain embodiments, the memory may also include memory that is remote from the one or more processors, such as network attached memory that is accessed via RF circuitry or external ports and a communications network, which may be the internet, one or more intranets, local area networks, wide area networks, storage area networks, and the like, or suitable combinations thereof. The memory unit also includes a memory controller. The memory controller may control access to the memory by other components in the mobile robot, such as the CPU and peripheral interfaces.

The processing unit is used for executing the at least one program so as to coordinate the storage unit, the display unit and the interface unit to execute the operation of displaying the map segments.

The processing unit comprises at least one processor, and the at least one processor is connected with the at least one memory, and is used for executing and realizing at least one embodiment described in the map building method when the at least one program is run. In an embodiment, the processor is operatively coupled with a memory and/or a non-volatile storage device. More specifically, the processor may execute instructions stored in the memory and/or non-volatile storage to perform operations in the computing device, such as retrieving a map segment and/or transmitting the map segment to an electronic display (i.e., display unit). As such, the processor may include one or more general purpose microprocessors, one or more special purpose processors, one or more field programmable logic arrays, or any combination thereof.

The display unit comprises a display screen and a controller of the display screen. Wherein the controller may be integrated with the processor in the processing unit or configured separately (e.g., GPU). The display screen can also be a touch screen so as to send corresponding operation instructions to the processing unit when the display screen is interactively operated with a user. Wherein the operation instruction comprises: and instructing the mobile robot to execute an operation related instruction, and/or displaying an operation result, an operation process, at least one interface information in the map segment and the like of the mobile robot to execute the operation to the user.

Please refer to fig. 7, which is a schematic diagram illustrating an architecture of a ue according to an embodiment. The client 2 includes: an interface module 21 and a display module 22. The interface module 21 is configured to coordinate an interface unit, a storage unit, and a processing unit in an electronic device, and is configured to be in communication connection with the mobile robot to obtain a plurality of map segments. The display module 22 is configured to display the plurality of map segments according to segment display rules by coordinating a display unit, a processing cloud, and a storage unit of the electronic device. Wherein the map segments are constructed by at least one map construction method, and the map segments comprise map segments which are constructed independently based on the interruption condition detected by the mobile robot. According to the generation examples of the map segments and the fusion example of the two map segments, the map segment displayed by the client is the map segment being constructed by the mobile robot, the map segment not synthesized yet, or the synthesized map segment.

In some practical applications, a user operates a main interface displayed by a display module of a client to generate an instruction for displaying a map constructed by the mobile robot, and the interface module communicates with the mobile robot to obtain a plurality of map segments stored by the mobile robot.

In still other implementations, the client stores a plurality of map segments provided by the mobile robot, and the interface module communicates with the mobile robot to update the stored plurality of map segments when the user operates the main interface presented by the client.

The display module displays a plurality of map segments in a display interface. In order to facilitate the user to identify each map segment, the display module displays each map segment according to a preset segment display rule. Wherein the fragment display rule examples include at least one of: and gradually filling and displaying each region picture segment to obtain a rule, and displaying the plurality of map segments by using different display colors.

Wherein, gradually filling and displaying each region picture segment to obtain a rule comprises: and displaying according to the generation time of each map segment in the map segments, or displaying the sequence numbers corresponding to the map segments according to the construction sequence of the map segments. The client displays each map segment in a dynamic display form, where the dynamic display form is, for example: a GIF-format animation, a FLASH-based animated video, etc.

Wherein the rule for displaying the plurality of map segments with different display colors comprises: and adopting preset type colors and adopting display rules of different colors by using adjacent map segments. Wherein the adjacent map segments include: the two map segments have a joining point, and/or the joining portion of the two map segments is greater than a preset adjacent threshold, and the like.

Taking the display of three map segments on the display screen as an example, according to the position of the mobile robot detecting the interruption condition, the displayed first map segment is connected with the third map segment, the third map segment is connected with the second map segment, the first map segment and the second map segment are displayed in the same color C1, and the third map segment is displayed in another color C2.

Taking the example of displaying two map segments on the display screen, wherein the two map segments are respectively a new map segment after being fused and a map segment after correcting the third map segment, and the displayed new map segment and the corrected third map segment adopt different colors of C1 and C2. The new map segment is obtained by the mobile robot through the steps of dividing the first map segment into a first map segment and dividing the second map segment into a second map segment.

It should be noted that the above examples of the display rules may be used in combination, for example, different colors are adopted, and the map segments are gradually filled and displayed in the display interface in sequence.

The map segment displayed may be a full map segment or a partial map segment depending on the size of the presentation interface displayed on the display screen in the electronic device.

The map segments presented by the display module may all be map segments that have already been constructed. In order to display a plurality of map segments in real time during the map construction of the mobile robot, when the display module in the client displays at least one constructed map segment, a second map segment currently constructed by the mobile robot is also displayed in the display interface through the interface module, so that a user can see the process of constructing the second map segment by the mobile robot, and the displayed second map segment is different from other constructed map segments according to the segment display rule.

In some practical applications, the client may also perform the steps S100 to S110 performed by the mobile robot. In this way, the client may perform the fusion operation on the map segments that have not been fused by the mobile robot, which is not described in detail herein. To this end, the client comprises a construction module for performing the steps S100-S110, namely for selecting a first map segment and a second map segment constructed using a first construction map manner from a plurality of map segments; and fusing the first map segment and the second map segment into a ground picture according to the pose deviations of the two map positions describing the same physical space in the first map segment and the second map segment. And the display module of the client displays the map segment fused by the construction module. For this purpose, the mobile robot may send the individually constructed map segments to the electronic device, and the construction module of the client performs the fusion operation of the map segments.

In still other applications, the steps S100-S110 may be further performed by a cloud server system, wherein the mobile robot is in communication with the cloud server system, and the cloud server system may be further in communication with the client, so as to enable the display of the picture segments on the client through the cloud server system. The mobile robot uploads a plurality of map segments constructed according to at least one map construction mode to the cloud server system, the cloud server system executes the steps S100-S110, and when the client displays the map segments, one or more map segments required to be displayed currently are obtained from the cloud server system.

In order to reduce the system resources occupied by the client when executing the method, the frequency of executing the method by the client is lower than the refreshing frequency of the currently displayed map segment. In this way, when the client displays a map segment being constructed by the mobile robot, the user can see the construction process of the corresponding map segment and the display change of the two map segments from before fusion to after fusion. Wherein the display change comprises: structural changes of the map segments before and after the two map segments perform the fusion operation, display changes generated according to display rules, and the like.

In still other applications, when the mobile robot determines the current location using the at least one map segment, the location may also be presented in a corresponding map segment of the client via a communication connection involving the mobile robot and the client.

The present application also provides a computer-readable and writable storage medium storing a computer program that, when executed, implements at least one embodiment described above for a method of constructing a map, such as the embodiment described in any one of fig. 5 and corresponding thereto. Or at least one embodiment described for the positioning method, such as the embodiment described in fig. 6 and any of the corresponding is implemented.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application.

In the embodiments provided herein, the computer-readable and writable storage medium may include read-only memory, random-access memory, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, a USB flash drive, a removable hard disk, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable-writable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are intended to be non-transitory, tangible storage media. Disk and disc, as used in this application, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

In one or more exemplary aspects, the functions described in the computer program of the methods described herein may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. The steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module, which may be located on a tangible, non-transitory computer-readable and/or writable storage medium. Tangible, non-transitory computer readable and writable storage media may be any available media that can be accessed by a computer.

The flowcharts and block diagrams in the figures described above of the present application illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The above embodiments are merely illustrative of the principles and utilities of the present application and are not intended to limit the application. Any person skilled in the art can modify or change the above-described embodiments without departing from the spirit and scope of the present application. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical concepts disclosed in the present application shall be covered by the claims of the present application.