阅读说明：本技术 基于图像处理控制土壤工作装置的方法及相关系统 (Method for controlling soil working device based on image processing and related system ) 是由 西尔韦奥·雷韦利 于 2018-05-29 设计创作，主要内容包括：本发明涉及基于图像处理来控制土壤工作装置的方法(100)。这种土壤工作装置包括运动构件(201)和工作构件(202)。该方法包括以下步骤：通过安装在工作装置上的数字图像获取装置(203)获取(101)土壤的至少一个数字图像；通过电子处理单元(204)处理(102)至少一个数字图像,该至少一个数字图像是通过经训练的神经网络(300；400；500)对数字图像执行至少一个卷积运算而获取的；通过电子处理单元,基于这种处理,获取(103)至少一个合成土壤描述符；通过电子处理单元,基于合成土壤描述符,生成(104)运动构件或工作构件的至少一个控制信号。(The invention relates to a method (100) for controlling a soil working device based on image processing. The soil working device comprises a moving member (201) and a working member (202). The method comprises the following steps: acquiring (101) at least one digital image of the soil by means of a digital image acquisition device (203) mounted on the working device; processing (102), by an electronic processing unit (204), at least one digital image, the at least one digital image being acquired by performing at least one convolution operation on the digital image by means of a trained neural network (300; 400; 500); -acquiring (103), by the electronic processing unit, at least one synthetic soil descriptor based on such processing; generating (104), by the electronic processing unit, at least one control signal of the moving or working member based on the synthetic soil descriptor.)

1. A method (100) for controlling a soil working device comprising a movement member (201) and a working member (202) based on image processing, wherein the method comprises the steps of:

-acquiring (101) at least one digital image of the soil by means of a digital image acquisition device (203) mounted on said working device;

-processing (102), by an electronic processing unit (204), said at least one digital image being obtained by performing at least one convolution operation on said digital image by means of a trained neural network (300; 400; 500);

-acquiring (103), by said electronic processing unit, at least one synthetic soil descriptor based on said processing;

-generating (104), by the electronic processing unit, at least one control signal of the moving member or of the working member based on the synthetic soil descriptor.

2. A method (100) for controlling an earth working device according to claim 1, wherein the generating step (104) comprises the step of generating at least one control signal for the moving member (201) and at least one control signal for the working member (202).

3. A method (100) for controlling a soil working device according to claim 1, comprising the further step of training (600; 700; 800) the neural network (300; 400; 500) with at least an image of the soil that is operable or operable with the working member (202).

4. A method (100) for controlling a soil working device according to claim 1, further comprising the step of generating data and/or parameters describing the nature or structure of the soil based on the aforementioned synthetic soil descriptor.

5. A method (100) for controlling a soil working device according to claim 4, wherein the descriptive parameter includes a boundary or discontinuity between two soil regions having different characteristics.

6. A method (100) for controlling a soil working device according to claim 5, wherein the boundary or discontinuity portion comprises a boundary between soil areas of plant species having different heights, or a boundary between an area previously cut or worked and an area not yet cut or worked.

7. A method (100) for controlling an earth working device according to claim 1, wherein the neural network (300; 400; 500) comprises at least one convolution layer and the at least one convolution operation is done by feed forward of the neural network.

8. A method (100) for controlling a soil working device as claimed in claim 7, wherein the synthetic soil descriptor is a soil classification and the convolutional neural network (300) comprises at least the following layers:

-an input layer (301) configured to receive an entire digital image or at least one down-sampling of the digital image;

-at least one convolutional layer (conv 1);

-at least one fully connected (303a) layer;

an output layer (304) having at least one neuron configured to enable differentiation between at least two soil classifications.

9. A method (100) for controlling a soil working device as claimed in claim 7, wherein the synthetic soil descriptor is a soil semantic segmentation and the convolutional neural network (400) comprises at least the following layers:

-an input layer (401) configured to receive an entire digital image or at least one down-sampling of the digital image;

-at least one convolutional layer (conv 1);

-at least one deconvolution layer (deconv 1);

-an output layer (404) configured to enable semantic segmentation of the soil image into at least two classifications.

10. A method (100) for controlling a soil working device as claimed in claim 7, wherein the synthetic soil descriptor is a specific action taken by the working device based on the characteristics of the soil being constructed, and the convolutional neural network (500) comprises at least the following layers:

-an input layer (501) configured to receive an entire digital image or at least one down-sampling of the digital image;

-at least one convolutional layer (conv 1);

-at least one fully connected (503a) layer;

-an output layer (504) having at least one neuron configured to enable distinguishing between at least two specific actions performed on the soil.

11. A method (100) for controlling a soil working device as claimed in claim 3, wherein the step of training (600) the neural network (300) comprises the steps of:

-defining (601) a position and an orientation of the digital image acquisition device (203) mounted on the working device,

-obtaining (602) in input a plurality of digital images of the soil to be worked in different working phases of the soil,

-classifying (603) the plurality of digital images by associating a label with each acquired digital image;

-initializing (605) the neural network by randomly or by default associating weights with connections;

-training the network (606) by a back propagation method to train at least one layer of the neural network by modifying the weights associated with the network based on the labels of a plurality of classified digital images.

12. A method (100) for controlling a soil working device as claimed in claim 3, wherein the step of training (700) the neural network (400) comprises the steps of:

-defining (701) the position and orientation of the digital image acquisition device (203) mounted on the working device;

-acquiring (702) in input a plurality of digital images of the soil to be worked in different working phases of the soil;

-segmenting (703) the plurality of digital images by associating the same identifier with each portion of the image;

-initializing (705) the neural network by randomly or by default associating weights with connections;

-training (706) the network by a back propagation method to train at least one layer of the neural network by modifying the weights associated with the network based on semantic classification of a plurality of segmented digital images.

13. A method (100) for controlling a soil working device as claimed in claim 3, wherein the step of training (800) the neural network (500) comprises the steps of:

-defining (801) a position and an orientation of the digital image acquisition device (203) mounted on the working device, the working device being movable by an operator;

-acquiring (802) in input a plurality of digital images of the soil to be worked in different working phases of the soil;

-recording (803), based on the characteristics of the soil, specific actions to be taken by the working device, each specific action being associated with the plurality of images acquired;

-initializing (805) the neural network by randomly or by default associating weights with connections;

-training the network (806) by a back-propagation method to train at least one layer of the neural network by modifying the weights associated with the network based on the particular action associated with a plurality of acquired digital images.

14. A method (100) for controlling a soil working device as claimed in any one of claims 11, 12 and 13, wherein the training step further includes the step of increasing (604; 704; 804) the number of images available for training by processing each selected raw image, the processing operation being selected from the group consisting of:

-a rotation of the image,

-a down-sampling selection of the image,

-correction of at least one color channel of said image.

15. A method (100) for controlling a soil working device as claimed in claim 14 when dependent on claim 13 wherein the training step further comprises the step of controlling the consistency of each image of the plurality of digital images with other images of the same classification acquired.

16. A method (100) for controlling a soil working device according to claim 11, wherein the synthetic soil descriptor is a soil classification and the step of generating (104) at least one control signal for the moving member or the working member based on the soil classification comprises the steps of:

-selecting (901) a predetermined sequence of motion instructions associated with the soil classification obtained based on the processing performed by the trained neural network (300);

-sending (902) the sequence of motion instructions to the motion member (201) of the working device or the working member (202) of the working device.

17. A method (100) for controlling an earth working device according to the preceding claim, wherein the movement instructions are selected from the group consisting of:

-maintaining the direction of the light beam,

-a change of direction of the movement of the support,

-stopping/starting the moving member (201) or the working member (202).

18. A method (100) for controlling a soil working device according to claim 12, wherein the synthetic soil descriptor is a soil semantic segmentation and the step of generating (104) at least one control signal for the motion member or the working member based on the segmentation comprises the steps of:

-selecting (921) at least one pair of semantic classifications at a boundary of the soil where the working device is expected to remain aligned;

-identifying (922) the at least one pair of semantic classifications on the acquired image;

-if the classes of said pair are adjacent, calculating (923), by means of a regression method, a parameter characterizing the separation curve or boundary of said two classes;

-determining (924) a sequence of motion commands or a sequence of work commands based on the calculated parameters characterizing the separation curve;

-sending (925) the sequence of movement instructions to the movement member (201) of the work device or to the work member (202) of the work device.

19. A method (100) for controlling an earth working device according to the preceding claim, wherein the movement instructions are selected from the group consisting of:

-rotation instructions for correcting the orientation of the working device by making it parallel to a line tangent to the separation curves of the two classifications;

-translation instructions for correcting the position of the work apparatus by keeping it on the separation curves of the two classifications;

-instructions for reversing the direction of the device and confining it within the working area;

-an instruction to avoid an obstacle.

20. A method (100) for controlling a soil working device according to claim 13, wherein the synthetic soil descriptor is a specific action taken by the working device based on the characteristics of the constructed soil, and the step of generating (104) at least one control signal includes the step of sending (931) the specific action to be taken to the moving member (201) or the working member (202) of the working device, the step being obtained based on the processing performed by the trained neural network (500).

21. A method (100) for controlling an earth working device according to the preceding claim, wherein the action is a classification of a discontinuous action selected from the group consisting of:

-the speed is increased by a speed increase,

-the speed of the reduction is reduced,

-changing the direction of movement,

-stopping the movement of the device,

-starting the movement of the movable part,

-activating the working member (202),

-stopping the working member (202).

22. A method (100) for controlling an earth working device according to claim 20, wherein the action is a continuous action.

23. A method (100) for controlling an earth working device according to claim 20, wherein the action is selected from the group consisting of:

-a rotational action directed towards the motion member (201) to correct the travel angle,

-an acceleration or deceleration action directed towards the moving member (201),

-starting/stopping the working member (202).

24. A method (100) for controlling a soil working device according to claim 1, further comprising a step of approving the at least one control signal of the moving member or the working member generated based on the synthetic soil descriptor by an operator or by a program defined by the operator.

25. A system (1000) comprising:

-a soil working device comprising a moving member (201) and a working member (202),

-digital image acquisition means (203) mounted on said working means for acquiring at least one digital image of said soil;

-an electronic processing unit (204) connected to the digital image acquisition device and to the motion member (201) and the work member (202) of the soil working device,

the electronic processing unit (204) is configured to perform the steps of the method according to any one of claims 1 to 24.

26. The system (1000) according to claim 25, wherein the electronic processing unit comprises at least one processor (205) and a memory block (206, 207) associated with the processor for storing instructions, the processor and the memory block being configured to perform the steps of the method according to any one of claims 1 to 24.

27. The system (1000) of claim 25, wherein the soil working device is selected from the group consisting of:

-a lawn-mower adapted to be moved by a user,

-a harvesting machine for harvesting the grain,

-a plough,

-a grain harvester for the grain,

-a vegetable harvester for harvesting the vegetables,

-a hay harvester for harvesting hay from the hay harvester,

-passing it through any machine whose visual perception of the soil is altered.

28. The system (1000) of claim 25, wherein the processing unit (204) includes an input/output interface (208), the input/output interface (208) being connected to the at least one processor (205) and memory block (206, 207) to allow an operator in the vicinity of the system to interact directly with the processing unit.

29. The system (1000) of claim 25, further comprising a wired or wireless communication interface (209) for connecting the processing unit (204) to a data communication network (210) to allow an operator to remotely interact with the processing unit.

Technical Field

The present invention relates generally to the field of control of soil working machines, such as mowers, harvesters, plows, and the like. In particular, the present invention is directed to a method for controlling a soil working device based on image processing through a convolutional neural network. The invention also relates to a system comprising a soil working device for carrying out the above method.

Background

There are currently commercially available lawn or robotic mowers for residential use that are configured to be autonomously (i.e., without operator guidance) operable. Furthermore, in the commercial field, the need for harvesters adapted for autonomous or semi-autonomous operation and, in general, agricultural soil working machines that minimize human intervention is growing.

Hitherto, various systems and related control methods are known which allow to control the movement of a mower so as to limit its operating range within the relevant working area and to direct it to the field to be cut.

Some robotic lawnmowers for residential use move along randomly selected trajectories. For such robotic lawnmowers, a continuous boundary-delimiting means (e.g., a peripheral cable) is employed to control the robot itself from leaving the intended work area.

Other robotic mowers on the market are equipped with sensors suitable for detecting large numbers of objects. Such sensors provide a large presence/absence condition at a certain height from the ground to the control unit of the lawn mower. The mower processing unit interprets this information as the presence/absence of grass.

Known machines operating on large surfaces, mainly for commercial or agricultural use, are generally equipped with an onboard GPS antenna, and optionally operatively associated with one or more GPS antennas mounted on the ground (differential GPS). The machine is equipped with a processing unit configured to compare the current machine coordinates with preset coordinates defining the periphery of the work area and thus able to keep the machine in the specified area. In contrast, by comparing the current coordinates of the machine with the historical coordinates, such a processing unit may set a machine movement trajectory that does not overlap with a trajectory already traveled.

In the residential, commercial and agricultural fields, it is well known to detect the position of a robotic lawnmower or an automated field work machine by using radio frequency devices or beacons known to those skilled in the art of wireless communication. More radio frequency beacons are placed at key points in the area to be operated. The processing unit equipped with the mower or machine can determine its unique position based on the calculation of the distance from the aforementioned beacon and thus limit the movement of the mower or machine in the stored periphery and generate a movement trajectory that does not overlap with the trajectory already travelled.

All the known solutions described above have the following drawbacks: there is always a need to use infrastructure, i.e. peripheral wires, satellite antennas or beacons, outside the robotic lawnmower or working machine in order to properly control and confine such machines in the work area. These are therefore solutions that require an initial infrastructure laying step and the related configuration of such a structure and its maintenance. This solution is therefore complex and for the reasons mentioned above, its portability is limited.

To date, the control by image processing has not found a specific application, since the proposed method is not robust enough to manage a plurality of disturbances present in the actual operating environment. These disturbances include: a shadow of the object; the shade of the mower itself; reflected light and refraction; presence of dry vegetables; inflorescence; also due to movement of the vehicle, the camera is under-or over-exposed; different perceptions caused by weather phenomena; generating different perceptions due to the direction of travel of the mower; grains having an off-nominal height; perspective distortion due to soil unevenness or vehicle inclination; plants that are inclined or swaying by wind; plants or grasses covered by leaf parts; a plant or herb that is different from the expected or infected; flood areas or puddles; areas without plants or grass; the presence of a three-dimensional obstacle; incomplete obstacles, such as high pressure towers; a flat obstacle; a human or animal present in the direction of travel.

Also, mention is made in particular of lawnmowers for residential use, and it should also be noted that there are also various disturbances in the garden, which, as is well known, is a dynamic environment that constantly changes with the succession of seasons and due to frequent updates and maintenance.

Disclosure of Invention

The object of the present invention is to devise and provide a method and a related system for controlling an earth working device that allow to overcome at least partially the drawbacks of the methods described above with reference to the known solutions.

This object is achieved by a method for controlling a soil working device according to claim 1. In particular, the invention relates to a method of controlling a soil working device based on image processing performed by a trained convolutional neural network.

The main aspect of the invention is the application of a neural network that implements a series of subsequent convolutions by internally constructing a hierarchical representation (not necessarily interpretable) of the image features. The network provides a composite descriptor of the soil configured to allow the soil working device to perform complex operations in an autonomous manner.

The method of the invention ensures a double advantage.

Since the modeling of the soil is not explicitly coded by humans, but automatically learned by machines, the first advantage is the high cognitive power and high degree of generalization of the system implementing the method. This makes the system of the present invention robust against the above listed disturbances, thereby overcoming the limitations of the prior art.

In particular, the applicant has verified that the system is robust to accidental events, the presence of accidental plants, variations in lighting and light and shadow games.

A second advantage is that the system of the invention can autonomously perform soil-related movements that were hitherto unthinkable, whether by means of a movement member or a cutting member. The element that determines this ability is the composite descriptor of the soil, which carries the information required by the control member. For example, as described below, the soil working device may distinguish between various obstacles to perform the best possible action. Or, for example, it may follow the previous cutting line in a robust and safe manner compared to an obstacle or accident. Or, for example, it may perform a continuous action directly related to the acquired image.

Preferred embodiments of such a control method are described in the dependent claims 2-24.

The invention also relates to a system implementing the aforementioned method according to claim 25.

Drawings

Further features and advantages of the control method and of the related system according to the invention will become apparent from the following description of preferred embodiments, given by way of illustrative and non-limiting example, with reference to the accompanying drawings, in which:

fig. 1 shows a flow chart of a method for controlling an image-processing-based soil working device according to the invention;

figure 2 shows a block diagram of a soil working system based on image processing implementing the method of figure 1;

fig. 3A shows a logic diagram of a first embodiment of a neural network comprising a convolution stage, employed in the method of the invention and configured to return a soil classification starting from a digital image;

figures 3B and 3C show in tables examples of coding different soil classifications;

figure 4A shows a logical diagram of a second embodiment of a neural network comprising convolution and deconvolution stages employed in the method of the present invention and configured to return semantically segmented images starting from the acquired digital image;

FIG. 4B shows an example of semantic segmentation of four acquired digital images;

fig. 5A shows a logic diagram of a third embodiment of a neural network comprising a convolution stage, employed in the method of the invention and configured to return the desired action starting from the acquired digital image;

FIG. 5B shows an example of a classification of coding actions in a table;

FIG. 6 shows a flow chart of a training method of the neural network of FIG. 4A;

FIG. 7 shows a flow chart of a training method of the neural network of FIG. 5A;

FIG. 8 shows a flow chart of a training method of the neural network of FIG. 6A;

figures 9A to 9B show a flow chart of a method of controlling the movement of a soil working device based on image processing performed by the neural network of figure 3A, and a corresponding application scenario, respectively;

10A-10B illustrate a flow chart of a method of controlling movement of a soil working device based on image processing performed by the neural network of FIG. 4A, and a corresponding application scenario, respectively;

fig. 11A to 11B respectively show a flow chart of a method for controlling the movement of a soil working device based on image processing performed by the neural network of fig. 5A, and a corresponding application scenario.

In the foregoing drawings, the same or similar elements are denoted by the same reference numerals.

Detailed Description

Referring to fig. 1, a method for controlling a soil working device according to the present invention is designated by reference numeral 100. In particular, the invention relates to a method for controlling a soil working device based on image processing. Hereinafter, this control method of the working device 100 is also simply referred to as a control method.

Referring to FIG. 2, a system including the soil working device described above is generally indicated by reference numeral 1000.

Such soil working devices are for example selected from the group consisting of: mowers, harvesters, plows, grain harvesters, vegetable harvesters, hay harvesters, and any similar machine that passes through soil that alters the visual perception of the soil itself.

The working device of the system 1000 comprises a suitable moving member 201 and a working member 202.

The above-mentioned moving member 201 includes, for example, a propulsion unit powered by a corresponding fuel, a steering unit, a stability control and safety control unit, and a power supply unit (e.g., a battery). Further, such a moving member 201 may also include a suitable input/output communication interface for receiving command signals and return signals indicative of movement.

The work member 202 comprises a suitable work tool that is movable by a corresponding motor, and further includes other input/output communication interfaces for receiving command and return signals indicative of work performed.

Further, the system 1000 includes a digital image acquisition device 203 mounted on the working device to acquire at least one digital image of the soil. Such image acquisition means are for example realized by one or more cameras 203. Such a camera 203 may acquire more or less of the surrounding soil based on its number, its position, and the geometry of the lens characterizing its field of view. Such cameras 203 and similar image capture devices may receive grayscale images or preferably color-coded visible spectrum (e.g., RGB) images. Such an image capture device may be configured to operate in the infrared and ultraviolet spectra, or to complete optical information using channels dedicated to depth (e.g., RGB-D).

The system 1000 further comprises an electronic processing unit 204, the electronic processing unit 204 being connected to the digital image acquisition device 203 and to the aforementioned movement member 201 and working member 202 of the soil working device.

The electronic processing unit 204 comprises at least one processor 205 and a memory block 206, 207 associated with the processor for storing instructions. In particular, such memory blocks 206, 207 are connected to the processor 205 through a data communication line or bus 211 (for example PCI) and are made up of a service memory 206 of the volatile type (for example SDRAM type) and a system memory 207 of the non-volatile type (for example SSD type).

In more detail, such electronic processing unit 204 comprises input/output interface means 208 connected to at least one processor 205 and to memory blocks 206, 207 through a communication bus 211, to allow an operator in the vicinity of the system 1000 to interact directly with the processing unit itself. Such input/output interface means 208 comprise, for example, a touch display operating from a data input/output and information display interface.

Further, the electronic processing unit 204 is associated with a wired or wireless data communication interface 209, which wired or wireless data communication interface 209 is configured to connect such processing unit 204 to a data communication network, such as the internet 210, to allow an operator to remotely interact with the processing unit 204 or to receive updates.

Referring to fig. 1, the operational steps of method 100 for controlling an earth-working device based on image processing implemented by system 1000 are described in more detail below.

In one embodiment, the electronic processing unit 204 of the system 1000 is arranged to execute code of an application program implementing the method 100 of the present invention.

In a particular embodiment, the processor 205 is configured to load and execute the code of the application implementing the method 100 of the invention in the memory blocks 206, 207.

In a more general embodiment, the control method 100 comprises a first step 101 of acquiring at least one digital image of the soil by means of a digital image acquisition device 203 (i.e. a camera) mounted on the working device.

The method 100 then comprises a step 102 of processing the acquired at least one digital image by means of the electronic processing unit 204.

Advantageously, this processing comprises performing at least one convolution operation on the acquired digital image by means of the trained neural network 300, 400, 500.

It should be noted that such a neural network 300, 400, 500 for performing the method comprises at least one convolutional layer. Further, the at least one convolution operation is accomplished by feed forward of the neural network. Examples of such convolutional neural networks 300, 400, 500 are shown in fig. 3A, 4A, and 5A, respectively. Each of these three examples of convolutional neural networks returns a particular form of soil composition descriptor: 300 the network returns a global classification of the image; 400, returning semantic segmentation to the network; 500 network return action. These networks will be described in more detail below.

Advantageously, the control method 100 of the invention comprises a step 103 of acquiring at least one synthetic soil descriptor by means of an electronic processing unit 204, based on the electronic processing unit.

Method 100 also comprises a step 104 of generating, by electronic processing unit 204, at least one signal to control one of the moving member 201 or the working member 202 of the above-mentioned working device, based on such a synthetic soil descriptor.

In a different embodiment of the method of the invention, the above-mentioned generating step 104 comprises the step of generating at least one control signal of the moving member 201 and at least one control signal of the working member 202. In other words, the method of the invention allows simultaneous control of the moving member 201 and the working member 202.

The control method 100 further comprises the step of generating data and/or parameters describing the nature or structure of the soil itself based on the synthetic soil descriptors described above.

In more detail, such descriptive parameters include a boundary or discontinuity between two soil regions having different characteristics.

In even more detail, such boundaries or discontinuities include boundaries between soil areas (including plant species of different heights) or between previously cut or worked soil areas and areas that have not been cut or worked (e.g., cut grass, uncut grass).

Referring to fig. 3A, 3B, and 3C, in a first embodiment of the present invention, such a synthetic soil descriptor is a soil classification. In this case, the convolutional neural network 300 or the first convolutional network 300 includes at least the following layers:

an input layer 301 configured to receive an entire digital image or at least one down-sampling of such a digital image acquired by the camera 203;

-at least one convolutional layer conv 1;

-at least one fully connected layer 303 a;

an output layer 304 having at least one neuron configured to distinguish between at least two soil classifications.

In more detail, the network 300 operating on the basis of soil classification comprises a convolution block 302 consisting of three concatenated convolution layers conv1, conv2, conv3 and three maximum pooling layers MP1, MP2, MP3, of a type known to the person skilled in the art. Each max pooling layer MP1, MP2, MP3 is directly connected to the output of the respective convolutional layer. Such a maximally pooling layer performs discretization operations of the image samples with the aim of adjusting their size.

It is well known that in convolutional layers of neural networks, each neuron is connected to only some neighboring neurons in the previous layer. Each neural connection uses the same set of weights (and local connection layout). In contrast, in a fully connected layer of the network, each neuron is connected to each neuron of the previous layer, and each connection has its own weight.

In the example of fig. 3A, the neural network 300 includes two fully connected layers 303A and 303 b. These two layers are similar to a convolution with a kernel covering the entire input layer of the neural network 300. Thus, these are two other convolutions configured to provide global meaning to the input layer.

According to a preferred aspect of the invention, the method 100 has been implemented to autonomously guide a robotic lawnmower in a work area. In particular, step 102 is accomplished by a neural network 300, wherein the input layer consists of 35643 neurons, i.e., a 3 × 109 × 109 neuron matrix (where 3 are color channels and 109 × 109 is the resolution of the input image). Experimentally, this resolution is particularly suitable for turf identification, since the input image is small enough to be processed quickly by commercially available low-cost electronic processing units (e.g., OdridXU 4 or NVIDIATX2), but still large enough to carry the information content needed to identify the turf features. The kernel size of the first convolution layer conv1 in fig. 3A is4 × 4 pixels and spans 1 pixel. Such a layer conv1 comprises 20 output filters of size 106 x 106. The maximum pooling layer MP of FIG. 3A has a kernel equal to 2 × 2 and a span equal to 2, whereby the size of each filter is reduced to 53 × 53. By performing subsequent convolution and pooling operations using similar parameters, progressively smaller filters are obtained. Once the convolution step 302 is strictly complete, the filter is connected to two fully connected layers 303a and 303 b. According to a preferred aspect, the neural network 300 includes an exit layer known to those skilled in the art to facilitate generalization. The number of neurons in the output layer 304 is equal to the number of different grass soils. In one embodiment, the value of such neurons of the output layer 304 is normalized using the softmax operator.

Fig. 3B and 3C show two examples in a table encoding different soil classifications. Each classification is associated with a neuron of the output layer 304. In particular, the table in fig. 3B associates a different numerical code for each soil classification, which can be distinguished by the control method 100 starting from the acquired image through the neural network 300.

It may be noted that in addition to encoding the presence/absence of mowing on the soil and the presence or absence of items in mowing, encoding obstacles and/or other specific items and possible fault conditions of the camera 203 are included.

In an application example of the control method 100 of the invention, the use of only two categories associated with the traversability of the soil itself is provided. In this case, the structure of the neural network 300 remains substantially unchanged, except that the output layer 304 preferably has a single neuron. Such activation of neurons, for example, indicates the complete presence of a weed in the image of the soil area acquired by camera 203. Vice versa, the lack of neuronal activation indicates that the soil area is not completely populated with vegetation and obstructions. Referring to the table in fig. 3C, only the presence/absence of grass to be cut is coded in a binary manner.

It should be noted that the neural network 300 may be adapted for different types of grass or grain, as well as for different agricultural soils, and thus used and applied in the control method 100 to the associated system 1000. For example, in the case of a harvester applied in a mature wheat field, such a neural network 300 returns whether the harvester meets wheat ahead of it or faces an area (e.g., a road or a grid) without wheat.

Referring to fig. 4A, in a second embodiment of the control method 100, the synthetic soil descriptor is a soil semantic segmentation. In this case, the convolutional neural network 400 or the second convolutional neural network 400 includes at least the following layers:

an input layer 401 configured to receive at least one down-sampling of the entire digital image or of the digital image acquired by the camera 203;

-at least one convolutional layer conv 1;

-at least one deconvolution layer deconv 1;

an output layer 404 configured to enable semantic segmentation of the soil image into at least two classifications.

In more detail, the network 400, which operates based on the semantic segmentation of the soil, includes a respective convolution block 402, which is composed of a plurality of convolution layers (e.g., cascaded n layers, conv1,.., convn), separated by n-1 maximum pooling layers MP1,.., MPn-1.

Furthermore, neural network 400 includes a respective deconvolution block 403, formed by a plurality of deconvolution layers (e.g., cascaded n layers, deconv1,.. and deconvn) of a type known to those skilled in the art, spaced by n-1 non-pooled layers, UNP1,.. and UNPn-1. Such a second deconvolution block 403 performs the inverse function to that performed by the convolution block 402, i.e. it is configured to reconstruct a semantically segmented image starting from the image segmented by the convolution block 402.

Referring again to the preferred and non-limiting aspects of mowing, a neural network of FCN type (fully convoluted) 400 has been implemented, with the convoluted portion unchanged relative to the neural network 300. According to another variation of step 102 of the control method 100, the output of the network is clearer relative to a simple turf soil classification and is given by a segmented image in different types of grass or obstacles that the mower may encounter during operation. The deconvolution block operates in the reverse of the previously described manner, sharing its basic characteristics: the image is thus reconstructed as a segmentation at the original resolution (which is 109 × 109 in the case of the mower described above).

Fig. 4B shows an example of semantic segmentation performed on the acquired four digital images I1, I2, I3, I4. In particular, the first image I1 shows a boundary or discontinuity L1 between two soil areas having different characteristics, for example, area a being tall grass and area B being short grass/mowed. Reference ML denotes a part of the soil working apparatus photographed by the camera 203. In response to the acquired first image I1, the method of the invention provides a corresponding first semantically segmented image IS1 by the method of the invention via the neural network 400.

Similarly, a second semantically segmented image IS2, a third semantically segmented image IS3 and a fourth semantically segmented image IS4 are provided by the method of the present invention through the neural network 400 in response to the acquired second image I2, third image I3 and fourth image I4, respectively, by the method of the present invention.

In particular, the second semantically segmented image IS2 comprises two areas B, short grass/mowed, which are separated from the area C without grass. The third semantically segmented image IS3 comprises the region B short grass/mowed in the vicinity of the working device, where the leaf element F IS present. Such a third image IS3 also comprises a horizon area OR, which IS separate from area B. The fourth semantically segmented image IS4 comprises an area B-short grass/mowed, in which an obstacle OS IS present.

It should be noted that such a neural network 400 may be suitable for different types of grass or cereals, and for different agricultural soils, and is therefore used in the control method 100 and applied to the relevant system 1000. For example, applied to a harvester in a mature wheat field, such a neural network 400 returns an image that indicates with two different color codes the portion of the field where the wheat is harvested and the portion that still must be harvested. Such information may be used in subsequent steps of the control method 100 for analysis-type processing.

Referring to fig. 5A, in a third embodiment of the present invention, a synthetic soil descriptor is a specific action taken by a working device based on characteristics of soil being constructed. In this case, the convolutional neural network 500 or the third convolutional neural network 500 is substantially similar to the convolutional network 300 and includes at least the following layers:

an input layer 501 configured to receive at least one down-sampling of the entire digital image or the acquired digital image;

-at least one convolutional layer conv 1;

at least one fully connected layer 503 a;

an output layer 504 with at least one neuron configured to enable distinguishing between at least two specific actions performed on the soil.

In more detail, the network 500 comprises a respective convolution block 502, which consists of three cascaded convolution layers conv1, conv2, conv3 and three maximum pooling layers MP1, MP2, MP 3. Each maximally pooled layer is directly connected to the output of the corresponding convolutional layer.

In the example of fig. 5A, the neural network 500 includes two fully connected layers 503a and 503 b. Referring again to a preferred and non-limiting example of mowing, each output neuron of network 500 (normalized with respect to other output neurons using appropriate softmax operations) represents a particular action of the mower. Thus, the classification does not represent the visible attributes of the grass, but rather the actual operation that can be directly attributed to the acquired image. This approach is defined as "pixel-to-action" or "end-to-end". As will be described in detail below, this method minimizes the complexity of the subsequent steps of the control method 100, which has returned primitives for action commands to be sent to the moving member 201 or the working member 202.

Fig. 5B shows in a table an example of encoding the motion classification to be performed based on the processing performed on the acquired image by the neural network 500. In particular, such a table associates a unique digital code with each action to be performed by the moving member 201 of the work device (e.g. keeping straight, stopping movement, etc.) after the processing of the digital image acquired by the neural network 500.

It should be noted that the control method 100 of the present invention also includes another step related to the training 600, 700, 800 of each of the three examples of neural networks 300, 400, 500 described above.

Specifically, referring to fig. 6, illustrative details of a training method 600 of the first neural network 300 are described.

In particular, such a method 600 includes an initial step 601 of defining the position and orientation of the digital image acquisition device 203 mounted on the work apparatus.

In the case of a lawn mower, according to an exemplary and non-limiting aspect of the present invention, a camera 203 with bayer RGB filters, a dynamic range of 69.5dB and a 180 degree lens is installed. It is placed along the axis of symmetry of the mower at a height of 26 cm from the ground. This camera is forward with an inclined plane of about-30 degrees, so that it shoots the soil mainly in front of the mower. A down-sampling of the selected image size of 109 x 109 pixels indicates that a portion of the soil is approximately isosceles trapezoid in shape with a long base equal to 76cm, a height of 73cm and a short base equal to 24cm, due to the viewing angle. The frame area is about 0.36 square meters and therefore contains a portion of soil sufficient to identify its characteristics.

Method 600 then includes acquiring 602 a plurality of input digital images of soil to be treated in different working phases of the soil.

In an exemplary mower, the mower is moved continuously for 34 days (about 8 hours per day) on different types of turf soil, under different obstacles, under different lighting and weather conditions ((from dawn to dusk, in storm rain, and in clear sky.) specifically, 3 images are acquired per second, for a total of about 300 million images.

Further, a plurality of classifications 603 of the acquired digital images is provided. This classification is performed by associating an appropriate label or code with each digital image acquired. In the particular case of a lawn mower, the images are manually sorted using the sorting of FIG. 3B.

At this point, the method 600 provides initialization 605 of the neural network 300 by randomly or by default associating weights of the neural connections with weights of the neural network.

The following step 606 of training the network 300 occurs by a back propagation method known to those skilled in the art. For the mower, the SGD (random gradient descent) method is used. In particular, at least one stage of the neural network 300 is then trained by modifying weights associated with the network based on the labels of the plurality of classified digital images.

Referring to fig. 7, the training method 700 of the second neural network 400 is described in detail.

In particular, such a method 700 includes an initial step 701 of defining the position and orientation of the digital image acquisition device 203 mounted on the work device.

The method 700 then includes a step 702 of acquiring a plurality of input digital images of the soil to be treated at different work stages.

Further, a step 703 of segmenting the plurality of acquired digital images by associating the same identifier (color) with each portion of the image is included. In the case of a lawn mower, each image used for training is colored with a uniform shade over the different semantic classifications determined. Thus, a table has been defined that is configured to uniquely assign a classification to each of the selected colors.

In this regard, the method 700 provides for initialization 705 of the neural network 400, the initialization 705 being performed by randomly or by default associating weights of neural connections with weights of the neural network.

The following step 706 of training the network 400 occurs by a back propagation method known to those skilled in the art. In particular, the weights associated with the network are then modified based on the identifiers of the plurality of segmented digital images.

Referring to FIG. 8, a training method 800 of the third neural network 500 is described in more detail.

In particular, this method 800 includes an initial step 801 of defining the position and orientation of the digital image acquisition device 203 mounted on the work device. In this case, the working device is moved by the operator.

Method 800 then includes a step 802 of acquiring a plurality of input digital images of the soil to be treated at different stages of operation.

Further, method 800 includes a step 803 of recording specific actions to be taken by the work device based on the characteristics of the soil. In particular, each specific action is associated with an acquired image of the aforementioned plurality of images.

In this regard, the method provides for initialization 805 of the neural network 500 by randomly or by default associating weights of the neural connections with weights of the neural network.

The following step 806 of training the network 500 occurs through a back propagation method. In particular, the weights associated with the network are modified based on specific actions that have been associated with the plurality of acquired digital images.

In a preferred embodiment of the method of the present invention, each of the three examples of training neural networks 300, 400, 500 further comprises the step 604, 704, 804 of increasing the number of images available for training by performing further processing operations on the acquired raw images. This is achieved, for example, by performing a rotation of each image, by selecting a down-sampling of the image or by correcting at least one chrominance channel for each image.

In the exemplary and non-limiting case of a lawn mower, the number of 300 ten thousand initial images that have been acquired has increased to as many as 600 ten thousand images by performing only a horizontal flip of each image. With further expedients such as brightness and contrast correction, the total number of images trained can reach about 2000 ten thousand.

The advantage achieved by adding 604, 704, 804 steps is that the neural network 300, 400, 500 can be trained using a greater number of images and thus the learning of the network itself is improved.

In a particular embodiment of the method of the present invention, the above-mentioned steps 600, 700, 800 of training the network 300, 400, 500 further comprise the step of controlling the consistency of each image of said plurality of digital images with other images of the same classification acquired. Thus, it is avoided that images erroneously acquired by the camera 203, which may be inconsistent with the detected soil condition, are provided to the neural network. This increases the accuracy of the classification.

It should be noted that downstream of the training step, the method 100 of the invention comprises the step of the above-mentioned step 104 further adapted to assign, for each of the three examples of neural networks 300, 400, 500, at least one control signal for generating the moving member 201 or the working member 202 based on the soil classification.

In particular, with reference to fig. 9A, in the case where the synthetic soil descriptor is a soil classification, the step 104 of generating at least one control signal, indicated by reference numeral 900, comprises:

-a step 901 of selecting a predetermined sequence of movement instructions or a predetermined sequence of work instructions, each predetermined sequence of movement instructions or work instructions being associated with a soil classification obtained on the basis of the processing performed by the trained neural network 300;

step 902, sending such a sequence of movement commands to the moving member 201 of the work device, or sending a sequence of work commands to the working member 202 of the work device.

In the first case, such motion instructions are selected from: maintaining the direction; change direction, stop/start moving member 201 or working member 202.

In the exemplary and non-limiting case of a robotic lawnmower, each soil classification of fig. 3B generates a selection 901 of specific movement instructions and work instructions. For example, classification 000 "weedy" would select a "hold direction" action. For example, classification 009 "grass with hose" gives rise to a choice of "direction-keeping" movement action and "stop cutting member" working action: thus, the mower can cover the entire working area without being restricted by the hose, but at the same time interrupt the cutting to avoid damaging the blade. For example, classification 500 "paving" selects a motion action "change direction": thus, the mower remains on the lawn.

Based on such movement instructions, an example of a trajectory T and soil working device of, for example, a lawn mower configured to implement the method 100 of the present invention using a first neural network 300 is shown in fig. 9B, the first neural network 300 being freely movable in the populated area R starting from a travel start ST. Such a residential area R is delimited, for example, by a perimeter wall 10, a living structure 11, an amusement structure 12, bushes 13 and trees 9.

Referring to fig. 9B, the applicant has verified that the robotic lawnmower moved and controlled according to the control method of the present invention moves pseudo-randomly in the work area, remaining within the work area without the aid of external infrastructure. Furthermore, compared to the solutions on the current market, mowers react to obstacles in a more intelligent way. According to a preferred aspect of the invention, some of the instruction sequences may be user defined. For example, referring to fig. 3B, the category 003 "grass with slate" can be arbitrarily defined by the user: it may be a traversable "holding direction" or a non-traversable "changing direction". The modes in which a user can interact with the system 1000 are further defined below.

It should be noted that the role of the user is not to train the system to accommodate the new obstacle classification. The user simply associates each obstacle that has been trained for vehicle recognition with the most convenient control action. In other words, the user does not perform training of the soil working device, but only defines his preferences.

Referring to fig. 10A, where the synthetic soil descriptor is a semantic segmentation of soil, the step 104 of generating at least one control signal is generally indicated by reference numeral 920, the step 104 comprising the steps of:

-selecting 921 at least one pair of semantic classifications at their boundaries, the working devices being expected to remain aligned;

-identifying 922 at least one pair of semantic classifications on the semantically segmented image;

if the classes of a pair are adjacent, a step 923 is performed, in which the parameters characterizing the separation curves or boundaries of the two classes are calculated by a regression method;

-determining 924 a sequence of movement instructions or work instructions based on the calculated parameter, the parameter characterizing a separation curve between the two soil classifications;

sending 925 such movement command sequences to the movement member 201 of the work device or sending work command sequences to the work member 202 of the work device.

In particular, in the second case, the above movement command is selected from the group consisting of:

-a rotation command to correct the orientation of the work device by making it parallel to a line tangent to the two sorting separation curves;

-a translation command to correct the position of the work device by keeping it on the two sorted separation curves;

-instructions for reversing the direction of travel of the device and keeping it within the work area;

-an instruction to avoid an obstacle.

Generally, if the machine is far from the boundary between two soil classifications, the control method implemented by system 1000 sends instructions that bring the machine itself to such boundary (translational motion). Once the boundaries are reached, the control method is configured to align the machine with such boundaries by performing an angular correction to align the machine with such boundaries.

In the exemplary case of a robotic lawnmower, it is desirable to advance it along the previous cutting line, cutting the lawn on a parallel strip. By selecting the semantic classifications "mowed" and "unhatched", the images I1 and I1S of fig. 4B appear to be two adjacent classifications in the semantically segmented image IS1, and are separated by a boundary line L1. The set of points at the boundary between the two classifications can be approximated with a straight line, for example, by linear regression. Such a straight line is characterized by an offset and an angular coefficient. If the mower IS perfectly aligned with the boundary between mowed and uncut, the line will be virtually vertical (zero angle coefficient in the selected coordinate system of image IS 1). Conversely, if the mower IS misaligned relative to the boundary L1, such a straight line will be tilted (positive or negative angle coefficient in the selected coordinate system of image IS 1). The selected movement command causes the mower to rotate proportionally to the factor, keeping it constant along the previous cut line, and causing it to proceed in order. In a particular example, if the coefficient is negative, the mower will rotate clockwise, and vice versa. The offset of the line represents the deviation of the robotic lawnmower from the boundary, according to the selected coordinate system. If the straight line is located primarily in the right half of the field of view, the instructions are selected to cause the machine to translate to the right to align itself with the bezel. If the straight line is located primarily in the left half of the field of view, the instructions are selected to cause the machine to translate to the left to align itself with the bezel.

Based on such movement instructions, applicants have verified that a soil working device, such as a harvester moving in accordance with the control method 100 of the present invention employing neural network 400, follows a trajectory T' as shown in fig. 10B. It should be noted that such a harvester can move freely on the work surface R 'starting from the travel start ST'. Such a work surface R' is delimited, for example, by service lines and various structures 15. The harvester keeps the cutting line which is followed in the previous stroke aligned and parallel during each operation, and field operation is carried out in sequence. Each time the end of the zone is detected, it reverses the direction of travel (180 degrees of rotation) and realigns itself with the boundary between harvested and non-harvested wheat.

Referring to fig. 11A, in the case of a composite soil descriptor, which is a specific measure taken by the working device according to the characteristics of the frame soil, the above-described step 104 generates at least one control signal, generally indicated by reference numeral 930, including a step 931, which step 931 sends a specific action to be taken to the moving member 201 or the working member 202 of the working device, the specific action being obtained based on processing performed by the trained neural network 500 (pixel-to-action type network).

In a first embodiment, such specific action is a classification of a discrete action selected from the group consisting of: increasing speed, decreasing speed, randomly changing direction, stopping motion, starting a working member, stopping a working member. In other words, the action is associated with the determined grass classification that characterizes the soil.

In a second embodiment, such a specific action is a regression with respect to the input image, and thus the start of each output neuron shows the value of the continuous action. For example, the continuous action is: rotating the traveling direction by X degrees; setting the running power to Y; the rotational speed of the working member 202 is brought to a value of Z revolutions per second.

In one embodiment, the above actions are selected from the group consisting of:

-a rotational action directed to the moving member (201) to correct the direction of travel,

an acceleration or deceleration action directed towards the moving member 201;

start/stop working member 202.

Based on such movement instructions, the soil working device (e.g., lawn mower) is configured to implement the method 100 of the present invention using a neural network 500, which neural network 500 follows the example of the trajectory T "shown in fig. 11B, moving freely in the above-described residential zone R starting from the travel start ST". Such a residential area R is similar to the residential area described with reference to fig. 9B. As shown in fig. 9B, the mower first proceeds along the peripheral border line 10 of the populated area R. The mower then aligns itself with the previous cutting line, advancing in a concentric path.

In another embodiment, the method 100 of the present invention further comprises the steps of: the above-mentioned at least one control signal of the moving member 201 or the working member 202 generated based on the synthetic soil descriptor is approved by the operator or by an operator-defined program. In other words, an operator on or remote from the working device may supervise the type of control action sent to the device in order to approve (or reject) its sending to the action member 201 and the working member 202.

It should be noted that the method for generating the synthetic soil descriptor and the control signal generated therewith is implemented entirely on the working device and does not require communication with the outside world. In this case, the communication is only intended to allow the user to approve or disapprove the determined action to be performed on the vehicle.

As set forth above, the method 100 of controlling a soil working device and the associated system 1000 of the present invention provide several advantages and achieve the intended purposes.

In particular, in order to allow correct control and confinement of a lawn mower or more generally an earth working machine within a precise working area, the control method 100 of the invention, implemented by software or hardware in the processing unit 204 of the system, does not require the use of infrastructure external to the machine itself, as in known solutions. Thus, the proposed solution ensures a high degree of portability of the system 1000.

Furthermore, the method 100 of the present invention allows the lawnmower to display the operating scene and move accordingly, and therefore, the rest is limited to the work area, avoids obstacles, and works in sequence.

The method 100 of the invention ensures that the mower, starting from the at least one digital image acquired, identifies the presence of grass, characteristics of different soil types, such as grass height or the presence of items other than grass.

The method of the invention is based on a 'deep learning' method, which is very different from the traditional statistical modeling.

In fact, in conventional statistical modeling, each attribute of an image is first analyzed by a person, who selects the appropriate model and explicitly encodes the logic flow that operates on the image.

For example, according to one method, statistics of an image (or a portion thereof) are compared to statistics of known surfaces using multivariate statistical analysis, with appropriate selection a priori and storage in memory. In this way, a mapping (binary or fuzzy) of the image is generated.

Alternatively, a small neural network may be resorted to identify low-level features (texture and color) of the image.

However, real images of the soil have so many and interrelated features that all cases cannot be manually modeled using traditional statistical models. Furthermore, without an overall interpretation, the local features themselves are typically not indicative: for example, a given texture may adopt completely different semantics depending on the context in which it is inserted. The use of the method and system of the invention allows to overcome the limitations of traditional statistical models, since they allow to abstract from the above-mentioned countless cases to understand the remarkable characteristics and features of the soil.

Furthermore, a trained neural network 300, 400, 500 and its associated preferential training methods 600, 700, 800 are described, which are dedicated to identifying characteristics of different soil types from the perspective of the working machine in different cutting environments, allowing an efficient and compact control method to be implemented, thereby reducing manufacturing costs.

To meet contingent requirements, a person skilled in the art may modify and adapt the embodiments of the method and system of the invention and replace elements with other functionally equivalent ones without thereby departing from the scope of the appended claims. Each feature described as belonging to a possible embodiment may be implemented independently of the other embodiments described.