阅读说明：本技术 设备的工作功率的控制方法、装置及存储介质 (Method and device for controlling working power of equipment and storage medium ) 是由 李江昀 刘博伟 魏冬 皇甫玉彬 于 2021-09-10 设计创作，主要内容包括：本公开涉及一种设备的工作功率的控制方法、装置及存储介质,上述方法包括：获取实时监控图像；将所述实时监控图像输入图像语义分割模型,输出目标对象的占地面积矩阵,其中,所述图像语义分割模型已通过训练,学习并保存有输入的图像与输出的所述占地面积矩阵之间的对应关系；将所述实时监控图像输入深度估计模型,输出所述目标对象的高度矩阵,其中,所述深度估计模型已通过训练,学习并保存有输入的图像与输出的所述高度矩阵之间的对应关系；将所述占地面积矩阵与所述高度矩阵中对应位置的元素的值相乘,并将相乘得到的多个元素乘积相加,得到所述目标对象的体积；根据所述目标对象的体积控制设备的工作功率。(The present disclosure relates to a method, an apparatus and a storage medium for controlling operating power of a device, wherein the method comprises: acquiring a real-time monitoring image; inputting the real-time monitoring image into an image semantic segmentation model, and outputting a floor area matrix of a target object, wherein the image semantic segmentation model is trained, learns and stores a corresponding relation between the input image and the output floor area matrix; inputting the real-time monitoring image into a depth estimation model, and outputting a height matrix of the target object, wherein the depth estimation model is trained, learns and saves the corresponding relation between the input image and the output height matrix; multiplying the occupied area matrix with the values of the elements at the corresponding positions in the height matrix, and adding the products of the multiplied elements to obtain the volume of the target object; and controlling the working power of the equipment according to the volume of the target object.)

1. A method of controlling operating power of a device, comprising:

acquiring a real-time monitoring image;

inputting the real-time monitoring image into an image semantic segmentation model, and outputting a floor area matrix of a target object, wherein the image semantic segmentation model is trained, learns and stores a corresponding relation between the input image and the output floor area matrix;

inputting the real-time monitoring image into a depth estimation model, and outputting a height matrix of the target object, wherein the depth estimation model is trained, learns and saves the corresponding relation between the input image and the output height matrix;

multiplying the occupied area matrix with the values of the elements at the corresponding positions in the height matrix, and adding the products of the multiplied elements to obtain the volume of the target object;

and controlling the working power of the equipment according to the volume of the target object.

2. The method of claim 1, wherein before the inputting the real-time monitoring image into an image semantic segmentation model, the method further comprises:

acquiring a historical monitoring image through first image acquisition equipment, and acquiring a depth image corresponding to the historical monitoring image through second image acquisition equipment;

and performing annotation processing on the historical monitoring image to obtain a semantic label corresponding to the historical monitoring image.

3. The method according to claim 2, wherein after the labeling processing is performed on the historical monitoring image to obtain the semantic tag corresponding to the historical monitoring image, the method further comprises:

training the image semantic segmentation model by using the historical monitoring image and the semantic label;

and taking the depth image as a label of the historical monitoring image, and training the depth estimation model by using the historical monitoring image and the depth image.

4. The method of claim 1, wherein the inputting the real-time monitoring image into an image semantic segmentation model and outputting a footprint matrix of target objects comprises:

inputting the real-time monitoring image into an image semantic segmentation network to obtain a first pixel matrix, wherein the image semantic segmentation model comprises: the image semantic segmentation network and the semantic conversion network;

and inputting the first pixel matrix into the semantic conversion network, and outputting the occupied area matrix.

5. The method of claim 1, wherein inputting the real-time monitoring image into a depth estimation model and outputting a height matrix of the target object comprises:

inputting the real-time monitoring image into a depth estimation network to obtain a second pixel matrix, wherein the depth estimation model comprises: the depth estimation network and the depth conversion network;

and inputting the second pixel matrix into the depth conversion network, and outputting the height matrix.

6. The method of claim 1, wherein the controlling the operating power of the device as a function of the target object volume comprises:

acquiring the density of the target object and the running speed interval of the equipment;

and calculating a mass of the target object from the density and the volume;

and controlling the working power according to the quality and the running speed interval.

7. The method of claim 3, comprising:

preprocessing an image before training the depth estimation model or after acquiring the real-time monitoring image using the historical monitoring image, wherein the image comprises: the historical monitoring image and the real-time monitoring image;

wherein preprocessing the image comprises: carrying out joint bilateral filtering processing on the image; and/or

And carrying out image sharpening processing on the image.

8. The method of claim 3, wherein prior to training the image semantic segmentation model, the method further comprises:

according to an extraction instruction of image edge information, adding one or more layers of attention networks in the image semantic segmentation model and the depth estimation model respectively through an ablation experiment, and adjusting layer sequence numbers of the attention networks in the models; and/or

And respectively reducing one or more layers of neural networks in the image semantic segmentation model and the depth estimation model through ablation experiments according to a real-time requirement instruction of model operation.

9. An apparatus for controlling operating power of a device, comprising:

the acquisition module is used for acquiring a real-time monitoring image;

the real-time monitoring image processing system comprises a first model module, a second model module and a third model module, wherein the first model module is used for inputting the real-time monitoring image into an image semantic segmentation model and outputting a occupation area matrix of a target object, and the image semantic segmentation model is trained and learns and stores the corresponding relation between the input image and the output occupation area matrix;

the second model module is used for inputting the real-time monitoring image into a depth estimation model and outputting a height matrix of the target object, wherein the depth estimation model is trained and learns and saves the corresponding relation between the input image and the output height matrix;

the multiplication module is used for multiplying the occupied area matrix and the values of the elements at the corresponding positions in the height matrix and adding a plurality of element products obtained by multiplication to obtain the volume of the target object;

and the control module is used for controlling the working power of the equipment according to the volume of the target object.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method of any one of claims 1 to 8.

Technical Field

The present disclosure relates to the field of artificial intelligence, and in particular, to a method and an apparatus for controlling operating power of a device, and a storage medium.

Background

In order to achieve an automated control of the transport device, it is an important direction to control the operating power of the transport device in dependence on the weight or volume of the transported goods. Because mass and volume are related, measuring the weight of a shipped item is equivalent to measuring the volume of the shipped item. In this direction, it is first of all to measure the weight or volume of the transported item, and there are two methods for measuring the weight or volume of the transported item in the prior art. One method is to adopt contact type weighing and carry a sensor on the transportation equipment, wherein, the sensor is a platform type special weighing sensor, when coal is transported, the weight of the coal on the transportation belt is applied on the metering roller so as to be transmitted to the weighing sensor to generate a signal which is linearly related to the weight; the other method is to adopt a laser ranging technology, specifically utilize three-dimensional data of transported objects on the transportation equipment acquired by a laser coal-drimeter, and utilize a digital interpolation technology to fit the surface data of the transported objects, so as to estimate the volume of the transported objects. Wherein, the laser coal drimeter is a non-contact weighing device.

In the course of implementing the disclosed concept, the inventors found that there are at least the following technical problems in the related art: when the equipment is controlled according to the volume of the transported object on the transportation equipment, a corresponding sensor or instrument needs to be deployed to calculate the volume of the transported object.

Disclosure of Invention

In order to solve the above technical problem or at least partially solve the above technical problem, embodiments of the present disclosure provide a method, an apparatus, and a storage medium for controlling operating power of a device, so as to solve at least the problem in the prior art that when a device is controlled according to a volume of a transported object on a transport device, a corresponding sensor or instrument needs to be deployed to calculate a volume of the transported object.

The purpose of the present disclosure is realized by the following technical scheme:

in a first aspect, an embodiment of the present disclosure provides a method for controlling operating power of a device, including: acquiring a real-time monitoring image; inputting the real-time monitoring image into an image semantic segmentation model, and outputting a floor area matrix of a target object, wherein the image semantic segmentation model is trained, learns and stores a corresponding relation between the input image and the output floor area matrix; inputting the real-time monitoring image into a depth estimation model, and outputting a height matrix of the target object, wherein the depth estimation model is trained, learns and saves the corresponding relation between the input image and the output height matrix; multiplying the occupied area matrix with the values of the elements at the corresponding positions in the height matrix, and adding the products of the multiplied elements to obtain the volume of the target object; and controlling the working power of the equipment according to the volume of the target object.

In an exemplary embodiment, before the inputting the real-time monitoring image into the image semantic segmentation model, the method further includes: acquiring a historical monitoring image through first image acquisition equipment, and acquiring a depth image corresponding to the historical monitoring image through second image acquisition equipment; and performing annotation processing on the historical monitoring image to obtain a semantic label corresponding to the historical monitoring image.

In an exemplary embodiment, after the labeling processing is performed on the historical monitoring image to obtain the semantic tag corresponding to the historical monitoring image, the method further includes: training the image semantic segmentation model by using the historical monitoring image and the semantic label; and taking the depth image as a label of the historical monitoring image, and training the depth estimation model by using the historical monitoring image and the depth image.

In an exemplary embodiment, the inputting the real-time monitoring image into an image semantic segmentation model and outputting a floor area matrix of a target object includes: inputting the real-time monitoring image into an image semantic segmentation network to obtain a first pixel matrix, wherein the image semantic segmentation model comprises: the image semantic segmentation network and the semantic conversion network; and inputting the first pixel matrix into the semantic conversion network, and outputting the occupied area matrix.

In an exemplary embodiment, the inputting the real-time monitoring image into a depth estimation model and outputting a height matrix of the target object includes: inputting the real-time monitoring image into a depth estimation network to obtain a second pixel matrix, wherein the depth estimation model comprises: the depth estimation network and the depth conversion network; and inputting the second pixel matrix into the depth conversion network, and outputting the height matrix.

In an exemplary embodiment, the controlling the operating power of the device according to the volume of the target object includes: acquiring the density of the target object and the running speed interval of the equipment; and calculating a mass of the target object from the density and the volume; and controlling the working power according to the quality and the running speed interval.

In one exemplary embodiment, includes: preprocessing an image before training the depth estimation model or after acquiring the real-time monitoring image using the historical monitoring image, wherein the image comprises: the historical monitoring image and the real-time monitoring image; wherein preprocessing the image comprises: carrying out joint bilateral filtering processing on the image; and/or performing image sharpening processing on the image.

In an exemplary embodiment, before training the image semantic segmentation model, the method further includes: according to an extraction instruction of image edge information, adding one or more layers of attention networks in the image semantic segmentation model and the depth estimation model respectively through an ablation experiment, and adjusting layer sequence numbers of the attention networks in the models; and/or reducing one or more layers of neural networks in the image semantic segmentation model and the depth estimation model respectively through ablation experiments according to the real-time requirement instruction of model operation.

In a second aspect, an embodiment of the present disclosure provides an apparatus for controlling operating power of a device, including: the acquisition module is used for acquiring a real-time monitoring image; the real-time monitoring image processing system comprises a first model module, a second model module and a third model module, wherein the first model module is used for inputting the real-time monitoring image into an image semantic segmentation model and outputting a occupation area matrix of a target object, and the image semantic segmentation model is trained and learns and stores the corresponding relation between the input image and the output occupation area matrix; the second model module is used for inputting the real-time monitoring image into a depth estimation model and outputting a height matrix of the target object, wherein the depth estimation model is trained and learns and saves the corresponding relation between the input image and the output height matrix; the multiplication module is used for multiplying the occupied area matrix and the values of the elements at the corresponding positions in the height matrix and adding a plurality of element products obtained by multiplication to obtain the volume of the target object; and the control module is used for controlling the working power of the equipment according to the volume of the target object.

In a third aspect, embodiments of the present disclosure provide an electronic device. The electronic equipment comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus; a memory for storing a computer program; a processor for implementing the method for controlling the operating power of the apparatus or the method for image processing as described above when executing the program stored in the memory.

In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium. The above-mentioned computer-readable storage medium has stored thereon a computer program which, when executed by a processor, implements a method of controlling the operating power of the apparatus or a method of image processing as described above.

Compared with the prior art, the technical scheme provided by the embodiment of the disclosure at least has part or all of the following advantages: acquiring a real-time monitoring image; inputting the real-time monitoring image into an image semantic segmentation model, and outputting a floor area matrix of a target object, wherein the image semantic segmentation model is trained, learns and stores a corresponding relation between the input image and the output floor area matrix; inputting the real-time monitoring image into a depth estimation model, and outputting a height matrix of the target object, wherein the depth estimation model is trained, learns and saves the corresponding relation between the input image and the output height matrix; multiplying the occupied area matrix with the values of the elements at the corresponding positions in the height matrix, and adding the products of the multiplied elements to obtain the volume of the target object; and controlling the working power of the equipment according to the volume of the target object. Because the occupied area matrix of the target object can be calculated through the image semantic segmentation model, the height matrix of the target object can be calculated through the depth estimation model, and the volume of the target object can be calculated according to the occupied area matrix and the height matrix, the technical means can solve the problem that in the prior art, when equipment is controlled according to the volume of transported goods on the transport equipment, corresponding sensors or instruments need to be deployed to calculate the volume of the transported goods, so that the cost in equipment control is reduced, and the efficiency of equipment control is improved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the related art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.

Fig. 1 is a block diagram schematically illustrating a hardware configuration of a computer terminal of a method of controlling an operating power of a device according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a method of controlling operating power of a device of an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow chart of model training of an embodiment of the present disclosure;

fig. 4 is a block diagram schematically illustrating the structure of a control device for operating power of an apparatus according to an embodiment of the present disclosure;

fig. 5 schematically shows a block diagram of an electronic device provided in an embodiment of the present disclosure.

Detailed Description

The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments. It should be noted that, in the present disclosure, the embodiments and features of the embodiments may be combined with each other without conflict.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.

The method embodiments provided by the embodiments of the present disclosure may be executed in a computer terminal or a similar computing device. Taking an example of the present invention running on a computer terminal, fig. 1 schematically shows a hardware block diagram of a computer terminal of a method for controlling operating power of a device according to an embodiment of the present disclosure. As shown in fig. 1, a computer terminal may include one or more processors 102 (only one is shown in fig. 1), wherein the processors 102 may include but are not limited to a processing device such as a Microprocessor (MPU) or a Programmable Logic Device (PLD) and a memory 104 for storing data, and optionally, the computer terminal may further include a transmission device 106 for communication function and an input/output device 108, it is understood by those skilled in the art that the structure shown in fig. 1 is merely illustrative and not a limitation to the structure of the computer terminal, for example, the computer terminal may further include more or less components than those shown in fig. 1, or have equivalent functions or different configurations than those shown in fig. 1.

The memory 104 can be used for storing computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the method for controlling the operating power of the device in the embodiment of the present disclosure, and the processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, so as to implement the method described above. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to a computer terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission device 106 is used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the computer terminal. In one example, the transmission device 106 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.

In the embodiment of the present disclosure, a method for controlling operating power of a device is provided, and fig. 2 schematically illustrates a flowchart of a method for controlling operating power of a device according to an embodiment of the present disclosure, and as shown in fig. 2, the flowchart includes the following steps:

step S202, acquiring a real-time monitoring image;

step S204, inputting the real-time monitoring image into an image semantic segmentation model, and outputting a floor area matrix of a target object, wherein the image semantic segmentation model is trained, and learns and stores a corresponding relation between the input image and the output floor area matrix;

step S206, inputting the real-time monitoring image into a depth estimation model, and outputting a height matrix of the target object, wherein the depth estimation model is trained, and learns and saves the corresponding relation between the input image and the output height matrix;

step S208, multiplying the occupied area matrix with the value of the element at the corresponding position in the height matrix, and adding the product of a plurality of elements obtained by multiplication to obtain the volume of the target object;

and step S210, controlling the working power of the equipment according to the volume of the target object.

By the method, the real-time monitoring image is obtained; inputting the real-time monitoring image into an image semantic segmentation model, and outputting a floor area matrix of a target object, wherein the image semantic segmentation model is trained, learns and stores a corresponding relation between the input image and the output floor area matrix; inputting the real-time monitoring image into a depth estimation model, and outputting a height matrix of the target object, wherein the depth estimation model is trained, learns and saves the corresponding relation between the input image and the output height matrix; multiplying the occupied area matrix with the values of the elements at the corresponding positions in the height matrix, and adding the products of the multiplied elements to obtain the volume of the target object; and controlling the working power of the equipment according to the volume of the target object. Because the occupied area matrix of the target object can be calculated through the image semantic segmentation model, the height matrix of the target object can be calculated through the depth estimation model, and the volume of the target object can be calculated according to the occupied area matrix and the height matrix, the technical means can solve the problem that in the prior art, when equipment is controlled according to the volume of transported goods on the transport equipment, corresponding sensors or instruments need to be deployed to calculate the volume of the transported goods, so that the cost in equipment control is reduced, and the efficiency of equipment control is improved.

The method for controlling the working power of the equipment is suitable for all transportation fields, particularly the industrial transportation field, for example, coal is conveyed by a conveyor belt in the coal mine industry, a target object is the coal, a floor area matrix of the coal on the conveyor belt is calculated through an image semantic segmentation model, a height matrix of the coal on the conveyor belt is calculated through a depth estimation model, the volume of the coal on the conveyor belt is calculated according to the floor area matrix and the height matrix, the working power of the conveyor belt is controlled according to the volume of the coal on the conveyor belt, and the equipment is the conveyor belt or conveyor belt equipment.

Before step S204, that is, before the real-time monitoring image is input into the image semantic segmentation model, the method further includes: acquiring a historical monitoring image through first image acquisition equipment, and acquiring a depth image corresponding to the historical monitoring image through second image acquisition equipment; and performing annotation processing on the historical monitoring image to obtain a semantic label corresponding to the historical monitoring image.

The first image acquisition device may be a monocular camera, a binocular camera, or the like, and the second image acquisition device may be a depth camera. Depth cameras are a new technology which has been developed in recent years, and compared with conventional cameras, a depth measurement is functionally added to the depth cameras, so that the surrounding environment and changes can be sensed more conveniently and accurately. The history monitoring image is labeled, and the area where the target object exists may be labeled as 1, and the area where the target object does not exist may be labeled as 0 in a preset area corresponding to the history monitoring image. The semantic label corresponding to the historical monitoring image is 1 or 0.

After the historical monitoring image is subjected to labeling processing to obtain a semantic label corresponding to the historical monitoring image, the method further comprises the following steps: training the image semantic segmentation model by using the historical monitoring image and the semantic label; and taking the depth image as a label of the historical monitoring image, and training the depth estimation model by using the historical monitoring image and the depth image.

It should be noted that the historical monitor images for training the image semantic segmentation model and the depth estimation model may be different images in different time periods. And dividing the historical monitoring image and the semantic label into a first training set and a first verification set according to a preset proportion, and training the image semantic segmentation model according to the first training set and the first verification set. Taking a conveyor belt in the coal mine industry as an example, after the image semantic segmentation model is trained, a region with coal on the conveyor belt can be identified, namely, a floor area matrix of the coal on the conveyor belt is identified, wherein the floor area matrix is a matrix of the area of the conveyor belt occupied by the coal. Specifically, the matrix position element corresponding to the area on the belt where coal is present is 1, and the matrix position element corresponding to the area on the belt where coal is absent is 0. And dividing the historical monitoring image and the depth image into a second training set and a second verification set according to a preset proportion, and training the depth estimation model according to the second training set and the second verification set. Taking a belt conveyor in the coal industry as an example, after the depth estimation model is trained, the height of the coal on the belt conveyor can be identified, that is, a height matrix of the coal on the belt conveyor is identified, wherein the height matrix is a matrix related to the height of the coal. In particular, the values of the elements of the matrix represent the height of the coal at the corresponding location on the conveyor belt. It should be noted that the distance from the image acquisition device to the coal surface is directly identified by the depth estimation model, because the distance from the image acquisition device to the conveyor belt is known, and the height of the coal on the conveyor belt is obtained by subtracting the distance from the image acquisition device to the coal surface from the distance from the device to the conveyor belt (the image acquisition device is arranged right above the conveyor belt, and the image acquisition device comprises the first image acquisition device and the second image acquisition device), and it can be understood that the height matrix of the coal on the conveyor belt is identified by the depth estimation model.

In step S206, the real-time monitoring image is input into an image semantic segmentation model, and a floor area matrix of a target object is output, including: inputting the real-time monitoring image into an image semantic segmentation network to obtain a first pixel matrix, wherein the image semantic segmentation model comprises: the image semantic segmentation network and the semantic conversion network; and inputting the first pixel matrix into the semantic conversion network, and outputting the occupied area matrix.

Taking a conveyor belt in the coal industry as an example for conveying coal, the first pixel matrix can be understood as that on the real-time monitoring image, the element of the position of the matrix corresponding to the area with coal on the conveyor belt is 1, and the element of the position of the matrix corresponding to the area without coal on the conveyor belt is 0. The semantic conversion network is a mapping layer, and can convert the actual area represented by an element in the first pixel matrix, and the effect of the semantic conversion network can be understood as multiplying a coefficient by the first pixel matrix to obtain the occupied area matrix, that is, the actual area of the device where the target object is located. The image semantic segmentation model in the embodiment of the disclosure is equivalent to adding a semantic conversion network to a common image semantic segmentation network.

In step S206, inputting the real-time monitoring image into a depth estimation model, and outputting a height matrix of the target object, including: inputting the real-time monitoring image into a depth estimation network to obtain a second pixel matrix, wherein the depth estimation model comprises: the depth estimation network and the depth conversion network; and inputting the second pixel matrix into the depth conversion network, and outputting the height matrix.

The depth estimation network is used for calculating the height of a target object on the real-time monitoring image, namely the second pixel matrix. Taking the example of a belt conveyor in the coal mining industry conveying coal, the second pixel matrix is a height matrix of the coal on the belt conveyor, wherein the values of the elements of the height matrix represent the height of the coal. The degree conversion network is a mapping layer that can convert the height of the coal calculated by the depth estimation model into the height of the coal actually conveyed by the conveyor belt. Similar to a map, the first pixel matrix and the second pixel matrix may be similarly understood as distances on the map, and the floor area matrix and the height matrix are actual distances.

In step S210, controlling the working power of the device according to the volume of the target object includes: acquiring the density of the target object and the running speed interval of the equipment; and calculating a mass of the target object from the density and the volume; and controlling the working power according to the quality and the running speed interval.

Taking a conveyor belt in the coal mine industry as an example, the density of raw coal is obtained. And calculating the mass of the coal according to the density and the volume of the raw coal, acquiring the running speed interval of the conveyor belt or the conveyor belt equipment, and calculating the working power of the conveyor belt equipment according to the mass of the coal and the running speed interval of the conveyor belt equipment so as to control the equipment.

Optionally, the images are preprocessed before the historical monitoring images are used for training the depth estimation model or after the real-time monitoring images are acquired, wherein the images include: the historical monitoring image and the real-time monitoring image; wherein preprocessing the image comprises: carrying out joint bilateral filtering processing on the image; and/or performing image sharpening processing on the image.

Taking a conveyor belt in the coal mine industry as an example, the environment under a coal mine is dark in light and has more dust particles compared with the outdoor environment. The method has the advantages that the images obtained under the coal mine have the characteristics of dark light and much interference of dust particles, and in order to improve the efficiency of training the models and recognizing the images by the models, a preprocessing method is added. The image is subjected to joint bilateral filtering processing to filter out interference components in the image, and the image sharpening processing can be performed to thicken or deepen edge contours in the image.

Before the training of the image semantic segmentation model, the method further comprises: according to an extraction instruction of image edge information, adding one or more layers of attention networks in the image semantic segmentation model and the depth estimation model respectively through an ablation experiment, and adjusting layer sequence numbers of the attention networks in the models; and/or reducing one or more layers of neural networks in the image semantic segmentation model and the depth estimation model respectively through ablation experiments according to the real-time requirement instruction of model operation.

The ablation experiment is to remove one layer of network in the model, and the effect of the removed network in the whole model is measured according to the effect of the rest network. In some special scenes, image edge information is very important, for example, a conveyor belt in the coal mine industry is used for conveying coal and calculating the volume of the coal, in the coal mine, the edge information in the acquired real-time monitoring image and the occupied area matrix are related to the height matrix, the image edge information is very important in calculating the volume of the coal, and one or more layers of attention networks are respectively added to the image semantic segmentation model and the depth estimation model. It can be determined through ablation experiments how many layers of the attention network are added, and how well the position of the attention network in the whole model is adjusted. In some special scenes, the requirement on the accuracy of image identification is low, the requirement on the real-time performance is high, the embodiment of the disclosure can determine the requirement on the real-time performance of image identification in the current scene according to a real-time performance requirement instruction, and further one or more layers of neural networks are respectively reduced in the image semantic segmentation model and the depth estimation model through ablation experiments, so that the operation speed of the model is increased.

In order to better understand the technical solutions, the embodiments of the present disclosure also provide an alternative embodiment for explaining the technical solutions.

Fig. 3 schematically illustrates a flowchart of model training according to an embodiment of the present disclosure, as shown in fig. 3:

s302: acquiring a historical monitoring image through first image acquisition equipment, and acquiring a depth image corresponding to the historical monitoring image through second image acquisition equipment;

s304: performing labeling processing on the historical monitoring image to obtain a semantic label corresponding to the historical monitoring image;

s306: performing data enhancement processing on the historical monitoring image and the semantic label;

s308: generating a first training set and a first verification set by using the historical monitoring image and the semantic label, and generating a second training set and a second verification set by using the historical monitoring image and the depth image;

s310: and training the image semantic segmentation model by using a first training set and a first verification set, and training the depth estimation model by using a second training set and a second verification set.

By the method, the real-time monitoring image is obtained; inputting the real-time monitoring image into an image semantic segmentation model, and outputting a floor area matrix of a target object, wherein the image semantic segmentation model is trained, learns and stores a corresponding relation between the input image and the output floor area matrix; inputting the real-time monitoring image into a depth estimation model, and outputting a height matrix of the target object, wherein the depth estimation model is trained, learns and saves the corresponding relation between the input image and the output height matrix; multiplying the occupied area matrix with the values of the elements at the corresponding positions in the height matrix, and adding the products of the multiplied elements to obtain the volume of the target object; and controlling the working power of the equipment according to the volume of the target object. Because the occupied area matrix of the target object can be calculated through the image semantic segmentation model, the height matrix of the target object can be calculated through the depth estimation model, and the volume of the target object can be calculated according to the occupied area matrix and the height matrix, the technical means can solve the problem that in the prior art, when equipment is controlled according to the volume of transported goods on the transport equipment, corresponding sensors or instruments need to be deployed to calculate the volume of the transported goods, so that the cost in equipment control is reduced, and the efficiency of equipment control is improved.

Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present disclosure or portions thereof that contribute to the prior art may be embodied in the form of a software product, where the computer software product is stored in a storage medium (such as a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, and an optical disk), and includes several instructions for enabling a terminal device (which may be a mobile phone, a computer, a component server, or a network device) to execute the methods of the embodiments of the present disclosure.

In this embodiment, a device for controlling operating power of an apparatus is further provided, where the device for controlling operating power of an apparatus is used to implement the foregoing embodiments and preferred embodiments, and details are not repeated after the description is given. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.

Fig. 4 is a block diagram schematically illustrating a structure of a device for controlling operating power of an apparatus according to an alternative embodiment of the present disclosure, and as shown in fig. 4, the device includes:

an obtaining module 402, configured to obtain a real-time monitoring image;

a first model module 404, configured to input the real-time monitoring image into an image semantic segmentation model, and output an occupied area matrix of a target object, where the image semantic segmentation model has been trained, learns, and stores a corresponding relationship between the input image and the output occupied area matrix;

a second model module 406, configured to input the real-time monitoring image into a depth estimation model, and output a height matrix of the target object, where the depth estimation model has been trained, and learns and stores a corresponding relationship between the input image and the output height matrix;

a multiplication module 408, configured to multiply the occupied area matrix with values of elements at corresponding positions in the height matrix, and add multiple multiplied element products to obtain a volume of the target object;

a control module 410 for controlling the operating power of the device in accordance with the volume of the target object.

By the method, the real-time monitoring image is obtained; inputting the real-time monitoring image into an image semantic segmentation model, and outputting a floor area matrix of a target object, wherein the image semantic segmentation model is trained, learns and stores a corresponding relation between the input image and the output floor area matrix; inputting the real-time monitoring image into a depth estimation model, and outputting a height matrix of the target object, wherein the depth estimation model is trained, learns and saves the corresponding relation between the input image and the output height matrix; multiplying the occupied area matrix with the values of the elements at the corresponding positions in the height matrix, and adding the products of the multiplied elements to obtain the volume of the target object; and controlling the working power of the equipment according to the volume of the target object. Because the occupied area matrix of the target object can be calculated through the image semantic segmentation model, the height matrix of the target object can be calculated through the depth estimation model, and the volume of the target object can be calculated according to the occupied area matrix and the height matrix, the technical means can solve the problem that in the prior art, when equipment is controlled according to the volume of transported goods on the transport equipment, corresponding sensors or instruments need to be deployed to calculate the volume of the transported goods, so that the cost in equipment control is reduced, and the efficiency of equipment control is improved.

The method for controlling the working power of the equipment is suitable for all transportation fields, particularly the industrial transportation field, for example, coal is conveyed by a conveyor belt in the coal mine industry, a target object is the coal, a floor area matrix of the coal on the conveyor belt is calculated through an image semantic segmentation model, a height matrix of the coal on the conveyor belt is calculated through a depth estimation model, the volume of the coal on the conveyor belt is calculated according to the floor area matrix and the height matrix, the working power of the conveyor belt is controlled according to the volume of the coal on the conveyor belt, and the equipment is the conveyor belt or conveyor belt equipment.

Optionally, the first model module 404 is further configured to obtain a historical monitoring image through a first image obtaining device, and obtain a depth image corresponding to the historical monitoring image through a second image obtaining device; and performing annotation processing on the historical monitoring image to obtain a semantic label corresponding to the historical monitoring image.

The first image acquisition device may be a monocular camera, a binocular camera, or the like, and the second image acquisition device may be a depth camera. Depth cameras are a new technology which has been developed in recent years, and compared with conventional cameras, a depth measurement is functionally added to the depth cameras, so that the surrounding environment and changes can be sensed more conveniently and accurately. The history monitoring image is labeled, and the area where the target object exists may be labeled as 1, and the area where the target object does not exist may be labeled as 0 in a preset area corresponding to the history monitoring image. The semantic label corresponding to the historical monitoring image is 1 or 0.

Optionally, the first model module 404 is further configured to train the image semantic segmentation model using the historical monitoring image and the semantic label; and taking the depth image as a label of the historical monitoring image, and training the depth estimation model by using the historical monitoring image and the depth image.

And dividing the historical monitoring image and the semantic label into a first training set and a first verification set according to a preset proportion, and training the image semantic segmentation model according to the first training set and the first verification set. Taking a conveyor belt in the coal mine industry as an example, after the image semantic segmentation model is trained, a region with coal on the conveyor belt can be identified, namely, a floor area matrix of the coal on the conveyor belt is identified, wherein the floor area matrix is a matrix of the area of the conveyor belt occupied by the coal. Specifically, the matrix position element corresponding to the area on the belt where coal is present is 1, and the matrix position element corresponding to the area on the belt where coal is absent is 0. And dividing the historical monitoring image and the depth image into a second training set and a second verification set according to a preset proportion, and training the depth estimation model according to the second training set and the second verification set. Taking a belt conveyor in the coal industry as an example, after the depth estimation model is trained, the height of the coal on the belt conveyor can be identified, that is, a height matrix of the coal on the belt conveyor is identified, wherein the height matrix is a matrix related to the height of the coal. In particular, the values of the elements of the matrix represent the height of the coal at the corresponding location on the conveyor belt.

Optionally, the second model module 406 is further configured to input the real-time monitoring image into an image semantic segmentation network to obtain a first pixel matrix, where the image semantic segmentation model includes: the image semantic segmentation network and the semantic conversion network; and inputting the first pixel matrix into the semantic conversion network, and outputting the occupied area matrix.

Taking a conveyor belt in the coal industry as an example for conveying coal, the first pixel matrix can be understood as that on the real-time monitoring image, the element of the position of the matrix corresponding to the area with coal on the conveyor belt is 1, and the element of the position of the matrix corresponding to the area without coal on the conveyor belt is 0. The semantic conversion network is a mapping layer, and can convert the actual area represented by an element in the first pixel matrix, and the effect of the semantic conversion network can be understood as multiplying a coefficient by the first pixel matrix to obtain the occupied area matrix, that is, the actual area of the device where the target object is located. The image semantic segmentation model in the embodiment of the disclosure is equivalent to adding a semantic conversion network to a common image semantic segmentation network.

Optionally, the second model module 406 is further configured to input the real-time monitoring image into a depth estimation network to obtain a second pixel matrix, where the depth estimation model includes: the depth estimation network and the depth conversion network; and inputting the second pixel matrix into the depth conversion network, and outputting the height matrix.

The depth estimation network is used for calculating the height of a target object on the real-time monitoring image, namely the second pixel matrix. Taking the example of a belt conveyor in the coal mining industry conveying coal, the second pixel matrix is a height matrix of the coal on the belt conveyor, wherein the values of the elements of the height matrix represent the height of the coal. The degree conversion network is a mapping layer that can convert the height of the coal calculated by the depth estimation model into the height of the coal actually conveyed by the conveyor belt. Similar to a map, the first pixel matrix and the second pixel matrix may be similarly understood as distances on the map, and the floor area matrix and the height matrix are actual distances.

Optionally, the control module 410 is further configured to obtain a density of the target object and an operation speed interval of the apparatus; and calculating a mass of the target object from the density and the volume; and controlling the working power according to the quality and the running speed interval.

Taking a conveyor belt in the coal mine industry as an example, the density of raw coal is obtained. And calculating the mass of the coal according to the density and the volume of the raw coal, acquiring the running speed interval of the conveyor belt or the conveyor belt equipment, and calculating the working power of the conveyor belt equipment according to the mass of the coal and the running speed interval of the conveyor belt equipment so as to control the equipment.

Optionally, the first model module 404 is further configured to pre-process the image before training the depth estimation model using the historical monitored image or after acquiring the real-time monitored image, wherein the image includes: the historical monitoring image and the real-time monitoring image; wherein preprocessing the image comprises: carrying out joint bilateral filtering processing on the image; and/or performing image sharpening processing on the image.

Taking a conveyor belt in the coal mine industry as an example, the environment under a coal mine is dark in light and has more dust particles compared with the outdoor environment. The method has the advantages that the images obtained under the coal mine have the characteristics of dark light and much interference of dust particles, and in order to improve the efficiency of training the models and recognizing the images by the models, a preprocessing method is added. The image is subjected to joint bilateral filtering processing to filter out interference components in the image, and the image sharpening processing can be performed to thicken or deepen edge contours in the image.

Optionally, the first model module 404 is further configured to add one or more layers of attention networks in the image semantic segmentation model and the depth estimation model respectively through an ablation experiment according to the extraction instruction of the image edge information, and adjust layer numbers of the attention networks in the models; and/or reducing one or more layers of neural networks in the image semantic segmentation model and the depth estimation model respectively through ablation experiments according to the real-time requirement instruction of model operation.

The ablation experiment is to remove one layer of network in the model, and the effect of the removed network in the whole model is measured according to the effect of the rest network. In some special scenes, image edge information is very important, for example, a conveyor belt in the coal mine industry is used for conveying coal and calculating the volume of the coal, in the coal mine, the edge information in the acquired real-time monitoring image and the occupied area matrix are related to the height matrix, the image edge information is very important in calculating the volume of the coal, and one or more layers of attention networks are respectively added to the image semantic segmentation model and the depth estimation model. It can be determined through ablation experiments how many layers of the attention network are added, and how well the position of the attention network in the whole model is adjusted. In some special scenes, the requirement on the accuracy of image identification is low, the requirement on the real-time performance is high, the embodiment of the disclosure can determine the requirement on the real-time performance of image identification in the current scene according to a real-time performance requirement instruction, and further one or more layers of neural networks are respectively reduced in the image semantic segmentation model and the depth estimation model through ablation experiments, so that the operation speed of the model is increased.

It should be noted that, the above modules may be implemented by software or hardware, and for the latter, the following may be implemented, but not limited to: the modules are all positioned in the same processor; alternatively, the modules are respectively located in different processors in any combination.

Embodiments of the present disclosure provide an electronic device.

Fig. 5 schematically shows a block diagram of an electronic device provided in an embodiment of the present disclosure.

Referring to fig. 5, an electronic device 500 provided in the embodiment of the present disclosure includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, where the processor 501, the communication interface 502 and the memory 503 complete communication with each other through the communication bus 504; a memory 503 for storing a computer program; the processor 501 is configured to implement the steps in any of the above method embodiments when executing the program stored in the memory.

Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the input/output device is connected to the processor.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:

s1, acquiring a real-time monitoring image;

s2, inputting the real-time monitoring image into an image semantic segmentation model, and outputting a occupation area matrix of a target object, wherein the image semantic segmentation model is trained, and learns and saves the corresponding relation between the input image and the output occupation area matrix;

s3, inputting the real-time monitoring image into a depth estimation model and outputting a height matrix of the target object, wherein the depth estimation model is trained, and learns and saves the corresponding relation between the input image and the output height matrix;

s4, multiplying the occupied area matrix and the value of the element at the corresponding position in the height matrix, and adding the product of multiple elements obtained by multiplication to obtain the volume of the target object;

s5, controlling the working power of the equipment according to the volume of the target object.

Embodiments of the present disclosure also provide a computer-readable storage medium. The computer-readable storage medium has stored thereon a computer program which, when executed by a processor, performs the steps of any of the method embodiments described above.

Alternatively, in the present embodiment, the storage medium may be configured to store a computer program for executing the steps of:

s1, acquiring a real-time monitoring image;

s2, inputting the real-time monitoring image into an image semantic segmentation model, and outputting a occupation area matrix of a target object, wherein the image semantic segmentation model is trained, and learns and saves the corresponding relation between the input image and the output occupation area matrix;

s3, inputting the real-time monitoring image into a depth estimation model and outputting a height matrix of the target object, wherein the depth estimation model is trained, and learns and saves the corresponding relation between the input image and the output height matrix;

s4, multiplying the occupied area matrix and the value of the element at the corresponding position in the height matrix, and adding the product of multiple elements obtained by multiplication to obtain the volume of the target object;

s5, controlling the working power of the equipment according to the volume of the target object.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and this embodiment is not described herein again.

It will be apparent to those skilled in the art that the modules or steps of the present disclosure described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. As such, the present disclosure is not limited to any specific combination of hardware and software.

The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure, and various modifications and changes may be made to the present disclosure by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the principle of the present disclosure should be included in the protection scope of the present disclosure.