阅读说明：本技术 图像处理装置、自动分析系统以及图像处理方法 (Image processing apparatus, automatic analysis system, and image processing method ) 是由 柿下容弓 服部英春 坂诘卓 铃木洋一郎 于 2020-01-28 设计创作，主要内容包括：为了提供能够根据由一个照相机取得的图像来判定容器或采集对象的状态的图像处理装置、自动分析系统以及图像处理方法,而具备：图像取得部,其取得上方图像,所述上方图像是从上方拍摄收纳采集对象的容器而得到的图像,所述采集对象包括试样、试剂、反应溶液；区域计算部,其根据所述上方图像计算所述容器的边缘区域或所述采集对象的上表面区域；以及状态判定部,其基于所述边缘区域或所述上表面区域来判定所述容器或所述采集对象的状态。(In order to provide an image processing device, an automatic analysis system, and an image processing method capable of determining the state of a container or a collection object from an image acquired by one camera, the image processing device is provided with: an image acquisition unit that acquires an upper image obtained by imaging a container containing a collection object including a sample, a reagent, and a reaction solution from above; an area calculation unit that calculates an edge area of the container or an upper surface area of the collection object from the upper image; and a state determination section that determines a state of the container or the collection object based on the edge region or the upper surface region.)

1. An image processing apparatus is characterized by comprising:

an image acquisition unit that acquires an upper image obtained by imaging a container containing a collection object including a sample, a reagent, and a reaction solution from above;

an area calculation unit that calculates an edge area of the container or an upper surface area of the collection object from the upper image; and

a state determination section that determines a state of the container or the collection object based on the edge region or the upper surface region.

2. The image processing apparatus according to claim 1,

the region calculating section calculates the edge region,

the state determination unit determines the type of the container based on the outer diameter and the inner diameter of the edge region.

3. The image processing apparatus according to claim 2,

the area calculating section further calculates the upper surface area,

the state determination section determines the amount of the collection object based on the kind of the container and the size of the upper surface area or the distance from the upper surface area to the edge area.

4. The image processing apparatus according to claim 1,

the region calculating section calculates the edge region and the upper surface region,

the state determination unit determines whether the container is tilted or not based on a degree of coincidence between the center of the edge region and the center of the upper surface region.

5. The image processing apparatus according to claim 3,

the state determination unit determines the inclination angle of the container based on the height of the collection object obtained using the type of the container and the size of the upper surface area and a center distance that is a distance between the center of the edge area and the center of the upper surface area.

6. The image processing apparatus according to claim 5,

the state determination unit corrects the center distance based on a ratio of a minor diameter of the upper surface region to an inner diameter of the edge region when the upper image is captured using a non-telecentric lens.

7. The image processing apparatus according to claim 1,

the image processing apparatus further includes:

a display unit that displays the selected image; and

and a training signal acquisition unit that displays an operation screen for inputting a training signal for the image displayed on the display unit.

8. The image processing apparatus according to claim 7,

the display unit further displays a screen indicating the number of selected images.

9. The image processing apparatus according to claim 1,

the region calculation unit performs convolution processing and pooling processing using an extended kernel created by inserting a predetermined number of zeros between elements of the kernel, thereby dividing the top image into a plurality of regions and calculating the edge region or the top surface region.

10. The image processing apparatus according to claim 9,

the region calculation unit determines the number of zeros to be inserted into the kernel, based on a stride, which is an amount of shift of the expanded kernels when the convolution processing and the pooling processing are executed.

11. An automatic analysis system comprising the image processing apparatus according to claim 1,

the automatic analysis system is provided with:

an acquisition unit that acquires the acquisition target from the container; and

and a control unit that controls the collection unit based on the determination result of the state determination unit.

12. The automated analysis system of claim 11,

the control unit stops the collecting unit when the state determining unit determines that the type of the container is unknown, or determines that the amount of the collecting object is smaller than a threshold, or determines that the inclination angle of the container is equal to or larger than a threshold.

13. The automated analysis system of claim 11,

the control unit adjusts the height and angle of the collecting unit based on the type of the container, the amount of the collecting object, and the inclination angle of the container determined by the state determining unit.

14. An image processing method is characterized by comprising the following steps:

an image acquisition step of acquiring an upper image obtained by imaging a container accommodating a collection target including a sample, a reagent, and a reaction solution from above;

a region calculation step of calculating an edge region of the container or an upper surface region of the acquisition object from the upper image; and

a state determination step of determining a state of the container or the collection object based on the edge region or the upper surface region.

15. An image processing method for dividing an image into a plurality of regions,

the method comprises the following steps:

creating an expanded kernel by inserting a predetermined number of zeros between elements of the kernel used in the convolution processing and pooling processing; and

performing convolution processing and pooling processing using the expanded kernel.

Technical Field

The present invention relates to a technique for determining a state of a container for storing a collection target such as a sample, a reagent, or a reaction solution or a state of the collection target in an automatic analysis system including an immunoassay device or the like.

Background

In an automatic analysis system including an immunoassay device or the like, the state of color development or light emission generated from a reaction solution that reacts a sample with a reagent is measured in order to analyze components and the like of the sample such as blood and urine. Samples, reagents, reaction solutions, and the like for analysis are stored in containers such as test tubes, and collected from the containers by a collecting unit such as a dispensing probe. In a container for storing a sample, a reagent, a reaction solution, or the like to be collected by a collecting unit, there are a plurality of types having different inner diameters and lengths, and it is desirable to automatically determine the type of the container in a state where a plurality of containers are mixed.

Patent document 1 discloses the following: the type of the container is determined by comparing a pattern obtained by imaging the container from both the opening direction and the side direction with a standard pattern stored in a memory.

Documents of the prior art

Patent document

Patent document 1: japanese laid-open patent publication No. 2004-151025

Disclosure of Invention

Problems to be solved by the invention

However, in patent document 1, in order to determine the type of the container, a plurality of cameras for imaging the container from both the opening direction and the side direction are required, and the hardware cost is high. In addition, patent document 1 only determines the type of container, and does not consider the determination of the state of the container or the collection target, such as the inclination of the container or the amount of the collection target.

Therefore, an object of the present invention is to provide an image processing apparatus, an automatic analysis system, and an image processing method that can determine the state of a container or a collection target from an image acquired by one camera.

Means for solving the problems

In order to achieve the above object, the present invention provides an image processing apparatus including: an image acquisition unit that acquires an upper image obtained by imaging a container containing a collection object including a sample, a reagent, and a reaction solution from above; an area calculation unit that calculates an edge area of the container or an upper surface area of the collection object from the upper image; and a state determination section that determines a state of the container or the collection object based on the edge region or the upper surface region.

Further, the present invention is an automatic analysis system including the image processing apparatus, the automatic analysis system including: an acquisition unit that acquires an acquisition object from the container; and a control unit that controls the collection unit based on the determination result of the state determination unit.

Further, the present invention is an image processing method including the steps of: an image acquisition step of acquiring an upper image obtained by imaging a container accommodating a collection target including a sample, a reagent, and a reaction solution from above; a region calculation step of calculating an edge region of the container or an upper surface region of the acquisition object from the upper image; and a state determination step of determining a state of the container or the acquisition object based on the edge region or the upper surface region.

Further, the present invention is an image processing method for dividing an image into a plurality of regions, the method including the steps of: inserting a predetermined number of zeros between elements of a kernel (kernel) used in convolution processing and pooling processing to create an expanded kernel; and performing convolution processing and pooling processing using the expanded kernel.

Effects of the invention

According to the present invention, it is possible to provide an image processing apparatus, an automatic analysis system, and an image processing method that can determine the state of a container or a collection target from an image acquired by one camera.

Drawings

Fig. 1 is a diagram showing an example of a hardware configuration of an automatic analysis system.

Fig. 2 is an example of a functional block diagram of embodiment 1.

FIG. 3 is a diagram showing an example of the processing flow of example 1.

Fig. 4 is a diagram showing an example of the result of the area division processing.

Fig. 5 is a diagram illustrating calculation of the edge area of the container.

Fig. 6A is a diagram showing an example of a table for determining the type of container.

Fig. 6B is a diagram showing an example of a table for determining the length of the container and the like according to the type of the container.

Fig. 7 is a diagram illustrating a concept of convolution (convolution) processing.

FIG. 8 is a view showing the concept of pooling (pooling).

Fig. 9 is a diagram showing an example of an area division process by FCN (full volumetric network).

Fig. 10 is a diagram showing an example of the high resolution area division processing by the FCN.

Fig. 11 is a diagram showing an example of a kernel in Convolution processing and expanded Convolution (scaled Convolution) processing.

Fig. 12 is a diagram showing an example of a kernel in Pooling and expanded Pooling (scaled Pooling).

Fig. 13 is a diagram showing the concept of the expansion process.

Fig. 14 is a diagram showing an example of a region segmentation procedure using an extended kernel.

Fig. 15 is a diagram illustrating region segmentation using the expanded core.

FIG. 16 is a diagram showing an example of the processing flow of example 2.

Fig. 17 is a diagram illustrating calculation of the upper surface area of the acquisition target.

Fig. 18A is a diagram showing an example of a non-telecentric upper image.

Fig. 18B is a diagram showing an example of a telecentric upper image.

Fig. 18C is a diagram showing an example of a telecentric oblique image.

Fig. 19 is a diagram showing an example of a table for obtaining the distance from the edge of the container to the upper surface of the collection target.

FIG. 20 is a diagram showing an example of the processing flow of example 3.

Fig. 21A is a diagram illustrating tilt determination based on a non-telecentric upper image.

Fig. 21B is a diagram illustrating tilt determination based on the telecentric upper image.

Fig. 22A is a diagram illustrating calculation of a tilt angle based on a non-telecentric upper image.

Fig. 22B is a diagram illustrating calculation of the tilt angle based on the telecentric overhead image.

Fig. 23 is an example of a functional block diagram of embodiment 4.

FIG. 24 is a diagram showing an example of a process flow of example 4.

Fig. 25 is a diagram showing an example of a screen displaying a list of images to which training signals are given.

Fig. 26 is a diagram showing an example of an operation screen for specifying an area to which a training signal is applied.

Fig. 27 is a diagram showing an example of an operation screen for applying a training signal.

Detailed Description

Hereinafter, preferred embodiments of an image processing apparatus, an automatic analysis system, and an image processing method according to the present invention will be described with reference to the drawings. In the following description and the drawings, the same reference numerals are given to components having the same functions and configurations, and redundant description is omitted.

Example 1

An example of the hardware configuration of the automatic analysis system 100 according to the present embodiment will be described with reference to fig. 1. The automatic analysis system 100 includes an image processing device 101, an imaging unit 111, a control unit 121, an acquisition unit 122, and an analysis unit 123. The image processing apparatus 101 is a so-called computer, and specifically includes an input unit 102, an arithmetic unit 103, a memory 104, and an output unit 105, and is connected to an imaging unit 111 and a control unit 121. The image processing apparatus 101 may be provided in a system other than the automatic analysis system 100. Hereinafter, each part will be explained.

The imaging unit 111 is a device that images the container 112 and the collection object 113 stored in the container 112, and is, for example, a camera. The container 112 is a test tube or the like, and has a bottom surface at one end of a cylindrical shape or a tapered shape and an opening at the other end. The container 112 has a plurality of types having different inner diameters and lengths. The collection target 113 is a liquid or powder, such as a sample such as blood or urine, a reagent that reacts with the sample, or a reaction solution that reacts the sample with the reagent. The imaging unit 111 is provided on the opening side of the container 112, i.e., above the container 112, and images the container 112 and the collection target 113 from above the container 112. An image captured from above the container 112 by the imaging unit 111 is referred to as an upper image. A telecentric or non-telecentric lens is mounted on the image pickup unit 111.

The input unit 102 is an interface for receiving data of the upper image from the imaging unit 111 or a recording device in which the upper image is recorded, or receiving an operation signal generated by an operator operating a keyboard, a mouse, or the like. The upper image may be a still image such as BMP, PNG, or JPEG, or a frame image extracted at a predetermined interval from a moving image such as MPEG or h.264.

The arithmetic Unit 103 is a device that executes various processes in the image Processing apparatus 101, and is, for example, a CPU (Central Processing Unit), an FPGA (Field-Programmable Gate Array), or the like. The functions executed by the arithmetic unit 103 will be described later with reference to fig. 2.

The memory 104 is a device for storing programs, parameters, coefficients, processing results, and the like executed by the arithmetic unit 103, and is an HDD, RAM, ROM, flash memory, or the like.

The output unit 105 is an interface for transmitting the processing result of the arithmetic unit 103 to the outside, and outputs the processing result to, for example, a display device such as a monitor or a printer, a recording device for recording the processing result, and a control device for controlling other devices. Fig. 1 illustrates a case where the processing result is output to the control unit 121, which is an example of a control device that controls another device.

The control Unit 121 is a device that controls the operation of the acquisition Unit 122 based on the Processing result output from the output Unit 105, and specifically, is a CPU (Central Processing Unit) or the like.

The collecting unit 122 is a device for collecting the collection object 113 from the container 112 by the control unit 121, and is, for example, a dispensing probe or the like.

The analysis unit 123 is an apparatus for performing analysis using the collection object 113, and is, for example, an immunoassay apparatus. The analysis result of the analysis unit 123 is displayed on a display or the like.

An example of the functional block diagram of the present embodiment will be described with reference to fig. 2. These functions may be configured by dedicated hardware or software that runs on the arithmetic unit 103. The present embodiment includes an image acquisition unit 201, a region calculation unit 202, and a state determination unit 203. Hereinafter, each part will be explained.

The image acquisition unit 201 acquires the upper image 212 captured by the imaging unit 111. The top image 212 includes the rim 213 and inner wall 214 of the container 112, the upper surface 215 of the collection object 113, the mixture 216, and the fixation portion 217. The mixture 216 is air bubbles, lipids, and the like present inside or on the upper surface of the collection target 113. The fixing portion 217 is a bracket or the like that fixes the container 112. Regarding the parts contained in the top image 212, reference is also made to the side view 211.

The area calculation unit 202 calculates any one of the areas, for example, the area of the edge 213 of the container 112 or the area of the upper surface 215 of the collection object 113, based on the upper image 212 acquired by the image acquisition unit 201. In order to calculate the area, threshold processing using a luminance value of an image, area division processing using FCN (full volume Networks), and the like are executed.

The state determination unit 203 determines the state of the container 112 and the collection object 113, for example, the type of the container 112, the inclination of the container 112, and the amount of the collection object 113, based on any one of the areas calculated by the area calculation unit 202.

An example of the processing flow of the present embodiment will be described with reference to fig. 3. In the present embodiment, the type of the container 112 is determined based on the area of the edge 213 of the container 112 calculated from the upper image 212.

(S301)

The image acquisition unit 201 acquires the upper image 212 via the input unit 102. The acquired upper image 212 may be an image captured by the imaging unit 111 or an image recorded in a recording device or the like, not shown.

(S302)

The area calculation unit 202 calculates the area of the edge 213 of the container 112 from the upper image 212 acquired in S301. In order to calculate the region of the edge 213, any one of threshold processing using the luminance value of the upper image 212, region division processing using FCN, and the like is executed. The area division process using FCN will be described later with reference to fig. 7 to 15.

An example of a region-divided image 401 obtained by performing region division processing on the upper image 212 will be described with reference to fig. 4. The region division image 401 includes an edge region 402, an inner wall region 403, an upper surface region 404, and a background region 405. The rim area 402 is the area of the rim 213 of the receptacle 112. The inner wall region 403 is the region of the inner wall 214 of the container 112, which is the region between the rim 213 of the container 112 and the acquisition object 113. The upper surface area 404 is the area of the upper surface 215 of the acquisition object 113, and the mixture 216 is also contained in the upper surface area 404. The background region 405 is the entire region outside the edge region 402, and the fixed portion 217 is also included in the background region 405.

(S303)

The state determination unit 203 determines the type of the container 112 based on the calculated edge area 402 in S302. This step will be described with reference to fig. 5 and 6.

The state determination unit 203 creates the binarized image 501 of the edge based on the calculation result in S302. The binarized image 501 of the edge includes the edge region 402 and a region 503 other than the edge region 402. For example, a luminance value of 0 is given to the edge region 402, and a luminance value of 255 is given to the other region 503.

The state determination unit 203 extracts the boundary of the edge region 402 by applying a sobel filter or the like to the binarized image 501 of the edge, and creates a boundary image 502 of the edge. Since the edge 213 of the container 112 has a circular ring shape, the boundary image 502 of the edge includes an inner boundary 505 that is a boundary on the inner wall side of the container 112 and an outer boundary 506 that is a boundary on the outer wall side.

The state determination unit 203 calculates the major axes of the inner boundary 505 and the outer boundary 506 by fitting an ellipse to the inner boundary 505 and the outer boundary 506, and determines the thickness of the container 112 from the major axes of the inner boundary 505 and the outer boundary using the following equation, assuming that the major axes are the inner diameter of the container 112.

[ numerical formula 1]

D＝(Lo-Lc)/2

Where D is the thickness of the container 112, Lo is the major axis of the outer boundary 506, and Lc is the major axis of the inner boundary 505.

In addition, the reason why the long diameter is used instead of the short diameter is to reduce the influence of the inclination of the container 112. That is, as the container 112 is tilted from the standing state, the short diameter of the inner boundary 505 and the outer boundary 506 becomes shorter, while the long diameter is less likely to change. The ellipse fitting may use all points of the inner boundary 505 or the outer boundary 506, and may also use at least 3 extracted points 508 arbitrarily extracted from the inner boundary 505 or the outer boundary 506. The extracted points 508 are points that reach the inner boundary 505 or the outer boundary 506, for example, by scanning in the vertical and horizontal directions from the center of gravity 507 obtained by arithmetically averaging the coordinates of the edge region 402.

The state determination unit 203 determines the type of the container 112 by comparing the inner diameter and the thickness of the container 112, which are determined from the major diameters of the inner boundary 505 and the outer boundary 506, with the table illustrated in fig. 6A. For example, when the inner diameter of the container 112 is 60 pixels and the thickness is 10 pixels, it is determined that the type of the container is α. The inner diameter and the thickness obtained correspond to which item in the table, and may be determined from the absolute value of the difference between the value obtained and the numerical value of each item. For example, when the obtained inner diameter is 98 pixels and the thickness is 6 pixels, the inner diameter is 100 pixels and the thickness is 6 pixels are selected from the table, and the type of the container 112 is determined to be δ.

The table illustrated in fig. 6A is prepared in advance by measuring the inner diameters and thicknesses of a plurality of types of containers 112, and is stored in the memory 104 or the like. The blank column in the table indicates an unknown container, and when the inner diameter and the thickness obtained in this step correspond to the blank column, it is determined that the container 112 included in the upper image 212 is an unknown container.

(S304)

The output unit 105 outputs the type of the container 112 determined in S303. The result of the determination is transmitted to the control unit 121, for example, and the control unit 121 controls the acquisition unit 122 based on the result of the determination. For example, when it is determined that the container 112 is an unknown container, the control unit 121 may stop the collecting unit 122. When the container 112 is an unknown container, the collision of the collecting unit 122 with the container 112 can be avoided by stopping the collecting unit 122.

The control unit 121 may adjust the height of the collecting unit 122 according to the type of the container 112. For example, the height of the collecting unit 122 may be adjusted based on the length of the container 112 obtained by comparing the type of the container 112 as the determination result with the table illustrated in fig. 6B. The table illustrated in fig. 6B is prepared in advance by measuring the inner diameter, thickness, length, and presence/absence of a taper of a plurality of types of containers 112, stored in the memory 104, and the like, and referred to as necessary.

The output unit 105 may output the result of the determination to a display or a printer, and may present the type of the container 112 to the operator, or may display a message prompting the change of the container 112 if the container is unknown. The output unit 105 may output the type of the container 112 to another image processing apparatus, and may output an image on which the correction process corresponding to the type of the container 112 is performed. The correction processing corresponding to the type of the container 112 is, for example, processing for normalizing the image size according to the inner diameter of the container 112 or processing for correcting the contrast according to the type of the container 112.

Through the above-described processing flow, the image processing apparatus 101 can determine the type of the container 112, which is one of the states of the container 112, from the upper image 212 acquired by one imaging unit 111. In addition, since the automatic analysis system 100 can appropriately control the acquisition unit 122 according to the determination result of the image processing apparatus 101, more accurate analysis can be performed without reducing the inspection efficiency.

Here, the area division processing using the FCN will be described. FCN is one of DNN (Deep Neural Network), and is a region division process of dividing an image into regions by constituting all layers by CNN (Convolutional Neural Network). The CNN is composed of 3 processes of Convolution (Convolution), Pooling (floating), and Activation (Activation). The convolution processing is processing for creating a map of feature quantities from input data, the pooling processing is processing for extracting a representative value from the input data, and the activation processing is processing for applying a nonlinear function to the input data.

An example of the operation of the convolution processing will be described with reference to fig. 7. As shown in fig. 7, in the convolution processing, while the positions at which the luminance values of the plurality of pixels are obtained are slid from the upper left to the lower right of the input data Ic, the output data Oc is calculated by performing multiplication of the luminance values of the plurality of obtained pixels by the coefficient Wc and addition of the addition coefficient Bc.

Equation 2 shows an example of a calculation equation for convolution processing.

[ numerical formula 2]

The input data Ic is data having 3 dimensions of the channel ch, the vertical position y, and the horizontal position x, and the output data Oc is data having 3 dimensions of the feature quantity number d, the vertical position y, and the horizontal position x. The multiplication coefficient Wc is a coefficient having 4 dimensions of the feature quantity number d, the channel ch, the vertical position fy, and the horizontal position fx, and is also referred to as a kernel, and the addition coefficient Bc is a coefficient having the dimension of the feature quantity number d. In addition, the size of the core is represented by a core height Fy and a core width Fx, and the amount of core sliding every 1 time is represented by a vertical direction step (stride amount) Sy and a horizontal direction step Sx.

The multiplication coefficient Wc and the addition coefficient Bc are previously obtained by machine learning using a random gradient descent method or the like using a training image, which is an image in which each region is accurately divided, as input data, and are stored in the memory 104 as coefficients for the region division processing. That is, machine learning is performed using a training signal that accurately indicates which of the edge region 402, the inner wall region 403, the upper surface region 404, and the background region 405 each pixel of the plurality of upper images 212 belongs to.

An example of the operation of the pooling process will be described with reference to fig. 8. As shown in fig. 8, in the pooling process, while sliding the positions at which the luminance values of the plurality of pixels are obtained from the upper left to the lower right of the input data Ip by a predetermined step, the output data Op is calculated by extracting a representative value from the obtained luminance values of the plurality of pixels. The representative value is, for example, the maximum value or the average value of the luminance values of a plurality of pixels, and the representative value extracted in the present embodiment is the maximum value.

Equation 3 represents an example of the calculation equation for the pooling process.

[ numerical formula 3]

Op(ch，y，x)＝max(Ip(ch，y*Sy+fy，x*Sx+fx)

where fy∈[0，Fy]，fx∈[0，Fx]

Here, the input data Ip and the output data Op are data having 3 dimensions of the channel ch, the vertical direction position y, and the horizontal direction position x. In the pooling process, similarly to the convolution process, the acquired luminance values of the plurality of pixels are multiplied by a coefficient called a kernel, the size of the kernel is represented by Fy and Fx, and the stride of the kernel is represented by Sy and Sx. In the pooling process, Sy and Sx are often set to Fy and Fx, and all elements of the kernel are 1. The kernel used in the pooling process is referred to as a P-kernel, and the kernel used in the convolution process is referred to as a C-kernel.

The activation processing is processing for applying a nonlinear function of expression 4 or expression 5 to the convolution-processed output data Oc or the pooling-processed output data Op, and for example, processing for identifying whether or not a certain pixel is the edge region 402.

[ numerical formula 4]

[ numerical formula 5]

ReLU(a)＝max(0，a)

The CNN combines the convolution processing, pooling processing, and activation processing to obtain a feature quantity and an identification result. The FCN is a region divider composed of CNNs only, and can input an image of an arbitrary size.

Fig. 9 shows an example of the FCN-based area division processing. The input image 901 is an image input to the FCN, and has a width and a height of 8 pixels. The 1 st kernel 902 represents an initial processing object area in each layer. The shaded portion indicates the processing object position. The 2 nd core 903 represents a 2 nd processing object area in each layer. The amount of offset between the 1 st core 902 and the 2 nd core 903 is determined by the strides Sy and Sx.

The layer 1 output data 904 is a result of applying convolution processing to the input image 901, and the layer 2 output data 905 is a result of applying pooling processing to the layer 1 output data 904. The layer 3 output data 906 is a result of applying convolution processing to the layer 2 output data 905, and the layer 4 output data 907 is a result of applying pooling processing to the layer 3 output data 906. Also, the region division result 908 indicates a recognition result of which position in the input image the 4 th-layer output data 907 corresponds to. Further, the kernel sizes Fy and Fx and the strides Sy and Sx of the convolution processing and the pooling processing are shown in fig. 9.

As shown in fig. 9, each time the processing with the stride of 2 or more is executed, the size of the output data of each layer is reduced in accordance with the stride. The 4 th layer output data 907 as the final output is reduced to 2 × 2 pixels, applied to the input image 901 only at 4 as indicated by the black squares of the region segmentation result 908. That is, only the recognition result of the black squares of the region division result 908 is obtained, and the white squares are in a state where the recognition result is not obtained, and the low resolution region division processing is performed.

The high-resolution region division process will be described with reference to fig. 10. In fig. 10, images having different acquisition positions are input to the FCN, and the result of region division for each image is acquired. That is, the 1 st input image 1001, the 2 nd input image 1002, the 3 rd input image 1003, and the 4 th input images 1004 and … are input, and the 1 st region division result 1005, the 2 nd region division result 1006, the 3 rd region division result 1007, and the 4 th region division results 1008 and … are obtained. Note that the 1 st input image 1001 is the same image as the input image 901 of fig. 9, the 2 nd input image 1002 is an image obtained by sliding the input image 901 by 1 pixel to the right, the 3 rd input image 1003 is an image obtained by sliding the input image 901 by 2 pixels to the right, and the 4 th input image 1004 is an image obtained by sliding the input image 901 by 1 pixel to the down. Due to the sliding of the pixels, a portion where no pixel exists is filled with the luminance value 0 or the luminance values of the adjacent pixels. The obtained plurality of region division results are integrated to obtain an integrated region division result 1009. Although the region division processing of fig. 10 can obtain a high-resolution region division result, the number of times of the region division processing increases, for example, 16 times in the example of fig. 10, and thus the processing time also increases.

Therefore, in the present embodiment, the kernel used in the convolution processing and pooling processing of the FCN is changed by the extension (scaled) processing, and the high-resolution area division result is obtained by the 1-time area division processing. The expansion process is a process of expanding a core by inserting a predetermined number of zeros between elements of the core, and the core after the expansion process is referred to as an expanded core or a D core. Further, Convolution processing and Pooling processing using the D kernel are referred to as "expanded Convolution (scaled Convolution) processing" and "expanded Pooling (scaled Pooling) processing".

An example of the D kernel in the extended convolution processing will be described with reference to fig. 11. C core 1101 is an example of a core used for convolution processing, and is composed of 9 elements a to i. The DC core 1102 is an example of a core for the extended convolution process, and is configured by inserting 1 zero between elements of the C core 1101. Values obtained by adding 1 to the number of inserted zeros are referred to as expansion numbers Dy and Dx, and DC core 1102 in fig. 11 has values Dy and Dx of 2 and 2.

Equation 6 shows an example of a calculation equation for the dilation convolution process.

[ numerical formula 6]

In expression 6, if Dy and Dx are 1 and 1, expression 2 is obtained.

An example of the D core in the expansion pooling process will be described with reference to fig. 12. The P kernel 1201 is an example of a kernel for pooling, and 9 total elements are 1. The DP kernel 1202 is an example of a kernel for expanding pooling, and is configured by inserting 1 zero between elements of the P kernel 1201. Similarly to the dilation convolution process, the dilation amounts Dy and Dx are also 2 and 2 in the DP core 1202 of fig. 12.

An example of the calculation formula of the extended pooling process is shown in equation 7.

[ number formula 7]

0p(ch，y，x)＝max(Ic(ch，y*Sy+fy*Dy，x*Sx+fx*Dx)

where fy∈[O，Fy]，fx∈[O，Fx]

In expression 7, if Dy and Dx are 1 and 1, expression 3 is obtained.

The dilation convolution process and dilation pooling process will be described with reference to fig. 13. The 1 st core 1301 and the 2 nd core 1302 are cores for convolution processing or pooling processing. In addition, the 1D core 1303 and the 2D core 1304 are cores for expansion convolution processing or expansion pooling processing. The kernel sizes Fy and Fx are both 3, and the strides Sy and Sx are both 2.

The convolution processing or the pooling processing corresponds to the expansion convolution processing or the expansion pooling processing in the case where both the expansion amounts Dy and Dx are 1. In addition, the expansion amounts Dy and Dx of the 1D core 1303 and the 2D core 1304 are both 2.

In addition, the processing range of each core may include an area other than the input data. For example, in the case of the 2 nd kernel 1302, the 1 st line corresponds to an area outside the input data. In the present embodiment, the region other than the input data is filled with the luminance value 0. The FCN using the extended convolution processing and the extended pooling processing is referred to as DFCN (extended FCN).

An example of a process flow for converting the FCN model into the DFCN model will be described with reference to fig. 14. In the processing flow of fig. 14, the FCN model is input, and the DFCN model for calculating the region division results in the vertical direction Ky and the horizontal direction Kx pixel units is output. As Ky and Kx approach 1, the resolution is higher, and as the values become larger, the resolution is lower, but the processing speed becomes higher. Here, Ky and Kx are 1 submultiples of the product Psy and Psx of the step sizes of all layers in the FCN model. The FCN model is composed of any one of a convolutional layer, a pooling layer, and an active layer. The convolution layer, the pooling layer and the activation layer are respectively subjected to convolution processing, pooling processing and activation processing.

In S1401, a variable L is initialized. The variable L indicates an index of a layer to be processed.

In S1402, the type of layer is determined. If the layer type is a convolutional layer or a pooling layer, the process proceeds to S1403, and if the layer type is an active layer, the process proceeds to S1406.

In S1403, the type of layer is transformed. If the L-th layer is a convolutional layer, the layer is converted to an extended convolutional layer, and if the L-th layer is a pooling layer, the layer is converted to an extended pooling layer. The parameters such as the size of the kernel of the expanded convolutional layer or the expanded pooling layer after the conversion, the number of types of the kernels, the coefficient of the kernel, and the pooling size are the same as the set values of the convolutional layer or the pooling layer before the conversion.

In S1404, the amount of expansion of the L-th layer is determined. An example of the order of determining the vertical expansion Dy will be described. The horizontal expansion Dx is similarly determined by replacing the subscript with y for x. First, the product PSy of the vertical step Sy from layer 0 to layer L-1 of the input FCN model is calculated. Next, the greatest common divisor GCy of the product PSy of the step length Sy and the step length Ky in the vertical direction of the region division result is obtained. The product PSy of the step Sy and the greatest common divisor GCy are substituted into the following formula to determine the vertical expansion Dy of the L-th layer.

[ number formula 8]

Dy＝PSy/GCy

And, the step Ky is updated by the following expression.

[ numerical formula 9]

Ky＝Ky/GCy

In S1405, the stride for the L-th level is determined. An example of the procedure for determining the vertical stride length Sy will be described. The horizontal stride Sx is similarly determined by replacing the index from y to x. The greatest common divisor GCy calculated at 1404 is substituted into the following expression to determine the vertical stride Sy of the L-th layer.

[ numerical formula 10]

Sy＝GCy

In S1406, the variable L is incremented. That is, the processing object is switched to the next layer.

In S1407, it is determined whether or not the conversion processing for all layers is completed. For example, if the variable L is equal to or greater than the number of layers constituting the FCN model, it is determined to be complete, and if not, it is determined to be incomplete. If completed, the process flow ends, and if not completed, the flow proceeds to S1402.

Through the above processing flow, the FCN model can be converted into a DFCN model that can obtain a high-resolution region segmentation result at high speed.

Fig. 15 shows an example of the DFCN-based region segmentation process. The input image 1501 is an image input to the DFCN, and has a width and a height of 8 pixels. The 1 st kernel 1502 indicates an initial processing target area in each layer. The shaded portion indicates the processing object position. The 2 nd core 1503 represents the 2 nd processing object region in each layer.

The layer 1 output data 1504 is the result of applying the dilation convolution processing to the input image 1501, and the layer 2 output data 1505 is the result of applying the dilation pooling processing to the layer 1 output data 1504. The layer 3 output data 1506 is the result of applying the dilation convolution process to the layer 2 output data 1505, and the layer 4 output data 1507 is the result of applying the dilation pooling process to the layer 3 output data 1506. Since no reduction in resolution occurs by the processing of each layer, the layer 4 output data 1507 becomes a region division result having the same resolution as the input image 1501.

In order to obtain a high-resolution area division result by FCN, as shown in fig. 10, it is necessary to perform area division processing for each image having a different acquisition position, and since some processing is repeated, the processing time increases. In the DFCN-based region segmentation process, since the process is repeated at once in the FCN with high resolution, the region segmentation result with high resolution can be obtained at high speed.

In addition, although the case where the FCN model is converted into the DFCN model is described in fig. 14, the DFCN model may be constructed from the beginning. Further, the image recognizer based on DNN, which recognizes an input image of a certain size as a recognition target, may be converted into FCN and further into DFCN. Machine learning for region segmentation may be performed by any model of DNN-based image recognizer, FCN, DFCN.

Example 2

In example 1, the case where the type of the container 112 is determined based on the edge area 402 of the container 112 calculated from the upper image 212 is described. The determination result output by the image processing apparatus 101 is not limited to the type of the container 112. In the present embodiment, a case will be described in which the amount of the collection object 113 is determined based on the edge area 402 of the container 112 and the upper surface area 404 of the collection object 113 calculated from the upper image 212. Note that the hardware configuration and functional block diagram of the present embodiment are the same as those of embodiment 1, and therefore, the description thereof is omitted.

An example of the processing flow of the present embodiment will be described with reference to fig. 16. Since S301 to S303 are the same as those in embodiment 1, descriptions thereof are omitted, and S1604 to S1606 after S303 are described.

(S1604)

The region calculation unit 202 calculates the upper surface region 404 of the acquisition object 113 from the upper image 212 acquired in S301. In the calculation of the upper surface area 404, threshold processing using the luminance value of the upper image 212, area division processing using FCN or DFCN, and the like are executed in the same manner as in S302.

(S1605)

The state determination unit 203 determines the amount of the collection object 113 based on the edge region 402 calculated in S302 and the upper surface region 404 calculated in S1604. This step will be described with reference to fig. 17 and 18. First, the upper surface area 404 will be described with reference to fig. 17.

The state determination unit 203 creates a binarized image 1701 of the upper surface of the acquisition object 113 based on the calculation result in S1604. The binarized image 1701 on the upper surface includes an upper surface region 404 and a region 1704 other than the upper surface region 404. The upper surface area 404 is given a luminance value of 0, for example, and the other areas 1704 are given a luminance value of 255.

The state determination unit 203 extracts the boundary of the upper surface region 404 by applying a sobel filter or the like to the binarized image 1701 of the upper surface, and creates a boundary image 1702 of the upper surface. The boundary image 1702 of the upper surface includes an upper surface boundary 1705. The state determination unit 203 performs ellipse fitting on at least 3 extraction points 1706 extracted arbitrarily from the upper surface boundary 1705 to calculate the major axis and the minor axis of the upper surface area 404. Even if the container 112 is inclined, the short diameter of the upper surface area 404 is less likely to change from the long diameter, and therefore the short diameter is used in the subsequent processing.

The determination of the amount of the collection target 113 will be described with reference to fig. 18. Note that a case where the lens of the imaging unit 111 is non-telecentric and a case where the lens is telecentric will be described. In the case where the lens is non-telecentric, the size of the object becomes smaller as the distance from the image pickup unit 111 to the object becomes longer, whereas in the case where the lens is telecentric, the size of the object becomes the same regardless of the distance from the image pickup unit 111 to the object.

The non-telecentric case will be described with reference to fig. 18A. Fig. 18A shows a side view or an upper image when the object 113 is acquired in a large amount and a small amount, that is, a side view 1801 when the amount is large and an upper image 1802 when the amount is large, a side view 1804 when the amount is small, and an upper image 1805 when the amount is small. As is clear from comparison of the large-volume upper image 1802 and the small-volume upper image 1805, the area of the upper surface 1803 in the large-volume case is larger than the area of the upper surface 1806 in the small-volume case. Therefore, the state determination unit 203 determines the amount of the collection object 113 based on the minor axis of the upper surface area 404.

Specifically, first, the short diameter of the upper surface area 404 is compared with the table illustrated in fig. 19, and the distance H from the edge 213 of the container 112 to the upper surface 215 of the collection object 113 is obtained. The table of fig. 19 is prepared in advance by measuring the short diameter or the distance H of the upper surface area 404 when different amounts of the collection objects 113 are stored in the plurality of types of containers 112, and is stored in the memory 104 or the like. Next, the type of container 112 determined based on the edge area 402 is compared with the table of fig. 6B, and the length G and the inner diameter D of the container 112 are determined. Then, the distance H, the length G, and the inner diameter D are substituted into the following equation to obtain the quantity Q of the collection object 113.

[ numerical formula 11]

Equation 11 is an equation in which the inner diameter D of the container 112 is fixed regardless of the distance H. When the container 112 has a tapered shape or the like and the inner diameter D of the container 112 changes according to the distance H, the amount Q of the collection object 113 is obtained by using equation 11 by storing the relationship between the distance H and the inner diameter D in the memory 104 or the like in advance and referring to the relationship as necessary.

The case of telecentricity will be described with reference to fig. 18B. Fig. 18B also shows a side view 1801 in the case of a large amount, an upper image 1802 in the case of a large amount, a side view 1804 in the case of a small amount, and an upper image 1805 in the case of a small amount, as in fig. 18A. As is clear from comparison of the large-volume upper image 1802 and the small-volume upper image 1805, in the case of telecentricity, the areas of the large-volume upper surface 1803 and the small-volume upper surface 1806 are equal, and the amount of the acquisition object 113 cannot be determined based on the minor axis of the upper surface region 404.

Therefore, in the case of telecentricity, as shown in fig. 18C, the amount of the acquisition target 113 is determined based on the upper image captured by tilting the imaging unit 111. That is, as is apparent from a comparison of the large-volume upper image 1802 and the small-volume upper image 1805, when the acquisition object 113 is large, the distance 1807 from the edge to the upper surface is shorter than when the amount is small, and therefore the amount of the acquisition object 113 is determined based on the distance 1807 from the edge to the upper surface. Specifically, when the distance 1807 from the edge to the upper surface is H' and the inclination angle of the imaging unit 111 with respect to the vertical direction is Ψ, the distance H from the edge 213 of the container 112 to the upper surface 215 of the collection object 113 is obtained by the following equation, and the amount Q of the collection object 113 is obtained by the equation 11.

[ numerical formula 12]

In the case where the container 112 has a tapered shape or the like, the amount Q of the collection object 113 is obtained by using equation 11 with reference to the relationship between the distance H and the inner diameter D stored in the memory 104 or the like in advance as necessary, as in the case of non-telecentric.

(S1606)

The output unit 105 outputs the amount of the collection target 113 determined in S1605. The result of the determination is transmitted to the control unit 121, for example, and the control unit 121 controls the acquisition unit 122 based on the result of the determination. For example, the control unit 121 may stop the acquisition unit 122 when it is determined that the amount of the acquisition target 113 has not reached a predetermined amount. The output unit 105 may output the result of the determination to a display or a printer to display the amount of the collection target 113 to the operator, or may display a message indicating that the amount of the collection target 113 has not reached a predetermined amount.

Through the above-described processing flow, the image processing apparatus 101 can determine the amount of the pickup object 113, which is one of the states of the pickup object 113, from the upper image 212 acquired by one imaging unit 111. In addition, since the automatic analysis system 100 can appropriately control the acquisition unit 122 according to the determination result of the image processing apparatus 101, more accurate analysis can be performed without reducing the inspection efficiency.

Example 3

The case of determining the type of the container 112 is described in example 1, and the case of determining the amount of the collection object 113 is described in example 2. In the present embodiment, a case will be described in which the inclination of the container 112 is determined based on the edge area 402 of the container 112 and the upper surface area 404 of the collection object 113 calculated from the upper image 212. Note that the hardware configuration and functional block diagram of the present embodiment are the same as those of embodiment 1, and therefore, the description thereof is omitted.

An example of the processing flow of the present embodiment will be described with reference to fig. 20. Since S301 to S302 are the same as those in embodiment 1 and S1604 is the same as that in embodiment 2, descriptions thereof are omitted, and S2005 to S2006 after S1604 are described.

(S2005)

The state determination unit 203 determines the inclination of the container 112 based on the edge region 402 calculated in S302 and the upper surface region 404 calculated in S1604. In the determination of the inclination of the container 112, the presence or absence of the inclination or the inclination angle is determined.

The determination of whether or not the container 112 is tilted will be described with reference to fig. 21. In the present embodiment, a case where the lens of the imaging unit 111 is non-telecentric and a case where the lens is telecentric will be described.

The non-telecentric case will be described with reference to fig. 21A. Fig. 21A shows a side view or an upper image of the container 112 when standing upright and when tilted, that is, a side view 2101 and an upper image 2102 when standing upright, a side view 2105 when tilted, and an upper image 2106 when tilted. In the upright-time upper image 2102 and the inclined-time upper image 2106, the center 2103 of the edge area 402 and the center 2104 of the upper surface area 404 are obtained based on the edge area 402 calculated in S302 and the upper surface area 404 calculated in S1604.

As is clear from comparison between the upright upper image 2102 and the inclined upper image 2106, the center 2103 of the upright edge area 402 coincides with the center 2104 of the upper surface area 404, whereas the center 2103 does not coincide with the center 2104 when inclined, and the distance between the center 2103 and the center 2104 increases as the inclination increases. Therefore, the state determination unit 203 determines whether or not the container 112 is tilted based on the distance between the center 2103 of the edge region 402 and the center 2104 of the upper surface region 404. That is, if the distance between center 2103 of edge region 402 and center 2104 of upper surface region 404 is equal to or greater than a predetermined threshold value, it is determined that container 112 is tilted, and if it is smaller than the threshold value, it is determined that container is not tilted.

The case of telecentricity will be described with reference to fig. 21B. Fig. 21B also shows a side view 2101 when standing, an upper image 2102 when standing, a side view 2105 when tilting, and an upper image 2106 when tilting, as in fig. 21A. In the case of non-telecentricity, the center 2103 of the edge region 402 and the center 2104 of the upper surface region 404 are obtained, and in the case of telecentricity, whether or not the container 112 is tilted is determined based on the distance between the center 2103 and the center 2104. When the container 112 has a tapered shape, the presence or absence of the inclination of the container 112 may be determined based on the distance between the center 2103 and the center 2104.

In addition to the distance between the center 2103 of the edge region 402 and the center 2104 of the upper surface region 404, whether or not the container 112 is tilted may be determined based on the inner wall region 403 of the container 112. For example, since the shape of the inner wall region 403 changes when the container 112 is tilted as compared with when the container 112 is standing upright, whether the container 112 is tilted or not may be determined using the width and area of the inner wall region 403. The outer wall area of the container 112 may be used for determining whether or not the container 112 is tilted, not limited to the inner wall area 403.

Determination of the tilt angle of the container 112 will be described with reference to fig. 22. In the present embodiment, a case where the lens of the imaging unit 111 is non-telecentric and a case where the lens is telecentric will be described.

The non-telecentric case will be described with reference to fig. 22A. Fig. 22A shows a side view 2201 when the container 112 is tilted and an upper image 2202 when tilted. In order to obtain the tilt angle of the container 112 in the tilted side view 2201, the distance in the horizontal direction and the distance in the vertical direction between the center 2203 of the edge 213 of the container 112 and the center 2204 of the upper surface 215 of the collection object 113 may be calculated. That is, the inclination angle θ of the container 112 with respect to the vertical direction is obtained from the inter-center horizontal distance 2205, which is the distance between the center 2203 and the center 2204 in the horizontal direction, and the inter-center vertical distance 2206, which is the distance in the vertical direction, by the following equation.

[ numerical formula 13]

Where Sh is the inter-center horizontal distance 2205 and H is the inter-center vertical distance 2206.

However, since the image captured by the imaging unit 111 is the upper image 2202 when tilted, it is necessary to obtain the inter-center horizontal distance 2205 and the inter-center vertical distance 2206 from the upper image 2202 when tilted in order to obtain the tilt angle θ of the container 112 using equation 13. First, the center-to-center vertical distance 2206, that is, the distance H from the edge 213 of the container 112 to the upper surface 215 of the acquisition object 113 is obtained by comparing the short diameter of the upper surface area 404 with the table illustrated in fig. 19 as described in example 2.

Next, the center-to-center horizontal distance 2205 is obtained from the top image 2202 at the time of the tilt. In the case of non-telecentricity, since the size of the subject becomes smaller as the distance from the imaging unit 111 to the subject becomes longer, it is preferable to perform processing by correcting the length on the upper surface 215 of the collection object 113 to the length on the edge 213 of the container 112. Specifically, the inter-area-center distance 2209, which is the distance between the center 2207 of the edge area 402 and the center 2208 of the upper surface area 404 in the upper image 2202 at the time of the tilt, is preferably corrected based on the ratio of the minor diameter of the upper surface area 404 to the inner diameter of the edge area 402. For example, Sh as the inter-center horizontal distance 2205 is calculated by the following equation.

[ numerical formula 14]

Where F1 is the 1 st center-to-center distance 2212, which is the distance from the lens center 2210 to the center 2208 of the upper surface area 404, and F0 is the 2 nd center-to-center distance 2211, which is the distance from the lens center 2210 to the center 2207 of the edge area 402. In addition, B0 is the inner diameter 2213 of edge region 402 and B1 is the minor diameter 2214 of upper surface region 404. According to equation 14, after the 2 nd inter-center distance 2211, which is the length on the upper surface 215 of the acquisition object 113, is corrected to the length on the edge 213 of the container 112, the difference from the 1 st inter-center distance 2212 is calculated, and the absolute value thereof is calculated as the inter-center horizontal distance 2205, which is Sh. That is, Sh as the center-to-center horizontal distance 2205 is calculated by correcting the size of the subject that varies according to the distance from the imaging unit 111 to the subject.

The calculated Sh and the distance H obtained before are substituted into equation 13, thereby obtaining the inclination angle θ of the container 112.

The case of telecentricity will be described with reference to fig. 22B. Fig. 22B also shows a side view 2201 of the container 112 when tilted and an upper image 2202 when tilted, as in fig. 22A. In the case of telecentricity, the inter-center horizontal distance 2205 is obtained as an inter-area center distance 2209 which is a distance between the center 2207 of the edge area 402 and the center 2208 of the upper surface area 404 in the oblique upper image 2202. Since the center-to-center vertical distance 2206 is obtained in the same manner as in example 2, the inclination angle θ of the container 112 is obtained by substituting the center-to-center horizontal distance 2205 and the center-to-center vertical distance 2206 into equation 13.

As shown in fig. 18C, imaging may be performed while the imaging unit 111 is tilted, and the tilt angle of the imaging unit 111 when the inner wall region 403 of the container 112 is no longer included in the upper image may be set as the tilt angle θ of the container 112.

(S2006)

The output unit 105 outputs the inclination of the container 112 determined in S2005. The result of the determination is transmitted to the control unit 121, for example, and the control unit 121 controls the acquisition unit 122 based on the result of the determination. For example, the control unit 121 may stop the collecting unit 122 when it is determined that the container 112 is tilted or the tilt angle of the container 112 exceeds a predetermined threshold value. Alternatively, the position and inclination of the collecting section 122 may be adjusted according to the inclination angle of the container 112.

The output unit 105 may output the result of the determination to a display or a printer, and may present the presence or absence of the inclination of the container 112 and the inclination angle to the operator, or may display a message to prompt attention when the ratio of the number of inclined containers 112 exceeds a predetermined threshold value. Further, the output unit 105 may output the tilt angle of the container 112 to another image processing apparatus, or may output an image on which correction processing is performed in accordance with the tilt angle of the container 112 at the same time. The correction processing corresponding to the tilt angle of the container 112 is, for example, processing of correcting distortion of the image due to the tilt of the container 112 by performing coordinate conversion of the image in accordance with the tilt angle of the container 112.

According to the above-described processing flow, the image processing apparatus 101 can determine the inclination of the container 112, which is one of the states of the container 112, from the upper image 212 acquired by one imaging unit 111. In addition, since the automatic analysis system 100 can appropriately control the acquisition unit 122 according to the determination result of the image processing apparatus 101, more accurate analysis can be performed without reducing the inspection efficiency.

Example 4

In embodiments 1 to 3, the case where the states of the container 112 and the collection object are determined based on the edge area 402 of the container 112 and the upper surface area 404 of the collection object 113 calculated from the upper image 212 is described. The image processing apparatus 101 needs an appropriate training signal for machine learning in order to perform region segmentation processing on the upper image 212 with high accuracy using FCN, DFCN, or the like. In the present embodiment, a case where a training signal for machine learning is generated efficiently will be described. The hardware configuration of this embodiment is the same as that of embodiment 1, and therefore, the description thereof is omitted.

An example of the functional block diagram of the present embodiment will be described with reference to fig. 23. These functions may be configured by dedicated hardware or software that runs on the arithmetic unit 103. The present embodiment includes an image acquisition unit 201, a region calculation unit 202, a likelihood acquisition unit 2301, an image selection/display unit 2302, a training signal acquisition unit 2303, and a learning unit 2304. The following describes the image acquisition unit 201 and the parts other than the area calculation unit 202 described in embodiment 1.

The likelihood acquisition unit 2301 acquires the likelihood of which region, for example, the edge region 402, the inner wall region 403, the upper surface region 404, or the like each pixel in each region calculated by the region calculation unit 202 is. The likelihood is normalized so that the total sum is 1.0 for each pixel, and the maximum likelihood in each pixel is called the maximum likelihood. The pixels with large maximum likelihood have high certainty factor of the corresponding region, and the pixels with small maximum likelihood have low certainty factor.

The image selection/display unit 2302 selects an image used for providing a training signal based on, for example, the maximum likelihood of each pixel in the image, and displays the selected image on a display or the like.

The training signal acquisition unit 2303 acquires a training signal from a training image that is an image including the training signal. The training signal is given to the region segmentation process performed by another image processing apparatus by the operation of the operator.

The learning unit 2304 performs machine learning using a training signal included in a training image, and calculates or updates parameters such as coefficients used by the region calculating unit 202 in the region segmentation process.

An example of the processing flow of the present embodiment will be described with reference to fig. 24. In the present embodiment, an image with relatively low certainty of each divided region is selected and displayed, and machine learning is performed using a training signal given to the displayed image by an operator.

(S2401)

The image selection/display unit 2302 selects an image to which a training signal is to be applied from an image group recorded in a recording device or the like based on the likelihood of each pixel in the image, and displays the selected image on a display or the like. For example, an image in which the average value of the maximum likelihood of each pixel is smaller than a predetermined threshold value, that is, an image in which the certainty of each region is relatively low is selected.

An example of a screen displayed in this step will be described with reference to fig. 25. The screen shown in fig. 25 is an image list showing the number of images to which a training signal is given and the number of images to which a training signal is not given, among the plurality of images selected in this step, for each type of container 112. When an arbitrary cell in the image list, for example, a cell displaying unfinished 100 of the container β is selected by a cursor of a mouse or the like, an image corresponding to the cell, that is, an image including the container β may be displayed.

The operator can confirm the number of images to which the training signal is given and the number of images to which the training signal is not given for each type of container 112 on the basis of the screen, and can determine which type of container 112 the training signal is preferably given to, and therefore, the task of giving the training signal can be performed efficiently. The displayed image list is not limited to the image list classified for each category of the container 112.

(S2402)

The training signal acquisition unit 2303 acquires a training signal for the image displayed in S2401 based on the operation of the operator. An example of a screen used in this step will be described with reference to fig. 26 and 27.

The screen shown in fig. 26 is an example of an operation screen for designating a region to which a training signal is to be applied, and includes an image display unit 2601, an image selection unit 2602, a correction tool 2603, a save button 2604, and a read button 2605.

The image display unit 2601 displays an image corresponding to the selection by the image selection unit 2602. The image selecting unit 2602 selects the condition of the image displayed on the image display unit 2601. In fig. 26, the original image and the region division result are selected by the image selecting unit 2602, and the result of the region division of the original image by the region calculating unit 202 is displayed on the image display unit 2601.

In the correction tool 2603, a region in which the training signal is to be corrected is selected. For example, when an arbitrary pixel of the image displayed on the image display unit 2601 is selected in a state where the liquid surface, that is, the upper surface of the collection object 113 when the liquid is selected by the correction tool 2603, the selected pixel is given a training signal of the upper surface region 404. The image display unit 2601 selects pixels in an image displayed thereon with a mouse or the like.

The save button 2604 is a button that is pressed when the corrected training signal is saved. That is, when the save button 2604 is pressed, the training signal acquisition section 2303 acquires the training signal given via the correction tool 2603. In addition, the original training signal is acquired for the pixels that are not corrected by the correction tool 2603.

The read button 2605 is a button to be pressed when another image is selected. By pressing the read button 2605, for example, the image displayed on the image display portion 2601 can be updated, or a screen as illustrated in fig. 25 can be displayed.

The screen shown in fig. 27 is an example of an operation screen for applying a training signal, and includes an upper image display portion 2701, a container type display portion 2702, a training signal input portion 2703, a save button 2604, and a read button 2605. The save button 2604 and the read button 2605 have the same functions as those of the screen in fig. 26, and therefore, the description thereof is omitted.

The upper image display portion 2701 displays an upper image 212 which is an image captured by the imaging portion 111. Fig. 27 shows a state in which a mixture 216 of bubbles, lipids, and the like floats on the upper surface 215 of the collection target 113.

The container type display portion 2702 displays the type of the container 112 determined from the area calculated by the area calculation portion 202, together with the inside diameter and the thickness. Further, the length, print contents, print position, presence or absence of screws around the edges, ribs on the bottom surface, and the like of the container 112 may be displayed. The operator may also correct various information displayed on the container type display portion 2702.

The training signal input unit 2703 inputs a training signal to be given to an arbitrary area of an image displayed on the upper image display unit 2701. For example, a training signal corresponding to the upper surface region 404 is input to bubbles, lipids, and the like in an image displayed on the upper image display portion 2701. The training signal input unit 2703 is not limited to the text box illustrated in fig. 27, and may be in the form of presentation options such as radio buttons and pull-down menus.

The use of the screens shown in fig. 26 and 27 assists the operator in providing the training signal, and therefore, the task of providing the training signal can be performed more efficiently.

(S2403)

It is determined whether or not more than a certain number of new training images have been collected. If more than a certain number of training images are collected, the process proceeds to S2404, and if the certain number is not reached, the process returns to S2401.

(S2404)

The learning unit 2304 updates the coefficient by machine learning using a predetermined number of or more training images. The coefficient is updated by, for example, a random gradient descent method. The initial value of the coefficient may be the coefficient stored in the memory 104 or may be a random number.

(S2405)

The learning unit 2304 evaluates the updated coefficient. For example, image groups with positive solution information are prepared in advance for evaluating the coefficients, and the image groups are subjected to region division processing using the coefficients updated in S2404 and the coefficients before updating, respectively, and the updated coefficients are evaluated according to whether or not the positive solution rate is increased.

(S2406)

It is determined whether or not to overwrite coefficients in the memory 104. The evaluation result of S2405 may be used to determine whether or not a coating is present. If the coefficient is covered, the process proceeds to S2407, and if not, the flow of the process ends.

(S2407)

The learning unit 2304 overwrites the coefficients in the memory 104 or stores the coefficients before and after updating.

With the above-described processing flow, since an appropriate training signal is efficiently given to an image with a relatively low certainty of the region division result, the image processing apparatus 101 can efficiently perform appropriate machine learning. In addition, an appropriate training signal can be given to bubbles, foreign matter, and the like. Since the accuracy of the region segmentation process can be improved by appropriate machine learning, the image processing apparatus 101 can more accurately determine the states of the container 112 and the collection object 113, and the automatic analysis system 100 can perform more accurate analysis.

A plurality of embodiments of the present invention have been described above. The present invention is not limited to these examples, and various modifications are possible. For example, the above-described embodiments have been described in detail to explain the present invention easily and understandably, and are not limited to having all of the structures described. In addition, a part of the structure of one embodiment may be replaced with the structure of another embodiment. Further, the structure of another embodiment may be added to the structure of one embodiment. In addition, a part of the configuration of each embodiment can be added, deleted, or replaced with another configuration.

Description of the reference numerals

100: automatic analysis system, 101: image processing apparatus, 102: input unit, 103: an arithmetic unit, 104: memory, 105: output unit, 106: bus, 111: imaging unit, 112: a container, 113: acquisition subject, 121: control unit, 122: acquisition unit, 123: analysis unit, 201: image acquisition unit, 202: region calculation unit, 203: state determination unit, 211: side view, 212: upper image, 213: rim of container, 214: inner wall of container, 215: upper surface of the object is acquired, 216: mixture, 217: fixing part, 401: region-segmented image, 402: edge area, 403: inner wall area, 404: upper surface area, 405: background region, 501: binary image of edge, 502: boundary image of edge, 503: other regions, 505: inner boundary, 506: outer boundary, 507: center of gravity of edge region, 508: extraction point, 901: input image, 902: core 1, 903: kernel 2, 904: layer 1 output data, 905: layer 2 output data, 906: layer 3 output data, 907: layer 4 output data, 908: region segmentation result, 1001: 1 st input image, 1002: input image 2, 1003: input image No. 3, 1004: input image No. 4, 1005: region 1 segmentation result, 1006: 2 nd area segmentation result. 1007: region 3 segmentation result, 1008: region 4 segmentation result, 1009: integrate region segmentation results, 1101: c kernel, 1102: DC kernel, 1201: p kernel, 1202: DP kernel, 1301: core 1, 1302: kernel 2, 1303: 1D kernel, 1304: 2D kernel, 1501: input image, 1502: core 1, 1503: kernel 2, 1504: layer 1 output data, 1505: layer 2 output data, 1506: layer 3 output data, 1507: layer 4 output data, 1701: binarized image of upper surface, 1702: boundary image of upper surface, 1704: other regions, 1705: upper surface boundary, 1706: extraction points, 1801: side view when large, 1802: top image at large, 1803: upper surface when large, 1804: side view when small, 1805: upper image at small amount, 1806: upper surface when small, 1807: distance from edge to upper surface, 2101: side view when upright, 2102: upper image when standing, 2103: center of edge area, 2104: center of upper surface area, 2105: side view when tilted, 2106: upper image at tilt, 2201: side view when tilted, 2202: upper image at tilt, 2203: center of edge, 2204: center of upper surface, 2205: center-to-center horizontal distance, 2206: center-to-center vertical distance, 2207: center of edge region, 2208: center of upper surface area, 2209: inter-area-center distance, 2210: lens center, 2211: distance between centers 2, 2212: distance between centers 1, 2213: inner diameter of edge area, 2214: minor axis of upper surface region, 2301: likelihood acquisition unit, 2302: image selection/display section, 2303: training signal acquisition unit, 2304: learning unit, 2601: image display unit, 2602: image selecting unit, 2603: correction tool, 2604: save button, 2605: read button, 2701: upper image display portion, 2702: container type display portion, 2703: a training signal input unit.