阅读说明：本技术 用于自动药物分配分析的计算机成像预处理 (Computer imaging pre-processing for automated drug dispensing analysis ) 是由 C·R·马克森 P·J·沙阿 C·G·勒默斯 于 2018-11-21 设计创作，主要内容包括：一种方法包括拍摄由收纳器容纳的药物的第一图像。该方法包括基于第一图像创建第二图像。该方法包括确定更可能对应于该收纳器的第二图像的像素的第一子集。该方法包括通过针对每个点进行以下步骤来沿着第一轴线处理第二图像：限定在该点处与第一轴线垂直相交的线,以及计数沿着该线定位的多少个像素在第一子集中。该方法包括确定计数的第一和第二局部最大值。该方法包括基于局部最大值的位置来估计该收纳器的第一边缘和第二边缘的位置。该方法包括基于第一边缘和第二边缘来限定椭圆,以及通过进一步处理排除所限定的椭圆之外的第一图像的区域。(A method includes capturing a first image of a medicament contained by a receptacle. The method includes creating a second image based on the first image. The method includes determining a first subset of pixels of a second image that are more likely to correspond to the receptacle. The method comprises processing the second image along the first axis by, for each point: defining a line that perpendicularly intersects the first axis at the point, and counting how many pixels located along the line are in the first subset. The method includes determining first and second local maxima of the count. The method includes estimating the positions of the first and second edges of the receptacle based on the position of the local maximum. The method comprises defining an ellipse based on the first edge and the second edge, and excluding regions of the first image outside the defined ellipse by further processing.)

1. A machine, comprising:

a camera configured to capture a first image of the medicament contained by the container;

memory hardware that stores code; and

processor hardware that executes the code stored by the memory hardware, wherein the code implements:

a color processing module configured to create a second image based on the first image, wherein each pixel of the second image is identified by a single value;

a pixel identification module configured to divide pixels of the second image into mutually exclusive first and second subsets of pixels, wherein the pixels of the first subset are more likely to correspond to the receptacles than the pixels of the second subset;

a scanning module configured to process the second image along a first axis by, for each point along the first axis:

defining a line perpendicular to the first axis and intersecting the first axis at that point;

counting how many pixels in the second image located along the line are in the first subset; and

recording the count;

a maximum detection module configured to determine a first local maximum of the count and a second local maximum of the count;

an ellipse calculation module configured to:

estimating a position of a first edge of the receptacle based on a position of the first local maximum along the first axis;

estimating a position of a second edge of the receptacle based on a position of the second local maximum along the first axis; and

defining an ellipse based on the first edge and the second edge; and

an output module configured to output a processed image based on the first image, wherein the processed image indicates that regions of the first image outside of the defined ellipse are excluded from further processing.

2. The machine of claim 1, wherein the color processing module is configured to, for each pixel of the first image, generate a corresponding pixel of the second image that excludes any other pixel of the first image.

3. The machine of claim 2, wherein the color processing module is configured to, for each pixel of the first image, generate the corresponding pixel of the second image by calculating a grayscale value for that pixel.

4. The machine of claim 2, wherein the color processing module is configured to, for each pixel of the first image, generate the corresponding pixel of the second image by:

setting the green value of the corresponding pixel as the sum of the green value of the pixel and the blue value of the pixel minus the red value of the pixel;

setting a green value of the corresponding pixel to zero; and

setting the blue value of the corresponding pixel to zero.

5. The machine of claim 1, wherein the output module is configured to include metadata in the processed image, wherein the metadata identifies regions of the first image excluded by further processing.

6. The machine of claim 1, wherein the output module is configured to set areas of the first image excluded by further processing to black.

7. The machine of claim 1, wherein:

the maximum detection module is configured to:

determining the first local maximum within a first region of the second image; and

determining the second local maximum within a second region of the second image;

the first region and the second region do not overlap, and

the size of the third region of the second image is not zero, does not overlap with the first region, and does not overlap with the second region.

8. The machine of claim 1, wherein:

the scanning module is configured to process the second image along a second axis by, for each point along the second axis:

defining a line perpendicular to and intersecting the second axis at the point;

counting how many pixels in the second image located along the line are in the first subset; and

recording the count;

the code also implements a second maximum detection module configured to determine a first local maximum of the count for the second axis and a second local maximum of the count for the second axis; and is

The ellipse calculation module is configured to:

estimating a position of a third edge of the receptacle based on a position of the first local maximum along the second axis;

estimating a position of a fourth edge of the receptacle based on a position of the second local maximum along the second axis; and

defining the ellipse based on the first edge, the second edge, the third edge, and the fourth edge.

9. The machine of claim 8, wherein:

the first axis is parallel to a first edge of the first image; and is

The second axis is perpendicular to the first axis.

10. The machine of claim 1, wherein the ellipse calculation module is configured to: estimating a position of the second edge of the receptacle as an edge of the first image in response to the second local maximum being less than the first local maximum by more than a predetermined amount.

11. A method, comprising:

capturing a first image of the medication received by the receiver;

creating a second image based on the first image, wherein each pixel of the second image is identified by a single value;

dividing pixels of the second image into mutually exclusive first and second subsets of pixels, wherein the pixels of the first subset are more likely to correspond to the receptacles than the pixels of the second subset;

processing the second image along a first axis by, for each point along the first axis:

defining a line perpendicular to the first axis and intersecting the first axis at that point;

counting how many pixels in the second image located along the line are in the first subset; and

recording the count;

determining a first local maximum of the count and a second local maximum of the count;

estimating a position of a first edge of the receptacle based on a position of the first local maximum along the first axis;

estimating a position of a second edge of the receptacle based on a position of the second local maximum along the first axis;

defining an ellipse based on the first edge and the second edge; and

outputting a processed image based on the first image, wherein the processed image indicates that regions of the first image outside of the defined ellipse are excluded from further processing.

12. The method of claim 11, wherein creating the second image comprises, for each pixel of the first image, generating a corresponding pixel of the second image that excludes any other pixel of the first image.

13. The method of claim 12, wherein creating the second image comprises, for each pixel of the first image, generating the corresponding pixel of the second image by:

setting the green value of the corresponding pixel as the sum of the green value of the pixel and the blue value of the pixel minus the red value of the pixel;

setting a green value of the corresponding pixel to zero; and

setting the blue value of the corresponding pixel to zero.

14. The method of claim 11, further comprising at least one of:

including metadata in the processed image, the metadata identifying regions of the first image excluded by further processing; and

setting areas of the processed image excluded by further processing to black.

15. The method of claim 11, wherein:

determining the first local maximum within a first region of the second image;

determining the second local maximum within a second region of the second image;

the first region and the second region do not overlap, and

the size of the third region of the second image is not zero, does not overlap with the first region, and does not overlap with the second region.

16. The method of claim 11, further comprising:

processing the second image along a second axis by, for each point along the second axis:

defining a line perpendicular to and intersecting the second axis at the point;

counting how many pixels in the second image located along the line are in the first subset; and

recording the count;

determining a first local maximum of the count for the second axis and a second local maximum of the count for the second axis;

estimating a position of a third edge of the receptacle based on a position of the first local maximum along the second axis;

estimating a position of a fourth edge of the receptacle based on a position of the second local maximum along the second axis; and

defining the ellipse based on the first edge, the second edge, the third edge, and the fourth edge.

17. The method of claim 16, wherein:

the first axis is parallel to a first edge of the first image; and is

The second axis is perpendicular to the first axis.

18. The method of claim 11, further comprising: estimating a position of the second edge of the receptacle as an edge of the first image in response to the second local maximum being less than the first local maximum by more than a predetermined amount.

19. The method of claim 11, further comprising:

obtaining prescription information corresponding to the receptacle, wherein the prescription information specifies a desired medication;

providing the processed image to a trained machine learning model to identify a probability of a drug being present in the receptacle;

determining a likelihood that at least one medication in the receptacle is not the desired medication based on the identified medication probabilities; and

initiating a remedial action for the receiver in response to the likelihood exceeding a predetermined threshold.

20. A non-transitory computer-readable medium storing instructions, the instructions comprising:

capturing a first image of the medication received by the receiver;

creating a second image based on the first image, wherein each pixel of the second image is identified by a single value;

dividing pixels of the second image into mutually exclusive first and second subsets of pixels, wherein the pixels of the first subset are more likely to correspond to the receptacles than the pixels of the second subset;

processing the second image along a first axis by, for each point along the first axis:

defining a line perpendicular to and intersecting the first axis at the point;

counting how many pixels in the second image located along the line are in the first subset; and

recording the count;

determining a first local maximum of the count and a second local maximum of the count;

estimating a position of a first edge of the receptacle based on a position of the first local maximum along the first axis;

estimating a position of a second edge of the receptacle based on a position of the second local maximum along the first axis;

defining an ellipse based on the first edge and the second edge; and

outputting a processed image based on the first image, wherein the processed image indicates that regions of the first image outside of the defined ellipse are excluded from further processing.

Technical Field

The present disclosure relates to image processing, and more particularly to image pre-processing that excludes irrelevant features by further image processing steps.

Background

Pharmacies may process and fill a large number of prescriptions according to a prescription order. Large pharmacies may use automated systems to process and fill prescriptions. Errors in compounding a prescription, such as dispensing errors or incorrectly combining multiple medications, are difficult to detect. Manual inspection is very expensive and also suffers from all the vulnerabilities of human ability. For example, a person's attention may be distracted, a person's attention may be reduced over time, and the human eye may be insensitive to certain differences in the dispensed medication.

However, it is very difficult to train a computer to recognize a medication dispensing error. Identifying the myriad of ways in which medications can be dispensed, and the myriad of angles at which individual items of medication can be arranged in a dispensed product, presents significant challenges to computer algorithms. In addition, computer algorithms can have difficulty distinguishing between the medication being dispensed and the package to which the medication is being dispensed. For these reasons, many pharmacies do not use an automated check for each dispensed medication, but rely on manual spot checks of a small subset (usually random) of the dispensed medications.

The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

Disclosure of Invention

A machine includes a camera configured to capture a first image of a medicament received by a receiver. The machine comprises: memory hardware that stores code; and processor hardware that executes the code stored by the memory hardware. The code implements a color processing module configured to create a second image based on the first image, wherein each pixel of the second image is identified by a single value. The code implements a pixel identification module configured to divide pixels of the second image into mutually exclusive first and second subsets of pixels. The first subset of pixels is more likely to correspond to the receptacle than the second subset of pixels. The code implements a scanning module configured to process the second image along a first axis by, for each point along the first axis: defining a line perpendicular to the first axis and intersecting the first axis at that point; counting how many pixels in the second image located along the line are in the first subset; and records the count. The code implements a maximum detection module configured to determine a first local maximum of the count and a second local maximum of the count. The code implements an ellipse calculation module configured to: estimating a position of a first edge of the receptacle based on a position of the first local maximum along the first axis; estimating a position of a second edge of the receptacle based on a position of the second local maximum along the first axis; and defining an ellipse based on the first edge and the second edge. The code implements an output module configured to output a processed image based on the first image. The processed image indicates that regions of the first image outside the defined ellipse are excluded from further processing.

In other features, the color processing module is configured to, for each pixel of the first image, generate a corresponding pixel of the second image that excludes any other pixel of the first image. In other features, the color processing module is configured to, for each pixel of the first image, generate the corresponding pixel of the second image by calculating a grayscale value for that pixel. In other features, the color processing module is configured to, for each pixel of the first image, generate the corresponding pixel of the second image by: setting the green value of the corresponding pixel as the sum of the green value of the pixel and the blue value of the pixel minus the red value of the pixel; setting a green value of the corresponding pixel to zero; and setting the blue value of the corresponding pixel to zero.

In other features, the output module is configured to include metadata in the processed image. The metadata identifies regions of the first image that are excluded by further processing. In other features, the output module is configured to set areas of the first image excluded by further processing to black. In other features, the maximum detection module is configured to determine the first local maximum within a first region of the second image and to determine the second local maximum within a second region of the second image. The first region and the second region do not overlap. The size of the third region of the second image is not zero, does not overlap with the first region, and does not overlap with the second region.

In other features, the scanning module is configured to process the second image along a second axis by, for each point along the second axis: defining a line perpendicular to and intersecting the second axis at the point; counting how many pixels in the second image located along the line are in the first subset; and records the count. The code also implements a second maximum detection module configured to determine a first local maximum of the count for the second axis and a second local maximum of the count for the second axis. The ellipse calculation module is configured to: estimating a position of a third edge of the receptacle based on a position of the first local maximum along the second axis; estimating a position of a fourth edge of the receptacle based on a position of the second local maximum along the second axis; and defining the ellipse based on the first edge, the second edge, the third edge, and the fourth edge.

In other features, the first axis is parallel to a first edge of the first image and the second axis is perpendicular to the first axis. In other features, the ellipse calculation module is configured to estimate the location of the second edge of the receptacle as an edge of the first image in response to the second local maximum being less than the first local maximum by more than a predetermined amount.

A method includes capturing a first image of a medication received by a receptacle. The method includes creating a second image based on the first image, wherein each pixel of the second image is identified by a single value. The method includes dividing pixels of the second image into mutually exclusive first and second subsets of pixels. The first subset of pixels is more likely to correspond to the receptacle than the second subset of pixels. The method comprises processing the second image along a first axis by, for each point along the first axis: defining a line perpendicular to the first axis and intersecting the first axis at that point; counting how many pixels in the second image located along the line are in the first subset; and records the count. The method includes determining a first local maximum of the count and a second local maximum of the count. The method comprises estimating a position of a first edge of the receptacle based on a position of the first local maximum along the first axis. The method comprises estimating a position of a second edge of the receptacle based on a position of the second local maximum along the first axis. The method includes defining an ellipse based on the first edge and the second edge. The method includes outputting a processed image based on the first image. The processed image indicates that regions of the first image outside the defined ellipse are excluded from further processing.

In other features, creating the second image includes, for each pixel of the first image, generating a corresponding pixel of the second image that excludes any other pixel of the first image. In other features, creating the second image includes, for each pixel of the first image, generating the corresponding pixel of the second image by: setting the green value of the corresponding pixel as the sum of the green value of the pixel and the blue value of the pixel minus the red value of the pixel; setting a green value of the corresponding pixel to zero; and setting the blue value of the corresponding pixel to zero.

In other features, the method includes including metadata in the processed image. The metadata identifies regions of the first image that are excluded by further processing. In other features, the method includes setting regions of the processed image excluded by further processing to black. In other features, the first local maximum is determined within a first region of the second image. Determining the second local maximum within a second region of the second image. The first region and the second region do not overlap. The size of the third region of the second image is not zero, does not overlap with the first region, and does not overlap with the second region.

In other features, the method includes processing the second image along a second axis by, for each point along the second axis: defining a line perpendicular to and intersecting the second axis at the point; counting how many pixels in the second image located along the line are in the first subset; and records the count. The method includes determining a first local maximum of the count for the second axis and a second local maximum of the count for the second axis. The method comprises estimating a position of a third edge of the receptacle based on a position of the first local maximum along the second axis. The method includes estimating a position of a fourth edge of the receptacle based on a position of the second local maximum along the second axis. The method includes defining the ellipse based on the first edge, the second edge, the third edge, and the fourth edge.

In other features, the first axis is parallel to a first edge of the first image and the second axis is perpendicular to the first axis. In other features, the method includes estimating the location of the second edge of the receptacle as an edge of the first image in response to the second local maximum being less than the first local maximum by more than a predetermined amount.

A non-transitory computer-readable medium stores instructions comprising taking a first image of a medication received by a receiver. The instructions include creating a second image based on the first image, wherein each pixel of the second image is identified by a single value. The instructions include dividing pixels of the second image into mutually exclusive first and second subsets of pixels. The first subset of pixels is more likely to correspond to the receptacle than the second subset of pixels. The instructions include processing the second image along a first axis by, for each point along the first axis: defining a line perpendicular to the first axis and intersecting the first axis at that point; counting how many pixels in the second image located along the line are in the first subset; and records the count. The method includes determining a first local maximum of the count and a second local maximum of the count. The instructions include estimating a position of a first edge of the receptacle based on a position of the first local maximum along the first axis. The instructions include estimating a position of a second edge of the receptacle based on a position of the second local maximum along the first axis. The instructions include defining an ellipse based on the first edge and the second edge. The instructions include outputting a processed image based on the first image. The processed image indicates that regions of the first image outside the defined ellipse are excluded from further processing.

Further areas of applicability of the present disclosure will become apparent from the detailed description, claims, and drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.

Drawings

The present disclosure will become more fully understood from the detailed description and the accompanying drawings.

FIG. 1 is a functional block diagram of an exemplary embodiment of a system for a large pharmacy.

FIG. 2 is a functional block diagram of an exemplary pharmacy fulfillment device that may be deployed within the system of FIG. 1, according to an exemplary embodiment.

FIG. 3 is a functional block diagram of an exemplary order processing device that may be deployed within the system of FIG. 1, according to an exemplary embodiment.

FIG. 4 is a graphical representation of a side view of a hypothetical dispensing system and a functional block diagram.

Fig. 5A is an exemplary image prior to preprocessing according to the principles of the present disclosure.

Fig. 5B is an exemplary image after applying image pre-processing according to the principles of the present disclosure to the image of fig. 5A.

Fig. 6 is a graphical example of a tile-by-tile analysis of an image.

FIG. 7 is a plot of exemplary data acquired by analyzing an image patch by patch.

Fig. 8 is a functional block diagram of exemplary processing hardware according to the principles of the present disclosure.

Fig. 9 is a functional block diagram of an exemplary pre-processing module according to the principles of the present disclosure.

FIG. 10 is a flow chart of an advanced pharmacy process.

11A and 11B together form a flow chart of an exemplary pre-processing operation in accordance with the principles of the present disclosure.

Fig. 12 is a functional block diagram of a machine in the exemplary form of a computer system within which a set of processor-executable instructions, for causing the machine to perform the methodologies discussed herein, may be executed or stored.

In the drawings, reference numbers may be reused to identify similar and/or identical elements.

Detailed Description

Introduction to the design reside in

Exemplary systems and methods for computer imaging pre-processing are described for automated medication dispensing analysis, for example, in a pharmacy. Typically, prescription orders are generated for large pharmacies. A prescription order may include more than one prescription drug for fulfillment. Each prescription drug in the prescription order is an order component of the prescription order. Typically, the ordering component is a medicine bottle, a liquid bottle, a blister pack (blister pack), a unit of use pack, an injectable pack, a spray bottle, a test tube, an ampoule, a drop, an insulated box, a child resistant container, or other package containing a quantity of a prescribed medicine therein.

While the medicaments described in this disclosure will be referred to as prescription medicaments, the principles of this disclosure are also applicable to over-the-counter medicaments. Further, while various receptacles are capable of receiving prescription drugs, the present disclosure will focus on an exemplary container that is a cylindrical bottle. Generally, dispensing equipment dispenses prescription drugs according to a prescription order into vials (sometimes called vials). To ensure that the correct medication is dispensed into the vial, including not containing the accidental prescription medication (even a single), a photograph is taken from above the vial. The photographs are processed to determine the confidence that the prescription drug in the bottle is in line with the prescription order and does not contain any accidental prescription drugs.

Many factors affect image quality, such as lighting, the position of the camera relative to the vial, the angle of the camera relative to the vial, the cleanliness of the camera lens, and the like. Furthermore, the hardware setup may be different on different distribution lines and different distribution facilities. In addition, each image typically includes some area outside the bottle, or if the imaging area is reduced, portions of the bottle itself may be omitted in the image. Pre-processing may be performed to identify the edges of the vial before automatic image analysis can be performed. Once the edge of the bottle is identified, the image analysis may ignore the area outside the bottle. In addition, pre-processing may exclude some or all of the vials, so that image processing can only focus on prescription drugs.

The bottle can be displayed with the darkest pixels, typically, by enhancing contrast or performing other color processing. After the darkest pixels in the image are identified, the image is analyzed column by column. Knowing that the bottle will be circular or at least elliptical, the first and second thirds of the image are analyzed to attempt to identify the edges of the bottle. The left third of the image is analyzed to identify a possible left edge of the vial, and the right third is analyzed to identify a possible right edge of the vial.

The maximum number of dark pixels per column will appear at the edge of the bottle. Thus, the maximum number of dark pixels establishes the edge of the bottle. If the maximum number of pixels in the first and second thirds are very different, the edge with the low peak number of pixels may actually be outside the boundaries of the image. No edges in the image explain why the number of peaks of dark pixels is so low.

After analyzing the images column by column, the analysis switches to establishing the top and bottom edges of the vial row by row. After the four edges are defined, an ellipse is calculated based on the positions of the edges. All outer parts of the ellipse may be excluded by further analysis. Applying such pre-processing can significantly reduce the number of false positives in which machine vision systems deem unnecessary prescription drugs to have been dispensed into bottles. The pre-treatment may also reduce unrecognized false positives where there is an unwanted prescription drug in the vial.

High level block diagram

FIG. 1 is a block diagram of an exemplary embodiment of a system 100 for a large pharmacy according to an exemplary embodiment. Although system 100 is generally described as being deployed in a large pharmacy or fulfillment center (e.g., a mail order pharmacy, a direct delivery pharmacy, etc.), system 100 and/or its components may be deployed in other ways (e.g., in a small pharmacy, etc.). A large pharmacy may be one that is capable of filling prescriptions automatically, mechanically, manually, or a combination thereof. The system 100 may include a welfare manager device 102, a pharmacy device 106, and a user device 108 in communication with each other directly and/or through a network 104. The system may also include a storage device 110.

The welfare manager device 102 is a device operated by an entity at least partially responsible for creating and/or managing a pharmacy or drug welfare. Although such entity operating the welfare manager device 102 is typically a pharmacy welfare manager (PBM), other entities may operate the welfare manager device 102 on their own (i.e., PBM) or other entities. For example, the welfare manager device 102 may be operated by a health plan, a retail pharmacy chain, a drug wholesaler, a data analysis or other type of software-related company, and so forth. In some embodiments, a PBM providing pharmacy benefits may also provide one or more other benefits, including medical or health benefits, dental benefits, vision benefits, health benefits, radiology benefits, pet care benefits, insurance benefits, long-term care benefits, nursing home benefits, and the like. In addition to its PBM operation, the PBM may also operate one or more pharmacies. The pharmacy may be a retail pharmacy, a mail order pharmacy, or the like.

Certain operations of operating the PBM of the welfare manager device 102 may include the following activities and procedures. A member of a pharmacy benefit plan (or a person representative of a member) may obtain a prescribed medication from a pharmacist or an assistant pharmacist at a retail pharmacy (e.g., a brick and mortar store, etc.). The member may also obtain prescription drugs by mail order delivery from a mail order pharmacy, which may be the system 100. In some embodiments, the member may obtain the prescribed medication directly or indirectly through the use of a machine, such as a kiosk, vending machine, mobile electronic device, or other type of mechanical, electrical, electronic communication device, and/or computing device. Such machines may be filled with prescription drugs by the system 100 in prescription packages that may include a plurality of prescription components. The pharmacy benefit plan is managed by or through the benefit manager device 102.

The member may have a share of the prescription drug that reflects the member's responsibility for paying the amount of the pharmacy for the prescription drug. The money paid by the member to the pharmacy may come from the member's personal funds, the member's home Health Savings Account (HSA), the member's home Health Reimbursement Arrangement (HRA), the member's home flexible consumption account (FSA), etc. In some cases, the member's employer may pay or reimburse the share directly or indirectly for the member.

The amount of the share required by the member may vary from one pharmacy benefit plan to another with different plan initiators or customers and/or for different prescribed medications. The share of the affiliate may be a fixed share (e.g., $ 10, etc.), a common insurance (e.g., 10%, etc.), and/or a self-payment (e.g., $ 500 top annual prescription charge, etc.) for some prescription drugs, some types and/or categories of prescription drugs, and/or all prescription drugs. The share amount may be stored in storage 110 or determined by benefits manager device 102.

In some cases, the member may pay no share, or only a portion of the share of the prescribed medication. For example, if the general cost of a common version of a prescription drug is $ 4 and the fixed share for the prescription drug is $ 20, a member may only pay $ 4 to receive the prescription drug. In another example involving worker compensation claims, a member may not need to pay for a share of a prescription drug.

In addition, the share amount may also vary based on different delivery channels for the prescribed drugs to be received. For example, the share of receiving a prescription drug from a mail order pharmacy may be less than the share of receiving a prescription drug from a retail pharmacy.

After receiving the share (if any) from the member and distributing the prescribed medication to the member, the pharmacy claims to the PBM for the prescribed medication. After receiving the claims, the PBM (e.g., the welfare manager device 102) may perform certain sanction operations, including verifying membership of the member, identifying/reviewing formulas applicable to the member to determine any appropriate share amounts, co-insurance and prescription drug deductible amounts, and conducting Drug Use Reviews (DUR) on the member. Further, the PBM may provide a response to the pharmacy (e.g., pharmacy system 100) after performing at least some of the foregoing operations.

As part of the sanction, when the prescription drug is successfully sanctioned, the plan initiator (or PBM on behalf of the plan initiator) eventually reimburses for the pharmacy to fill the prescription drug.

The foregoing arbitration operations typically occur prior to receiving the share and dispensing the prescribed medication. However, in some cases, these operations may occur simultaneously, substantially simultaneously, or in a different order. In addition, more or less arbitration operations may be performed as at least part of the arbitration process.

The reimbursement amount paid by the intended initiator to the pharmacy and/or the amount of money paid by the member may be determined based at least in part on the type of pharmacy network that includes the pharmacy. In some embodiments, the amount may also be determined based on other factors. For example, if a member pays a pharmacy for a prescription drug without using a prescription or drug welfare provided by the PBM, the amount paid by the member may be higher than if the member used the prescription or drug welfare. In some embodiments, the amount received by the pharmacy for dispensing the prescription drug and the prescription drug itself may be higher than the amount received by the member for using the prescription or drug welfare. Some or all of the foregoing operations may be performed by executing instructions stored in welfare manager device 102 and/or an additional device.

Examples of network 104 include a mobile communications (GSM) network, a Code Division Multiple Access (CDMA) network, a third generation partnership project (3GPP), an Internet Protocol (IP) network, a Wireless Application Protocol (WAP) network, a WiFi network, or an IEEE 802.11 standard network, as well as various combinations thereof. Network 104 may comprise an optical network. The network 104 may be a local area network or a global communication network, such as the internet. In some embodiments, the network 104 may include a network dedicated to prescription orders, for example, a prescription network, such as an electronic prescription network operated by Surescripts of Arlington, Va.

Further, although the system shows a single network 104, multiple networks may be used. Multiple networks may communicate with each other in series to link the devices 102 and 110, or may communicate in parallel to link the devices 102 and 110.

The pharmacy device 106 may be a device associated with a retail pharmacy (e.g., an exclusive pharmacy, a grocery store with a retail pharmacy, or a general sales store with a retail pharmacy) or other type of pharmacy where a member attempts to obtain a prescription. The pharmacy may submit the claim to the PBM for resolution using the pharmacy device 106.

Additionally, in some embodiments, the pharmacy device 106 may enable information exchange between the pharmacy and the PBM, for example, to allow sharing of membership information such as medication history, which may allow the pharmacy to better serve the membership (e.g., by providing more informed treatment consultation and medication interaction information, etc.). In some embodiments, the welfare manager device 102 may track their prescribed medication fulfillment and/or other information when (or in conjunction with) a non-member or patient who has not yet identified himself as a member attempts to have a prescription filled by a pharmacy.

The pharmacy device 106 may include an order processing device 114, a pharmacy management device 116, and a pharmacy fulfillment device 112 that communicate with each other directly and/or over the network 104.

The order processing device 114 may receive information regarding filling the prescription and may direct the order components to one or more devices of the pharmacy fulfillment device 112 in the pharmacy. The pharmacy fulfillment device 112 may fulfill, distribute, aggregate, and/or package the order components of the prescription drug in accordance with one or more prescription orders indicated by the order processing device 114. The order processing device 114 may be deployed in the system 100 or may be otherwise used.

Typically, the order processing device 114 is a device located within or associated with a pharmacy to enable fulfillment of the prescription and dispensing of the prescribed medication by the pharmacy fulfillment device 112. In some embodiments, the order processing device 114 may be an external order processing device that is separate from the pharmacy and communicates with other devices within the pharmacy.

For example, the external order processing device may communicate with an internal pharmacy order processing device and/or other devices located within the system 100. In some embodiments, the external order processing device may have limited functionality (e.g., operated by a patient requesting fulfillment of a prescribed medication) while the internal pharmacy order processing device may have greater functionality (e.g., operated by a pharmacist).

The order processing device 114 may track prescription orders fulfilled by the pharmacy fulfillment device 112. The prescription order may include one or more prescription drugs to be filled by the pharmacy. The order processing device 114 may make pharmacy routing decisions and/or order consolidation decisions for a particular prescription order. The pharmacy routing decision includes which devices in the pharmacy are responsible for filling or otherwise processing certain portions of the prescription order. The order consolidation decision includes whether or not portions of a prescription order or prescription orders should be shipped together to the patient or patient's family. Order processing device 114 may also track and/or arrange graphics (literary) or paperwork (paperwork) associated with each prescription order or multiple prescription orders shipped together. In some embodiments, the order-processing device 114 may operate in conjunction with the pharmacy management device 116.

The order processing device 114 may include circuitry, a processor, memory for storing data and instructions, and communication functionality. The order processing device 114 is dedicated to performing the processes, methods, and/or instructions described herein. Other types of electronic devices specifically configured to implement the processes, methods, and/or instructions described herein may also be used.

In some embodiments, at least some of the functionality of the order-processing device 114 may be included in the pharmacy management device 116. The order-processing device 114 may be in a client-server relationship with the pharmacy management device 116, in a peer-to-peer relationship with the pharmacy management device 116, or in a different type of relationship with the pharmacy management device 116. The order processing device 114 and/or the pharmacy management device 116 may communicate with the storage device 110 directly (e.g., by utilizing local storage, etc.) and/or through the network 104 (e.g., by utilizing cloud configuration or software as a service, etc.).

The user device 108 is used by the device operator. The device operator may be a user (e.g., employee, contractor, benefit member, etc.) associated with the software development project. Other device operators may also operate the user device 108.

Examples of user devices 108 include set-top boxes (STBs), receiving cards, mobile phones, Personal Digital Assistants (PDAs), display devices, portable gaming machines, and computing systems, among others.

The storage device 110 may include: non-transitory storage (e.g., memory, hard drive, CD-ROM, etc.) in communication with the welfare manager device 102, the pharmacy device 106, and/or the user device 108, directly and/or over the network 104. The non-transitory storage may store order data 118, membership data 120, claims data 122, drug data 124, prescription data 126, and/or plan initiator data 128. In addition, the system 100 may include other devices that may communicate with each other directly or through the network 104.

Order data 118 may relate to prescription orders. The order data may include the type of prescription drug (e.g., drug name and strength, etc.) and the quantity of prescription drug, etc. The order data 118 may also include data for completing a prescription, such as prescription materials. Typically, the prescription materials include an electronic copy of information relating to the prescription drug, to contain or otherwise be associated with the fulfilled prescription. The prescription materials may include electronic information regarding medication interaction warnings, recommended usage, possible side effects, expiration dates, drug dates, and the like. The order data 118 may be used by a large fulfillment center to fulfill a pharmacy order.

In some embodiments, the order data 118 includes verification information associated with fulfillment of a prescription in a pharmacy. For example, the order data 118 may include the following videos and/or images: (i) prescription drug video and/or images taken before, during, and/or after dispensing; (ii) video and/or images of prescription containers (e.g., prescription vials and caps, prescription packages, etc.) for holding prescription drugs taken before, during, and/or after dispensing; (iii) video and/or images of packaging and/or packaging material taken before, during, and/or after dispensing for shipping or otherwise delivering the prescribed medication; and/or (iv) video and/or images of the fulfillment process within the pharmacy. Other types of verification information, such as bar code data read from pallets, boxes, trays, carts, etc. used to transport prescriptions within a pharmacy, may also be stored as order data 118.

The member data 120 includes information about members associated with the PBM. The information stored as member data 120 may include personal information, personal health information, protected health information, and the like. Examples of member data 120 include name, address, phone number, email address, prescription history, and the like. The affiliate data 120 can include a program originator identifier that identifies a program originator associated with the affiliate and/or an affiliate identifier that identifies the affiliate to the program originator. Affiliate data 120 can include an affiliate identifier that identifies a program originator associated with a patient and/or a patient identifier that identifies the patient to the program originator. Member data 120 can also include, for example, distribution preferences such as tag type, lid type, message preferences, language preferences, and the like.

The member data 120 may be accessed by various devices in the pharmacy (e.g., a large fulfillment center, etc.) to obtain information for fulfilling and shipping prescription orders. In some embodiments, at least a portion of the member data 120 may be accessed by or on behalf of the member for review, verification, or the like, or other purposes.

In some embodiments, the member data 120 may include information about people who are patients of the pharmacy but who are not members in the pharmacy welfare plan provided by the PBM. For example, these patients may obtain medications directly from the pharmacy through a private label service provided by the pharmacy, large fulfillment center, or otherwise. Generally, the terms member and patient are used interchangeably herein.

The claims data 122 includes information regarding pharmacy claims for one or more plan initiators that are adjudicated by the PBM under the medication welfare plan provided by the PBM. Typically, the claims data 122 includes identification of the customer who initiated the drug welfare plan for which the claim was made, and/or identification of the member who purchased the prescription drug that caused the claim, identification of the prescription drug filled by the pharmacy (e.g., national drug code, etc.), dispense date, general purpose index, GPI number, drug category, cost of the prescription drug provided under the drug welfare plan, share/co-insurance amount, rebate information, and/or membership, etc. Other information may also be included.

In some embodiments, other types of claims besides prescription drug claims may be stored in the claim data 122. For example, a member's medical claims, dental claims, health claims, or other types of health-related claims can be stored as part of the claims data 122.

In some embodiments, the claims data 122 includes claims that identify members associated with the claim. In some embodiments, the claim data 122 includes claims that have been de-identified (e.g., associated with a unique identifier but not a particular identifiable affiliate, etc.).

The drug data 124 may include a drug name (e.g., technical and/or generic name, etc.), other names used to know the drug, an active ingredient, an image of the drug (e.g., pill form), etc. The drug data 124 may include information related to a single drug or multiple drugs.

Prescription data 126 may include information about a prescription that may be filled by a prescriber on behalf of a patient, which may be a member of a pharmacy welfare plan, such as by a pharmacy. Examples of prescription data 126 include patient name, medication or treatment (e.g., laboratory examinations), dosage information, and the like. The prescription may be an electronic prescription, a scanned paper prescription, or otherwise. In some embodiments, the dosage information reflects frequency of use (e.g., once per day, twice per day, before each meal, etc.) and time of use (e.g., days, weeks, months, etc.).

In some embodiments, the order data 118 may be linked to associated member data 120, claims data 122, drug data 124, and/or prescription data 126.

The plan initiator data 128 includes information about the plan initiator of the PBM. Examples of plan initiator data 128 include company name, company address, contact name, contact phone number, contact email address, and the like.

Figure 2 illustrates a pharmacy fulfillment device 112, according to an exemplary embodiment. The pharmacy fulfillment device 112 may be used to process and fulfill prescriptions and prescription orders. After fulfillment, the fulfilled prescription is packaged for shipment.

The pharmacy fulfillment device 112 may include a device in communication with the welfare manager device 102, the order processing device 114, and/or the storage device 110, either directly or through the network 104. In particular, the pharmacy fulfillment device 112 may include a tray sizing and positioning (packaging) device 206, a loading device 208, an inspection device 210, a use unit device 212, an automatic dispensing device 214, a manual fulfillment device 216, an audit device 218, an imaging device 220, a capping device 222, an accumulation device 224, a packing device 226, a graphics device 228, a use unit packing device 230, and a mailing list device 232. In addition, the pharmacy fulfillment device 112 may include other devices that may communicate with each other directly or through the network 104.

In some embodiments, the operations performed by one of these devices 206 and 232 may be performed sequentially or in parallel with the operations of the other device, which may be coordinated by the order processing device 114. In some embodiments, the order processing device 114 tracks the prescription for the pharmacy based on the operations performed by the one or more devices 206 and 232.

In some embodiments, the pharmacy fulfillment device 112 may transport the prescription drug containers between multiple devices 206 and 232 at a large fulfillment center through the use of a tray, for example. The tray sizing and positioning device 206 may deploy a locator (puck) in the tray. The tray may be a shipping structure for a plurality of prescription containers and may include a plurality of cavities. Locators may be placed in one or more cavities in the tray by the tray sizing and positioning device 206. The positioner may include a receptacle sized and shaped to receive the prescription container. Such containers may be supported by the positioner during transport in the pallet. Different positioners may have receptacles of different sizes and shapes to accommodate different sized containers, which may be suitable for different prescriptions.

The placement of the locators in the tray may be determined by the order processing device 114 based on the prescription that the order processing device 114 decides to place. The placement logic may be implemented directly in the tray sizing and positioning device 206. After the prescription to be dispensed has been set, the locator of the appropriate size container appropriate for the prescription can be placed in the tray by a robotic arm or picker. After the positioner is configured in the tray, the tray sizing and positioning device 206 may activate the tray.

The loading device 208 may load prescription containers into the locators on the pallet via a robotic arm, pick and place mechanism, or the like. In one embodiment, the loading device 208 has a robotic arm or picker to grasp and move prescription containers into and out of the tray or positioner. The loading device 208 may also print labels appropriate for the containers to be loaded onto the tray and apply the labels to the containers. During these operations, the trays may be located on the conveyor assembly (e.g., at a large fulfillment center, etc.).

The inspection device 210 can verify that the containers in the tray are properly marked and properly positioned on the tray. Inspection device 210 may scan the labels on one or more containers on the pallet. The label of the container may be scanned or imaged, in whole or in part, by the inspection device 210. Such imaging may be performed after the container is lifted out of its positioner by a robotic arm, picker, or the like, or may be scanned or imaged while it remains in the positioner. In some embodiments, images and/or videos taken by inspection device 210 may be stored as order data 118 in storage device 110.

The use unit device 212 may temporarily store, monitor, mark, and/or dispense the use unit product. Typically, the use unit product is a prescription drug product that can be delivered to a patient or member without repackaging at the pharmacy. These products may include pills in containers, pills in blister packs, inhalers, and the like. The prescription drug products dispensed by the use unit device 212 may be individually or collectively packaged for shipment, or may be shipped in combination with other prescription drugs dispensed by other devices of a large fulfillment center.

At least some of the operations of the devices 206 and 232 may be indicated by the order processing device 114. For example, the manual fulfillment device 216, the review device 218, the automatic dispensing device 214, and/or the packaging device 226, among others, may receive instructions provided by the order processing device 114.

The automated dispensing device 214 may include one or more devices that dispense prescription drugs or medicines into prescription containers according to one or more prescription orders. Generally, the automatic dispensing device 214 may include mechanical and electronic components, in some embodiments with software and/or logic to facilitate the dispensing of medication that would otherwise be performed manually by a pharmacist and/or an assisting pharmacist. For example, the automated dispensing device 214 may include a large filler that quickly fills a variety of prescription drug types and a patch packaging machine that dispenses and packages the drug into the patch. The prescription drugs dispensed by the automated dispensing device 214 may be individually or collectively packaged for shipment, or may be shipped in combination with other prescription drugs dispensed by other devices of a large fulfillment center.

The manual fulfillment device 216 may provide for manual fulfillment of the prescription. For example, the manual fulfillment device 216 may receive or obtain a container and enable a pharmacist or an assistant pharmacist to fulfill the container. In some embodiments, the manual fulfillment device 216 provides the filled container to another device in the pharmacy fulfillment device 112 for engagement with other containers in the patient's or member's prescription order. Typically, manual fulfillment may include operations performed at least in part by a pharmacist or an assistant pharmacist. For example, a person may retrieve a supply of prescription drugs, may make observations, may count the amount of prescription drugs and place them in prescription containers, and so forth. Some portions of the manual fulfillment process may be automated through the use of machinery. For example, counting of capsules, tablets, or pills may be at least partially automated (e.g., by using a pill counter, etc.). The prescription drugs dispensed by manual fulfillment devices 216 may be individually or collectively packaged for shipment, or may be shipped in combination with other prescription drugs dispensed by other devices of a large fulfillment center.

The inspection device 218 may process the prescription containers to be reviewed by the pharmacist for appropriate pill counting, exception handling, prescription verification, and the like. Such as state or local legal requirements, fulfilled prescriptions may be manually reviewed and/or verified by a pharmacist. A pharmacist or other licensed pharmacy personnel who may dispense certain medications according to local and/or other laws may operate the review device 218 and visually inspect the prescription containers already containing the prescribed medication. The pharmacist may review, verify and/or evaluate the drug quantity, drug strength and/or drug interaction issues, or otherwise perform pharmacist services. The pharmacist may also handle containers that have been marked as exceptions, such as containers with unreadable labels, containers that have cancelled an associated prescription order, containers with defects, and the like. In an example, the manual review may be performed at a manual station.

After the container has been filled with the drug, the imaging device 220 may image the container. The imaging device 220 may measure the fill level of the drug in the container based on the obtained images to determine if the container is filled to the correct height given the type of drug and the number of pills in the prescription. Images of the pills in the container may also be acquired to detect the size of the pills themselves and the indicia thereon. The image may be transmitted to the order processing device 114 and/or stored as part of the order data 118 in the storage device 110.

The capping device 222 may be used to cap or otherwise seal prescription containers. In some embodiments, the capping device 222 may secure the prescription container with one type of cap depending on the patient's preferences (e.g., preferences regarding child resistance, etc.), the plan originator's preferences, the prescriber's preferences, etc. The capping device 222 may also etch messages into the caps, but the process may also be performed by subsequent equipment in a large fulfillment center.

The accumulation mechanism 224 accumulates the various containers of prescription drugs in the prescription order. The accumulation device 224 may accumulate prescription containers from various devices or areas of the pharmacy. For example, the accumulation mechanism 224 may accumulate prescription containers from the usage unit mechanism 212, the automatic dispensing mechanism 214, the manual fulfillment mechanism 216, and the audit mechanism 218 of a large fulfillment center. The accumulation mechanism 224 may be used to group prescription containers prior to shipping them to members or the like.

The graphics device 228 prints or otherwise generates graphics to include the order for the prescription drug. Graphics may be printed on a plurality of substrates such as paper, coated paper, printable polymers, or combinations thereof. The graphics printed by graphics device 228 may include information necessary to accompany the prescription drugs included in the prescription order, information related to the prescription drugs in the order, financial information related to the order (e.g., invoices or statements, etc.).

In some embodiments, the graphics device 228 folds or otherwise prepares graphics to include prescription medication orders (e.g., in shipping containers, etc.). In some embodiments, the graphic-to-text device 228 may be separate from the graphic-to-text device that prepares the graphic to include the prescription order.

The packaging device 226 packages the prescription order to prepare the shipping order. The packaging device 226 may case, bag, or otherwise package the filled prescription order for delivery. The packaging device 226 may further place an insert (e.g., a graphic or other paper, etc.) in a package received from a graphic device 228, etc. For example, bulk prescription orders may be packed in boxes, while other prescription orders may be packed in bags, which may be sealed bags of packaging.

The packaging device 226 may sort the boxes or bags for mailing in an efficient manner (e.g., sorted by shipping address, etc.).

The unit order is packaged using the unit packaging device 230 to prepare the shipping order. Using the unit packing apparatus 230 may include manually scanning the containers to be bagged for shipment to verify each container in the order. In an exemplary embodiment, the manual scan may be performed at a manual station. The pharmacy fulfillment device 112 may also include a mailing list device 232 to print mailing labels used by the packaging device 226 and may print shipping lists and packaging lists.

Although the pharmacy fulfillment device 112 in FIG. 2 is shown as including a single device 206 and 232, multiple devices may be used. The devices 206 and 232 may be the same type or model of device or may be different types or models of devices. When a plurality of devices are present, the plurality of devices may have the same device type or model, or may have different device types or models. The type of devices 206 and 232 shown in FIG. 2 are exemplary devices. In other configurations of the system 100, fewer, additional, or different types of devices may be included.

Further, multiple devices may share processing and/or memory resources. The devices 206 and 232 may be located in the same area or in different locations. For example, the device 206 and 232 may be located in a building or a group of adjacent buildings. Devices 206-232 may be interconnected (e.g., by conveyors, etc.), networked, and/or otherwise in contact with each other or integrated with each other (e.g., at a large fulfillment center, etc.). In addition, the functionality of a device may be divided between multiple discrete devices and/or combined with other devices.

FIG. 3 illustrates an order processing apparatus 114 according to an exemplary embodiment. The order processing device 114 may be used by one or more operators to generate prescription orders, make routing decisions, make prescription order consolidation decisions, use the system 100 to track graphics and/or view order status and other order related information. For example, a prescription order may include order components.

The order processing device 114 may receive instructions to fulfill the order without operator intervention. The order components may include prescription drugs that are fulfilled by the system 100 through the use of containers. The order processing device 114 may include an order verification subsystem 302, an order control subsystem 304, and/or an order tracking subsystem 306. Other subsystems may also be included in order processing device 114.

Order verification subsystem 302 may communicate with welfare manager device 102 to verify membership and review recipes to determine appropriate co-loads, co-insurance and self-payments for prescribed drugs and/or to execute DURs. Other communications between the order verification subsystem 302 and the welfare manager device 102 may be performed for various purposes.

The order control subsystem 304 controls various movements of the containers and/or trays, as well as various filling functions, during progression through the system 100. In some embodiments, the order control subsystem 304 may identify prescription drugs in one or more prescription orders that can be fulfilled by the automated dispensing apparatus 214. Order control subsystem 304 may determine which prescriptions are to be filled and may determine the trays of automatically filled containers to be filled.

The order control subsystem 304 may determine that an auto fill prescription for a particular drug is to be filled and may check an order queue waiting to fulfill other prescriptions that are to be filled with the same drug. The order control subsystem 304 may then send orders with similar automatic fill drug requirements together in a tray to the automated dispensing equipment 214. Since the devices 206-232 may be interconnected by a conveyor system or other container movement system, the order control subsystem 304 may control the various conveyors to transport the trays from the loading device 208 to the manual fulfillment device 216, such as from the teletext device 228 to transport paperwork as needed to fill the prescription.

The order tracking subsystem 306 may track prescription orders as fulfillment (or cessation) proceeds. The order tracking subsystem 306 may track, record, and/or update order history, order status, and the like. The order tracking subsystem 306 may store the data locally (e.g., in memory, etc.) or as part of the order data 118 stored in the storage 110.

Analytical device

In fig. 4, a dispensing device 404, such as an automated dispensing device 214, dispenses a medicament into a vial 408. For illustrative purposes only, the bottles 408 are positioned on a conveyor 412. The bottle 416 has been moved by the conveyor 412 from the dispensing apparatus 404 to a position where the camera 420 can take a picture of the bottle 416. The camera 420 may be implemented by the imaging device 220. In various implementations, the conveyor 412 may stop in order to take a picture. In other embodiments, the camera 420 may take a picture when the vial 416 is positioned substantially below the camera 420.

Processing hardware 424 analyzes the photographs to assess whether the correct medication is being dispensed into vial 416 and to determine the likelihood of including an unwanted medication in vial 416. If the likelihood of including an unwanted medication or a completely mismatching medication is sufficiently high, the processing hardware 424 may alert the operator interface 428. For example, operator interface 428 may include a light, a buzzer, or an electronic notification (e.g., a text message).

The packaging device 432 performs a variety of tasks, including capping the bottles 416 and placing the bottles 416 into a package that can hold a plurality of bottles. For example, the box may contain a large number of bottles and then be shipped to a distributor. In other embodiments, packaging device 432 packages bottle 416 with a small number of other bottles into a single order for delivery to a customer. If the processing hardware 424 determines that the vial 416 may have one or more unwanted drugs, the flow diverter 436 may divert the vial 416. Diverter 436 may move bottle 416 aside for human observation or may discard bottle 416. The dispensing apparatus 404 may then fill another bottle after the bottle 416 is transferred, or may await further input from a manual operator.

Example analysis

In fig. 5A, an example image of a bottle taken from a planar perspective, for example, by the camera 420 of fig. 4, is shown. Note that the right side of the bottle is cut away and the plastic ribs, such as at 450 and 452, associated with the bottle are shown. After performing pre-processing in accordance with the principles of the present disclosure, an image such as that shown in FIG. 5B may result. Most of the bottle has been removed and the ribs 450, 452 have been shortened to stub portions (stubs) 460, 462. This allows the image processing to focus on the drug in the vial and avoid false positives due to ribs and other components of the vial.

In fig. 6, a simplified graphical representation of image pre-processing is shown. First, image pixels that may correspond to bottles are identified. These may be referred to as candidate pixels. As described in more detail below, the darkest 25% of the pixels in the image may be selected as candidate pixels. In fig. 6, the candidate pixels are shown and form a partial ring. In a real image, it is unlikely that the candidate pixels will form such a sharply defined ring, and indeed, it is expected that some candidate pixels will not even be contiguous.

Assuming that the first third of the image along the horizontal axis will include the left side of the bottle and the second third of the image along the horizontal axis will include the right side of the bottle, each column of the image is analyzed separately to calculate how many candidate pixels are present in each column.

When the candidate pixels form a sharply defined ring, the columns at the inner edge of the ring will have the largest number of pixels in the first third of the image, as shown in FIG. 6. This column is indicated approximately by reference numeral 480. Whenever you leave column 480, the number of candidate pixels in each column is reduced.

In fig. 7, the graph shows the count of the number of candidate pixels on the Y-axis, where the X-axis is the horizontal coordinate (column number) of the image. In fig. 7, peak level 484 is determined for counts within the left third of the image, and peak level 488 is determined for counts within the right third of the image. If the difference between peak levels 484 and 488 is too large, it may indicate that the side of the image with the lower peak level does not include the edge of the bottle. For example, looking at FIG. 6, the right side of the bottle is missing, so the peak count for each column of candidate pixels in the right third of the image is never as high as in the left third of the image.

Processing hardware

In fig. 8, a functional block diagram of the processing hardware 500 includes a pre-processing module 504, the pre-processing module 504 receiving images from the camera 420 and providing processed images to an assignment analysis module 508. The dispensing analysis module determines whether there is a possibility of misdispensing the medication and reports the possibility to the quality analysis module 512 and to the remedial action module 516. Quality analysis 512 keeps track of apparent dispensing errors on the fly. The remedial action module 516 may activate the operator interface 428 and/or the diverter 436.

In fig. 9, a functional block diagram of an example implementation of the pre-processing module 504 includes a color adjustment module 604. The color adjustment module 604 receives a copy of the image as does the output module 608. The color adjustment module 604 adjusts the color of the image to make the bottle of pills itself easier to recognize. For example, the color adjustment module 604 may increase the contrast of the image. In various embodiments, the color adjustment module may set the red, green, and blue values of each pixel according to the formulas described below.

The pixel identification module 612 identifies pixels that may be part of a bottle based on the color adjusted image. The multi-axis scan module 616 outputs counts of these possible pixels along multiple axes. For example, the count of possible pixels for each column of pixels is provided to the first maximum detection module 620, while the count of possible pixels in each row of the image is provided to the second maximum detection module 624.

The first maximum detection module 620 may determine a maximum value within a first region of the image and a maximum value within a second region of the image. These values may be referred to as h1 and h 3. For example, when the first maximum detection module 620 analyzes the horizontal axis, the first region may be the left-hand third of the image and the second region may be the right-hand third of the image. Thus, the first maximum detection module 620 does not analyze the remaining middle third of the image.

The second maximum detection module 624 may determine a maximum within the first region of the image and a maximum within the second region of the image. These values may be referred to as v1 and v 3. For example, when the first maximum detection module 620 analyzes the vertical axis, the first region may be the top third of the image and the second region may be the bottom third of the image. Thus, the second maximum detection module 624 does not analyze the remaining middle third of the image.

The ellipse calculation module 628 defines an ellipse based on the location of the maximum detected by the first maximum detection module 620 along the first axis, the first maximum detection module 620 defining endpoints of the first axis of the ellipse. The ellipse is also based on the position of the maximum detected by the second maximum detection module 624 along the second axis, the second maximum detection module 624 defining the endpoints of the second axis of the ellipse. The longer of the two axes of the ellipse is called the major axis, while the shorter of the two axes of the ellipse is called the minor axis.

If the difference between the two maxima along the axis is greater than a predetermined value, it can be assumed that the lower limit of the maxima is not the edge of the bottle, but the edge of the bottle is at the edge of the image. The end points of the respective axes of the ellipses will thus be set as the edges of the image. The predetermined value may be 20% of the larger of the two maximum values, or may be 20% of the size of the image in a direction perpendicular to the axis.

In various embodiments, the ellipse defined by the ellipse calculation module 628 may be a circle, i.e., the major and minor axes of the ellipse are equal in length. For example, the ellipse calculation module 628 may average the major and minor axes of the ellipse to determine the diameter of the circle. In other implementations, the ellipse calculation module 628 may select one of the major or minor axes as the diameter of the circle.

The ellipse calculation module 628 may calculate the boundary of the ellipse as follows:

center of bottle x and y coordinates: [ (h1+ h3)/2, (v1+ v3)/2]

Top center edge (T): [ (h1+ h3)/2, v1]

Bottom central edge (B): [ (h1+ h3)/2, v3]

Right central edge (R): [ h3, (v1+ v3)/2]

Left central edge (L) [ hi, (v1+ v3)/2]

Vial Radius (RAD): max [ (h1+ h3)/2, (v1+ v3)/2]

Upper left edge: [ (h1+ h3)/4, (v1+ v3)/2-sin (pi/4) < > RAD ]

Upper right edge: [ 3] (h1+ h3)/4, (v1+ v3)/2-sin (pi/4) < > RAD ]

Left lower edge: [ (h1+ h3)/4, (v1+ v3)/2+ sin (pi/4) < > RAD ]

Lower right edge: [ 3] (h1+ h3)/4, (v1+ v3)/2+ sin (pi/4) < > RAD ]

The output module 608 uses the computed ellipse to adjust the copy of the original image. The adjusting may include creating metadata identifying pixels that should be ignored in the processing, or may include adjusting pixels to be ignored. For example, the metadata may be embedded in the image file or may be provided in conjunction with the image file. Adjusting the pixels to be ignored may include setting the pixels to be ignored to a predetermined color, such as black. The output module 608 outputs the processed image to the assignment analysis module 508 of fig. 8.

Flow chart

In fig. 10, a flow chart of the overall operation of the quality control system is shown. Control begins at 704, where if medication has been dispensed into the vial at 704, control continues at 708; otherwise, control remains at 704. At 708, control captures an image from above the opened bottle. At 712, control preprocesses the image, for example, by removing regions outside the bottle.

For example, a machine learning (M L) model, such as a Convolutional Neural Network (CNN), is first trained on the pill images, which model captures features such as color, shape, size, and labeling.

At 720, if the likelihood of misdispensing the medication is greater than a predetermined threshold, control transfers to 724; otherwise, control transfers to 728. The trained CNN may output a set of probabilities indicating the likelihood of various pills being present in the bottle from the pre-processed image. For example, a trained CNN may output the 5 most likely pills present in the image and the probabilities that the pills are actually present.

In another embodiment, the CNN may be trained to recognize N pills (as an example, N may be 1500) and output an N-dimensional vector, where each element of the vector represents a likelihood that a corresponding pill is present in the bottle. Based on the prescription controlling the dispensing of the pills, a reference vector may be created for which the prescription drug value is 1.0 (indicating 100% likelihood of existence) and for which the remaining N-1 element values are zero. If the difference between the reference vector and the vector output from the CNN is greater than a predetermined distance threshold, control may determine that the likelihood of misdispensing the medication is sufficiently great to trigger a remedial action.

At 724, control performs remedial action based on the possibility of the mediation of the misdispensing of the medication. For example, remedial action may include notifying an operator, forcing manual inspection of the bottles and/or discarding and restocking the bottles.

Control then continues at 732. At 732, quality statistics are updated based on the identified likelihood of misdispensing the medication and based on any feedback from the manual review. These quality statistics can be used to determine false positives in image processing and to identify problems with the dispensing device. Control then returns to 704. At 728, control caps and packs the bottles for dispensing. Control then continues at 732.

11A and 11B together form a flowchart of example operations of preprocessing in accordance with the principles of the present disclosure. In FIG. 11A, control begins at 800 when an image to be pre-processed is received. At 800, the control adjusts the color of the image copy to increase the contrast. For example, for each pixel, the red component of the pixel may be set equal to the sum of the green and blue values minus the red value, while both the green and blue values are set to zero. In other words, the color of the pixel encoded in the red channel is reduced to a single value. A single value of color, the magnitude of which indicates the brightness of the pixel on a black and white continuum, may also be referred to as a gray scale.

At 804, control identifies pixels that are most likely to be outside of the bottle. For example, all pixels in an image may be ordered by their darkness, and the darkest 25% of the pixels may be identified as possible outer pixels. At 808, control selects a first axis. For example, as shown in fig. 6, the first axis may be a horizontal axis. In other embodiments, the first axis is a vertical axis or an axis that is not perpendicular to either the horizontal or vertical axis. At 812, control selects the first third of the image along the selected axis. For example, when the selected axis is a horizontal axis, the first third of the image may be defined as the left-hand third of the image.

At 816, control resets a variable named Max to 0, at 820, within the selected third of the image, control selects the first row of pixels perpendicular to the selected axis, e.g., if the selected third is the left-hand third of the image, the first row of pixels is the left-most column of pixels, at 824, control counts the number of possible outside pixels in the selected row, at 828, control transfers to 832 if the count is greater than the current value of Max, otherwise, control transfers to 832, otherwise, control transfers to 836, because Max is reset to 0 at 816, the first count will become the next Max, at 832, a Max variable is set equal to the count, and a variable Max L oc (the position corresponding to the maximum) is set equal to the number of selected rows, then control continues at 836.

At 836, if there are additional lines in the selected one-third of the image, control transfers to 840, otherwise control transfers to 844, at 840 control selects the next line of pixels and continues at 824, at 844, if the last third of the image has been selected, control transfers to 848, otherwise control transfers to 852, at 852, the last third of the image has not been selected, which means the first third of the image has been selected, therefore, the value of Max L oc is recorded as the focus location for the first third of the image, the value of Max is recorded as the peak within the first third of the image 856.

At 848, the last third of the image has been selected, so the value of Max L oc is recorded as the location of interest for the last third of the image, and the value of Max is recorded as the peak value for the last third of the image control then continues at 860 in FIG. 11B control determines whether the peak value for the last third of the image is less than 80% of the peak value for the first third of the image at 860 if so control transfers to 864 to handle the difference, otherwise control transfers to 864 control transfers to 868 at 864 the relatively smaller value of the last third peak indicates that there is no edge of the vial in the last third of the image, so the edge of the vial is assumed to be at the extreme end of the last third, and then the last third is set to the end of the last third of the image 872 control then continues at 872.

At 868, control determines whether the peak value of the first third of the image is less than 80% of the peak value of the second third of the image. If so, control transfers to 876 to process the difference; otherwise, control transfers to 872. At 876, the relatively small of the first third peak indicates that the edge of the bottle is outside the first third of the image, and thus, the focus position of the first third is set to the very beginning of the first third of the image. Control then continues at 872.

At 872, if the second axis is selected, control transfers to 880; otherwise, control transfers to 884. At 884, the second axis has not been selected, and thus, the first and second third positions are recorded relative to the first axis. These positions correspond to defined edges of the bottle along the first axis. Control continues at 888 where a second axis (e.g., a vertical axis) is selected at 888. Control then returns to 812 in FIG. 11A.

At 880, the second axis has been selected, and thus, the first and second third positions of the second axis are recorded. These values correspond to the bottle rim determined along the second axis. Control continues at 892 where control determines a mathematically defined ellipse having a first diameter along the first axis from a first third position to a second third position determined for the first axis. The ellipse also has a second diameter along the second axis from a front one-third position to a rear one-third position determined for the second axis. Control continues at 896 where a mathematically defined ellipse is used to mask regions of the original image that are to be excluded from further processing at 896. For example, all pixels outside the defined ellipse within the original image may be set to black. Control then ends.

Prototype machine

Fig. 12 illustrates a block diagram of a specific-purpose machine in the example form of a computer system 900 within which a set of instructions, for causing the machine to perform any one or more of the methodologies, processes, operations, or methods discussed herein, may be executed. The device 206 and 232 may include, for example, one or more of the functions of the computer system 900. These apparatus and systems are dedicated to performing one or more of the methods, processes, operations, or methods discussed herein.

In an example embodiment, the machine operates as a standalone device or may be connected (e.g., networked, etc.) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client computer in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

The example computer system 900 includes a processor 902 (e.g., a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or both, etc.), a main memory 904 and a static memory 906, which are coupled for communication with each other via a bus 908. the computer system 900 also includes a video display unit 910 (e.g., a liquid crystal display (L CD), or a Cathode Ray Tube (CRT), etc.) the computer system 900 also includes an alphanumeric input device 912 (e.g., a keyboard, etc.), a cursor control device 914 (e.g., a mouse, etc.), a drive unit 916, a signal generation device 918 (e.g., a speaker, etc.), and a network interface device 920.

The drive unit 916 includes a computer-readable medium 922 on which is stored one or more sets of instructions (e.g., instructions 924, etc.) embodying any one or more of the methodologies or functions described herein. The instructions 924 may also reside, completely or at least partially, within the main memory 904 and/or within the processor 902 during execution thereof by the computer system 900, the main memory 904 and the processor 902 also constituting non-transitory computer-readable media. When the instructions 924 are loaded, the processor 902 is a machine that is dedicated only to the current process and method. The instructions 924 may further be transmitted or received over a network 926 via the network interface device 920.

Conclusion

In the foregoing detailed description, it can be seen that various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. The methods of the present disclosure should not be construed as reflecting the intent: the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter may lie in less than all features of a single disclosed embodiment. Thus the following claims are hereby incorporated into the detailed description, with each claim standing on its own as a separate embodiment.

The foregoing description is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses. The broad teachings of the disclosure can be implemented in a variety of forms. Therefore, while this disclosure includes particular examples, the true scope of the disclosure should not be so limited since other modifications will become apparent upon a study of the drawings, the specification, and the following claims. It should be understood that one or more steps within a method may be performed in a different order (or simultaneously) without altering the principles of the present disclosure. Furthermore, although each of the embodiments is described above as having certain features, any one or more of those features described with respect to any of the embodiments of the present disclosure may be implemented in and/or combined with the features of any of the other embodiments, even if the combination is not explicitly described. In other words, the described embodiments are not mutually exclusive and substitutions of one or more embodiments with one another are still within the scope of the present disclosure.

As used herein, the terms "based on" or "using" reflect open-ended terms that may reflect other elements in addition to those explicitly recited. The present disclosure makes reference to robotics and words of similar meaning. A robot may be a machine that is capable of automatically performing a complex series of actions. These complex series of actions may include picking, orienting, positioning and/or releasing prescription components, pills, containers or other structures. The robot may be dedicated to a single series of actions or may perform multiple series of actions. The robot may include a processor that receives instructions and then executes the instructions to control its motion. In another example, the robot may resemble a human and replicate certain human actions and functions, may move positions, have articulated arms, have gripping structures that replicate fingers without damaging the container, and the like.

Various terms are used to describe spatial and functional relationships between elements (e.g., between modules) including "connected," engaged, "" interface, "and" coupled. Unless explicitly described as "direct," when a relationship between a first element and a second element is described in the above disclosure, the relationship includes a direct relationship where no other intermediate element exists between the first element and the second element, and an indirect relationship where one or more intermediate elements exist (spatially or functionally) between the first and second elements. As used herein, at least one of the phrases A, B and C should be interpreted using a non-exclusive logical OR to represent a logic (a OR B OR C), and should not be interpreted to represent "at least one of a, at least one of B, and at least one of C. "

In the drawings, the direction of the arrows generally illustrates the flow of information (e.g., data or instructions) of interest to the illustration, as indicated by the arrows. For example, when element a and element B exchange various information, but the information transmitted from element a to element B is related to the illustration, an arrow may point from element a to element B. This one-way arrow does not mean that no other information is sent from element B to element a. Further, for information sent from element a to element B, element B may send a request or receipt acknowledgement for the information to element a.

In this application, including the following definitions, the term "module" or the term "controller" may be replaced by the term "circuit". The term "module" may refer to or be part of processor hardware (shared, dedicated, or group) that executes code, or includes memory hardware (shared, dedicated, or group) that stores code executed by the processor hardware.

The functionality of any given module of the present disclosure may be distributed among multiple modules connected via AN interface circuit.

As used above, the term code may include software, firmware, and/or microcode, and may refer to programs, routines, functions, classes, data structures, and/or objects. Shared processor hardware comprises a microprocessor that executes some or all of the code in multiple modules. Group processor hardware includes a microprocessor that, in combination with other microprocessors, may execute some or all of the code in one or more modules. References to multiple microprocessors include multiple microprocessors on discrete dies, multiple microprocessors on a single die, multiple cores of a single microprocessor, multiple threads of a single microprocessor, or a combination of the above.

Shared memory hardware includes a single memory device that stores some or all code from multiple modules. The group storage hardware comprises a storage device that, in combination with other storage devices, may store some or all of the code from one or more modules.

The term memory hardware is a subset of the term computer-readable medium. As used herein, the term computer-readable medium does not encompass transitory electrical or electromagnetic signals propagating through a medium (e.g., on a carrier wave); the term computer-readable medium is therefore considered tangible and non-transitory. Non-limiting examples of non-transitory computer-readable media are non-volatile storage (e.g., flash memory devices, erasable programmable read-only memory devices, or mask read-only memory devices), volatile storage (e.g., static random access memory devices or dynamic random access memory devices), magnetic storage media (e.g., analog or digital tapes or hard drives), and optical storage media (e.g., CDs, DVDs, or Blu-ray discs).

The apparatus and methods described herein may be partially or fully implemented by a special purpose computer created by configuring a general purpose computer to perform one or more specific functions embodied in a computer program. The functional blocks and flow diagram components described above serve as software specifications, which may be converted into a computer program by the routine work of a technician or programmer.

The computer program includes processor-executable instructions stored on at least one non-transitory computer-readable medium. The computer program may also comprise or rely on stored data. The computer programs may include a basic input/output system (BIOS) that interacts with the hardware of the special purpose computer, a device driver that interacts with specific devices of the special purpose computer, one or more operating systems, user applications, background services, background applications, and the like.

The computer program may include (i) descriptive text to be parsed, such as HTM L (HyperText markup language), XM L (extensible markup language), or JSON (JavaScript Object Notification), (ii) assembly code, (iii) compiled from a source by a compilerFor example only, the source code may be code generated using techniques including C, C + +, C #, Objective-C, Swift, Haskell, Go, SQ L, R, L isp, etc,Fortran、Perl、Pascal、Curl、OCaml、HTM L5 (Hypertext Markup L angle 5 modified version), Ada, ASP (active Server Page), PHP (PHP: Hypertext preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Ada, ASP (active Server Page), PHP (PHP: Hypertext preprocessor), Ada, Adiffel, Smalltalk, Adlang, Ruby, Adm, Ad,VisualL ua, MAT L AB, SIMU L INK and

no element recited in the claims is intended to be a means-plus-function element within the meaning of 35u.s.c. § 112 (f). Unless an element is explicitly recited using the phrase "means for … …" or the phrases "operation against … …" or "step against … …" in the context of a method claim.