阅读说明：本技术 测序系统中试剂冷却液不稳定性和流动池加热器故障的预测 (Prediction of reagent coolant instability and flow cell heater failure in sequencing systems ) 是由 G·艾克 于 2019-01-03 设计创作，主要内容包括：所公开的技术检测冷却液系统的不稳定性,从而减少假警报。在低于预定阈值的稳定操作周期的预定时间窗内测试冷却液温度传感器数据的平滑时间序列。如果稳定温度操作周期小于预定的稳定性测量值,或者如果稳定的试剂冷却液温度已经超过阈值,则所公开的技术预测试剂冷却液是不稳定的。所公开的技术在系统中在多个循环中检测流动池加热器故障。测试流动池加热器温度传感器数据的时间序列,以确定最近处理循环中的点的计数是否记录为在阈值以上。如果在两个连续循环中超过阈值的点的计数小于预定计数,则所公开的技术确定流动池加热器发生故障。(The disclosed technique detects instability of the coolant system, thereby reducing false alarms. The smoothed time series of coolant temperature sensor data is tested within a predetermined time window of a stable operating cycle below a predetermined threshold. The disclosed technique predicts that the reagent coolant is unstable if the stable temperature operating period is less than a predetermined stability measurement, or if the stable reagent coolant temperature has exceeded a threshold. The disclosed technology detects flow cell heater faults in multiple cycles in a system. The time series of flow cell heater temperature sensor data is tested to determine if the count of points in the most recent processing cycle is recorded to be above the threshold. The disclosed technique determines that the flow cell heater is malfunctioning if the count of points exceeding the threshold in two consecutive cycles is less than a predetermined count.)

1. A method of detecting coolant system instability, comprising:

applying a smoothing function to the time series of coolant temperature sensor data and reducing transient oscillations to produce a smoothed time series of coolant temperature sensor data;

determining that a smoothed time series of the coolant temperature sensor data in a predetermined time window does not reach a stable temperature operating criterion in a time interval during which temperature readings in the smoothed time series vary by more than a predetermined rate of temperature change from interval to interval; and

generating an instability notification when the smoothed time series of coolant temperature sensor data does not reach the stable temperature operating criteria within a time interval exceeding a predetermined percentage within the predetermined time window.

2. The method of claim 1, further comprising:

determining the predetermined rate of temperature change based on devices located at a plurality of locations and operated by a plurality of independent operators, comprising:

causing a configuration of the device to record and report temperature sensor readings;

collecting a log of the temperature sensor readings;

analyzing a time series of the temperature sensor readings in an instance of equipment in which a coolant system has failed and determining the predetermined rate of temperature change; and

storing the predetermined rate of temperature change for determining the coolant system is unstable.

3. The method of claim 1, further comprising:

updating the predetermined rate of temperature change based on devices located at a plurality of locations and operated by a plurality of independent operators, comprising:

causing a configuration of the device to record and report temperature sensor readings;

collecting a log of the temperature sensor readings and a service log after the instability notification;

analyzing a time series of the temperature sensor readings in the device instance for which the coolant system generated the notification and a service subsequent to the notification;

determining an update to the predetermined rate of temperature change based on the analysis; and

storing the updated predetermined rate of temperature change for determining the coolant system is unstable.

4. The method of claim 1, further comprising:

accessing, in a cloud-based active maintenance analyzer, a log of temperature sensor readings from a particular coolant system; and

executing the application, the determining, and generating a notification from the cloud-based proactive maintenance analyzer.

5. The method of claim 4, further comprising filtering notifications for duplicate notifications and submitting filtered notifications to a customer relationship management system for tracking.

6. The method of claim 4, further comprising filtering the notification for duplicate notifications and submitting the filtered notification to an operator of a sequencer that includes the coolant system.

7. The method of claim 2, further comprising determining the predetermined rate of temperature change based on equipment located at a plurality of locations and operated by a plurality of independent operators, wherein the plurality of locations comprises at least 50 locations and the plurality of independent operators comprises at least 20 independent operators.

8. The method of claim 1, wherein the smoothing function is applied by a derivative filter.

9. The method of claim 1, wherein applying the smoothing function eliminates transient oscillations that produce a rate of temperature change of 0.125 degrees celsius per minute or greater.

10. The method of claim 1, wherein the time series data represents coolant temperature sensor data between 4 hours and 48 hours.

11. The method of claim 1, further comprising automatically presenting smoothed coolant system temperature sensor data for viewing by a user when presenting a system instability determination.

12. The method of claim 1, further comprising comparing the average and median temperatures of the stable operating cycle and reporting a severity level 1 error when the average and median temperature changes exceed a first threshold.

13. The method of claim 1, further comprising comparing the average and median temperatures of the stable operating cycle and reporting a severity level 2 error when the average and median temperature changes exceed a second threshold.

14. A system comprising one or more processors coupled to a memory loaded with computer instructions for detecting coolant system instability, the instructions when executed on the processors implement operations comprising:

applying a smoothing function to the time series of coolant temperature sensor data and reducing transient oscillations to produce a smoothed time series of coolant temperature sensor data;

determining that a smoothed time series of the coolant temperature sensor data in a predetermined time window does not reach a stable temperature operating criterion in a time interval during which temperature readings in the smoothed time series vary by more than a predetermined rate of temperature change from interval to interval; and

generating an instability notification when the smoothed time series of coolant temperature sensor data does not reach the stable temperature operating criteria within a time interval exceeding a predetermined percentage within the predetermined time window.

15. The system of claim 14, further implementing operations comprising:

determining the predetermined rate of temperature change based on devices located at a plurality of locations and operated by a plurality of independent operators, comprising:

causing a configuration of the device to record and report temperature sensor readings;

collecting a log of the temperature sensor readings;

analyzing a time series of the temperature sensor readings in an instance of equipment in which a coolant system has failed and determining the predetermined rate of temperature change; and

storing the predetermined rate of temperature change for determining the coolant system is unstable.

16. The system of claim 14, further implementing operations comprising:

updating the predetermined rate of temperature change based on devices located at a plurality of locations and operated by a plurality of independent operators, comprising:

causing a configuration of the device to record and report temperature sensor readings;

collecting a log of the temperature sensor readings and a service log after the instability notification;

analyzing a time series of the temperature sensor readings in the device instance for which the coolant system generated the notification and a service subsequent to the notification;

determining an update to the predetermined rate of temperature change based on the analysis; and

storing the updated predetermined rate of temperature change for determining the coolant system is unstable.

17. The system of claim 14, further implementing operations comprising:

accessing, in a cloud-based active maintenance analyzer, a log of temperature sensor readings from a particular coolant system; and

executing the application, the determining, and generating a notification from the cloud-based proactive maintenance analyzer.

18. The system of claim 17, further implementing operations comprising:

and filtering the notifications aiming at the repeated notifications, and submitting the filtered notifications to a client relationship management system for tracking.

19. The system of claim 17, further implementing operations comprising:

the notification is filtered for duplicate notifications and the filtered notification is submitted to an operator of a sequencer that includes the coolant system.

20. A set of one or more non-transitory computer-readable storage media collectively imprinted with computer program instructions for detecting coolant system instability, the instructions, when executed on one or more processors, implementing a method comprising:

applying a smoothing function to the time series of coolant temperature sensor data and reducing transient oscillations to produce a smoothed time series of coolant temperature sensor data;

determining that a smoothed time series of the coolant temperature sensor data in a predetermined time window does not reach a stable temperature operating criterion in a time interval during which temperature readings in the smoothed time series vary by more than a predetermined rate of temperature change from interval to interval; and

generating an instability notification when the smoothed time series of coolant temperature sensor data does not reach the stable temperature operating criteria within a time interval exceeding a predetermined percentage within the predetermined time window.

21. The set of one or more non-transitory computer-readable storage media of claim 20, wherein the smoothing function is applied by a derivative filter.

22. The set of one or more non-transitory computer-readable storage media of claim 20, wherein applying the smoothing function eliminates transient oscillations that produce a rate of temperature change of 0.125 degrees celsius per minute or more.

23. The set of one or more non-transitory computer-readable storage media of claim 20, wherein the time series data represents coolant temperature sensor data between 4 hours and 48 hours.

24. The set of one or more non-transitory computer-readable storage media of claim 20, implementing a method further comprising:

the average and median temperatures of the stable operating cycle are compared and a missing fault is reported when the average and median temperature changes exceed a first threshold.

25. The set of one or more non-transitory computer-readable storage media of claim 20, implementing a method further comprising:

the average and median temperatures of the stable operating cycle are compared and a false alarm is reported when the average and median temperatures vary by more than a second threshold.

Background

The subject matter discussed in the background section should not be considered prior art merely as a result of its mention in the background section. Similarly, the problems mentioned in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognized in the prior art. The subject matter in the background section merely represents different approaches, which may themselves correspond to embodiments of the claimed technology.

The disclosed technology relates to sequencing systems, including systems that use sequencing-by-synthesis techniques to sequence nucleotides. Sequencing to identify nucleotides in a molecule is a lengthy process that takes several days to complete. All subsystems of the sequencer need to operate without error so that the resulting base calls are useful for downstream analysis. Predicting a corresponding failure in the operation of a sequencer before and during a sequencing run is a challenge. Sensors in the sequencing system generate readings for controlling the operating conditions of the various components. These readings are used in the control loop to change the future state of the system but are not available to the operator. Even if the operator can obtain sensor readings, the problem of predicting a sequencer failure is not adequately addressed because appropriate sensor values are not readily apparent to the operator.

The subsystems of a sequencer may be affected by external factors, including their operating environment. The sensor readings do not identify whether the abnormal sensor readings are due to an unstable or malfunctioning sub-system or an external factor. The influence of external factors is often temporary and when external factors are removed, subsystem performance returns to normal levels. It is desirable to provide a solution to identify whether sensor readings are overrun due to subsystem instability or failure, or due to external factors.

Drawings

The drawings are included for illustrative purposes and are merely intended to provide examples of possible structures and process operations for one or more embodiments of the present disclosure. These drawings in no way limit any changes in form and detail that may be made by one skilled in the art without departing from the spirit and scope of the disclosure. A more complete understanding of the subject matter may be derived by referring to the detailed description and claims when considered in conjunction with the following figures, wherein like reference numbers refer to similar elements throughout the figures.

FIG. 1 shows an architectural level schematic of a system in which a reagent coolant instability prediction system predicts coolant system instability and a flow cell heater failure prediction system detects flow cell heater failures, both of which are determined from sequencing hardware sensor metrics newly collected from a sequencing system.

FIG. 2 illustrates subsystem components of the reagent coolant instability prediction system and the flow cell heater failure prediction system of FIG. 1.

Fig. 3 shows an example of a time series of coolant temperature sensor data before and after filtering noise data.

FIG. 4 is a flow chart illustrating the process steps for predicting reagent coolant system instability by the reagent coolant fault prediction system of FIG. 1.

FIG. 5 shows an example time series of flow cell heater temperature sensor data before and after a flow cell heater failure.

FIG. 6 is a flowchart of process steps for detecting a flow cell heater fault by the flow cell heater fault prediction system of FIG. 1 with and without setpoint data.

FIG. 7 illustrates an example user interface for presenting results of active monitoring of a sequencing system to predict hardware failures.

FIG. 8 is a simplified block diagram of a computer system that may be used to implement the reagent coolant instability prediction system and the flow cell heater failure prediction system of FIG. 1.

Detailed Description

The following detailed description refers to the accompanying drawings. Example embodiments are described to illustrate the disclosed technology, not to limit its scope, which is defined by the claims. Those of ordinary skill in the art will recognize a variety of equivalent variations on the description that follows.

Introduction to

Sequencing By Synthesis (SBS) is one of several commonly used techniques for sequencing nucleotides in DNA or RAN molecules. The machines that perform sequencing are complex systems that include complex subsystems that operate at specific temperatures in the sequencing process steps. The cost of acquiring and operating a sequencer is high. During sequencing, the subsystems of the sequencer may be subject to internal and external instabilities.

In SBS processing cycles, complementary nucleotides are added one at a time to a nucleotide sequence fragment (also referred to as a molecule or insert) of the DNA to be sequenced. Sequencing of nucleotides in a molecule is performed in hundreds of cycles. A library of molecules to be sequenced is prepared on a slide or flow cell prior to the start of the sequencing cycle. The molecules are arranged in small regions within a plurality of channels on a flow cell. One cycle includes chemical, image capture and image processing operations. Subsystems, including optical, mechanical and chemical subsystems, operate in each cycle to identify complementary nucleotides attached to the molecule. Identification of added nucleotides is massively parallel, as there are millions or billions of molecular clusters on a flow cell. A sequencing run involves hundreds of cycles of the sequencing process, which may take several days to complete. Sometimes, the results of an entire sequencing run are discarded because they do not meet the minimum quality requirements for downstream analysis. Therefore, if subsystem failures affect the quality of sequencing results, early prediction is required.

The disclosed technology involves modifying the sequencer to expose selected data from sensors used by the internal control loop that have not been previously collected or analyzed. The selection of exposure and collection of sensor data requires careful analysis of the subsystems and sensors used in the control loop.

Advances in this technology include analyzing newly collected sensor data and identifying features in the time series data that can be used to predict faults.

Enabling the collection of selected sensor data from many machines in different environments with different classes of users will support a refinement of the prediction method. Analyzing the various data should enable the development team to reduce false alarms that corrupt the confidence in the predictions without missing significant events.

Sensor data collection and analysis during sequencing runs will enable operators to abort a potentially failing sequencing run, or schedule preventative maintenance between runs.

Notably, the predetermined detection parameters and filters are intended to distinguish between error conditions and temporary and transient fluctuations caused by external factors, so that false alarms do not result in the operation that should be successful being cancelled. For example, the reagent coolant subsystem maintains precise temperatures of reagents for sequencing runs. If the door of the room in which the sequencer operates is opened in summer, warm air from the outside will increase the room temperature. When air enters the reagent coolant chamber, the temperature reading recorded by the sensor is higher than usual. Such transient fluctuations should not generate false status alarms. In this example, an unstable or underperforming reagent coolant system would alarm after filtering out transient temperature fluctuations caused by external factors. In another example, the disclosed technology uses temperature data from multiple sequencing cycles to alert operators to a flow cell heater failure. A flow cell heater that heats too slowly may be detected from the temperature sensor data, taking into account the cycle set point or derived threshold. Failure of the flow cell heater to heat as expected may indicate a heater failure and/or result in potential unsuccessful operation.

Analysis of newly collected sensor data during the sequencing process can generate alarms and alerts for predicted failures of subsystems and potentially failing sequencing runs. This will reduce down time and improve customer satisfaction.

Environment(s)

We describe an early prediction system for reagent coolant failures and flow cell heater failures in sequencing systems that applies to extended optical base recognition processes. The four nucleotides in a DNA molecule are adenine (A), cytosine (C), guanine (G) and thymine (T). Base recognition refers to the determination of the nucleotide base (A, C, G, T) that each cluster adds to a molecule in one cycle of a sequencing run. The system is described with reference to fig. 1, fig. 1 showing an architectural level diagram of the system according to an embodiment. Because fig. 1 is an architectural diagram, certain details are intentionally omitted to enhance clarity of the description. The discussion of FIG. 1 is organized as follows. First, elements of the drawings are described, and then their interconnections are described. The use of the elements in the system will then be described in more detail.

Fig. 1 includes a system 100. This paragraph names the label portion of the system 100. The figure shows a sequencing system (or sequencer) 185, sequencing system operator 113, technician 119, Customer Relationship Management (CRM) system 167, service alert database 141, alert status database 114, and service solutions database 143. The system 100 also includes a sequencing hardware sensor reading and Q-value database 151, a configuration engine 117, and an alarm service 121. These components facilitate the reagent coolant instability prediction system 131 and the flow cell heater failure prediction system 141. The database 151, the alarm service 121, the reagent coolant instability prediction system 131, the flow cell heater failure prediction system 141 can be implemented as a cloud-based active maintenance analyzer 111.

The disclosed techniques are applicable to a variety of sequencing systems 185, also known as sequencers and sequencing platforms. Network 155 interconnects sequencing system 185, operator 113, CRM system 167, technician 119, configuration engine 117, alarm state database 114, alarm service 121, reagent coolant instability prediction system 131, flow cell heater failure prediction system 141, and database 151. The CRM system 167 communicates with the service alert database 141 and the service solutions database 143 to send alerts to the operator 113 and the technician 119. The solution for the technician post-service alert is stored in the service solution database 143. CRM system 167 may also be packaged in a customer relationship module.

The sequencing system 185 may use the Sequencing By Synthesis (SBS) technique of Illumina or another sequencing technique. Illumina inc. is a manufacturer of sequencing system 185, providing various sequencing systems, including but not limited to hisefx^TM、HISEQ2500^TM、HISEQ3000^TM、HISEQ4000^TM、NOVASEQ6000^TMAnd MISEQDX^TM. These sequencing machines include a control computer, a monitor and some major subsystems including flow cells, fluidics and reagents, optics and image acquisition and processing modules. These sequencing systems apply SBS techniques to the base recognition cycles in the sequencing run. The sequencing system 185 is used in a wide variety of physical environments, from laboratories in large research facilities to high school classrooms. Many sources of signal noise affect sequencers that operate in different environments. Sequencing machine operators have varied levels of skill, from well-trained researchers in research laboratories to high school teachers and students who use leased equipment. Some models of sequencers are not highly insulated and therefore may be affected by weather conditions and door and window openings.

The sequencing run proceeds through hundreds of processing cycles, for example 200 to 600 cycles or 300 to 1000 cycles. Depending on the platform, a 300 cycle sequencing run may take three days to complete. Sometimes, one run is divided into two reads, also referred to as a double-ended run. One cycle includes chemical, image acquisition, and image processing steps. In chemical processing, one complementary nucleotide is added per molecule in a cluster of molecules arranged in a channel of a flow cell. Some subsystems will be described in the following paragraphs.

The fluidic subsystem contains a fluidic pump that delivers reagents to a flow cell and then to a waste container. A reagent is a compound or substance added to a flow cell in a chemical process. The tube racks in the reagent subsystem hold sufficient quantities of reagents for the entire sequencing run. The reagent coolant holds the reagent rack and maintains the internal temperature at around 4 ℃. It will be appreciated that in other sequencing systems, the reagent coolant may be maintained at a different temperature range.

The flow cell subsystem may include a flow cell stage that holds the flow cell in place during a sequencing run. Some stages hold two flow cells. The heater heats the flow cell to the appropriate reaction temperature during the sequencing cycle.

The optical subsystem includes an optical assembly capable of imaging the flow cell to identify A, C, G and the T base using the fluorescently labeled complementary nucleotide. The excitation laser beam excites the fluorescent label. Images are acquired using a camera and processed to identify bases. In other embodiments of the sequencer, CMOS semiconductor sensors covered by nanopores have been used as the substrate for flow cells, replacing overhead cameras.

Sequencing systems and subsystems use a number of sensors in the control loop. The system software has been updated to record selected sensor readings that were previously used only for the inner control loop. The sequencing system may be retrofitted (or initially configured), for example by deploying a software patch, to collect and/or record sensor readings that were previously used only for internal control. The collected sensor readings may be sent to a cloud-based data active maintenance analyzer 111 or stored locally in a sequencer or enterprise network.

In one embodiment, the cloud-based active maintenance analyzer 111 aggregates the collected sensor readings. The platform is directly integrated with the provided sequencer. Instrument operational data can be transmitted from the sequencing system 185 to the cloud-based active maintenance analyzer 111 over the network 155. In another embodiment, a local version of the cloud-based proactive maintenance analyzer 111 can be capable of data storage and analysis on-site through an installed local server. Operational data for a particular sequencing run of the sequencer is stored as a data set of time series data. The operational data may be stored as a time series of quality data, such as the Q of the cycle and other metrics, including intensity and phasing/pre-phasing. The quality data can be used as a dependent variable for independent sensor reading analysis.

The collected data may be used to establish or update predetermined detection parameters and filters. For example, a cloud-based active maintenance analyzer collects and analyzes time series and quality data to set or update predetermined detection parameters. The active maintenance analyzer may also periodically update predetermined detection parameters, combine the collected time series data with service solution data to distinguish between correct and false alarms, and indicate how the alarms were resolved. Time series data from devices that fail without warning may also be considered when updating the predetermined detection parameters. Missing faults and false alarms can be identified using service solution data from the CRM system and used to refine predetermined detection parameters and corresponding time series filtering.

The sequencing system 185 reports sensor readings during or after the sequencing process. The sequencing system also reports data relating to quality. The collection of sensor and/or quality readings may be referred to as a log. The collected sensor readings and quality data are stored in database 151, sequencing hardware sensor readings, and Q-value database 151. The database 151 may store a time series of sensor readings from the base recognition cycle tissue of each sequencing system. The database 151 may also store the quality value of the base recognition cycle for each sequencing system as a dependent variable. The Q value is a commonly used quality value for predicting the probability of base recognition errors. Details of Q values are found in technical Specifications Quality scales for Next Generation queue ringing (2011) < links https:// www.illumina.com/documents/products/technologies _ Q-scales. A high Q value indicates that base recognition is more reliable and less likely to be incorrect. In one embodiment, database 151 stores reagent coolant temperatures and flow cell heater temperatures reported by the sensors.

In addition to the Q value, several examples of quality metrics are as follows. For example, the chemical processing subsystem generates phasing and a predetermined amount of phasing. The term "phasing" describes the situation where one molecule in a cluster of molecules lags behind other molecules in the same cluster by at least one base during sequencing. This result may be due to incomplete chemical reactions. The term "predetermined phase" describes the situation where one molecule jumps at least one base forward compared to other molecules in the same cluster of molecules. One reason for the pre-phasing is the addition of one non-terminating nucleotide followed by the addition of a second nucleotide in the same sequencing cycle. Increasing phasing or predetermined phasing confuses the luminescent signals in the clusters, thereby reducing the accuracy of the identification. Thus, phasing and predetermined phase measurements may be used with sensor time series data to set or update predetermined detection parameters.

The optical subsystem produces intensity measurements that can be used as quality data. Some sequencers use a camera to acquire images of clusters on a flow cell in a sequencing cycle. Image acquisition includes intensity measurements of cycles in a sequencing run. The process of determining the intensity values of clusters in a sequenced image is called intensity extraction. To extract the intensity, a portion of the image containing the molecular clusters is used to calculate the background of the molecular clusters. The background signal is subtracted from the cluster signal to determine the intensity. The sequencing hardware sensor readings and Q-value database 151 may store one or more imaging performance metrics as dependent variables.

The configuration engine 117 may be used to deliver a software patch that adapts the sequencing system and exposes sensor readings for collection and recording. The newly collected sensor reading data is analyzed to determine predetermined detection parameters for sensor readings of different sequencing system components or subsystems. After the predetermined detection parameters are determined, sensor readings from the sequencing system are tested against the predetermined detection parameters to predict a corresponding subsystem failure. Further details of the configuration engine 117 and the alert service 121 are presented in the description of the subsystem components shown in FIG. 2. The reagent coolant subsystem and the flow cell heater subsystem are two example subsystems of a sequencing system that have been adapted by the disclosed techniques to collect sensor readings.

The reagent coolant system cools the reagent stored in the test tube rack within the housing to a cold temperature, such as about 4 degrees celsius for one chemical process. The reagents used in the sequencing system are cooled until used for chemical processing. The reagent coolant cannot compensate for fluctuations in ambient temperature, which may expose the stored reagent to a higher than desired temperature environment for a long period of time, thereby damaging the reagent. Reagent coolant instability prediction system 131 uses reagent coolant temperature data reported by temperature sensors in the reagent coolant to identify instabilities in reagent coolant operation. In one embodiment, the software reports the sensor readings in the reagent coolant every five minutes. It should be understood that in other embodiments, the temperature sensor data may be reported at intervals greater than or less than 5 minutes, such as in the range of 1 to 30 minutes or 30 seconds to 1 hour. The data reported by the temperature sensor in the reagent coolant may be noisy due to the mechanical systems used in the operation of the coolant subsystem. The temperature of the coolant subsystem is affected by external factors, such as the environment in which the sequencer operates, and the operation of the reagent coolant subsystem. The reagent coolant instability prediction system 131 analyzes the time series of coolant temperature sensor data to determine whether the reagent coolant system is unstable. More details are given in the description of the subsystem assembly in fig. 2.

The flow cell heater and cooling fluid heat and cool the flow cell and reagents, respectively, to temperatures required for chemical processing to attach and remove fluorescent labels, which are imaged and converted to base calls. The chemical treatments were carried out at different temperatures. In one sequencing cycle embodiment, the flow cell temperature is raised from an initial value of 20 ℃ to 55 ℃ for a short period of time, and then to 60 ℃ for a short period of time. Before imaging, the temperature of the flow cell was reduced to 20 ℃. The temperature increase and decrease are repeated in the next sequencing cycle. Flow cell heater failure prediction system 141 analyzes a time series of flow cell heater temperature sensor data to determine whether a flow cell heater has failed. Details of the reagent coolant instability prediction system 131 and the flow cell heater failure prediction system 141 are given in the description of the subsystem components in fig. 2.

The alarm service 121 generates a service alarm when a fault prediction system, such as the reagent coolant instability prediction system 131 and the flow cell heater fault prediction system 141, indicates an approaching hardware fault. The CRM system 167 forwards an alert that enables the operator 113 and/or technician 119 to establish a service call to service the sequencing system 185. The alerts are stored in a service alerts database 141. The status of the alarms is maintained in the alarm status database 114 to manage upgrades to service requests in a scheduled manner, for example, according to a service level agreement. The service solution database 143 includes details of equipment services performed by the technician. The missed faults and false alarms may be used to adjust predetermined detection parameters. Missing faults may be used as false negatives and false alarms may be used as false positives. For example, in flow cell heater fault prediction, a false positive may indicate that a threshold above ambient temperature may need to be increased. For false negatives, the threshold may need to be lowered.

Completing the description of FIG. 1, the components of system 100 are all communicatively coupled to network 123 as described above. The actual communication path may be a point-to-point path over a public and/or private network. Communications may be conducted over various networks, such as a private network, a VPN, MPLS circuit, or the Internet, and may use appropriate Application Programming Interfaces (APIs) and data exchange formats, such as representational State transfer (REST), JavaScript object notation (JSON), extensible markup language (XML), Simple Object Access Protocol (SOAP), Java Message Service (JMS), and/or Java platform Module systems. All communications may be encrypted. Communications are typically conducted over networks such as LANs (local area networks), WANs (wide area networks), telephone networks (public switched telephone networks (PSTNs)), Session Initiation Protocols (SIPs), wireless networks, point-to-point networks, star networks, token ring networks, hub networks, the internet (including the mobile internet), via protocols such as EDGE, 3G, 4G LTE, Wi-Fi, and WiMAX. The engine or system components of FIG. 1 are implemented by software running on different types of computing devices. Example devices are workstations, servers, computing clusters, blade servers, and server farms. In addition, a variety of authorization and authentication techniques, such as username/password, open authorization (OAuth), Kerberos, security keys, digital certificates, and the like, may be used to secure communications.

System assembly

FIG. 2 is a high-level block diagram of the component configuration engine 117, the alarm service 121, the reagent coolant instability prediction system 131, and the flow cell heater failure prediction system 141. These systems are computers implemented using a variety of different computer systems, as shown below in the description of FIG. 8. When implemented, the illustrated components may be combined or further separated.

Configuration engine

The development team, responsible for the so-called active alarm generation platform, investigates which data from the sensors used in the sequencer control loop can be logged and used to produce the primary indicator that a fault is imminent. The sequencing system includes a number of sensors and software that can be updated to record a modest number of readings. New signals from the closed loop may be identified and analyzed to generate a primary indicator of a fault.

For example, temperature time series data determined by the development team from the reagent coolant may yield a primary indicator that the coolant is about to fail and that the reagent is about to fail. The development team investigated which signals the sensors embedded in the sequencer were exposed to. After the signal to be acquired is identified, the sequencer is adapted (and may be configured) to expose the signal. Typically, the configuration engine 117 may be used to provide patches to the sequencer.

Configuration engine 117 includes patch application engine 211 to deploy a software program as a patch or update to an existing software program running on a computer that controls the operation of the sequencer machine. The subsystems are controlled by a computer. Subsystems of the sequencing system include sensors that generate sensor readings that are used to control the loop during operation of the sequencer. The new system can be built with equivalent programming.

The newly deployed software patch supports the collection and recording of sensor data. For example, the patch application engine 211 may install a software patch to collect temperature sensor readings from the reagent coolant for the instability prediction system 131. Similarly, a software patch may be applied to collect flow cell heater sensor readings for the failure prediction system 141. This portion of the technology may also be packaged in a sensor exposure module. The configuration engine 117 can retrofit a sequencer such that previously unrecorded data from sensors in the sequencer can be exposed for active maintenance.

The configuration engine 117 includes a detection parameter predetermination and update engine 212. Reliable prediction of impending hardware failure includes signal analysis of collected and/or recorded sensor readings. The update engine 212 processes at least selected log data exposed from closed-loop control. Data that has not been previously recorded may be collected from a plurality of geographically dispersed sequencing machines. The data may be time-stamped or ordered for association, or may be associated at the time of collection. Data from multiple machines in independent operation improves the reliability of the primary indicator of instrument failure.

Detection parameter predetermination and update engine 212 implements analysis prototyped by the development team to predetermine detection parameters and filters to apply to time series data prior to sequencing. Examples of analyses that may be used include regression analysis, logistic regression, threshold fitting to minimize cost functions, and machine learning (if there are enough samples of faults). The smoothed rate of change is one of the signal characteristics that can be analyzed. An analysis is performed on the primary indicator to determine a trend of change of the primary indicator to predict an impending failure. The detection parameter predetermination and update engine 212 can repeatedly analyze sensor readings in failed component instances to predetermine detection parameters. One example of such an analysis is determining a predetermined rate of temperature change in an instance of equipment in which a corresponding failure of the coolant system is imminent.

To improve the quality of maintenance prediction alarms and reduce the number of false alarms, the detection parameter predetermination and update engine 212 may use the service solution data to update predetermined detection parameters after a service call by the technician 119. The service solution data may include information such as false positive indications of replacement or alarms of failed or failed components. Existing optimization techniques, such as gradient descent or re-application of the above analysis, may be used to update the predetermined detection parameters to reduce the number of missed faults and false alarms.

After collection of the service call record over a period of time (e.g., one month, three months, or one to twelve months), the predetermined detection parameters may be periodically updated. When the detection parameters predetermine and update engine processes temperature data from the reagent coolant in the sequencing system, an updated portion of the engine may also be packaged in the threshold adjustment module. When the detection parameters are predetermined and the update engine processes temperature data from a flow cell in the sequencing system, the updated portion of the engine can also be packaged in the temperature margin adjustment module.

Alert service

An operational alarm may be generated when a failure prediction system, such as the reagent coolant instability prediction system 131 and the flow cell heater failure prediction system 141, predicts an impending failure. The alert is passed to the alert service 121. The alert service 121 includes an alert generator component 213 that implements, for example, service alert subscription and publication functionality. The alert is sent to the operator 113 and/or technician 119. A Customer Relationship Management (CRM) system can implement alerts and track follow-up through solutions.

The filtering can be applied to alarms that occur repeatedly over multiple cycles of a single run and over multiple sequencing runs, particularly for laboratories with high sequencing system 185 utilization. The alarm filtering engine 214 filters the repetitive alarms. In one embodiment, the system maintains an alarm status database 114 to upgrade service alarms in a planned manner. The CRM system 167 updates the status of the alert by successive states, such as the creation of service tickets, the scheduling of service access, and the completion of device services. The alert service 121 may escalate the service alert if the service operation is not completed within the required service time.

The alert service 121 may generate more than one type of alert, such as an instrument alert and a run alert. Instrument alarms are long-lived, typically spanning multiple runs, and once an alarm is generated, it will remain active until resolved. The instrument alarm may require replacement or repair of parts. Examples of instrument alarms include reagent coolant instability, flow cell heater failure, or laser power failure. On the other hand, run alarms may be specific to a sequencing run. In some cases, the operator 113 can take action on such an alert. For example, an operator may terminate a sequencing run upon receiving an alert identifying a flow cell misalignment on a sequencer flow cell rack. This can save processing time for failed runs and sequencing operation costs.

Reagent coolant instability prediction system components

The block diagram presents example components of two failure prediction systems 131 and 141 that predict instability of the reagent coolant and malfunctioning flow cell heaters and/or coolants. The time series preparer component 221 is common to both systems 131 and 141. Component 221 prepares a time sequence according to sequencing hardware metrics. Timing data is collected from sensors in the sequencing system subsystem. The time series preparer 221 may also be packaged in a log collection module. In one embodiment, the collected data is uploaded to the cloud-based active maintenance analyzer 111 and stored in the sequencing hardware sensor readings and Q-value database 151. Examples of reagent coolant and flow cell heater temperature sensor time series data are shown in fig. 3 and 5. Details of the components specific to the reagent coolant instability prediction system 131 and the flow cell heater failure prediction system 141 are given in the following paragraphs.

The reagent coolant instability prediction system 131 also includes a data smoother 231, a time series tester 241, a severity level identifier 251, and a reagent coolant system stability predictor 261. The reagent coolant temperature sensor data is chronologically sequenced in ascending order, if necessary, to prepare a time series. The time series is provided as input to a data smoother component 231. As described above, the temperature data of the reagent coolant is noisy. The data smoother module 231 filters out transient oscillations in the coolant temperature sensor data time series. This portion of the disclosed techniques may also be packaged in a time series smoothing module. In one embodiment, the data smoother module 231 applies a derivative filter with a cutoff value of 0.125 ℃ per minute to filter the transient oscillations and generate a smoothed time series of coolant temperature sensor data. Alternatively, a filter may be applied to eliminate transient oscillations that produce a rate of change of 0.250 degrees celsius per minute or greater. Alternatively, the smoothing function may eliminate transient oscillations based on a predetermined rate of temperature change of greater than or equal to 0.0625 degrees celsius per minute. The filter may have an upper limit built in, such as 5.0 degrees celsius per minute, but this is not required.

The reagent coolant prediction system 131 may be implemented as part of the cloud-based active maintenance analyzer 111. As described above, the log of temperature sensor data from the reagent coolant is analyzed by the configuration engine 117 to predetermine the detection parameters. The time series component 241 predicts the instability of the coolant system using predetermined detection parameters. The time series tester component 241 tests a smooth time series of coolant temperature sensor data within a predefined time window of a stable temperature operating cycle. The time series tester component may also be packaged in a temperature instability detection module. The period of stable temperature operation is defined as the period of time in which the absolute value of the rate of change of the temperature readings in the smoothed time series varies by less than a predetermined rate of temperature change. In one embodiment, the absolute temperature rate of change for steady operation is less than 0.05 ℃ per minute. In another embodiment, a higher value, such as 0.25 ℃ per minute, may be used, or a lower value, such as 0.01 ℃ per minute, may be used. If the total number of cycles of stable temperature operation within the predetermined time window is less than the predetermined stability measure, the reagent coolant system stability predictor component 261 determines that the coolant system is unstable and reports whether the temperature is rising rapidly (i.e., faster than the threshold). The number of cycles of the stabilizing operation may be expressed as a predetermined percentage. Component 261 notifies alarm service 121 that the reagent coolant system is unstable. The reagent coolant system stability predictor component 261 and the alarm service 121 may also be packaged together in a temperature instability alarm module.

Severity level identifier component 251 compares the average and median temperatures of the stable coolant system to two thresholds to determine severity level 1 and severity level 2 errors. In one embodiment, the configuration engine 117 analyzes temperature sensor readings collected from reagent coolants in the sequencing system to set thresholds. For example, for HISEXX^TM、HISEQ3000^TMAnd HISEQ4000^TMThe result of this analysis by the sequencing system was a threshold of 9 ℃ for the severity level 1 problem and a threshold of 7.5 ℃ for the severity level 2 problem. It should be understood that different thresholds may be set for severity levels 1 and 2. When the severity level identifier 251 determines that the coolant system has a problem with severity level 1 or severity level 2, it notifies the alert service 121, and the alert service 121 may then generate an alert.

Flow cell heater failure prediction system assembly

FIG. 2 also shows the components of the flow cell heater and/or coolant prediction system 141, including a setpoint data splitter 233, a data analyzer 243 without a setpoint data component, a data analyzer 253 with a setpoint data component, and a flow cell heater fault predictor 263. The time series preparer component 221 retrieves the flow cell heater temperature sensor data from the sequencing hardware sensor readings and the Q-value database 151. In one embodiment, time series preparer component 221 separates temperature data of a-side and B-side of the flow cell subsystem. In this implementation, the time series of each side is tested separately.

The temperature sensor data of the flow cell heater is chronologically viewed as a time series. Flow cell heater temperature sensor data can be defined in a sequencing process cycle. The processing cycle, also referred to as a base recognition cycle, includes a plurality of sub-cycles of chemical processing. In one embodiment, the duration of the base recognition cycle is about 15 minutes and the duration of the chemical treatment sub-cycle is about 5 minutes.

In one embodiment, the temperature is reported from the flow cell about every minute during a chemical sub-cycle during the base recognition cycle. It should be appreciated that in other embodiments, the samples may be reported at a higher or lower sampling rate, for example in the range of 15 seconds to 3 minutes.

During a chemical processing sub-cycle, on one sequencer, the temperature is raised from an initial temperature (e.g., about 20 ℃) to a higher temperature (e.g., about 55 ℃) for a short period of time, then raised to a higher temperature (e.g., about 60 ℃) for another short period of time, and then dropped back to the initial temperature. These three temperature levels are referred to as set points.

In one embodiment, the sampling interval of the temperature sensor reading is longer than the hold duration of a particular temperature point during a chemical sub-cycle. In this embodiment, no temperature readings are taken at the higher temperatures (55 ℃ and 60 ℃) for a small portion of the chemical sub-cycle. Thus, before the component 243 or 253 tests the temperature sensor data for a processing cycle, it is checked whether a sufficient number of temperature sensor data readings are available. In one embodiment, at least 5 readings are required in one processing cycle before testing the data. Alternatively, depending on the chemical duration and sensor reporting frequency, at least 3 readings or 3 to 10 readings may be required.

The flow cell heater fault prediction system 141 can be implemented as part of the cloud-based active maintenance analyzer 111. The sensor data may be analyzed with or without reporting setpoint data. If setpoint data is available for the flow cell heater temperature sensor, then component 253 analyzes the temperature sensor data using the setpoint data. There may be more than one temperature set point. The setpoint data separator module 233 separates the setpoint data time series from the flow cell heater temperature sensor data time series. Otherwise, if setpoint data is not available, a component 243 of the data analyzer, referred to as no setpoint data, analyzes the temperature sensor data using the operating heater threshold. The assembly tests a time series of flow cell heater temperature sensor data to count measured temperature sensor data points recorded in the most recent processing cycle that are above an operating heater threshold. The threshold is determined based on the likelihood of making a sensor measurement during a particular temperature interval. In one embodiment, the threshold is 31 ℃, considerably above the ambient point, although the threshold may be set up to 54 ℃, just below the second set point, without significant change in operation. A threshold from 10 deg.c above ambient temperature to a third set point may be used. Multiple thresholds may be used instead of tracking one threshold heating towards the second set point.

The threshold may be established by data analysis without accessing the design parameters of the system. When the temperature sensor data does not include setpoint data, the predetermined threshold analyzer for predicting flow cell heater failure is one of the detection parameters set by the configuration. The configuration analyzer may use logs of flow cell heater sensor data from sequencers located at multiple locations and operated by multiple independent operators to determine thresholds and/or margins above and/or below ambient temperature. In one embodiment, the configuration analyzer determines a first predetermined margin, also referred to as a threshold, above ambient temperature. A time series from a temperature sensor in the flow cell heater is tested to determine if samples in the time series are above an ambient temperature by a first predetermined margin. If the data in the temperature time series does not exceed the ambient temperature by a first predetermined margin, the flow cell heater may fail. More than one consecutive time series corresponding to a sequencing cycle may be tested to predict flow cell heater failure.

The flow cell may also be cooled to below ambient temperature during the sequencing cycle. To predict a fault in which the flow cell cools below ambient temperature, the configuration analyzer may determine a second predetermined margin, also referred to as a threshold, below ambient temperature. The configuration analyzer may use logs of flow cell heater sensor data from sequencers located at multiple locations and operated by multiple independent operators to determine the second margin and/or threshold. The time series of temperature sensors in the flow cell heater are tested to determine whether the samples in the time series are below the ambient temperature by a second predetermined margin. If cooled at the beginning of the cycle, the test can be performed early in the base recognition cycle, prior to a predetermined count of sensor measurements during the cycle. If the data in one or more consecutive sequencing cycles does not fall below the ambient temperature by a second predetermined margin, below a second threshold, a flow cell heater cooling failure can be predicted.

In one cycle, the number of sensor measurements that meet one or more thresholds may be counted. If the count of satisfactory temperature sensor data points in the evaluated process cycle is less than the predetermined count threshold, then a test is also applied to obtain a count of flow cell heater temperature data points for a previous (or subsequent) process cycle immediately before (or after) the most recent process cycle. If the second count of satisfactory temperature data points in the previous process cycle is less than the predetermined count threshold, the flow cell heater fault predictor 263 determines that the flow cell heater is malfunctioning and requires servicing in addition to the unsatisfactory first count. In one embodiment, the value of the predetermined count threshold is set to 5. The predetermined count threshold may range from 1 to 1000 or higher depending on the chemical duration and sensor reporting frequency. Component 263 notifies alarm service 121, and alarm service 121 sends an alarm to a technician. The alarm service 121 may also be packaged in a temperature margin fault alarm module. The data analyzer and flow cell heater fault predictor component 263 without the setpoint data component 243 may be packaged together as a temperature margin detection module.

The data analyzer 253 with setpoint data compares the flow cell heater temperature data in the most recent processing cycle to the setpoint data. Like the threshold analysis, if the temperature data is outside the predetermined allowable range of the setpoint data for the most recent processing cycle, the temperature data for the previous cycle immediately preceding the most recent processing cycle is tested. If the flow cell heater temperature data point for two consecutive cycles is outside of a predetermined allowable range of the setpoint data, the flow cell heater is determined to be faulty. In one embodiment, the allowed range is defined to be within 2 ℃ of the setpoint data. As described above for the threshold, a predetermined count of unsatisfied temperature data points may be used.

Reagent coolant instability prediction data and flow charts

Fig. 3 shows a time series of coolant temperature sensor data collected from eight sequencers M1-M8. The horizontal axis label represents data reported for six days. The legend in the upper right hand corner of graph 311 shows the serial numbers (Sn1 through Sn8) of the eight machines reporting sensor data. As described above, the data is noisy. Several factors can cause noise in the data, such as the operation of the mechanical system for cooling and condensation of reagent coolant dripping on the temperature sensor. External factors may also cause temperature changes, such as a door to a room in which the sequencing system is operating, which remains open when the external temperature is above room temperature. Transient oscillations in temperature, sometimes referred to as high frequency, are removed from the time series of coolant system temperature sensor data by applying a filter. Even if the amplitude of the signal is low, the high frequency signal has a high derivative, and thus causes a problem in signal processing. A derivative or other filter with a frequency cut-off threshold may be applied to eliminate high frequency or transient oscillations in the coolant temperature sensor data. The derivative filter also removes noise signals having frequencies above a cut-off threshold. FIG. 351 shows the cleaning temperature profile of sequencer M1 with sequence number Sn 1. In one embodiment, a derivative filter with a cutoff of 0.125 ℃ per minute is used to filter out noise in the smoothing time series 361. In another embodiment, a higher cut-off value is used, such as 0.5 ℃ per minute. More generally, the smoothing filter may smooth out oscillations having a predetermined rate of temperature change greater than or equal to 0.0625 degrees celsius per minute and less than or equal to 0.50 degrees celsius per minute.

The steady state period of the coolant system is represented by the relatively flat horizontal portion of the smooth line on the graph. Configuration engine 117 analyzes a time series log of temperature sensor readings in a sequencing system in which a coolant system fails to determine a predetermined rate of temperature change to predict an unstable coolant system. The predetermined inspection parameters may be periodically updated using a service log of the plurality of machines. The period of stable temperature operation is defined as the period of time in which the absolute value of the rate of change of the temperature readings in the smoothed time series varies by less than a predetermined rate of temperature change. In one embodiment, the absolute temperature rate of change for steady operation is less than 0.05 ℃ per minute. In another embodiment, a higher value may be used, such as 0.25 ℃ per minute, or a lower value may be used, such as 0.01 ℃ per minute, or in the range of 0.01 ℃ to 0.25 ℃ per minute. The stability criterion should not overlap with the smoothing filter parameters, otherwise the filter will stabilize all data analysis.

The disclosed techniques may analyze the number of cycles that the coolant system is operating steadily over a time window to predict an unstable coolant system. In one embodiment, the coolant system is considered to be in a steady state operating state if the total time of the steady state period is at least 14 hours within a 24 hour time window. In other embodiments, the time series of coolant temperature sensor data for a shorter time window may be analyzed to identify periods of steady state, such as 1 to 20 hours. In such an embodiment, the stability of the coolant system is predicted by testing a plurality of shorter time series. The configuration engine 117 analyzes a time series log of temperature sensor readings in a sequencing system in which the coolant system has failed to determine a predetermined number of steady state periods within a certain time window to predict an unstable coolant system.

Graph 361 shows that in the first half of the second day (9 months and 2 days), the temperature is rising and crosses the upper limit of 9 ℃. If the temperature rise is caused by an external factor, no warning should be given that the coolant system is unstable. It is assumed that the temperature rise is due to external factors, such as the door being open and warm air entering the room. When external factors are removed, such as the door being closed, the temperature will drop. If this occurs in a relatively short period of time, the reagent is less likely to be destroyed.

The disclosed technique distinguishes the effects of external factors on coolant system instability, thereby reducing false alarms. In one embodiment, the disclosed technique includes a predetermined detection parameter that defines how long the coolant system is allowed to operate above an upper temperature limit (9 ℃) before the alarm is issued. In such embodiments, the disclosed technique observes a reversal of the trend in the temperature profile 361. If analysis of the data in the graph indicates that the temperature is dropping toward the upper limit (9℃.), the disclosed technique determines the expected time at which the coolant system temperature will fall back into the normal operating temperature range. The total time the coolant system is expected to remain above the upper limit (9 ℃) is compared to the time allowed to operate above the upper reagent coolant limit. If the total expected time to exceed the upper limit is less than the allowed time to exceed the upper limit, no alarm will be issued. Reagent coolant instability prediction system 131 uses the detection parameters set by configuration engine 117 to test the time series data collected from the temperature sensors in the coolant system. The process steps may also be illustrated by the flowchart 400.

FIG. 4 is an example flow diagram illustrating one embodiment of a reagent coolant system stability prediction process 400. The process begins at step 401 with temperature sensor data from the sequencing hardware sensor readings and the Q-value database 151 being given as inputs at step 411. As described above, the data includes a time series of coolant temperature sensor data. At step 421, the time series data is arranged in chronological order. A derivative filter is applied to remove the noise data at step 431. A stable operating period of the coolant system within a predetermined time window is identified at step 341. At step 451, the count of stable operating periods is compared to a threshold. If the count is less than the threshold, an alert is sent to the alert service 121 that the coolant system needs service (step 450). If the number of cycles of the stabilizing operation is greater than the threshold value, errors of severity level 1 and severity level 2 of the coolant temperature sensor data are tested using the corresponding threshold values in step 361. The results of the severity level test are reported at step 371. The process ends at step 381.

Flow cell heater fault prediction data and flow charts

Fig. 5 includes a plot 511 of an example time series of flow cell heater temperature sensor data for a sequencing run completed within three days, along with an accompanying set point time series. At the beginning of the treatment cycle, the temperature of the flow cell was about 20 ℃. As the chemical treatment in the cycle progresses, the temperature of the flow cell rises to 55 ℃ for a short time and to 60 ℃ for another short time. At the end of the cycle, the temperature of the flow cell dropped back to 20 ℃ and stayed at 20 ℃ until the next cycle of chemical treatment. This pattern of temperature ramping and cooling of the flow cell repeats with each treatment cycle. As shown in graph 511, there are three setpoint data time series. The set point 1 data time series 523 corresponds to a temperature level of 20 ℃, the set point 2 data time series 615 corresponds to 55 ℃ and the set point 3 data time series 513 corresponds to 60 ℃.

Graph 511 shows that the flow cell heater works normally at the beginning of a sequencing run. As the processing cycle progresses, the flow cell temperature follows normal operation (517) that rises and falls according to the setpoint data. The current setpoint data is intended to float up and down over time as a time series throughout the process. In the figure, the three setpoints look like a continuous line, since the three day data is plotted on a short horizontal axis, but the current setpoints actually float up and down. However, the flow cell heater fails in the middle of the first day of operation, as indicated by label 519 on the graph. After the flow cell heater fails, the temperature of the flow cell is maintained at ambient level (521) and does not follow a ramp up and cool down to three set points. As shown in graph 551, failure of the flow cell heater resulted in A, G, T and a failure of the subsequent base recognition of the C base. The intensity of the four channels corresponding to the four bases drops sharply at the same time as the flow cell heater fails. Note that the temperature sensor data time series 517 and 521 as shown in graph 511 represent data from both flow cells on side a and side B. The simultaneous failure of both flow cells may be due to an upstream error (e.g., power failure, control board failure, etc.). For flow cell heater time series data that does not include setpoint data, the predetermined detection parameters determined by configuration engine 117 are used to determine flow cell heater failures. Two examples of such predetermined detection parameters include a first predetermined margin and a second predetermined margin, as explained above in the system description of the flow cell heater fault prediction system (FIG. 2). The process of testing the flow cell heater temperature time series data using either the setpoint data or the predetermined detection parameters is given in the flow chart below.

FIG. 6 is an example flow diagram illustrating one embodiment of a flow cell heater and/or coolant failure prediction process. The process starts in step 601. In step 613, hardware metrology data is given as input. As described above, the hardware metrics include flow cell heater temperature sensor data time series and setpoint data time series. At step 623, the setpoint data time-series is separated from the temperature sensor data time-series. At step 633, flow cell heater temperature sensor data for the most recent processing cycle is identified. If there are sufficient data points in the most recent process cycle (step 643), then the flow cell heater failure prediction process continues at step 653, otherwise steps 633 and 643 are repeated in the previous process cycle immediately prior to the most recent process cycle. In one embodiment, at step 543, at least five flow cell heater temperature sensor data points of a process cycle are required to satisfy the condition of sufficient data points.

At step 653, it is determined whether setpoint data is available. If setpoint data is available, the time series of most recently cycled flow cell heater temperature sensor data is tested at step 655. In step 663, the temperature data is tested to see if it is within a predetermined allowable range of the setpoint data. If the data value is within the predefined allowable range, control moves to step 662, which indicates that the flow cell heater is operating properly and does not require any service. Otherwise, control moves to step 673. If setpoint data is not available, the time series of flow cell heater temperature sensor data is tested using a threshold that uses a first predetermined margin above ambient temperature, as defined in FIG. 2. If the count of data points is above the threshold, the flow cell heater does not require any service (step 662). Otherwise, the above process of testing the temperature data points of a process cycle is repeated for a previous process cycle immediately preceding (or following) the most recent process cycle. If the test fails in two consecutive process cycles, it is determined that the flow cell heater needs servicing (step 683). The process is complete at step 685.

FIG. 7 is an example user interface (721) that may be used to present service alerts for a sequencing system. The results may also indicate the number of alarms that resulted in hardware replacement (725) and the number of unique sequencing tools for which alarms were generated (729). The monthly distribution of alarms may also be presented graphically (763). These alarms are expected to reduce unplanned downtime of the sequencing system.

Computer system

Fig. 8 is a simplified block diagram of a computer system 800, the computer system 800 may be used to implement the reagent coolant fault prediction system 131 of fig. 1 to detect coolant system instability. A similar computer system 900 may be used to implement the flow cell heater failure prediction system 141 of fig. 1 to detect flow cell heater failures over multiple cycles. Computer system 800 includes at least one Central Processing Unit (CPU)872, which communicates with a number of peripheral devices through a bus subsystem 855. These peripheral devices may include a storage subsystem 810 including, for example, a memory device and file storage subsystem 836, user interface input devices 838, user interface output devices 876, and a network interface subsystem 874. The input and output devices allow a user to interact with the computer system 800. Network interface subsystem 874 provides an interface to external networks, including interfaces to corresponding interface devices in other computer systems.

In one embodiment, reagent coolant failure prediction system 131 of FIG. 1 is communicatively linked to storage subsystem 810 and user interface input device 838. In another embodiment, flow cell heater fault prediction system 141 of FIG. 1 may be communicatively linked to storage subsystem 810 and user interface input device 838.

The user interface input device 838 may include a keyboard; a pointing device such as a mouse, trackball, touchpad, or tablet; a scanner; a touch screen incorporated into the display; audio input devices such as voice recognition systems and microphones; and other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways to input information into computer system 800.

User interface output devices 876 may include a display subsystem, a printer, a facsimile machine, or a non-visual display such as an audio output device. The display subsystem may include an LED display, a Cathode Ray Tube (CRT), a flat panel device such as a Liquid Crystal Display (LCD), a projection device, or some other mechanism for creating a visual image. The display subsystem may also provide non-visual displays such as audio output devices. In general, use of the term "output device" is intended to include all possible types of devices and ways to output information from computer system 800 to a user or to another machine or computer system.

Storage subsystem 810 stores programming and data structures that provide the functionality of some or all of the modules and methods described herein. These software modules are typically executed by the deep learning processor 878.

The deep learning processor 878 may be a Graphics Processing Unit (GPU) or a Field Programmable Gate Array (FPGA). The deep learning processor 878 may be hosted by a deep learning Cloud Platform, such as the Google Cloud Platform^TM、Xilinx^TMAnd Cirrascale^TM. Examples of deep learning processor 878 include Google's Tensor Processing Unit (TPU)^TMGX4 rack-mounted series^TMGX8 rack-mounted series^TMEqual-frame solution, Yingweida DGX-1^TMMicrosoft Stratix V FPGA^TMGraphcore's Intelligent Processor Unit (IPU)^TMHigh-pass Snapdragon processor^TMZeroth platform^TMVolta of great, English^TMDRIVE PX of great britain^TMJETSON TX1/TX2 MODULE of NVIDIA^TMIntel Nirvana^TMMovidius VPU^TMFuji Tong DPI^TMARM dynamicIQ^TM、IBM TrueNorth^TMAnd so on.

Memory subsystem 822, used in storage subsystem 810, may include a number of memories, including a main Random Access Memory (RAM)832 for storing instructions and data during program execution and a Read Only Memory (ROM)834 for storing fixed instructions. The file storage subsystem 836 may provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. Modules implementing the functionality of certain embodiments may be stored by the file storage subsystem 836 in the storage subsystem 910 or in other machines accessible to the processor.

Bus subsystem 855 provides a mechanism for the various components and subsystems of computer system 800 to communicate with one another as intended. Although bus subsystem 855 is shown schematically as a single bus, alternative embodiments of the bus subsystem may use multiple buses.

The computer system 800 itself may be of various types, including a personal computer, portable computer, workstation, computer terminal, network computer, television, mainframe, server farm, widely distributed group of loosely networked computers, or any other data processing system or user device. Due to the ever-changing nature of computers and networks, the description of computer system 800 depicted in FIG. 8 is merely a specific example for purposes of illustrating specific embodiments of the disclosed technology. Many other configurations of computer system 800 are possible with more or fewer components than the computer system shown in FIG. 8.

The foregoing description is presented to enable the manufacture and use of the disclosed technology. Various modifications to the disclosed embodiments will be readily apparent, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the disclosed technology. Thus, the disclosed technology is not limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein. The scope of the disclosed technology is defined by the following claims.

Description of the preferred embodiments

Reagent coolant instability prediction system

The disclosed technology relates to detection of coolant system instability, which reduces false alarms.

The disclosed technology may be practiced as a system, method, or article of manufacture. One or more features of the embodiments may be combined with the basic embodiments. Non-mutually exclusive implementations are taught as combinable. One or more features of the embodiments may be combined with other embodiments. The present disclosure alerts the user of these options periodically. The omission of some embodiments that repeat the recitation of these options should not be taken as limiting the combination of the teachings in the preceding sections — these recitations are therefore incorporated by reference into each of the embodiments below.

A first system embodiment of the disclosed technology includes one or more processors and a memory coupled to the processors. The memory is loaded with computer instructions for detecting an instability of the coolant system, the computer instructions configured to generate fewer false alarms than simple threshold alarms. When executed on the processor, the computer instructions apply a smoothing function to the time series of coolant temperature sensor data to reduce transient oscillations. Transient oscillations in temperature are sometimes referred to as high frequency oscillations. Application of this function produces a smooth time series of coolant temperature sensor data. The system tests a smooth time series of coolant temperature sensor data within a predefined time window of a stable temperature operating cycle. The temperature readings in the smoothed time series vary by less than a predetermined rate of temperature change. The system determines that the coolant system is unstable when less than 50% of the time window is stable, and reports a service demand when the total number of cycles of stable temperature operation is less than a predetermined stability measurement.

The system embodiments and other systems disclosed optionally include one or more of the following features. The system may also include features described in connection with the disclosed methods. For the sake of brevity, alternative combinations of features of the system are not separately enumerated. Features applicable to the systems, methods, and articles of manufacture are not repeated for the set of base features of each legal class. The reader will understand how the features identified in this section can be readily combined with the basic features in other legal categories.

The system determines a predetermined rate of temperature change based on devices located at a plurality of locations and operated by a plurality of independent operators. The system includes logic that causes the device configuration to record and report temperature sensor readings and store a collection log of the temperature sensor readings. The system includes analyzing a time series of temperature sensor readings in an instance of the equipment in which the coolant system failed and determining a predetermined rate of temperature change. The predetermined rate of temperature change is stored for use in determining the coolant system is unstable.

The system includes updating the predetermined rate of temperature change based on devices located at a plurality of locations and operated by a plurality of independent operators. The system includes logic that causes the device to configure the recording and reporting of temperature sensor readings. The system collects and stores a log of temperature sensor readings and a service log after an instability notification. The system includes analyzing a time series of temperature sensor readings in an instance of the device for which the coolant system generated the notification and a service subsequent to the notification. The system determines an update to a predetermined rate of temperature change based on an analysis of the time series of temperature sensor readings and the service log data after the notification. The system stores the updated predetermined rate of temperature change for use in determining the coolant system is unstable.

The system includes a cloud-based active maintenance analyzer for accessing a log of temperature sensor readings for a particular coolant system. The cloud-based active maintenance analyzer applies a smoothing function, determines that a smoothed time series of coolant temperature sensor data does not meet a stable temperature operating standard within a predefined time window, and generates a notification.

The system filters out duplicate notifications and submits the filtered notifications to the customer relationship system for tracking. The system filters out duplicate notifications and submits the filtered notifications to the operator of the sequencer, including the coolant system.

The sequencing system can be located in at least 50 multiple locations. The sequencing system can be operated by at least 20 independent operators.

The system may require higher stability, applying predetermined stability measurements for 75% or 90% time windows. The time window may be between 4 and 48 hours. One choice of time window may be about 24 hours. Another option is 6 to 36 hours.

The system may apply a smoothing function to the time series data using a derivative filter. The smoothing function may be tuned to eliminate transient oscillations that produce a rate of temperature change of 0.125 or 0.25 degrees celsius per minute or more. Or the smoothing function may be tuned to eliminate transient oscillations that produce a rate of temperature change greater than or equal to 0.625 degrees celsius per minute and less than or equal to 0.50 degrees celsius per minute.

The system may use a temperature change criterion of less than 0.010, 0.05 or 0.25 degrees celsius per minute as the predetermined stability measurement, or within a range between any of these criteria.

The system may automatically attach a report of system instability determinations and smooth coolant system temperature sensor data in a graph or table for viewing by a user.

The system includes comparing the average temperature and the median temperature during stable operation and reporting a severity level 1 error above a first threshold. The system also includes reporting a severity level 2 error if the average and median temperatures during steady operation are above a second threshold.

The system includes applying a derivative filter that removes transient oscillations at an absolute rate of temperature change of at least 0.125 degrees celsius per minute. The system includes testing a smoothed time series of coolant temperature sensor data over a predetermined time window of a stable temperature operating cycle in which temperature readings in the smoothed time series vary by less than a predetermined absolute rate of temperature change of 0.05 degrees celsius per minute.

Other embodiments may include a non-transitory computer readable storage medium storing instructions executable by a processor to perform the system functions described above. Yet another embodiment may include a method of performing the functions of the system described above.

A second system embodiment of the disclosed technology includes one or more processors and a memory coupled to the processors. The memory contains computer instructions for detecting and alerting a technician that the cooling system of the sequencer is unstable. The alarm system includes a time series smoothing module that receives temperature sensor data from sensors exposed in a coolant system of the sequencer and generates a smoothed temperature time series. The temperature instability detection module receives the smoothed temperature time series. The temperature instability detection module reports changes between smooth consecutive data in the smooth temperature time series that exceed a predetermined temperature change as instability and determines a degree of instability. The system includes a temperature instability alert module that receives a report of a degree of instability and generates an alert to a technician when the degree of instability exceeds a predetermined threshold.

The system embodiments and other systems disclosed optionally include one or more of the following features. The system may also include features described in connection with the disclosed methods. For the sake of brevity, alternative combinations of features of the system are not separately enumerated. Features applicable to the systems, methods, and articles of manufacture are not repeated for the set of base features of each legal class. The reader will understand how the features identified in this section can be readily combined with the basic features in other legal categories.

The system includes a sensor exposure module on the sequencer that exposes a temperature sensor in the coolant system and reports temperature sensor data from the exposed temperature sensor. The system includes a log collection module that receives temperature sensor data from a plurality of devices including a sequencer. The collection module makes temperature sensor data from a coolant system of the sequencer available to the time series smoothing module.

The system may update various predetermined detection parameters for use by the alarm system. Three examples of predetermined detection parameter updates are as follows.

The system includes a threshold update component that modifies a predetermined threshold. The threshold update component also includes a log collection module and a threshold adjustment module. The log collection module receives temperature sensor data from a plurality of devices including a sequencer. The log collection module makes temperature sensor data from a coolant system of the sequencer available to the threshold adjustment module. The threshold adjustment module receives the new temperature sensor data, modifies the predetermined threshold based on the new temperature sensor data, and stores the modified predetermined threshold for use by the temperature instability alert module.

The system includes a threshold update component that modifies a predetermined temperature change. The threshold update component also includes a log collection module and a threshold adjustment module. The log collection module receives temperature sensor data from a plurality of devices including a sequencer. The log collection module makes temperature sensor data from a coolant system of the sequencer available to the threshold adjustment module. The threshold adjustment module receives the new temperature sensor data, modifies the predetermined temperature change based on the new temperature sensor data, and stores the modified predetermined temperature change for use by the temperature instability detection module.

The system includes a threshold update component that modifies a smoothing module parameter. The threshold update component also includes a log collection module and a threshold adjustment module. The log collection module receives temperature sensor data from a plurality of devices including a sequencer. The log collection module makes temperature sensor data from a coolant system of the sequencer available to the threshold adjustment module. The threshold adjustment module receives new temperature sensor data, modifies parameters of the smoothing module based on the new temperature sensor data, and stores the modified parameters for smoothing for use by the time series smoothing module.

The threshold update component and system also includes a customer relationship module and a threshold adjustment module. The customer relationship module tracks alarms, faults, and solutions for a plurality of devices including a sequencer. The threshold adjustment module also receives fault and solution data from the customer relationship module. When modifying the parameters used by any of the time series smoothing module, the temperature instability detection module, or the temperature instability alert module, it distinguishes between missing faults and false alarms.

Other embodiments may include a non-transitory computer readable storage medium storing instructions executable by a processor to perform the system functions described above. Yet another embodiment may include a method of performing the functions of the system described above.

A first method embodiment of the disclosed technology includes detecting an instability of the coolant system, which reduces false alarms. The method includes applying a smoothing function to the time series of coolant temperature sensor data to reduce transient oscillations. The application of the filter produces a smooth time series of coolant temperature sensor data. The method includes testing a smoothed time series of coolant temperature sensor data within a predetermined time window of a stable temperature operating cycle. The temperature readings in the smoothed time series vary by less than a predetermined rate of temperature change. Finally, the method determines that the coolant system is unstable and reports a service requirement when the total number of cycles of stable temperature operation is less than a predetermined stability measurement.

The method embodiments and other methods disclosed optionally include one or more of the following features. The method may also include features described in connection with the disclosed system. The reader will understand how the features identified in this section can be readily combined with the basic features in other legal categories.

The method includes determining a predetermined rate of temperature change based on devices located at a plurality of locations and operated by a plurality of independent operators. The method includes having the configuration of the device record and report temperature sensor readings and storing a collection log of the temperature sensor readings. The method includes analyzing a time series of temperature sensor readings in an instance of the equipment in which the coolant system failed and determining a predetermined rate of temperature change. The predetermined rate of temperature change is stored for use in determining the coolant system is unstable.

The method includes updating the predetermined rate of temperature change based on devices located at a plurality of locations and operated by a plurality of independent operators. The method includes causing a configuration of the device to record and report temperature sensor readings. The method includes collecting and storing temperature sensor readings and a service log after an instability notification. The method includes analyzing a time series of temperature sensor readings in an instance of the device for which the coolant system generated the notification and a service subsequent to the notification. The method includes determining an update to the predetermined rate of temperature change based on an analysis of the time series of temperature sensor readings and the service log data after the notification. The updated predetermined rate of temperature change is stored for use in determining the coolant system is unstable.

The method includes accessing a log of temperature sensor readings from a particular coolant system. The method includes applying a smoothing function to determine that a smoothed time series of coolant temperature sensor data in a predefined time window does not meet a stable temperature operating criterion, and generating a notification.

The method includes filtering out duplicate notifications and submitting the filtered notifications to a customer relationship management system for tracking. The method includes filtering out duplicate notifications and submitting the filtered notifications to an operator of a sequencer that includes a coolant system.

The sequencing system can be located in at least 50 multiple locations. The sequencing system can be operated by at least 20 independent operators.

Higher stability may be required using this method, applying predetermined stability measurements for 75% or 90% time windows. The time window may be between 4 and 48 hours. One choice of time window may be about 24 hours. Another option is 6 to 36 hours.

The method may include applying a smoothing function to the time series data using a derivative filter. The smoothing function may be tuned to eliminate transient oscillations that produce a rate of temperature change of 0.125 or 0.25 degrees celsius per minute or higher. Or it may be tuned to eliminate transient oscillations that produce a rate of temperature change greater than or equal to 0.625 degrees celsius per minute and less than or equal to 0.50 degrees celsius per minute.

The method may include using a temperature change criterion of less than 0.010, 0.05 or 0.25 degrees celsius per minute as the predetermined stability measurement, or within a range between any of these criteria.

The method may include automatically appending, in graph or table form, a report of the system instability determination along with smoothed coolant system temperature sensor data for viewing by a user.

The method includes comparing the average temperature and the median temperature for the stable operating period and reporting a severity level 1 error above a first threshold. The method also includes reporting a severity level 2 error if the average and median temperature of the stable operating period is above a second threshold.

The method includes applying a derivative filter that removes transient oscillations of an absolute rate of temperature change of at least 0.125 degrees celsius per minute. The system includes testing a smoothed time series of coolant temperature sensor data over a predetermined time window of a stable temperature operating cycle in which temperature readings in the smoothed time series vary by less than a predetermined absolute rate of temperature change of 0.05 degrees celsius per minute.

Each of the features discussed in this particular implementation section of the system implementation are equally applicable to this method implementation. As noted above, not all system features are repeated here and should be considered repeated by reference.

Other embodiments may include a set of one or more non-transitory computer-readable storage media collectively storing computer program instructions executable by one or more processors to detect an instability of a coolant system. When executed on one or more processors, the computer program instructions implement the method, including detecting an instability of the coolant system to reduce false alarms. The method includes applying a smoothing function to the time series of coolant temperature sensor data to reduce transient oscillations. The application of the filter produces a smooth time series of coolant temperature sensor data. The method includes testing a smoothed time series of coolant temperature sensor data within a predetermined time window of a stable temperature operating cycle. The temperature readings in the smoothed time series vary by less than a predetermined rate of temperature change. Finally, the method determines that the coolant system is unstable and reports a service requirement when the total number of cycles of stable temperature operation is less than a predetermined stability measurement. Yet another embodiment may include a system comprising a memory and one or more processors operable to execute instructions stored in the memory to perform the first method described above.

Computer-readable medium (CRM) embodiments of the disclosed technology include one or more non-transitory computer-readable storage media that, when executed on one or more processors, are impressed with computer program instructions to carry out the above-described methods.

This CRM implementation includes one or more of the following functions. CRM embodiments may also include features described in connection with the systems and methods disclosed above. The method includes determining a predetermined rate of temperature change based on devices located at a plurality of locations and operated by a plurality of independent operators. The method includes having the configuration of the device record and report temperature sensor readings and storing a collection log of the temperature sensor readings. The method includes analyzing a time series of temperature sensor readings in an instance of the equipment in which the coolant system failed and determining a predetermined rate of temperature change. The predetermined rate of temperature change is stored for use in determining the coolant system is unstable.

CRM implemented methods include updating a predetermined rate of temperature change based on devices located at multiple locations and operated by multiple independent operators. The method includes causing a configuration of the device to record and report temperature sensor readings. The method includes collecting and storing temperature sensor readings and a service log after an instability notification. The method includes analyzing a time series of temperature sensor readings in an instance of the device for which the coolant system generated the notification and a service following the notification. The method includes determining an update to the predetermined rate of temperature change based on an analysis of the time series of temperature sensor readings and the service log data after the notification. The updated predetermined rate of temperature change is stored for use in determining the coolant system is unstable.

A CRM implemented method includes accessing a log of temperature sensor readings from a particular coolant system. The method includes applying a smoothing function to determine that a smoothed time series of coolant temperature sensor data in a predefined time window does not meet a stable temperature operating criterion, and generating a notification.

A CRM-implemented method includes filtering out duplicate notifications and submitting the filtered notifications to a customer relationship management system for tracking. The method includes filtering out duplicate notifications and submitting the filtered notifications to an operator of a sequencer that includes a coolant system.

The sequencing system can be located in at least 50 multiple locations. The sequencing system can be operated by at least 20 independent operators.

Higher stability may be required using this method, applying predetermined stability measurements for 75% or 90% time windows. The time window may be between 4 and 48 hours. One choice of time window may be about 24 hours. Another option is 6 to 36 hours.

Such a CRM-implemented method may include applying a smoothing function to the time series data using a derivative filter. The smoothing function may be tuned to eliminate transient oscillations that produce a rate of temperature change of 0.125 or 0.25 degrees celsius per minute or higher. Or it may be tuned to eliminate transient oscillations that produce a rate of temperature change greater than or equal to 0.625 degrees celsius per minute and less than or equal to 0.50 degrees celsius per minute.

CRM implemented methods may include using temperature change criteria of less than 0.010, 0.05, or 0.25 degrees celsius per minute as the predetermined stability measurement, or within a range between any of these criteria.

CRM implemented methods may include automatically appending a report of system instability determinations, in graph or table form, along with smoothed coolant system temperature sensor data for viewing by a user.

A CRM implemented method includes comparing the average and median temperatures for a stable operating cycle and reporting a severity level 1 error above a first threshold. The method also includes reporting a severity level 2 error if the average and median temperature of the stable operating period is above a second threshold.

A CRM implemented method includes applying a derivative filter that removes transient oscillations having an absolute rate of change of temperature of at least 0.125 degrees celsius per minute. The system includes testing a smoothed time series of coolant temperature sensor data over a predetermined time window of a period of stable temperature operation, during which period temperature readings in the smoothed time series vary by less than a predetermined absolute rate of temperature change of 0.05 degrees celsius per minute.

A second method embodiment of the disclosed technology includes detecting that a sequencer has an unstable coolant system. The method includes receiving temperature sensor data obtained from a sensor exposed to a coolant system of a sequencer. The method includes applying a smoothing function to the temperature sensor data to produce a smoothed temperature time series. The method includes determining a change between smooth consecutive data in a smooth temperature time series that exceeds a predetermined temperature change. The method includes determining a degree of instability based on the determined change. The method includes generating an alert indicating that the sequence has an unstable coolant system when the degree of instability exceeds a predetermined threshold.

Other methods implemented and disclosed by the method optionally include one or more of the following features. The method may also include features described in connection with the disclosed system. The reader will understand how the features identified in this section can be readily combined with the basic features in other legal categories.

The temperature sensor data is determined based on sensors located at a plurality of locations and operated by a plurality of independent operators. The method includes configuring the device to record and report temperature sensor readings. The method includes collecting a log of temperature sensor readings. The method includes analyzing a time series of temperature sensor readings in an instance of the equipment in which the coolant system failed and determining a predetermined temperature change. The method includes storing a predetermined temperature change for determining a degree of instability.

The method also includes receiving temperature sensor data from a plurality of devices including a sequencer. The method includes receiving new temperature sensor data from a plurality of devices. The method includes modifying the predetermined threshold based on the new temperature sensor data and storing the modified predetermined threshold to generate an alert.

The method also includes receiving temperature sensor data from a plurality of devices including a sequencer. The method includes receiving new temperature sensor data from a plurality of devices. The method includes modifying the predetermined temperature change based on the new temperature sensor data. The method includes storing the modified predetermined temperature change for determining a change that exceeds the predetermined temperature change.

The method includes threshold updating, including receiving temperature sensor data from a plurality of devices including a sequencer. The method includes receiving new temperature sensor data from a plurality of devices. The method includes modifying parameters of the smoothing function based on the new temperature sensor data and storing the modified parameters of the smoothing function.

The method includes tracking alarms, faults, and solutions for a plurality of devices including a sequencer. The method includes receiving fault and solution data from a customer relationship module. The method includes distinguishing between missed faults and false alarms when modifying parameters of a smoothing function, determining a degree of instability, or generating an alarm.

The smoothing function is applied by a derivative filter. Applying a smoothing function may eliminate transient oscillations that produce a rate of change of 0.125 degrees celsius per minute or higher.

The method includes comparing the average and median temperatures of the stable operating cycle and reporting a first degree of instability when the average and median temperatures change beyond a first threshold.

The method includes comparing the average and median temperatures of the stable operating cycle and reporting a second degree of instability when the average and median temperatures change beyond a second threshold.

System embodiments of the technology include one or more processors coupled to a memory loaded with computer instructions that, when executed by the one or more processors, cause the system to perform a method according to any one of the methods described above. Each of the features discussed in the above section for the specific embodiments of the second method embodiment are equally applicable to this system embodiment.

CRM implementations of the technology include a non-transitory computer-readable storage medium imprinted with computer program instructions. The instructions, when executed on one or more processors, implement a method according to any of the methods described above.

Each of the features discussed in this particular implementation section of the system implementation are equally applicable to the CRM implementation. As noted above, not all system features are repeated here and should be considered repeated by reference.

Flow cell heater fault prediction system

The disclosed technology relates to detecting flow cell heater faults over multiple cycles in a system without a set point.

A first system embodiment of the disclosed technology includes one or more processors and a memory coupled to the processors. The memory is loaded with computer instructions for detecting flow cell heater failure over a plurality of cycles in a system without a setpoint. When executed on a processor, computer instructions test a time series of flow cell heater temperature sensor data in a base recognition cycle to determine whether the most recent or immediately recent base recognition cycle has enough flow cell heater temperature sensor data points to evaluate. In some embodiments, a count of pool heater temperature sensor data points sufficient to be evaluated corresponds to a particular time in a base recognition cycle at which the flow pool heater temperature is deemed to exceed the ambient operating temperature by a first predetermined margin. The instructions also perform determining whether a most recent flow cell heater temperature sensor data in the evaluation loop exceeds an ambient operating temperature by a first predetermined margin. When the evaluated loop flow cell heater temperature sensor data does not exceed the operating temperature by a first predetermined margin, it is determined whether the flow cell heater temperature sensor data in a successive loop immediately following the evaluated loop exceeds the ambient operating temperature by a first predetermined margin. Then, in the evaluated cycle and the successive cycles, when the evaluated cycle flow cell heater temperature sensor data does not exceed the operating temperature by a first predetermined margin, determining that the flow cell heater is malfunctioning and reporting a service demand.

The system embodiments and other systems disclosed optionally include one or more of the following features. The system may also include features described in connection with the disclosed methods. For the sake of brevity, alternative combinations of features of the system are not separately enumerated. Features applicable to the system, method, and article of manufacture are not repeated for the set of base features of each legal class. The reader will understand how the features identified in this section can be readily combined with the basic features in other legal categories.

The system determines a first predetermined margin based on equipment located at a plurality of locations and operated by a plurality of independent operators. The system includes logic that causes the device configuration to record and report temperature sensor readings and store a collection log of the temperature sensor readings. The system includes logic for analyzing a time series of temperature sensor readings in an instance of the device where the flow cell heater failed and determining a first predetermined margin. A first predetermined temperature margin is stored for determining whether the flow cell heater is malfunctioning.

The system updates the first predetermined margin based on equipment located at a plurality of locations and operated by a plurality of independent operators. The system includes logic that causes the configuration of the device to record and report temperature sensor readings and a service log after reporting service requirements. The system stores the collected logs. The system includes analyzing a time series of temperature sensor readings in flow cell heater health and malfunctioning equipment instances, and a service log after service demand reporting. The system determines an update to the first predetermined margin based on the analysis.

The system includes a cloud-based active maintenance analyzer to access a log of temperature sensor readings from a particular flow cell heater. The cloud-based active maintenance analyzer performs applications of testing, determining, and reporting of service requirements of the cloud-based active maintenance analyzer.

The system filters out duplicate notifications and submits the filtered notifications to a customer relationship management system for tracking. The system filters out duplicate notifications and submits the filtered notifications to a sequencer operator that includes a flow cell heater system.

The system determines whether a count of pool heater temperature sensor data points corresponding to a time in the base recognition cycle at which the flow pool heater temperature should exceed the ambient operating temperature is sufficient to be evaluated beyond a first predetermined margin.

On the lower temperature side, when the flow cell is assumed to be cooled below ambient temperature, the instructions may further perform determining whether the evaluated one or more cell heater temperature sensor data points obtained prior to counting are below ambient operating temperature minus a second predetermined margin. When the evaluated loop flow cell heater temperature sensor data is less than the operating temperature by a second predetermined margin, it is determined whether flow cell heater temperature sensor data acquired prior to counting in a successive loop immediately following the evaluated loop is less than the ambient operating temperature by the second predetermined margin. Then, in the evaluated cycle and the continuous cycle, when the evaluated cycle flow cell heater temperature sensor data is less than the operating temperature by a second predetermined margin, a flow cell cooling failure is determined and a service demand is reported.

The system determines a second predetermined margin based on equipment located at a plurality of locations and operated by a plurality of independent operators. The system includes logic that causes the device configuration to record and report temperature sensor readings and store a collection log of the temperature sensor readings. The system includes analyzing a time series of temperature sensor readings in an instance of the device where the flow cell heater failed and determining a second predetermined margin. A second predetermined temperature margin is stored for determining whether the flow cell heater is malfunctioning.

The system updates the second predetermined margin based on equipment located at a plurality of locations and operated by a plurality of independent operators. The system includes logic that causes the device configuration to record and report temperature sensor readings and a service log after reporting service requirements. The system stores the collected logs. The system includes analyzing a time series of temperature sensor readings in flow cell heater health and malfunctioning equipment instances, and a service log after service demand reporting. The system determines an update to the second predetermined margin based on the analysis.

Other embodiments may include a non-transitory computer readable storage medium storing instructions executable by a processor to perform the system functions described above. Yet another embodiment may include a method of performing the functions of the system described above.

A second system embodiment includes an alarm system for detecting and alerting technicians of a flow cell temperature control system failure of a sequencer. The temperature detection module analyzes a time series of flow cell temperature sensor data across a base recognition cycle. The temperature margin detection module determines whether the most recent or immediately recent base recognition cycle has enough flow cell temperature sensor data points to evaluate. It also determines whether the temperature sensor data in the evaluation cycle exceeds an ambient operating temperature by a first predetermined margin. Once the evaluated cycle flow cell temperature sensor data fails to exceed the ambient operating temperature by a first predetermined margin, flow cell temperature sensor data in successive cycles immediately before or after the evaluated cycle is determined. The temperature margin detection module sets a first fault condition if the flow cell temperature sensor data in successive cycles does not exceed the ambient operating temperature by a first predetermined margin. The system also includes a temperature margin fault alarm module that receives the determination of the first fault condition and generates a flow cell heater alarm to a technician.

The system embodiments and other systems disclosed optionally include one or more of the following features. The system may also include features described in connection with the disclosed methods. For the sake of brevity, alternative combinations of features of the system are not separately enumerated. Features applicable to the system, method, and article of manufacture are not repeated for the set of base features of each legal class. The reader will understand how the features identified in this section can be readily combined with the basic features in other legal categories.

The temperature margin detection module is further configured to determine a flow cell coolant fault by analyzing a time series of flow cell heater temperature sensor data across a base recognition cycle. The system determines whether the most recent or immediately recent base recognition cycle has a flow cell temperature sensor data point to be evaluated during the flow cell cooling sub-cycle. The system determines whether the temperature sensor data in the evaluation cycle is cooled to a second predetermined margin below the ambient operating temperature. When the evaluated loop flow cell temperature sensor data fails to fall below the ambient operating temperature by a second predetermined margin, the system determines the flow cell heater temperature sensor data in successive loops immediately before or after the evaluated loop. The system sets a second fault condition if the continuous loop temperature sensor data fails to fall below the ambient operating temperature by a second predetermined margin. The temperature margin fault alarm module receives a determination of a second fault condition and generates a flow cell coolant alarm to a technician.

The system includes a sensor exposure module on the sequencer that exposes a temperature sensor in the flow cell temperature control system. The sensor exposure module also reports temperature sensor data from the exposed temperature sensors. The log collection module receives temperature sensor data from a plurality of devices including a sequencer. The log collection module makes temperature sensor data from a flow cell temperature control system of the sequencer available to the temperature margin detection module.

The system includes updating the temperature margin. The log collection module receives temperature sensor data from a plurality of devices including a sequencer. The log collection module makes temperature sensor data from a flow cell temperature control system of the sequencer available to the temperature margin adjustment module. The temperature margin adjustment module receives new temperature sensor data from a plurality of devices. It modifies the first predetermined margin based on the new temperature sensor data and stores the modified first predetermined threshold for use by the temperature margin fault alarm module.

The system includes updating the temperature margin. The log collection module receives temperature sensor data from a plurality of devices including a sequencer. The log collection module makes temperature sensor data from a flow cell temperature control system of the sequencer available to the temperature margin adjustment module. The temperature margin adjustment module receives new temperature sensor data from a plurality of devices. It modifies the second predetermined margin based on the new temperature sensor data and stores the modified second predetermined threshold for use by the temperature margin fault alarm module.

The system utilizes CRM data for temperature margin updates. The customer relationship module tracks alarms, faults, and solutions for a plurality of devices including a sequencer. The temperature margin adjustment module receives fault and solution data from the customer relationship module. When modifying the parameter implemented by the temperature margin adjustment module, it distinguishes between missed faults and false alarms.

Other embodiments may include a non-transitory computer readable storage medium storing instructions executable by a processor to perform the system functions described above. Yet another embodiment may include a method of performing the functions of the system described above.

A first method embodiment of the disclosed technology includes detecting flow cell heater failure over multiple cycles in a system without a setpoint. The method includes testing a time series of flow cell heater temperature sensor data defined during a processing cycle to determine how many points were recorded above a threshold during a most recent processing cycle. The threshold is determined based on the likelihood of measurements being made during a particular temperature interval. When the first point count recorded in the most recent processing cycle is less than the predetermined count threshold, the method repeats the test for the immediately preceding processing cycle prior to the most recent processing cycle and determines how many points were recorded above the threshold in the preceding processing cycle. The threshold is determined based on the likelihood of measurements being made during a particular temperature interval. The method determines that the flow cell heater is malfunctioning and reports a service requirement when a second count of points recorded in a previous processing cycle is less than a predetermined count threshold, in addition to the first count of points recorded in a previous processing cycle being less than the predetermined count threshold.

The method embodiments and other methods disclosed optionally include one or more of the following features. The method may also include features described in connection with the disclosed system. The reader will understand how the features identified in this section can be readily combined with the basic features in other legal categories.

The method includes determining a first predetermined margin value based on equipment located at a plurality of locations and operated by a plurality of independent operators. The method includes having the configuration of the device record and report temperature sensor readings and storing a collection log of the temperature sensor readings. The method includes analyzing a time series of temperature sensor readings in an instance of the device where the flow cell heater failed and determining a first predetermined margin. A first predetermined temperature margin is stored for determining whether the flow cell heater is malfunctioning.

The method includes updating the first predetermined margin value based on equipment located at a plurality of locations and operated by a plurality of independent operators. The method includes having the device configuration record and report temperature sensor readings and a service log after reporting service requirements. The method includes storing the collected logs. The method includes analyzing a time series of temperature sensor readings in flow cell heater health and malfunctioning equipment instances and a service log after reporting service requirements. The method includes determining an update to the first predetermined margin based on the analysis.

The method includes accessing a log of temperature sensor readings from a particular flow cell heater. The method includes executing an application that tests, determines, and reports service requirements from a cloud-based active maintenance analyzer.

The method includes filtering out duplicate notifications and submitting the filtered notifications to a customer relationship management system for tracking. The method includes filtering out duplicate notifications and submitting the filtered notifications to an operator of a sequencer that includes a flow cell heater system.

The method includes determining whether a count of cell heater temperature sensor data points corresponding to a time in a base recognition cycle at which a flow cell heater temperature is deemed to exceed an ambient operating temperature is sufficient to be evaluated beyond a first predetermined margin.

On the lower temperature side, when the flow cell is deemed to be cooling below ambient temperature, the instructions may further perform determining whether one or more cell heater temperature sensor data points in the evaluation obtained prior to counting are below ambient operating temperature minus a second predetermined margin. When the evaluated loop flow cell heater temperature sensor data fails to be less than the operating temperature by a second predetermined margin, it is determined whether flow cell heater temperature sensor data obtained prior to counting in consecutive loops immediately following the evaluated loop is less than the ambient operating temperature by the second predetermined margin. Then, in the evaluated cycle and the consecutive cycle, when the evaluated cycle flow cell heater temperature sensor data fails to be less than the operating temperature by a second predetermined margin, a flow cell cooling failure is determined and a service demand is reported.

The method includes determining a second predetermined margin value based on equipment located at a plurality of locations and operated by a plurality of independent operators. The method includes logic for configuring the device to record and report temperature sensor readings and to store a collection log of the temperature sensor readings. The method includes analyzing a time series of temperature sensor readings in an instance of the device where the flow cell heater fails and determining a second predetermined margin. A second predetermined temperature margin is stored for determining whether the flow cell heater is malfunctioning.

The method includes updating the second predetermined margin value based on equipment located at a plurality of locations and operated by a plurality of independent operators. The method includes having the device configuration record and report temperature sensor readings and a service log after reporting service requirements. The method includes storing the collected logs. The method includes analyzing a time series of temperature sensor readings in a flow cell heater health and malfunctioning equipment instance, and reporting a service log after service demand. The method includes determining an update to a second predetermined margin based on the analysis.

Other embodiments may include a set of one or more non-transitory computer-readable storage media collectively storing computer program instructions executable by one or more processors. When executed on one or more processors, computer program instructions implement the method, comprising detecting a flow cell heater failure in a plurality of cycles in a base recognition system. The method includes testing a time series of flow cell heater temperature sensor data defined during a processing cycle to determine how many points were recorded above a threshold during a most recent processing cycle. The threshold is determined based on the likelihood of measurements being made during a particular temperature interval. When the first point count recorded in the most recent processing cycle is less than the predetermined count threshold, the method repeats the test for the immediately preceding processing cycle prior to the most recent processing cycle and determines how many points were recorded above the threshold in the preceding processing cycle. The threshold is determined based on the likelihood of measurements being made during a particular temperature interval. The method determines that the flow cell heater is malfunctioning and reports a service requirement when a second count of points recorded in a previous processing cycle is less than a predetermined count threshold, in addition to the first count of points recorded in a previous processing cycle being less than the predetermined count threshold.

The method embodiments and other methods disclosed optionally include one or more of the following features. The method may also include features described in connection with the disclosed system. The reader will understand how the features identified in this section can be readily combined with the basic features in other legal categories.

A CRM-implemented method includes determining a first predetermined margin based on devices located at a plurality of locations and operated by a plurality of independent operators. The method includes configuring the device to record and report temperature sensor readings and storing a collection log of the temperature sensor readings. The method includes analyzing a time series of temperature sensor readings in an instance of the device where the flow cell heater failed and determining a first predetermined margin. A first predetermined temperature margin is stored for determining whether the flow cell heater is malfunctioning.

A CRM-implemented method includes updating a first predetermined margin based on equipment located at a plurality of locations and operated by a plurality of independent operators. The method includes having the device configuration record and report temperature sensor readings and a service log after reporting service requirements. The method includes storing the collected logs. The method includes analyzing a time series of temperature sensor readings in a flow cell heater health and malfunctioning equipment instance, and reporting a service log after a service demand. The method includes determining an update to the first predetermined margin based on the analysis.

A CRM implemented method includes accessing a log of temperature sensor readings from a particular flow cell heater. The method includes executing an application that tests, determines, and reports service requirements from a cloud-based active maintenance analyzer.

A CRM-implemented method includes filtering out duplicate notifications and submitting the filtered notifications to a customer relationship management system for tracking. The method includes filtering out duplicate notifications and submitting the filtered notifications to an operator of a sequencer that includes a flow cell heater system.

A CRM-implemented method includes determining whether a count of cell heater temperature sensor data points corresponding to a time in a base recognition cycle at which a flow cell heater temperature is deemed to exceed an ambient operating temperature is sufficient to be evaluated beyond a first predetermined margin.

On the lower temperature side, when the flow cell is deemed to be cooling below ambient temperature, the instructions may further perform determining whether the evaluated one or more cell heater temperature sensor data points obtained prior to counting are below ambient operating temperature minus a second predetermined margin. When the evaluated loop flow cell heater temperature sensor data is less than the operating temperature by a second predetermined margin, it is determined whether flow cell heater temperature sensor data acquired prior to counting in a successive loop immediately following the evaluated loop is less than the ambient operating temperature by the second predetermined margin. Then, in the evaluated cycle and the consecutive cycle, when the evaluated cycle flow cell heater temperature sensor data fails to be less than the operating temperature by a second predetermined margin, a flow cell cooling failure is determined and a service demand is reported.

A CRM-implemented method includes determining a second predetermined margin based on devices located at a plurality of locations and operated by a plurality of independent operators. The method includes logic for configuring the device to record and report temperature sensor readings and to store a collection log of the temperature sensor readings. The method includes analyzing a time series of temperature sensor readings in an instance of the device where the flow cell heater failed and determining a second predetermined margin. A second predetermined temperature margin is stored for determining whether the flow cell heater is malfunctioning.

A CRM-implemented method includes updating a second predetermined margin based on equipment located at a plurality of locations and operated by a plurality of independent operators. The method includes having the device configuration record and report temperature sensor readings and a service log after reporting service requirements. The method includes storing the collected logs. The method includes analyzing a time series of temperature sensor readings in a flow cell heater health and malfunctioning equipment instance, and reporting a service log after a service demand. The method includes determining an update to a second predetermined margin based on the analysis.

A second method embodiment of the disclosed technology includes detecting that a sequencer has a failed flow cell temperature control system. The method includes analyzing a time series of flow cell temperature sensor data across a base recognition cycle. This also includes determining whether the first base recognition cycle has enough flow cell temperature sensor data points to meet a count threshold. The method includes determining whether temperature sensor data in a first cycle exceeds an ambient operating temperature by a first predetermined margin. When the flow cell temperature sensor data in the first cycle fails to exceed the ambient operating temperature by a first predetermined margin, the method includes determining that the flow cell temperature sensor data in a second consecutive cycle immediately before or after the first cycle has sufficient flow cell temperature sensor data points to satisfy the count threshold. The method also includes determining that the flow cell temperature sensor data in the second consecutive cycle does not exceed the ambient operating temperature by a first predetermined margin. The method then responsively sets a first fault condition. The method includes generating a flow cell heater alarm in response to a first fault condition.

The method embodiments and other methods disclosed optionally include one or more of the following features. The method may also include features described in connection with the disclosed system. The reader will understand how the features identified in this section can be readily combined with the basic features in other legal categories.

The method includes determining a flow cell coolant fault by analyzing a time series of flow cell heater temperature sensor data across a base recognition cycle. This also includes determining that the first base recognition cycle has a flow cell temperature sensor data point to be evaluated during the flow cell cooling sub-cycle. The method includes determining whether the temperature sensor data in the first cycle is cooled to a second predetermined margin below an ambient operating temperature. When the flow cell temperature sensor data fails to cool to below the ambient operating temperature by a second predetermined margin in a first cycle, the method includes determining that the flow cell heater temperature sensor data in a second consecutive cycle immediately before or after the first cycle fails to cool to below the ambient operating temperature by the second predetermined margin. Thereafter, the method includes setting a second fault condition. The method includes generating a flow cell coolant alarm in response to a second fault condition.

The method includes exposing a temperature sensor in a flow cell temperature control system and reporting temperature sensor data from the exposed temperature sensor. The method includes receiving temperature sensor data from a plurality of devices including a sequencer. The method includes applying analysis of a time series of flow cell temperature sensor data across a plurality of base recognition cycles to temperature sensor data from a plurality of devices.

The method includes a temperature margin update including receiving temperature sensor data from a plurality of devices including a sequencer. The method also includes receiving new temperature sensor data from the plurality of devices. The method also includes modifying the first predetermined margin value based on the new temperature sensor data and storing the modified first predetermined margin value.

The method includes a temperature margin update including receiving temperature sensor data from a plurality of devices including a sequencer. The method also includes receiving new temperature sensor data from the plurality of devices. The method includes modifying the second predetermined margin value based on the new temperature sensor data and storing the modified second predetermined margin value.

The method utilizes CRM data in temperature margin updates, including tracking alarms, faults, and solutions for a plurality of devices including a sequencer. The method also includes receiving fault and solution data from the customer relationship module and distinguishing missed faults from false alarms when modifying parameters implemented by the temperature margin adjustment module.

System embodiments of the technology include one or more processors coupled to a memory loaded with computer instructions that, when executed by the one or more processors, cause the system to perform a method according to any one of the methods described above. Each of the features discussed in the detailed description of the second method embodiment above apply equally to this system embodiment.

CRM implementations of the technology include a non-transitory computer-readable storage medium imprinted with computer program instructions. The instructions, when executed on one or more processors, implement a method according to any of the methods described above.

Each of the features discussed in this detailed description of system embodiments are equally applicable to CRM embodiments. As noted above, not all system features are repeated here and should be considered repeated by reference.