阅读说明：本技术 用于缓存生物图像数据的系统和方法 (System and method for caching biometric image data ) 是由 P·博达 A·沙玛 S·Y·余 于 2019-09-03 设计创作，主要内容包括：本公开涉及数字病理系统和服务器系统,其各自可以包括预填充有生物图像数据的缓存,所述生物图像数据包括已预处理为目标文件格式的生物图像数据。可以在所述数字病理系统和/或所述服务器系统上用生物图像数据填充缓存,以检索生物图像数据,从而有助于在浏览器中或在所述数字病理系统上实时或近实时地可视化生物图像数据。当接收扫描的图像数据时,可以自动对服务器上的生物图像数据进行预处理。(The present disclosure relates to a digital pathology system and a server system, each of which may include a cache pre-populated with biological image data including biological image data that has been pre-processed into a target file format. A cache may be populated with biological image data at the digital pathology system and/or the server system to retrieve biological image data to facilitate real-time or near real-time visualization of biological image data in a browser or on the digital pathology system. The biological image data on the server may be automatically preprocessed when the scanned image data is received.)

1. A method of pre-processing biological image data on a server to facilitate transmission of the biological image data to a digital pathology analysis system, the method comprising:

(a) receiving biological image data having a first image file format, the biological image data derived from a scan of a microscope slide having one or more biological samples disposed thereon, the biological samples stained with hematoxylin and eosin or stained to identify the presence of one or more biomarkers;

(b) pre-processing the received biometric image data to generate one or more pre-processed image files, each of the generated one or more pre-processed image files being derived from the biometric image data in the received first image file format, and wherein each of the generated one or more pre-processed image files is provided in a second file format; and

(c) storing the generated one or more preprocessed image files provided in the second file format in one or more memories communicatively coupled with the server.

2. The method of claim 1, wherein the first image file format and the second image file format are the same.

3. The method of claim 2, wherein the preprocessing step generates a plurality of preprocessed image files.

4. The method of claim 3, wherein the plurality of preprocessed image files are a plurality of image tile files, and wherein the plurality of image tile files each include a file name that includes coordinates corresponding to a location of a single tile file within the received biometric image data having the first image file format.

5. The method of claim 2, wherein the pre-processing step generates a plurality of sets of pre-processed image files, wherein each set of pre-processed image files in the plurality of sets of pre-processed image files includes a plurality of image tile files corresponding to a single magnification level.

6. The method of claim 2, wherein the first file format and the second file format are compressed image file formats.

7. The method of claim 6, wherein the compressed image file format is selected from the group consisting of a JPEG file interchange format, a portable network graphics file format, and derivatives thereof.

8. The method of claim 1, wherein the first image file format and the second image file format are different.

9. The method of claim 8, wherein the first image file format is one of a RAW image file format or an uncompressed image file format; and wherein the second image file format is a compressed image file format.

10. The method of claim 9, wherein the preprocessing step comprises converting the received biometric image data in the RAW image file format or the uncompressed image file format and generating the one or more preprocessed image files in the compressed image file format.

11. The method of claim 10, wherein the preprocessing step generates a single preprocessed image file in the compressed image file format.

12. The method of claim 10, wherein the preprocessing step generates a plurality of preprocessed image files in the compressed image file format.

13. The method of claim 12, wherein the pre-processing step generates a plurality of sets of pre-processed image files, wherein each set of pre-processed image files in the plurality of sets of pre-processed image files includes a plurality of image tile files in the compressed image file format corresponding to a single magnification level.

14. The method of claim 12, wherein the plurality of preprocessed image files are a plurality of image tile files, and wherein the plurality of image tile files each include a file name that includes coordinates corresponding to a location of a single tile file within the received biometric image data having the first image file format.

15. The method of claim 12, wherein a file size of the received biometric image data having a first image file format is greater than an aggregate file size of all of the generated plurality of pre-processed image files of the compressed image file format.

16. The method of claim 9, wherein the compressed image file format is a target file format.

17. The method of claim 16, wherein the target file format is a browser-compatible file format.

18. The method of claim 1, further comprising receiving a request from the digital pathology analysis system for one or more stored preprocessed image files.

19. The method of claim 1, further comprising automatically transmitting the stored one or more pre-processed image files to a caching agent running on the digital pathology analysis system.

20. The method of claim 1, wherein the one or more memories comprise both volatile and non-volatile memories.

21. The method of claim 1, wherein the one or more memories are non-volatile memories.

22. The method of claim 21, wherein the stored one or more pre-processed image files are maintained in the one or more non-volatile memories for a predetermined amount of time.

23. The method of claim 22, wherein the predetermined amount of time ranges from 5 days to 15 days.

24. A method of managing biometric image data in a client cache within a digital pathology analysis system, the method comprising:

(a) performing user verification using a remote server, wherein the verification includes confirming that a caching agent is running on the digital pathology analysis system;

(b) receiving, from the remote server, a plurality of new biological image tiles derived from a scan of a microscope slide having one or more biological samples placed thereon, the biological samples stained with hematoxylin and eosin or stained to identify the presence of one or more biomarkers;

(c) removing at least a first portion of a pre-existing biological image tile from a client cache communicatively coupled with the digital pathology system based on a predetermined criterion; and

(d) storing the received new biometric image tile in the client cache.

25. The method of claim 24, wherein the predetermined criteria include storage allocation limits, cache image expiration, slide viewing period expiration, and case reassignment.

26. The method of claim 24, wherein an aggregate size of the removed first portions of the pre-existing biometric image tiles is approximately the same as a total size of the received new biometric image tiles.

27. The method of claim 24, wherein an aggregate size of the removed first portions of the pre-existing biometric image tiles is greater than a size of the received new biometric image tile.

28. The method of claim 24, wherein the received plurality of new biometric image tiles are pre-processed biometric image tiles having a compressed file format.

29. The method of claim 28, wherein the compressed file format is JPEG.

30. The method of claim 24, further comprising maintaining the client cache by removing a first portion of a stored new biometric image tile based on a predetermined criterion.

31. The method of claim 30, wherein the predetermined criteria include storage allocation limits, cache image expiration, slide viewing period expiration, and case reassignment.

32. The method of claim 30, further comprising removing a second portion of the stored new biometric image tile examined by a pathologist.

33. A method of managing biometric image data in a client cache within a digital pathology analysis system, the method comprising:

(a) maintaining a client cache communicatively coupled with the digital pathology analysis system, the client cache including biological image data of one or more patient cases, wherein the client cache is maintained by removing a first portion of pre-existing biological image data based on a predetermined criterion;

(b) performing authentication using a remote server;

(c) receiving a first request from the remote server to add new biometric image data to a client cache; and

(d) storing the new biometric image data in the client cache,

wherein the pre-existing biological image data and the stored new biological image data are derived from a scan of a microscope slide having one or more biological samples placed thereon, the biological samples being stained with hematoxylin and eosin or stained to identify the presence of one or more biomarkers.

34. The method of claim 33, wherein the predetermined criteria include storage allocation limits, cache image expiration, slide viewing period expiration, and case reassignment.

35. The method of claim 33, wherein the client cache is further maintained by removing a second portion of pre-existing biometric image data of the one or more patient cases examined by a pathologist assigned to the one or more patient cases.

36. The method of claim 33, wherein the client cache is pre-allocated on a storage subsystem communicatively coupled with the digital pathology analysis system.

37. The method of claim 33, wherein the verifying comprises confirming that a caching agent is running on the digital pathology analysis system.

38. The method of claim 33, further comprising receiving a second request from a viewer application or browser running on the digital pathology analysis system to retrieve at least a portion of the stored new biometric image data from the client cache.

39. The method of claim 38, wherein the stored new biometric image data is transmitted to the viewer application or the browser and displayed on a display screen of the digital pathology analysis system.

40. The method of claim 33, wherein the stored biometric image data comprises a plurality of pre-processed image tiles in a compressed image file format.

41. The method of claim 40, wherein the plurality of image tiles includes a first series of image tiles corresponding to biological image data scanned at a first magnification level and a second series of image tiles corresponding to biological image data scanned at a second magnification level.

42. A method of maintaining cached biometric image data in a client cache communicatively coupled with a digital pathology analysis system, comprising:

(a) accessing the biological image data, wherein the biological image data is derived from a scan of a microscope slide having one or more biological samples disposed thereon, the biological samples being stained with hematoxylin and eosin or stained to identify the presence of one or more biomarkers;

(b) receiving confirmation that a caching agent is running on the digital pathology analysis system;

(c) communicating at least a portion of the accessed biometric image data to at least partially fill a portion of available storage space within the client cache; and

(d) storing one or more URLs associated with at least a portion of the accessed biological image data communicated to the digital pathology analysis system.

43. The method of claim 42, wherein at least a portion of the accessed biometric image data is transferred to completely fill the available storage space on the client cache.

44. The method of claim 43, wherein at least a portion of the accessed biometric image data communicated to the client cache comprises a plurality of pre-processed image tiles in a compressed image file format.

45. The method of claim 44, wherein the plurality of pre-processed image tiles comprises a first series of pre-processed image tiles at a first magnification level and a second series of pre-processed image tiles at a second magnification level.

46. The method of claim 42, further comprising receiving an indication that sufficient available storage space exists within the client cache prior to transferring at least a portion of the accessed biometric image data.

47. The method of claim 42, further comprising receiving an indication of an amount of available storage space within the client cache.

48. The method of claim 47, wherein communicating at least a portion of the accessed biometric image data comprises: (i) determining a size of the accessed biometric image data; and (ii) select at least a portion of the accessed biometric image data based on available storage space within the client cache.

49. The method of claim 48, wherein the selection of the portion of the accessed biometric image data is based on a predetermined selection criterion.

50. The method of claim 49, wherein the predetermined selection criteria is a scan date of a single image within the accessed biometric image data.

51. The method of claim 42, wherein the accessed biological image data is distributed to a first pathologist on the digital pathology analysis system.

52. The method of claim 51, wherein the verifying step further comprises confirming an identity credential of the first pathologist.

53. The method of claim 42, wherein a server maintains a connection with the caching agent, and wherein the step of receiving confirmation that a caching agent is running on the digital pathology analysis system comprises confirming that the connection is established.

54. A method of displaying biological image data in an interface application on a digital pathology analysis system, comprising:

(a) receiving a first user input relating to a first patient case;

(b) contacting a remote server for one or more stored biological image tiles corresponding to the received first user input, wherein the stored biological image tiles are derived from a scan of a microscope slide having one or more biological samples placed thereon, the biological samples being hematoxylin and eosin stained or stained to identify the presence of one or more biomarkers;

(c) receiving location information of the one or more stored biometric image tiles corresponding to the first user input;

(d) retrieving the one or more stored biometric image tiles related to the first user input based on the received location information; and

(e) visualizing the retrieved one or more biometric image tiles related to the first user input.

55. The method of claim 54, further comprising determining whether one or more stored biometric image tiles are present within a client cache communicatively coupled with the digital pathology analysis system.

56. The method of claim 55, wherein the stored one or more biometric image tiles are retrieved from the client cache if the biometric image data is already maintained within the client cache.

57. The method of claim 55, wherein the stored one or more biometric image tiles are retrieved from the remote server if the image is removed from the client cache.

58. The method of claim 57, wherein the stored one or more biometric image tiles retrieved from the remote server are downloaded to the client cache.

59. The method of claim 58, wherein one or more biometric image tiles that are visualized are retrieved from the client cache.

60. The method of claim 55, wherein if the stored one or more biometric image tiles have not been maintained in the client cache, redirecting the interface to a storage location on the remote server.

61. The method of claim 54, wherein the stored one or more biological image tiles related to the first user input are retrieved from the remote server, and further comprising receiving a second user input related to the first patient case, and wherein one or more stored image tiles related to the second user input are retrieved from a client cache.

62. The method of claim 54, wherein the stored one or more biometric image tiles are pre-processed image tiles in an uncompressed format.

63. The method of claim 54, further comprising managing a client cache by removing pre-populated biometric image data that meets predetermined criteria.

64. The method of claim 54, wherein the location information includes one or more URLs.

Background

Digital pathology refers to the management and interpretation of pathology information in a digital environment. The scanning device is used to image a slide of a biological sample that may be stained, thereby generating a digital slide image, e.g., an entire slide image. The digital pathology software is capable of storing digital slides in a computer storage device, viewing and analyzing pathology information on a computer monitor.

Disclosure of Invention

The present disclosure is directed, inter alia, to a client-server system including a client user interface (GUI) with improved response time. Typically, an interface for visualization and analysis of image data runs on (i) a client system (e.g., a viewer application running locally on the client system) or (ii) a remote server, where the client system accesses an instance of the viewer application on the remote server through a browser (such as a Web browser). Image data, such as scanned images of biological samples stained with hematoxylin and eosin or stained to identify the presence of one or more biomarkers, may be communicated from a remote server (e.g., from a storage system communicatively coupled to the remote server) to a viewer application or browser. In some embodiments, the image data is in the form of image tiles (as defined herein) that may be passed to a viewer application or browser. Due to the amount of image tiles typically transmitted from a remote server (not to mention the size of the transmitted file), the image data may not be visualized in real-time in the viewer application or browser upon user input received by the viewer application. This may hinder the ability of the user to perform image data analysis in real time.

To better facilitate image visualization, in some embodiments, the present disclosure provides for automatic pre-processing of biological image data (e.g., histopathological image data or cytopathological image data) on a server, and communicating the pre-processed image data to a client system, e.g., a digital pathology analysis system. In some embodiments, the pre-processing of the image data includes converting the scanned image data from an input format (e.g., DICOM, BIF, or another uncompressed image file format) to a target format (e.g., a compressed image file format, such as JPEG). It is believed that the image data included in the input format is large, e.g., in excess of 2000 megabytes per image file, and converting the input file format to the target file format (where the data may be compressed) enables faster delivery of the image data to the client system. In some embodiments, the pre-processing of the image data includes segmenting the scanned image data file into multiple files (e.g., multiple image tile files, which may be in any input file format or target file format), again to facilitate faster transfer of the image data from the server to the client system. Servers are considered to have limited processing power and may be burdened if multiple requests for image data are received in real-time, such as from one or more client systems. By pre-processing the image data on the server prior to requesting the image data from the client system, the burden on server resources (such as processing resources) will be reduced, enabling improved delivery of the image data to the client system. For example, instead of generating image tiles in real-time in response to a user request or pre-fetch operation, the biometric image data may be pre-processed and the image tiles may be immediately available for transmission.

In some embodiments, the present disclosure also provides for automatic transfer of biometric image data, including pre-processed image data in a target file format, from a server to a client cache communicatively coupled to a client system (such as via a caching agent). In this manner, image data requested from the client system may be retrieved from a local client cache of pre-populated image data (such as pre-processed image data) without having to be transmitted over a network, which may be slow in view of the file sizes involved. In general, the systems and methods described herein provide for more robust and accurate review of image data, ultimately providing improved patient care. In addition, the systems and methods described herein provide a better user experience, enabling a user (e.g., a pathologist) to perform their duties faster and more accurately.

One aspect of the present disclosure is a method of pre-processing biological image data on a server to facilitate transmission of the biological image data to a digital pathology analysis system, the method comprising receiving biological image data having a first image file format, the biological image data resulting from a scan of a microscope slide having one or more biological samples disposed thereon, the biological samples being hematoxylin and eosin stained or stained to identify the presence of one or more biomarkers; pre-processing the received biological image data to generate one or more pre-processed image files, each of the generated one or more pre-processed image files being derived from the received biological image data in a first file format, and wherein each of the generated one or more pre-processed image files is provided in a second file format; and storing the generated one or more pre-processed image files provided in the second file format in one or more memories communicatively coupled to the server.

In some embodiments, the first image file format and the second image file format are the same. In some embodiments, the preprocessing step generates a plurality of preprocessed image files. In some embodiments, the plurality of preprocessed image files are a plurality of image tile files, and wherein the plurality of image tile files each include a file name that includes coordinates corresponding to a location of a single tile file in the received biometric image data having the first image file format. In some embodiments, the pre-processing step generates a plurality of sets of pre-processed image files, wherein each set of pre-processed image files in the plurality of sets of pre-processed image files includes a plurality of image tile files corresponding to a single magnification stage. In some embodiments, the first file format and the second file format are compressed image file formats. In some embodiments, the compressed image file format is selected from the group consisting of a JPEG file interchange format, a portable network graphics file format, and derivatives thereof.

In some embodiments, the first image file format and the second image file format are different. In some embodiments, the first image file format is one of a RAW image file format or an uncompressed image file format; and wherein the second image file format is a compressed image file format. In some embodiments, the preprocessing step comprises converting the received biometric image data in a RAW or uncompressed image file format; and generating the one or more preprocessed image files in a compressed image file format. In some embodiments, the pre-processing step generates a single pre-processed image file in the compressed image file format. In some embodiments, the preprocessing step generates a plurality of preprocessed image files in the compressed image file format. In some embodiments, the pre-processing step generates a plurality of sets of pre-processed image files, wherein each set of pre-processed image files in the plurality of sets of pre-processed image files comprises a plurality of image tile files in a compressed image file format corresponding to a single magnification stage. In some embodiments, the plurality of preprocessed image files are a plurality of image tile files, and wherein the plurality of image tile files each include a file name that includes coordinates corresponding to a location of a single tile file in the received biometric image data having the first image file format. In some embodiments, the file size of the received biometric image data having the first file format is greater than a total file size of all of the plurality of preprocessed image files of the generated compressed image file format. In some embodiments, the compressed image file format is a target file format. In some embodiments, the target file format is a browser-compatible file format.

In some embodiments, the method further comprises receiving a request from the digital pathology analysis system for the stored one or more preprocessed image files. In some embodiments, the method further comprises automatically transmitting the stored one or more pre-processed image files to a caching agent running on the digital pathology analysis system. In some embodiments, the one or more memories include volatile and non-volatile memories. In some embodiments, the one or more memories are non-volatile memories. In some embodiments, the stored one or more pre-processed image files are maintained in the one or more non-volatile memories for a predetermined amount of time. In some embodiments, the predetermined amount of time ranges from about 5 days to about 15 days.

Another aspect of the disclosure is a method of managing biometric image data in a client cache within a digital pathology analysis system, the method comprising: performing user authentication using a remote server, wherein the authentication includes confirming that a caching agent is running on the digital pathology analysis system; receiving, from the remote server, a plurality of new biological image tiles resulting from a scan of a microscope slide having one or more biological samples disposed thereon, the biological samples being stained with hematoxylin and eosin or stained to identify the presence of one or more biomarkers; removing at least a first portion of a pre-existing biological image tile from a client cache communicatively coupled to the digital pathology system based on a predetermined criterion; and storing the received new biometric image tile in the client cache.

In some embodiments, the predetermined criteria include storage allocation limits, buffer image expiration, slide viewing period expiration, and case reassignment. In some embodiments, the total size of the first portion of the removed pre-existing biometric image tile is approximately the same as the total size of the received new biometric image tile. In some embodiments, the total size of the first portion of the removed pre-existing biometric image tile is greater than the size of the received new biometric image tile. In some embodiments, the received plurality of new biometric image tiles are pre-processed biometric image tiles having a compressed file format. In some embodiments, the compressed file format is JPEG.

In some embodiments, the method further comprises maintaining the client cache by removing a first portion of a stored new biometric image tile based on a predetermined criterion. In some embodiments, the predetermined criteria include storage allocation limits, buffer image expiration, slide viewing period expiration, and case reassignment. In some embodiments, the method further comprises removing a second portion of the stored new biometric image tile examined by the pathologist.

Another aspect of the disclosure is a method of managing biometric image data in a client cache within a digital pathology analysis system, the method comprising: maintaining a client cache communicatively coupled to the digital pathology analysis system, the client cache including biological image data for one or more patient cases, wherein the client cache is maintained by removing a first portion of existing biological image data based on a predetermined criterion; performing authentication using a remote server; receiving a first request from a remote server to add new biometric image data to a client cache; and storing the received new biological image data in the client cache, and wherein the existing biological image data and stored new biological image data are derived from a scan of a microscope slide having one or more biological samples disposed thereon that are hematoxylin and eosin stained or stained to identify the presence of one or more biomarkers.

In some embodiments, the predetermined criteria include storage allocation limits, buffer image expiration, slide viewing period expiration, and case reassignment. In some embodiments, the client cache is maintained by removing a second portion of existing biometric image data of one or more patient cases that are reviewed by a pathologist assigned to the one or more patient cases.

In some embodiments, the client cache is pre-allocated on a storage subsystem communicatively coupled to the digital pathology analysis system. In some embodiments, the authenticating comprises: confirming that a caching agent is running on the digital pathology analysis system.

In some embodiments, the method further comprises receiving a second request from a viewer application or browser running on the digital pathology analysis system to retrieve at least a portion of the stored new biometric image data from the client cache. In some embodiments, the retrieved stored new biometric image data is transmitted to the viewer application or the browser and displayed on a display screen of the digital pathology analysis system. In some embodiments, the stored new biometric image data includes a plurality of pre-processed image tiles in a compressed image file format. In some embodiments, the plurality of image tiles includes a first series of image tiles corresponding to biological image data scanned at a first magnification level and a second series of image tiles corresponding to biological image data scanned at a second magnification level.

Another aspect of the disclosure is a method of maintaining cached biometric image data in a client cache communicatively coupled to a digital pathology analysis system, the method comprising: accessing biological image data, wherein the biological image data is derived from a scan of a microscope slide having one or more biological samples disposed thereon, the biological samples being stained with hematoxylin and eosin or stained to identify the presence of one or more biomarkers; receiving confirmation that a caching agent is running on the digital pathology analysis system; transferring at least partially accessed biometric image data to at least partially fill a portion of available memory space within a client cache; and storing one or more URLs related to at least a portion of the accessed biological image data communicated to the digital pathology analysis system.

In some embodiments, at least a portion of the accessed biometric image data is communicated to completely fill the available storage space on the client server. In some embodiments, at least a portion of the accessed biometric image data communicated to the client cache comprises a plurality of pre-processed image tiles in a compressed image file format. In some embodiments, the plurality of pre-processed image tiles comprises a first series of pre-processed image tiles at a first magnification level and a second series of pre-processed image tiles at a second magnification level. In some embodiments, the method further comprises receiving an indication that sufficient available storage space exists within the client cache prior to delivering at least a portion of the accessed biometric image data. In some embodiments, the method further comprises receiving an indication of an amount of available storage space within the client cache. In some embodiments, communicating at least a portion of the accessed biometric image data comprises: (i) determining a size of the accessed biological image data; and (ii) select at least a portion of the accessed biometric image data based on available memory space within the client cache. In some embodiments, the selection of the accessed biological image data portion is based on predetermined selection criteria. In some embodiments, the predetermined selection criteria is a date of scanning of a single image within the accessed biometric image data. In some embodiments, the accessed biological image data is distributed to a first pathologist on the digital pathology analysis system. In some embodiments, the authenticating step further comprises confirming the identity of the first pathologist. In some embodiments, the server maintains a connection with said caching agent, and wherein the step of receiving confirmation confirming that the caching agent is running on said client system comprises confirming that said connection is established.

Another aspect of the present disclosure is a method of displaying biological image data in an interface application on a digital pathology analysis system, the method comprising: receiving a first user input relating to a first patient case; contacting a remote server to obtain one or more stored biological image tiles corresponding to the received first user input, wherein the stored biological image tiles are derived from a scan of a microscope slide having one or more biological samples disposed thereon, the biological samples being hematoxylin and eosin stained or stained to identify the presence of one or more biomarkers; receiving location information of one or more stored biometric image tiles corresponding to a first user input; retrieving one or more stored biometric image tiles related to the first user input based on the received location information; and visualizing the retrieved one or more biometric image tiles related to the first user input.

In some embodiments, the method further comprises determining the presence of one or more biological image tiles stored within a client cache communicatively coupled to the digital pathology analysis system. In some embodiments, if the biometric image data is already maintained within the client cache, then the stored one or more biometric image tiles are retrieved from the client cache. In some embodiments, if the image is removed from the client cache, the stored one or more biometric image tiles are retrieved from the remote server. In some embodiments, the stored one or more biometric image tiles retrieved from the remote server are downloaded to the client cache. In some embodiments, one or more biometric image tiles of the visualization are retrieved from the client cache. In some embodiments, if one or more stored biometric image tiles are not maintained in the client cache, the interface is redirected to a storage location on the remote server. In some embodiments, the stored one or more biometric image tiles related to the first user input are retrieved from the remote server, and further comprising receiving a second user input related to the first patient case, and wherein the one or more stored image tiles are related to the second user input retrieved from the client cache. In some embodiments, the stored one or more biometric image tiles are pre-processed image tiles in an uncompressed format. In some embodiments, the method further comprises managing the client cache by removing pre-populated biometric image data that meets a predetermined criterion. In some embodiments, the location information includes one or more URLs.

Drawings

For a general understanding of the features of the present disclosure, refer to the accompanying drawings. In the drawings, like reference numerals are used to identify like elements throughout the figures.

The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided to the office upon request and payment of the necessary fee.

FIG. 1A illustrates a client-server system including a server 160 communicatively coupled to a client system 170, e.g., over a network, according to some embodiments.

FIG. 1B illustrates a client-server system including a server 160 communicatively coupled to a client system 170, the client system 170 including a viewer application and a caching agent, according to some embodiments.

FIG. 1C illustrates a client-server system including a server 160 communicatively coupled to a client system 170, the client system 170 including a browser and a cache agent, according to some embodiments.

FIG. 1D illustrates a client-server system including a server 160 communicatively coupled to a client system 170, the client system 170 including a browser and a caching proxy, according to some embodiments.

Figure 2A illustrates a server communicatively coupled to a scanning device, in accordance with some embodiments.

Fig. 2B illustrates a client system according to some embodiments, including a system of processing subsystems, storage subsystems, output devices, and input devices, each communicatively coupled by a bus, network, or other wired or wireless interconnect. According to some embodiments, the system may include a viewer application that runs locally on the client system and may cache content (e.g., image data) from a local client within the storage subsystem or access content from a remote server.

Fig. 2C illustrates a client system including a processing subsystem, a storage subsystem, an output device, and an input device, each communicatively coupled by a bus, network, or other wired or wireless interconnect, according to some embodiments. According to some embodiments, the system may include a browser that runs locally on the client system and may cache content (e.g., image data) from a local client within the storage subsystem or access content from a remote server.

FIG. 3 shows a flowchart that provides the general steps of creating a patient case and associating the created patient case with scanned image data and a pathologist user identification, according to some embodiments.

Fig. 4A shows a flowchart that provides the general steps of delivering content (e.g., image data or portions thereof) to a client cache managed by a caching agent running on a client system, according to some embodiments.

Figure 4B illustrates a flowchart that provides the general steps of performing authentication between a client system and a remote server, including a client system running a caching proxy, according to some embodiments.

FIG. 5 illustrates a flow diagram that shows the general steps of maintaining a client cache, such as facilitated by a cache agent running on a client system, in accordance with some embodiments.

FIG. 6 shows a flowchart that illustrates the general steps of receiving image data from a remote server, such as data pushed from the remote server into a client cache managed by a caching agent, in accordance with some embodiments.

FIG. 7 shows a flowchart that provides steps for visualizing image data corresponding to received user input, such as input received by a user interacting with a viewer application, either directly or indirectly through a browser, in accordance with some embodiments.

FIG. 8 shows a flowchart that provides steps for visualizing image data corresponding to received user input, such as input received by a user interacting with a browser, in accordance with some embodiments.

FIG. 9 illustrates a flowchart that provides steps for installing a client-side caching agent on a client system, according to some embodiments.

FIG. 10 illustrates a flowchart that provides steps for pre-processing image data on a server, such as pre-processing image data received in an input file format into a target file format, according to some embodiments.

Detailed Description

It will also be understood that, unless indicated to the contrary, in any methods claimed herein that include more than one step or action, the order of the steps or actions of the method need not be limited to the order in which the steps or actions of the method are expressed.

As used herein, the singular forms "a", "an" and "the" include plural referents unless the context clearly dictates otherwise. Likewise, the word "or" is intended to include "and" unless the context clearly indicates otherwise. The term "comprising" is defined as inclusive, e.g., "comprising A or B" means including A, B or A and B.

As used herein in the specification and claims, "or" should be understood to have the same meaning as "and/or" as defined above. For example, where items in a list are separated by "or" and/or "should be interpreted as inclusive, i.e., including at least one element from the list of elements or elements, but also including more than one element, and optionally including additional unlisted items. To the contrary, terms such as "only one" or "exactly one," or "consisting of," as used in the claims, are intended to mean that there is exactly one element from a number or list of elements. In general, the use of the term "or" only preceded by an exclusive term, such as "or", "one of", "only one of", or "exactly one", should be construed to mean an exclusive alternative (i.e., "one or the other, but not both"). The term "consisting essentially of as used in the claims shall have the ordinary meaning as used in the patent law.

As used herein, the terms "comprising," "including," "having," and the like are used interchangeably and have the same meaning. Similarly, "including," "comprising," "having," and the like are used interchangeably and have the same meaning. In particular, each term is defined consistent with the common U.S. patent statutes defining "including", such that each term is to be interpreted as an open-ended term in the sense of "at least the following", and also in a sense that it is not to be interpreted as excluding additional features, limitations, aspects, and the like. Thus, for example, a "device having components a, b, and c" means that the device includes at least components a, b, and c. Also, the phrase: by "a method involving steps a, b and c" is meant that the method comprises at least steps a, b and c. Further, although the steps and processes may be summarized herein in a particular order, those skilled in the art will recognize that the sequential steps and processes may vary.

As used herein in the specification and in the claims, with respect to a list of one or more elements, the phrase "at least one" should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each element specifically listed in the list of elements, nor excluding any combination of elements in the list of elements. This definition allows that, in addition to the elements specifically identified in the list of elements to which the phrase "at least one" refers, other elements are optionally present, whether related or not to the specifically identified elements. Thus, as a non-limiting example, "at least one of a and B" (or, equivalently, "at least one of a or B," or, equivalently, "at least one of a and/or B") can refer, in one embodiment, to at least one that optionally includes more than one a, but no B (and optionally includes elements other than B); in another embodiment, at least one is optionally comprised of more than one B, but no A (and optionally includes elements other than A); in yet another embodiment, at least one of the elements selectively includes more than one a, and at least one of the elements selectively includes more than one B (and optionally includes other elements), and so on.

As used herein, the term "biological sample," "tissue sample," "specimen," or similar terms refer to any sample, including biomolecules (e.g., proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Examples of other organisms include mammals (e.g., humans; veterinary animals such as cats, dogs, horses, cows, and pigs; and laboratory animals such as mice, rats, and primates), insects, annelid animals, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (e.g., tissue sections and needle biopsies of tissue), cellular samples (e.g., cytological smears, such as cervical smears or blood smears or obtained by microdissection), or cellular fractions, debris, or organelles (e.g., obtained by lysing cells and separating their components by centrifugation or other means). Other examples of biological samples include blood, serum, urine, semen, stool, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical biopsy or needle biopsy), nipple aspirate, cerumen, breast milk, vaginal secretion, saliva, a swab (e.g., a buccal swab), or any material that contains a biomolecule and is derived from a first biological sample. In certain embodiments, the term "biological sample" as used herein refers to a sample prepared from a tumor or a portion thereof obtained from a subject (e.g., a homogenized or liquefied sample).

As used herein, the term "cytopathology" refers to the study of disease at the cellular level, such as a sample of free cells or tissue fragments.

As used herein, the term "DICOM" refers to a standard for processing, storing, printing, and transmitting information in medical imaging. It includes file format definitions and network communication protocols. A network communication protocol is an application protocol that uses TCP/IP to communicate between systems. One of the goals of this standard is to homogenize the transmission of medical images and information between viewing and scanning sources to enable users of different imaging software and/or hardware to share information. DICOM files may be exchanged between two entities capable of receiving image and patient data in DICOM format. DICOM may integrate scanners, servers, workstations, printers, and networking hardware from multiple manufacturers into a Picture Archiving and Communication System (PACS) for storing and downloading digital images. Different devices carry DICOM conformance statements that clearly illustrate the DICOM classes they support. DICOM has been widely adopted by hospitals and is becoming more and more popular in small dentist and doctor offices. DICOM files typically contain images; therefore, they are commonly referred to as DICOM images. It will be understood, however, that a DICOM file need not necessarily include images. Rather, such files may include measurement or reporting data. Thus, a DICOM file may contain media data, such as video and audio data, or not contain any media data at all. In that case, the DICOM file may contain only metadata identifying the original modality, the operator, or the patient to be examined. The modality herein refers to any image generation device in medical imaging, and also includes generating images of histopathological or cytopathological slides using a scanner, such as those pathological slides stained with hematoxylin and eosin or stained to identify the presence of one or more biomarkers.

As used herein, the term "histopathology" refers to microscopic examination of tissue (e.g., biopsy specimen, surgical specimen) to study the manifestation of disease.

As used herein, the term "image data" encompasses raw image data acquired from a biological tissue sample, such as by an optical sensor or sensor array, or pre-processed image data. In particular, the image data may comprise a matrix of pixels. In some embodiments, the image data is stored in blocks of an image tile (e.g., a portion of the entire slide image has a size of about 500 pixels by about 500 pixels). In some embodiments, the image data is from a scan of a biological sample disposed on a substrate (e.g., a microscope slide), and the scan of the biological sample can include multiple images (e.g., 20x, 40x, etc.) of the same slide at different magnification levels. In some embodiments, first, at a low magnification level (e.g., suitable for thumbnail images); second, with an intermediate amplification stage; third, image data is captured at a high magnification level. In some embodiments, image data may be acquired for a plurality of magnification stages, and one or more image data files may be generated for each of the plurality of magnification stages. In some embodiments, the number of image tiles increases with magnification level or zoom level, i.e. an image tile is available for each of a predetermined number of available magnification levels, and those higher magnification levels will have a greater number of image tiles than lower magnification levels. In some embodiments, the number of tiles per zoom level increases, for example, linearly, quadratically, exponentially, or otherwise. For example, the lowest magnification image data (corresponding to the first magnification stage described above) may include one hundred tiles, while the intermediate magnification image data may include four hundred tiles.

As used herein, the term "slide" refers to any substrate of any suitable size (e.g., a substrate made in whole or in part of glass, quartz, plastic, silicon, etc.) upon which a biological specimen can be placed for analysis, and more particularly to a "microscope slide" such as a standard 3 inch by 1 inch microscope slide or a standard 75mm by 25mm microscope slide. Examples of biological specimens that may be placed on slides include, but are not limited to, cytological smears, thin tissue sections (e.g., from a biopsy), and biological specimen arrays, such as tissue arrays, cell arrays, DNA arrays, RNA arrays, protein arrays, or any combination thereof. Thus, in one embodiment, tissue sections, DNA samples, RNA samples, and/or proteins are placed on specific locations of the slide. In some embodiments, the term "slide" can refer to SELDI and MALDI chips, as well as silicon wafers.

As used herein, the term "stain," "stain," or similar term generally refers to any treatment of a biological specimen that detects and/or distinguishes the presence, location, and/or amount (e.g., concentration) of a particular molecule (e.g., lipid, protein, or nucleic acid) or a particular structure (e.g., normal or malignant cells, cytoplasm, nucleus, golgi body, or cytoskeleton) in the biological specimen. For example, staining may align specific molecules or specific cellular structures of a biological specimen with surrounding parts, and the intensity of staining may determine the amount of a specific molecule in the specimen. Staining may be used not only with bright field microscopes, but also with other viewing tools such as phase contrast microscopes, electron microscopes, and fluorescence microscopes for aiding in the observation of molecules, cellular structures, and organisms. Some staining by the system may allow the outline of the cells to be clearly visible. Other staining by the system may rely on specific cellular components (e.g., molecules or structures) that are stained and do not stain or stain relatively little to other cellular components. Examples of various types of staining methods performed by the system include, but are not limited to, histochemical methods, immunohistochemical methods, and other methods based on intermolecular reactions, including non-covalent binding interactions, such as hybridization reactions between nucleic acid molecules. Specific staining methods include, but are not limited to, primary staining methods (e.g., H & E staining, cervical staining, etc.), enzyme-linked immunohistochemistry methods, and in situ RNA and DNA hybridization methods, such as Fluorescence In Situ Hybridization (FISH).

As used herein, the term "user interface" refers to an interface (such as a graphical user interface, GUI) that allows a user (e.g., an end user, such as a historian and/or a pathologist) to enter commands and data and receive results. The terms "user interface" and "graphical user interface" are used interchangeably herein.

SUMMARY

Pathological images are typically very large, e.g., over 1000 megabytes, each over 2000 megabytes. For example, the high resolution slide image may have a resolution of 100,000 x 80,000 pixels. Such large images are often too large to be efficiently transmitted over a network, particularly over a network and through an internet service provider. To mitigate the effects of network latency in visualizing image data in a viewer application, image data is prefetched and transmitted from a remote server, such as in the form of image tiles (including as preprocessed image tiles as described herein), to the viewer application 120 based on user input. It is believed that the use of larger blocks of image data to be decomposed using tiles can greatly improve the efficiency of transmitting image data over a network, since only those tiles or portions of tiles corresponding to a particular selected magnification need be transmitted as compared to transmitting the entire slide image.

Prefetching refers to requesting image data from a remote storage system or database (such as a database of a remote server) prior to any particular user's request for image data so that the image data can be collected and cached on the device until the particular user requests the image data. In this manner, pre-fetching attempts to capture image data in the background before actually invoking the image data to build a visualization effect on the display, thereby reducing (or even eliminating) the need for the client system to request image data only after a user request. The pre-fetched image data is automatically identified, requested and stored in pre-allocated memory of the viewer application or browser for later use in constructing the visual display. In some embodiments, the pre-fetched data includes an image tile, such as a plurality of image tiles at a particular magnification level.

In some embodiments, the prefetching is based on a prediction of user input, e.g., using past operations, selections, and inputs of the user to anticipate further operations, selections, and inputs, i.e., image data for the next several levels of possible user input is also prefetched when the user input is received through the user interface. The viewer application or browser may conventionally pre-fetch image data comprising a plurality of image tiles based on user input and/or anticipated user input. When a user interacts with a visualization displayed in a viewer application or browser, the user may wish to pan to visualize other image data in the vicinity of any first visualized image data. Thus, a viewer application or browser uses the system to prefetch and store a sufficient amount of image data (e.g., image tiles) to provide a requested visualization, while caching other image data in the memory of the viewer application or browser enables efficient interaction of the user with the visualization. For example, if the viewer application or browser determines that the user is panning the image in a particular direction, the image tile data may be prefetched along a general path of the user's panning. Other methods of visualizing and pre-fetching tile data are described in U.S. patent nos. 9,569,463 and 8,970,618, the disclosures of which are incorporated herein by reference in their entirety.

As described above, prefetching of data is limited to the available memory resources allocated to the interface application, such as: a viewer application 120 or browser 195 running on the client system 170. Such memory allocation may not be sufficient to provide real-time visualization and analysis of image data based on user input. In some embodiments, the client cache and/or the server cache (each of which may include pre-processed image data) may be used to expand on the memory available to the viewer application 120 or the browser 195. As described herein, a greater number of image tiles may be stored in the client-side cache than may be comparably stored in memory allocated to the viewer application 120 or the browser 195. Indeed, the entire image of the patient case record, the entire patient case record, or even multiple patient case records (multiple images and related image tiles each having one or more magnification levels) may be stored in a client cache (such as one managed by the caching agent 140) for retrieval by the viewer application 120 or browser 195. This maintained caching of image data enables retrieval or pre-fetching of image data more quickly than retrieval of pre-fetched image data from a remote server over network 130.

To facilitate providing image data and expedite visualization of such image data, in some embodiments, a client cache (e.g., a cache coupled to a digital pathology analysis system) is provided that stores the image data in a storage subsystem communicatively coupled to the client system. In some embodiments, the client cache is managed by a caching agent that determines, for example, a set of image data received and stored such as those that are to be retained or discarded from the cache based on pre-established and/or user-configurable criteria. In some embodiments, the image data is automatically pushed from the server to the client system (such as through a "caching proxy" application running on the client system). In this manner, a client coupled to the client system caches pre-populated image data, such as when the server receives newly scanned image data or after it is first pre-processed by the server. In some embodiments, the client cache may improve visualization and analysis of requested and/or rendered image data because image data may be quickly prefetched or retrieved from a client cache coupled to the client system rather than from a remote server (i.e., locally retrieved from a pre-populated client cache rather than over a network). As a result, the delay is improved (see examples herein). It is also believed that this may also free up server resources as there is no need for the server to acquire and transmit data or as much data as possible. Further, it is believed that by using such client-side caching, users will not be limited by the available network or internet service provider bandwidth, thereby enabling faster and more efficient visualization and analysis of image data.

Another modality to improve image data transmission for visualization and analysis is through implementation of a server-side preprocessing routine, wherein received scanned image data, such as an input file format (e.g., DICOM, BIF, or another uncompressed image file format), is first converted to a target file format and then stored in a server cache or transmitted to a client system. In some embodiments, the target file format is a standard web browser readable format running on the client system, i.e., its format is a web browser recognizable format, without any further conversion or use of any other image reading application. In some embodiments, the input image data may also be broken into smaller files that may be stored in a server cache (and pre-processed image data may be pre-fetched as needed) or transmitted to the client system, either in real-time as requested by the user, or pre-transmitted to the client cache. Such pre-processing using image data on the server may more efficiently utilize server resources (e.g., pre-processing upon receipt of the image data, rather than real-time processing upon request), and may also more quickly transmit such data to the client system (e.g., using a compressed image file format, or multiple image files, each in a compressed image file format).

The systems and methods provided herein may be applied to the visualization and analysis of any type of image of a biological specimen, including histopathological specimens and/or cytopathological specimens. In some embodiments, the tissue sample is stained with hematoxylin and eosin or stained to identify the presence of one or more biomarkers. For example, the biological sample may be stained by applying one or more stains, and the resulting image or image data includes signals corresponding to each of the one or more stains. In some embodiments, the input image is a simplex image with only a single stain (e.g., stained with 3,3' -Diaminobenzidine (DAB)). In some embodiments, the biological sample may be stained in a multiplexed analysis of two or more stains (thereby providing a multiplexed image). In some embodiments, the biological sample is stained for at least two biomarkers. In some embodiments, the biological sample is stained for the presence of at least two biomarkers and is also stained with an initial stain (such as hematoxylin). In some embodiments, the biological sample is stained for the presence of at least one protein biomarker and at least two nucleic acid biomarkers (e.g., DNA, RNA, microRNA, etc.).

In some embodiments, the biological sample is stained with an immunohistochemistry assay for the presence of one or more protein biomarkers. For example, a biological sample can be stained for the presence of human epidermal growth factor receptor 2 protein (HER2 protein) (e.g., herceptist)^TM(DAKO, Glostrup Denmark) and HER2/neu (4B5) rabbit monoclonal primary antibody (Ventana, Tucson, Arizona).

In some embodiments, the biological sample is stained for the presence of Estrogen Receptor (ER), Progestin Receptor (PR), or Ki-67. In some embodiments, the biological sample is stained for the presence of EGFR or HER 3. Zamyly et al, "Current and Prospectral Biomarkers of Long Cancer", cancers (Basel), 11 months 2018; 9(11) describe examples of other protein biomarkers, the disclosure of which is incorporated herein by reference in its entirety. Examples of protein biomarkers described by Zanay include CEACAM, CYFRA21-1, PKLK, VEGF, BRAF, and SCC.

In some embodiments, the biological sample is stained for the presence of one or more nucleic acids, including mRNA, in an In Situ Hybridization (ISH) assay. U.S. patent No. 7,087,379, the disclosure of which is incorporated herein by reference in its entirety, describes a method of staining a sample with an ISH probe so that a single spot (or dot) representing a single gene copy can be observed and detected. In some embodiments, multiple target genes are analyzed simultaneously by exposing a cell or tissue sample to multiple nucleic acid probes that have been labeled with multiple different nucleic acid tags.

In some embodiments, the system of the present disclosure is adapted to facilitate visualization, analysis, and reporting of biological image data obtained from a subject (e.g., a biopsy sample obtained from a human patient). An exemplary client-server system 150 is depicted in fig. 1A-1D, including a server 160 and a client system 170 (e.g., a digital pathology analysis system) coupled together by a network 130, which network 130 may be, for example, the internet or an intranet. Server 160 includes a processor 161 coupled to a memory subsystem 162, a user interface 163, a mass storage device 164, and a network interface 165. In some embodiments, mass storage device 164 includes an amount of pre-allocated storage space for image data (such as pre-processed image data). In some embodiments, network interface 165 may be directly coupled to network 130, or may be coupled to network 130 through an Internet Service Provider (ISP)180 or other entity. In some embodiments, the server 160 may include additional modules, such as an image management module 188 (see fig. 1D). In some embodiments, the image management module 188 may include logic or software to load, read, and convert files including image data. In some embodiments, the image management module 188 may include a converter configured to take incoming scanned image data and automatically convert it for distribution to and viewing within an interface (e.g., a web browser).

Similarly, the client system 170 includes a processing subsystem 102 coupled to a memory subsystem 172, a user interface 171, a storage subsystem 104, and a network interface 112. In some embodiments, storage subsystem 104 may include pre-allocated space for storing image data, such as pre-processed image data received from server 160. The client 170 may be coupled to the network 130 directly or may be coupled to the network 130 through an ISP 185 or other entity. The user interface 171 of the client system may include a display 108, a keyboard and mouse, and/or other graphical input devices 106. Additionally or alternatively, the user interface 171 may be a touch screen, for example. In some embodiments, a user interface 171 is rendered by processing subsystem 102 and provided on display 108 to (i) facilitate visualization and analysis of imaging data and/or patient data; and/or (ii) retrieve imaging data, patient information, or other clinical information from a database or mass storage device 164 based on the received user input, such as one available over the network 130.

In some embodiments, server 160 and client system 170 each include a cache, i.e., memory space allocated within mass storage device 164 or storage subsystem 104, respectively. In some embodiments, the mass storage device 164 or a cache within the storage subsystem 104 is configured to store image data, such as pre-processed image data derived from an entire slide scan of a biological specimen disposed on a microscope slide.

In some embodiments, software, such as the viewer application 120 and the cache agent 140, runs on the client system 170, and a client interface or client portal (not shown) on the server 160 is used to facilitate communications between the client system 170 and the server 160 (see FIG. 1B). In some embodiments, the viewer application 120 runs remotely on the server 160 and an instance of the viewer application 120 runs within the browser 195 on the client system 170 (see FIG. 7C). As such, a user (e.g., a pathologist) may interact with elements of the system (e.g., configurable elements) so that image data may be analyzed and/or interpreted (e.g., an organizer or pathologist may select user configurable parameters, such as a menu bar tool and/or a viewer panel).

In some embodiments, the locally running viewer application 120 or browser 195 sends a request to an application running on the remote server 160 for the location of the image data file. In some embodiments, the location of the informational data file is stored in one or more URLs. The location of the image data file may be on a remote server, such as in a cache on the remote server, or may be in a local cache communicatively coupled to the client system. In any event, the transmitted location information (e.g., URL) provides the location of the appropriate image data file or image tile file for retrieval. In some embodiments, the URL provides additional parameters such as sharpness, contrast, saturation, color, and the like. Examples of URLs including parameters are as follows: https:// localhost/restapp/slide/imageFetcher/1012/116quality 2& roff 3072,0& rsiz 1536,1536& imgtype jpg & brightness 0& distortion 0& sharpness 0& color 0,0& transformype SRGB 0

Fig. 2A shows a server 160 (a computer or computing device) communicatively coupled to the scanning device 110. In some embodiments, server 160 may include software to instruct scanning apparatus 110 to perform certain operations using certain user-configurable parameters and to send acquired imaging data to processing subsystem 161 or mass storage device 164. In some embodiments, the scanning apparatus 110 can be communicatively coupled to the server 160, whereby scanned slides can be transmitted to the server 160 over the network 130 and stored on the mass storage device 164 for later retrieval, e.g., an entire slide image or a pre-processed image tile derived from the entire slide image can be stored on the server. For example, the scanning apparatus 110 may be used to acquire image data and store the scanned image data in a file located on the mass storage device 164 or other network server, whereby the file (including relevant metadata, patient case identifier, and user identification) may be later retrieved for visualization and analysis (such as from the client 170). In some embodiments, for example, the server 160 and the scanning device 110 are maintained separately, e.g., by different organizations, where slides may be scanned by one participant and transferred to the server 160 under the control of another participant.

Fig. 2B and 2C are block diagrams of a client system 170 according to embodiments of the present disclosure. Client system 170 may be implemented using any type of user-operable computing device, including desktop computers, laptop computers, tablet computers, handheld devices (e.g., smart phones, media players), and so forth. As described above, client system 170 may include a plurality of interconnected components, such as processing subsystem 102, storage subsystem 104, user input device 106, display 108, and network interface 112 that communicate via bus 114, as discussed in more detail below.

Processing subsystem 102 may include a single processor (which may have one or more cores), or may include multiple processors (each having one or more cores). In some embodiments, processing subsystem 102 may include one or more general-purpose processors (e.g., CPU), special-purpose processors (e.g., graphics processor, GPU), digital signal processors, or any combination of these and other types of processors. In some embodiments, some or all of the processors in the processing subsystem may be implemented using custom circuitry, such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA). In some embodiments, such integrated circuits execute instructions stored on the circuit itself. By way of example only, the processing subsystem 102 may execute instructions to receive and process image data stored within a local or network storage system and display the image data (e.g., display an entire slide scan image, or any enlarged portion of an entire slide scan image).

Storage subsystem 104 and/or memory subsystem 172 (and their counterparts on server 160) may include various memory units, such as system memory, Read Only Memory (ROM), and permanent storage. The ROM may store static data and instructions required by the processing subsystem 102 and other modules of the client system 170. The persistent storage device may be a read-write storage device. The persistent storage may be a non-volatile storage unit that stores instructions and data even when the client system 170 is powered down. In some embodiments, a mass storage device (e.g., a magnetic or optical disk or flash memory) may be used as the permanent storage device. Other embodiments may use a removable storage device (e.g., a flash drive) as the permanent storage device. The system memory may be a read-write memory device or a volatile read-write memory (e.g., dynamic random access memory). The system memory may store some or all of the instructions and data required by the processor during runtime.

The memory subsystem 104 and/or the memory subsystem 172 may include any combination of non-transitory computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash, programmable read-only memory), and so forth. Magnetic and/or optical disks may also be used. In some embodiments, the storage subsystem 104 may include removable storage media that may be read and/or written to; examples of such media include Compact Discs (CDs), read-only digital versatile discs (e.g., DVD-ROMs, dual-layer DVD-ROMs), read-only and readableMagnetic disks, ultra-high density optical disks, flash memory cards (e.g., SD cards, mini SD cards, micro SD cards, etc.), and the like. In some embodiments, the image data and/or patient data may be stored in one or more remote locations, such as cloud storage, and synchronized with other components of the client system 170. When the terms "memory" or "a memory" are used herein, they can refer to one or more memories, such as multiple memories.

In some embodiments, the storage subsystem 104 may store one or more software programs to be executed by the processing subsystem 102, such as the viewer application 120 or the caching agent 140. "software" generally refers to a sequence of instructions that, when executed by processing subsystem 102, cause client system 170 to perform various operations, thereby defining one or more specific machine implementations that perform and implement the operations of a software program. Thus, "software" may also include any other type of instructions readable and executable by firmware or embedded applications or processing subsystems 102. The software may be implemented as a single program or as a collection of separate programs or program modules that interact as desired. In some embodiments, programs and/or data may be stored in non-volatile memory and copied, in whole or in part, to volatile working memory during program execution. The processing subsystem 102 may retrieve program instructions to be executed and data to be processed from the storage subsystem 104 in order to perform various operations, including the operations described below.

The user interface 171 may be provided to the display device 108 and/or one or more other user output devices (not shown). The user interface 171 may include, for example, visualizations and other representations including images, menu bars, drop down menus, and/or panels resulting from scanning of stained biological samples (e.g., samples stained for the presence of one or more biomarkers or stained with hematoxylin and eosin). Input device 106 may include any device through which a user may provide signals to client system 170; client system 170 may interpret the signal as indicating a particular user request or information. In some embodiments, the input device 106 may include any or all of a keyboard touchpad, a touch screen (e.g., a touch-sensitive overlay on the display surface of the display 108), a mouse or other pointing device, a scroll wheel, a click wheel, a dial, a button, a switch, a keyboard, a microphone, and so forth.

The display 108 may display visualizations generated by the client system 170 (e.g., a viewer panel that includes representations of image data, conveys information to a user, or provides a contextual menu of user-selectable configuration options, etc.) and may include various image generation technologies such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), Light Emitting Diodes (LEDs) (including organic light emitting diodes, OLEDs), projection systems, etc., as well as supporting electronics (e.g., digital to analog or analog to digital converters, signal processors, etc.). Some embodiments may include a device that acts as both an input device and an output device, such as a touch screen. In some embodiments, other user output devices may be provided in addition to or in place of display 108.

In some embodiments, the user interface may provide a graphical user interface (such as through a browser or in a viewer application on the client system 170) in which visible image elements in certain areas of the display 108 are defined as active elements, interactive elements, or control elements selected by the user using the user input device 106. For example, the user may manipulate the user input device 106 to position an on-screen cursor or pointer over the control element, then "click" a button to indicate a selection, and select to send a signal to perform a specified action or routine. For example, a user may manipulate the user input device 106 to select an icon within a user interface (such as an icon in a viewer panel, menu bar, or within a drop-down menu), which will enable initiation of an operation or selection of a tool. In some embodiments, the user may manipulate the user input device 106 to select certain images within the patient case, zoom in on the images, pan the displayed images (i.e., change the field of view), tilt the images, rotate the images, and so forth. In some embodiments, the user may manipulate the user input device 106 to interact with a drop down menu to select one or more panels, including an interactive panel.

Additionally or alternatively, the user may touch a control element on the touch screen device (e.g., with a finger or stylus). In some embodiments, the user may speak one or more words associated with the control element (the words may be, for example, a tag on the element or a function associated with the element). In some embodiments, user gestures on the touch-sensitive device may be recognized and interpreted as input commands; these gestures may, but need not, be associated with any particular area on display 108. Other user interfaces may also be implemented.

Network interface 112 may provide data communication capabilities for client system 170. In some embodiments, the network interface 112 may include a Radio Frequency (RF) transceiver component, a GPS receiver component, and/or other components for accessing a wireless voice and/or data network (e.g., using cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE, 5G, WiFi (IEEE 802.11 family of standards), or other mobile communication technologies, or any combination thereof). In some embodiments, the network interface 112 may provide wired network connectivity (e.g., ethernet) in addition to or instead of a wireless interface. The network interface 112 may be implemented using a combination of hardware (e.g., antennas, modulators/demodulators, encoders/decoders, and other analog and/or digital signal processing circuitry) and software components.

Bus 114 may include various system buses, peripheral buses, and chipset buses that communicatively connect the numerous components of client system 170. For example, the bus 114 may communicatively couple the processing subsystem 102 with the storage subsystem 104. The bus 114 may also be connected to the input device 106 and the display 108. The bus 114 may also couple the processing subsystem 102 to a network through the network interface 112. In this manner, the client system 170 may be connected to a network 130 of multiple computer systems (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), an intranet, or a network of networks such as the Internet). The skilled person will appreciate that other components, such as a scanning device, a tissue processing system, may be connected to the bus 114.

Some embodiments include electronic components, such as microprocessors, storage devices, and memory storing computer program instructions in a computer-readable storage medium. Many of the features described herein can be implemented as processes specified as a set of program instructions encoded on a computer readable storage medium. When executed by one or more processing units, the program instructions cause the processing units to perform the various operations indicated in the program instructions. Examples of program instructions or computer code include machine code (e.g., produced by a compiler) and files including high-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.

With appropriate programming, processing subsystem 102 may provide various functionality for client system 170. For example, processing subsystem 102 may execute a browser that is in communication with remote server 160 and facilitates browsing and interpreting scanned images of biological samples. The browser may be configured to run an instance of a remote application, such as a viewer application, and may provide various functions, such as the ability to select user configurable options or user selectable panels, or the ability to control navigation and annotation of images (see fig. 2C). As another example, processing subsystem 102 can execute a viewer application 120 (i.e., such as a viewer application running locally in fig. 2B) having a user interface that facilitates viewing and interpreting scanned images of biological samples. The viewer application 120 may provide various functions, such as the ability to select user configurable options or user selectable panels, or the ability to control navigation and annotation of images.

In some embodiments, the viewer application 120 incorporates various interoperating modules (e.g., code blocks) that, when executed by one or more processors within the processing subsystem 102, implement aspects of interface operations. For example, viewer application 120 may include a content fetcher 122 (e.g., a fetcher adapted to retrieve URLs or image data from server 160 over network 130), a content renderer 124 (e.g., to decrypt received image data or patient data), a GUI renderer 126 (e.g., to continuously generate representations and visualizations based on user selections and commands and based on image data received by content fetcher 122), and a UI interpreter 128 (e.g., to interpret user selections and commands, such as pan commands, zoom commands, and select commands).

Referring to FIG. 2B, in some embodiments, the viewer application 120 is configured to communicate with the server 160, e.g., to send a request for picture location information, such as a URL, to the server 160 and to receive such image location information from the server 160. In some embodiments, the viewer application 120 may communicate with the caching agent 140 to access image data in the client cache based on the received location information (such as in one or more URLs). In some embodiments, if the client cache no longer includes the image data requested in the cache (e.g., the image data may have been automatically deleted from the cache according to user established criteria), the viewer application 120 (or caching agent 140) may be redirected to the server (or reverse proxy) to retrieve the image data.

Referring to fig. 2C, in some embodiments, the viewer application 120 instead runs on the server 160 (such as over the network 130) and the browser 195 runs on the client system 170, whereby the browser 195 may run an instance of the viewer application remotely or the browser may communicate with the viewer application or other software running on the remote server 160. In either case, the browser is configured to send a request for an image data location (such as a URL) to server 160. In some embodiments, based on the received file location information, the browser 195 may communicate with the caching agent 140 to access the image data in the client cache based on the received image location information. In some embodiments, if the client cache no longer includes the image data requested in the cache (e.g., the image data may have been automatically deleted from the cache based on user established criteria), the browser 195 (or caching agent 140) may be redirected to the server (or reverse proxy) to retrieve the image data.

In some embodiments, the cache proxy 140 may be communicatively coupled to the viewer application 120 or browser 195 of the client system 170, and also communicatively coupled to the server 160. In some embodiments, the cache agent 140 is separate from the viewer application 120 or browser 195 running locally, and in some embodiments the cache agent 140 is running at all times, that is, it is running even after the viewer application 120 or browser 195 is closed. In some embodiments, the caching proxy 140 communicates directly with the server 160, for example, via the HTTP protocol, and may itself be considered an HTTP server running on the client system 170.

In some embodiments, caching agent 140 is a client-side cache management application configured to manage image data (e.g., or a portion of a scanned image, such as an image tile as described herein) and to manage caching of the image data or image tiles according to various events or pre-established user criteria. In some embodiments, the caching agent 140 may receive a request at runtime from the viewer application 120 or the browser 195, such as a request to receive image data including pre-processed image data, based on the location information received from the server 160. For example, server 160 may be configured to automatically send newly received or scanned image data to client system 170 (such as after pre-processing the scanned image data), and caching agent 140 may manage communication with server 160 and manage receipt of the image data. In some embodiments, the caching agent 140 is configured to manage the client cache itself, e.g., determine which image datasets to retain and which to delete, such as based on pre-established user criteria (described further herein). In some embodiments, the caching agent 140 is configured to determine whether to retain or delete an image dataset that has been previously visualized and/or analyzed. In some embodiments, the client cache is a pre-allocated amount of storage space within the storage subsystem 104 of the client system 170.

In some embodiments, the caching agent 140 may optionally send data to the server to indicate that certain image data sets (or entire patient cases) have been reviewed by the user and/or image data that has been deleted or is to be deleted from the cache (such as a set date according to a user-configured cache expiration criterion). In some embodiments, caching agent 140 does not provide data to server 160.

In some embodiments, the caching agent 140 may be configured to facilitate the transfer of image data to the browser 195 or the viewer application 120. For example, the viewer application 120 may request the location of the required image data based on user input, and may request this information from the server 160 (e.g., it may request the URL of the image data), and the caching agent 140 may be used to retrieve the image data from the client cache, provided that the image data is retained by the caching agent 140.

Populating client side caches with image data

Fig. 3 shows a flow chart illustrating the acquisition of image data from a stained biological sample. In a first step (300), a pathology physician creates and enters a pathology worksheet. In some embodiments, the pathologist has an assigned user identification. In this manner, any worksheet created by the pathologist can be traced back to the user identification of the pathologist.

Next, the stained slide, such as a slide prepared by a histopathologist, is scanned (step 310). In some embodiments, the image data is obtained from a scanning device (such as VENTANA DP 200 scanner, available from Ventana Medical Systems, inc., Tucson, AZ) and the image data may be stored in a database, such as a networked database or on the mass storage device 164, for later visualization and analysis.

In some embodiments, the scanning device generates a high resolution image of each slide for subsequent display and viewing. In some embodiments, the scanning device produces a high resolution image of each slide at multiple magnification levels (e.g., 20x or 40 x). In some embodiments, image data of all slides related to the specimen and clinical detailed information of the specimen may be stored in a database along with metadata about the virtual slides. The scanned slide is associated with patient case identification information (and a pathologist user identification), including metadata. In some embodiments, the scanned image data is stored in a first file format (e.g., an uncompressed image file format). In some embodiments, the scanned image data (e.g., a scan of an entire slide image) is stored as a DICOM file, BIF file, or TIFF file. In some embodiments, a DICOM file or BIF file may include RAW image data or other uncompressed image data at multiple magnification levels.

Subsequently, the image data and associated patient case identification information from the scanning apparatus 110 are sent to the server 160 for storage (e.g., sent to the server 160 and stored within the mass storage device 164) (step 320 of fig. 3, step 1000 of fig. 10). In some embodiments, the scanned image data is stored as a separate file, i.e., a single file representing the scan of the entire slide image in the first file format is stored on the server 160. In some embodiments, image tiles are generated and sent to the client system at the request of the user.

In some embodiments, the scanned image data is automatically pre-processed on the server 160 after it is received from the scanning device 110, such as with the image management module 188 (see FIG. 10, step 1010). In some embodiments, the image management module 188 includes logic or software to load received image data, read the received data, convert the received image data from a first format to a second format, and/or perform other image processing operations, such as extracting portions of the received image data and outputting the extracted portions to a separate file.

In some embodiments, the scanned image data is pre-processed such that a single image file in a first format (e.g., an input file format) is converted to a single image file having a second format, such that the first image format and the second image format are different. For example, an entire slide image in a first format can be converted to an entire slide image in a second format. In some embodiments, the second image file format is a target file format. In some embodiments, the target file format is one that is pre-selected for use on the client system 170 or one that is compatible with a browser application, such as a standard web browser application running on a personal computer. In some embodiments, the second image file format is a compressed file format including, but not limited to, JPEG2000, and PNG. For example, a single DICOM, BIF, or TIFF image file may be converted to a single compressed image file, e.g., a single JPEG file.

Additionally or alternatively, where a single DICOM or BIF image file includes image data relating to multiple scan magnification levels, a single compressed image file may be generated for each magnification level. For example, if the BIF image includes entire slide image data at a magnification level of 20x and entire slide image data at a magnification level of 40x, two compressed image files (e.g., JPEG files) may be generated, one at 20x and the other at 40x, and both are full slide images. In some embodiments, these compressed image files may be sent to the user upon request (e.g., upon request, the compressed image files may be broken up into multiple tiles at a magnification and sent to the user in real-time).

In some embodiments, the scanned image data is pre-processed (step 1010 of FIG. 10) such that a single image file in a first format is segmented into a plurality of pre-processed image files, wherein each of the plurality of pre-processed image files may have the same or a different file format than the first file format. For example, a single image file of a first file format, such as the TIFF file format, may be divided into a plurality of pre-processed image files, such as 500 or more pre-processed image files, each having the TIFF file format or a different file format. As another example, a TIFF image file may be split into multiple pre-processed image files having a TIFF format, e.g., a single TIFF file may be split into 500 or more TIFF format pre-processed image files.

Additionally or alternatively, and again following the above example, the TIFF image file may be divided into multiple preprocessed image files having different file formats, such as compressed image file formats, including JPEG, JPEG2000, and PNG. In these embodiments, the TIFF image file may first be converted to a second file format (e.g., JPEG) and then divided into a plurality of pre-processed JPEG image files; alternatively, the TIFF image file may be divided into a plurality of preprocessed image files having a TIFF file format and then separately converted into JPEG files. In either case, the result is a plurality of preprocessed image files having different file formats, such as an object file format, including a format compatible with a Web browser.

In some embodiments, the pre-processing may generate multiple sets of pre-processed image files, and each separate set of image pre-processed image files may include multiple image tile files. For example, and as described above, an entire slide scan in DICOM format or BIF format may include image data corresponding to several magnification levels, e.g., 1x, 20x, and 40 x. In some embodiments, multiple image tile files may be generated for each magnification level within the original entire slide scan. For example, the first set of generated preprocessed image files may include a plurality of image tile files associated with a 20x magnification, while the second set of generated preprocessed image files may include a plurality of image tile files associated with a 40x magnification.

In some embodiments, after the pre-processed image files are generated, they are stored in one or more memories (step 1020), including but not limited to volatile and non-volatile memories. In some embodiments, the preprocessed image files are stored in pre-allocated storage space within a mass storage device 164 communicatively coupled to server 160. In some embodiments, the preprocessed image files each include a unique file name. In some embodiments, the unique file name includes coordinates of the location of the tile image derived from the entire slide image. In this manner, if a user requests tiles with coordinates (x1, y1), (x2, y1), and (x3, y1), the pre-processed tile image files corresponding to each of these locations may be easily retrieved based on the unique file name attributed to each image tile file.

In some embodiments, the preprocessed image file may then be sent to client system 170. In some embodiments, the preprocessed image file may be sent to client system 170 upon receiving user input requesting acquisition of certain image data (e.g., see step 710 of FIG. 710), e.g., transmitted in real-time when the user input requires a particular image tile. In some embodiments, after the pre-processing of the image is complete (e.g., after steps 1000-1020 are complete), the pre-processed image file may be pushed out to the client system 170. For example, the pre-processed image file may be passed to a caching agent 140 running on the client system 170 so that, upon receiving user input requesting a particular image tile, the image data may be retrieved locally from the storage subsystem of the client system, as described herein (see step 330 of FIG. 3; see also FIGS. 4A and 7). In some embodiments, the data is pushed to a client system that has been assigned a particular patient case identification and to a system used by a pathologist with a particular user identification. In other words, in some embodiments, the image data is pushed onto the computer of the particular pathologist to whom the patient case is assigned.

Fig. 4A shows the step of populating the client cache on client system 170 with newly received (i.e., newly generated image data). In some embodiments, if data has not been accessed from the scanning device 110 of a computer system coupled to the scanning device, the server 160 accesses newly generated image data that has been associated with a particular user identification (step 400). In some embodiments, the accessed scanned image data (such as in an input file format) may be pre-processed before being pushed to the caching agent 140 as described above.

Next, server 160 performs authentication with client system 170 (step 410) to ensure that client system 170 is running caching agent 140 (so that the managed client cache can be filled) and that the correct information is delivered to the correct computer and to the correct user (i.e., the correct pathologist for visualization and analysis). In some embodiments, server 160 sends an authentication request to client system 170 (see step 411 of FIG. 4B). After authenticating the request, in some embodiments, client system 170 sends its Internet Protocol (IP) address to server 160 (step 412 of fig. 4B), which server 160 verifies as belonging to the correct client system 170. Next, server 160 requests and client system 170 sends user identification data (step 413 of FIG. 4B). In some embodiments, once the correct system and user are identified, authentication has been completed. Of course, other identification data may be used for authentication purposes.

In some embodiments, authentication is performed only once, for example, after the caching agent 140 is installed on the client system 170 (see, e.g., FIG. 9). In these embodiments, authentication is still performed, rather than confirming the user identification credentials and/or the IP address of the client system 170, only that the caching agent is still running on the system.

After authentication, in some embodiments, server 160 sends all available image data to client system 170. In these embodiments, the caching agent 140 is provided the task of clearing space within the client cache coupled to the client system 170 to store all newly received image data. For example, if a 10GB preprocessed image tile file is to be transmitted from the server 160 to the client system 170, but only 5GB of space is available in the client cache, the caching agent 140 will delete 5GB of the existing image data in the client cache to make room for all image data received by the server. The maintenance of available cache space may be determined based on pre-established criteria such as those described herein.

In some embodiments, only a portion of all available image data is pushed to client system 170. For example, and in some embodiments, server 160 sends portions of the accessed image data based on whether the user has viewed the data. For example, if some image data has been previously transmitted but not viewed by the user before the cache expiration date of the previously transmitted image data (as described herein), the image data (or portions thereof) may be transmitted to the client cache again.

In some embodiments, server 160 transmits those portions of the image data based on predetermined selection criteria. In some embodiments, server 160 transmits the earliest of those portions of the image data. For example, if there are image scans starting from a first scan date and image scans starting from a second scan date in 10GB of image data available for push to client system 170, and the first scan date is before the second scan date, then the older image scan starting from the first scan date may be the image scan selected to be transmitted to the client cache. In some embodiments, the server 160 sends portions of the image data marked with tokens, e.g., those who must start the review out of order, those with higher priority, those images that belong to cases already under review (e.g., a new scan that provides new images for the reviewed cases), etc.

In some embodiments, server 160 may perform a determination of how much storage space is available within a client cache coupled to client system 170 (step 420). In some embodiments, the caching agent 140 running on the client system 170 may send a value representing the amount of pre-allocated storage space in the client cache, which is available at the time of the request by the server 160. In some embodiments, the server utilizes the value transmitted from the caching agent 140 (i.e., the value representing the amount of storage space available within the client cache) to determine how much of the available scanned image data received by the server 160 can be transmitted to the client cache. In some embodiments, server 160 selects portions of the image data to communicate to the client cache based on the available space in the client cache (step 430).

For example, if the image data available for delivery on the server includes 10GB of information and only 5GB of storage space is available in the client cache, the server 160 can determine which portions of the image data to send, i.e., which patient case, image, pre-processed image or portion thereof, image tile, etc., to send to the client system 170 based on the available space within the client cache. In some embodiments, the portion of image data selected for delivery may be from a single patient case or from multiple patient cases. For example, a series of pre-processed image tiles related to a first patient case may be transferred, and another series of pre-processed image files may be transferred from a second patient case. In some embodiments, the portion of image data selected for transfer may be pre-processed image tiles derived from certain magnifications within the scanned image file (e.g., only 40x magnification pre-processed image tiles are transferred, rather than three different sets of magnification pre-processed image tiles, given space constraints within the client cache). In some embodiments, the entire free available space of the client cache is filled with available image data from the server 160.

After at least a portion of the available image data remaining on the server is passed to the client cache, one or more URLs providing the location of the stored image data are generated and stored on server 160 (these URLs may be subsequently retrieved by a viewer application, such as by an instance of the viewer application running on the server or loaded in a browser on the client system). In some embodiments, the URL includes various image parameters including brightness, sharpness, contrast, color, and the like. Examples of suitable URLs are as follows: https:// localhost/restapp/slide/imageFetcher/1012/116quality ═ 2& roff ═ 3072,0& rsiz ═ 1536,1536& imgtype ═ jpg & brightness ═ 0& transmission ═ 0& sharpness ═ 0& color ═ 0,0& transformType ═ SRGB. In some embodiments, a portion of the URL may be used with the client proxy URL if an image is to be retrieved from the client cache. In some embodiments, the URL may be retrieved by the client system 170 (such as the viewer application 120 and/or the caching agent 140 running on the client system 170) so that image data (such as image data present within the client cache) may be retrieved for visualization and/or analysis.

In some embodiments, the caching agent 140 manages image data stored in a client cache that resides on the storage subsystem 104 of the client system 170 (see 610 of fig. 6). In some embodiments, the caching agent 140 maintains the client cache periodically, e.g., at fixed times or fixed intervals. In some embodiments, the caching agent 140 maintains client caches as needed. In some embodiments, the caching agent 140 deletes those cached image datasets from the client cache that have been checked by the user (see step 510 of fig. 5). For example, the caching agent 140 can delete cached image data related to a particular set of slides that have been checked. As another example, the caching agent 140 may be configured to delete all patient cases that the user has already checked.

In some embodiments, the caching agent 140 may be configured to delete cached image data based on pre-existing and user-established criteria (see step 520). For example, if the user fails to view the cached image data within a predetermined period of time, the cached image data may be deleted. As another example, the cached data itself may have a validity period regardless of whether the image data has been viewed by the user (e.g., a set time period of 3 days, 1 week, or 1 month). As yet another example, if a patient case has been reassigned from one user to another, the cached data may be deleted by the caching agent 140. As yet another example, once a certain storage space threshold is met, for example, if only a certain amount of space remains available in a pre-allocated client cache, older cache data may be deleted and newer data may be retained in anticipation of receiving new image data from server 160.

In some embodiments, the caching agent 140 running on the client system 170 is configured to receive new image data periodically, such as at a particular time of day. In some embodiments, the caching agent 140 is configured to receive new image data once it becomes available on the server 160 (i.e., once the server receives a new scanned image and/or patient case and is ready for transmission). Referring to FIG. 6, a request may be received from remote server 160 (step 620), and remote server 160 may check to see if caching agent 140 is installed and/or running on client system 170. In some embodiments, authentication is performed (step 630) (see also fig. 4B). After the authentication session is established between the caching agent 140 and the client system 170, the caching agent 140 may receive the stored image data for storage in the pre-allocated client cache (step 640).

Then, after having received a request for image data from the user, the client system 170 may access the image data within the client cache for visualization (see fig. 7 and 8). Referring to FIG. 7, in a first step, the viewer application 120 receives a first user input, e.g., a user input, related to the selection of image data, where additional tiles are needed to complete the panning operation (step 710). In some embodiments, the viewer application 120 runs on the client system 170. In some embodiments, the viewer application 120 runs on the remote server 160, but an instance of the viewer application is accessed via the browser 195 on the client system 170. If the viewer application is accessed through a browser, the browser may receive user inputs, and those user inputs may be transmitted remotely.

Subsequently, a request for image data is sent from viewer application 120 to remote server 160. In some embodiments, authentication is performed between client system 170 and remote server 160 (step 720), and client system 170 may signal to remote server 160 that caching agent 140 is running on client system 170. In some embodiments, the remote server 160 then sends one or more URLs to the viewer application 120, where each of the one or more URLs provides location information regarding a storage location of the requested image data (step 730). In some embodiments, the viewer application 120 determines whether the requested image data is present in the client cache (step 740). In some embodiments, the viewer application 120 may retrieve the image data directly from the client cache based on the provided URL and assuming that the image data is stored in the client cache (step 750). In some embodiments, the caching agent 140 retrieves image data from the client cache based on instructions received from the viewer application 120. In some embodiments, based on the provided URL, the viewer application 120 is redirected to a remote server (or a reverse proxy running on a remote server) for the requested image data (step 750) (in this example, if the image data was deleted by the caching agent 140 during client cache maintenance, the image data is not found in the cache). Once the viewer application 120 receives the image data, it is visualized (step 760). The process repeats as new user inputs (such as the second, third, and fourth user inputs) are received. The process is also repeated for each patient case that the user views.

Referring to FIG. 8, in some embodiments, a viewer application is running remotely on server 160 and an instance of the viewer application is loaded into a browser running locally on client system 170. In this manner, the user will use the viewer application in the same manner as if the viewer application were running locally on client system 170. In this regard, after performing authentication and checking to see if the caching agent 140 is running on the client system 170, user input into a viewer loaded into the browser will cause the viewer application to retrieve the requested image data. The remotely running viewer application will then send one or more URLs to the browser, which will attempt to retrieve the URLs, such as from within the client cache on the client system (and through the caching agent 140 running on the client system 170). If the image data is present in the local client cache, the browser will retrieve the image from the cache. Otherwise, the browser will be redirected to a remote server (e.g., a reverse proxy) that will retrieve the image data. In either case, the retrieved image data will be loaded and visualized.

In some embodiments, authentication is performed between the client system and the remote server, and the client system may send a signal to the remote server that the caching agent is running on the client system. In some embodiments, the remote server may then send one or more URLs to a browser running on the client system, where each of the one or more URLs provides location information regarding a storage location of the requested image data. In some embodiments, the browser determines whether the requested image data is present in the client cache. In some embodiments, the browser may retrieve the image data directly from the client cache (e.g., by a caching agent running on the client system) based on the provided URL and assuming that the image data is stored in the client cache. In some embodiments, based on the provided URL, the browser is redirected to a remote server (or a reverse proxy running on a remote server) for the requested image data (in this instance, if the image data has been deleted by the caching agent during client cache maintenance, the image data is not found in the cache). Once the browser receives the image data, it is visualized. The process repeats as new user inputs (such as the second, third, and fourth user inputs) are received. The process is also repeated for each patient case that the user views.

Other embodiments

Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their analogous structures, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions, which can be encoded on a computer storage medium for execution by, or to control the operation of, data processing apparatus. Any of the modules described herein may comprise logic that is executed by the processor. "logic," as used herein, refers to information having any form of instruction signals and/or data that may affect the operation of a processor. Software is an example of logic.

The computer storage medium may be or may be embodied in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or any combination of one or more of them. Furthermore, although the computer storage medium is not a propagated signal, it can be the source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage media may also be or be embodied in one or more separate physical components or media, such as multiple CDs, diskettes, or other storage devices. The operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

The term "programmed processor" encompasses various devices, apparatuses, and machines that process data, including by way of example a programmable microprocessor, a computer, a system on a chip, or a plurality or combination of the foregoing. The apparatus may comprise special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can include, in addition to hardware, code that creates an execution environment for the associated computer program, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The devices and execution environments may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

A computer program (also known as a program, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative languages, or programming languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. The computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform operations by operating on input data and generating output results. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and the processor of any one or more digital computers. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor that performs operations in accordance with instructions and one or more memory devices that store instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data (magnetic, magneto-optical disks, or optical disks). However, a computer does not require such a device. Further, the computer may be embedded in another device, e.g., a mobile telephone, a Personal Digital Assistant (PDA), a mobile audio or video player, a game controller, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a Universal Serial Bus (USB) flash drive), to name a few. Suitable means for storing computer program instructions and data include various forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be added by, or incorporated in, special purpose logic circuitry.

To facilitate interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., an LCD (liquid crystal display), an LED (light emitting diode) display, or an OLED (organic light emitting diode) display, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can effect input to the computer. In some embodiments, a touch screen may be used to display information and receive user input. Other kinds of devices may also be used to facilitate interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and may receive user input in any form, including acoustic, speech, or tactile input. In addition, a computer may enable interaction with a user by sending and receiving files to and from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks). For example, the network 20 of FIG. 1 may include one or more local area networks.

The computing system may include any number of clients and servers. Typically, the clients and servers are located remotely from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and the client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for displaying data to and receiving input from a user interacting with the client device). Data generated at the client device (e.g., as a result of user interaction) may be received at the server from the client device.

All U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, and non-patent publications referred to in this specification and/or listed in the application data sheet, are incorporated herein by reference, in their entirety. Aspects of the embodiments can be modified, if necessary to employ concepts of the various patents, applications and publications to provide yet further embodiments.

Although the present disclosure has been described with reference to a few illustrative embodiments, it should be understood that numerous other modifications and embodiments can be devised by those skilled in the art that will fall within the spirit and scope of the principles of this disclosure. More specifically, reasonable variations and modifications are possible in the component parts and/or arrangements of the subject combination arrangement within the scope of the foregoing disclosure, the drawings and the appended claims without departing from the spirit of the disclosure. In addition to variations and modifications in the described components and/or arrangements, alternative uses will also be apparent to those skilled in the art.