阅读说明：本技术 用于牙齿图像的方法和装置 (Method and apparatus for dental images ) 是由 M·K·小卡里尔 P·T·哈里斯 S·维尼琴科 S·布兰科 A·S·卡沙可夫 S·哈伦 于 2017-11-03 设计创作，主要内容包括：本文描述的是从一个或多个预定视角获得患者牙齿的图像或一组图像的方法和装置。这些方法和和装置可以包括使用覆盖图,所述覆盖图包括用于每个预定视角的牙齿的轮廓线。覆盖图可用于自动捕获、聚焦和/或照射牙齿。本文还描述了用于使用包括一组预定视图的患者牙齿的一系列图像的方法和装置,以确定患者是否是正畸手术的候选者。(Described herein are methods and apparatus for obtaining an image or set of images of a patient's teeth from one or more predetermined viewing angles. These methods and apparatus may include the use of an overlay that includes contours of the teeth for each predetermined viewing angle. The overlay can be used to automatically capture, focus, and/or illuminate the tooth. Also described herein are methods and apparatus for using a series of images of a patient's teeth including a set of predetermined views to determine whether the patient is a candidate for orthodontic surgery.)

1. A non-transitory computing device-readable medium having instructions stored thereon that are executable by a processor to cause the processor to:

using a mobile communication device having a camera to guide a user to take a series of images of a patient's teeth in a plurality of predetermined views by sequentially performing the following for each of the plurality of predetermined views: displaying an overlay including a tooth contour in one of the predetermined views from the plurality of predetermined views on a screen, wherein the overlay is displayed on top of an image of a patient's teeth;

transmitting the series of images from the mobile communication device to a remote location to determine whether the patient is a candidate for orthodontic treatment based on the series of images; and

displaying on a screen of the mobile communication device an indication of whether the patient is an orthodontic treatment candidate.

2. The medium of claim 1, wherein guiding the user comprises: for each predetermined view, an image of the patient's teeth is captured when the image of the patient's teeth substantially matches the overlay corresponding to the predetermined view.

3. The medium of claim 2, further comprising instructions to automatically determine when the image of the patient's tooth substantially matches the overlay by detecting an edge of the patient's tooth and comparing the detected edge to the overlay.

4. The medium of claim 2, further comprising instructions to automatically adjust the camera to focus the camera in a screen area within the overlay.

5. The medium of claim 2, further comprising instructions to select an overlay based on one or more images of the patient's teeth.

6. The medium of claim 2, further comprising instructions to automatically adjust the light emitted by the camera based on a brightness of light within an area of the screen within the overlay.

7. The medium of claim 2, wherein capturing the image of the patient's teeth when the image of the patient's teeth substantially matches the overlay corresponding to the predetermined view comprises automatically capturing the image.

8. The medium of claim 2, wherein capturing the image of the patient's teeth when the image of the patient's teeth substantially matches the overlay corresponding to the predetermined view comprises triggering a visual, audible, or visual and audible indication.

9. The medium of claim 1, wherein guiding a user comprises guiding a user to take the series of images, including guiding the user to take at least one of: front view, buccal view, maxillary view, and mandibular view.

10. The medium of claim 1, wherein transmitting the series of images from the mobile communication device to a remote location further comprises: receiving, in the mobile communication device, an indication of whether the patient is a candidate within 15 minutes of transmitting the series of images.

11. The medium of claim 1, wherein transmitting the series of images from the mobile communication device to a remote location further comprises: using the series of images, a manual determination is made as to whether the patient is a candidate for orthodontic treatment.

12. The medium of claim 1, wherein transmitting the series of images from the mobile communication device to a remote location further comprises: automatically determining whether the patient is a candidate for orthodontic treatment using the series of images.

13. The medium of claim 1, further comprising instructions to automatically identify each image in the series of images to indicate which view of a plurality of views each image comprises.

14. The medium of claim 1, further comprising instructions to automatically determine whether one or more of the series of images was taken using a mirror.

15. The medium of claim 1, further comprising instructions to receive, in the mobile communication device, an indication of the patient's primary dental problem and to aggregate the indication of the patient's primary dental problem with the series of images, wherein transmitting the series of images comprises transmitting the aggregated series of images and the patient's primary dental problem.

16. The medium of claim 1, further comprising instructions to instruct a user to retract a cheek of a patient with the cheek retractor.

17. The medium of claim 16, further comprising instructions to detect an indicium on the buccal retractor and identify the image based on the identified buccal retractor to indicate which of the plurality of views the image comprises.

18. The medium of claim 1, wherein the user is a patient.

19. A non-transitory computing device-readable medium having instructions stored thereon that are executable by a processor to cause the processor to:

using a mobile communication device having a camera to guide a user to take a series of images of a patient's teeth in a plurality of predetermined views by sequentially performing the following for each of the plurality of predetermined views:

displaying an overlay including a tooth contour in one of the predetermined views from the plurality of predetermined views on a screen, wherein the overlay is displayed on top of an image of a patient's teeth; and

capturing an image of the patient's teeth when the overlay substantially matches the patient's teeth in the view of the patient's teeth;

transmitting the series of images to a remote location to determine whether the patient is a candidate for orthodontic treatment based on the series of images; and

an indication of whether the patient is an orthodontic treatment candidate is displayed on a screen of the mobile communication device.

20. The medium of claim 19, wherein capturing comprises automatic capturing.

21. The medium of claim 19, wherein capturing comprises triggering a visual, audible, or visual and audible indication.

22. The medium of claim 19, further comprising instructions to automatically determine when the image of the patient's tooth substantially matches the overlay by detecting an edge of the patient's tooth and comparing the detected edge to the overlay.

23. The medium of claim 19, further comprising instructions to automatically determine when the patient dental image substantially matches the overlay by estimating an indication of a distance between an edge of the patient tooth and an edge of the overlay in the patient dental image.

24. The medium of claim 19, further comprising instructions to automatically adjust the camera to focus the camera in a screen area within an overlay.

25. The medium of claim 19, further comprising instructions to select an overlay based on one or more images of the patient's teeth.

26. The medium of claim 25, further comprising instructions to select a patient-specific overlay.

27. The medium of claim 19, further comprising instructions to automatically adjust the light emitted by the camera based on a light level in a screen area within the overlay.

28. The medium of claim 19, wherein guiding a user comprises guiding a user to take the series of images, including guiding the user to take at least one of: front view, buccal view, maxillary view, and mandibular view.

29. The medium of claim 19, wherein transmitting the series of images from the mobile communication device to a remote location further comprises: receiving, in the mobile communication device, an indication of whether the patient is a candidate within 15 minutes of transmitting the series of images.

30. The medium of claim 19, wherein transmitting the series of images from the mobile communication device to a remote location further comprises: using the series of images, a manual determination is made as to whether the patient is a candidate for orthodontic treatment.

31. The medium of claim 19, wherein transmitting the series of images from the mobile communication device to a remote location further comprises: automatically determining whether the patient is a candidate for orthodontic treatment using the series of images.

32. The medium of claim 19, further comprising instructions to automatically identify each image in the series of images to indicate which view of the plurality of views each image includes.

33. The medium of claim 19, further comprising instructions to automatically determine whether one or more of the series of images was taken using a mirror.

34. The medium of claim 19, further comprising instructions to receive, in the mobile communication device, an indication of a primary dental problem of the patient and to aggregate the primary dental problem with the series of images, wherein transmitting the series of images comprises transmitting the aggregated series of images and the primary dental problem of the patient.

35. The medium of claim 19, further comprising instructions to instruct a user to retract a cheek of a patient with the cheek retractor.

36. The medium of claim 35, further comprising instructions to detect an indicium on the buccal retractor and identify the image based on the identified buccal retractor to indicate which of the plurality of views the image comprises.

37. The medium of claim 19, wherein the user is a patient.

38. A non-transitory computing device readable medium having stored thereon instructions executable by a processor, the instructions when executed by the processor of a mobile communication device having a camera cause the processor to:

guiding a user to take a series of images of a patient's teeth with a camera in a plurality of predetermined views;

transmitting the series of images from the mobile communication device to a remote location to determine whether the patient is a candidate for orthodontic treatment based on the series of images;

displaying on a screen of the mobile communication device an image from the camera and an overlay comprising a tooth contour in one of the predetermined views from the plurality of predetermined views, wherein the overlay is displayed on top of the image from the camera; and

displaying on a screen of the mobile communication device an indication of whether the patient is an orthodontic treatment candidate.

39. The medium of claim 38, wherein the instructions cause the processor to guide the user to take a series of images of the patient's teeth by automatically adjusting the camera to focus the camera in a screen area within the overlay.

40. The medium of claim 38, wherein the instructions cause the processor to guide the user to take a series of images of the patient's teeth by selecting an overlay based on one or more images of the patient's teeth.

41. The medium of claim 38, wherein the instructions cause the processor to guide the user to take a series of images of the patient's teeth by automatically adjusting the light emitted by the camera based on the light level in the screen area within the overlay.

42. The medium of claim 38, wherein the instructions cause the processor to direct the user to take a series of images of the patient's teeth by: indicating when an overlay comprising a tooth gear profile in one of the predetermined views from the plurality of predetermined views is aligned with the view of the patient's teeth from the camera, wherein the overlay is displayed on top of the view from the camera.

43. The medium of claim 38, wherein the instructions cause the processor to direct the user to take a series of images of the patient's teeth by: automatically capturing an image of a patient's teeth when an overlay including a tooth gear profile in one of the predetermined views from the plurality of predetermined views is aligned with the view of the patient's teeth from the camera, wherein the overlay is displayed on an upper level of the view from the camera.

44. The medium of claim 38, wherein the instructions cause the processor to direct a user to take a series of images of a patient's teeth comprises directing the user to take at least one of: front view, buccal view, maxillary view, and mandibular view.

45. The medium of claim 38, wherein the instructions cause the processor to receive an indication in the mobile communication device of whether a patient is a candidate within 15 minutes of transmitting the series of images.

46. The medium of claim 38, wherein the instructions cause the processor to automatically identify each image in the series of images to indicate which view of the plurality of views each image includes.

47. The medium of claim 38, wherein the instructions cause the processor to automatically determine whether one or more of the series of images was taken using a mirror.

48. The medium of claim 38, wherein the instructions cause the processor to receive an indication of a major dental problem of the patient in the mobile communication device and to aggregate the major dental problem of the patient with the series of images, wherein the medium is configured to transmit the series of images as the aggregated series of images and the major dental problem of the patient.

49. The medium of claim 38, wherein the instructions cause the processor to instruct a user to retract a cheek of a patient with a cheek retractor.

50. The medium of claim 49, wherein the instructions further cause the processor to identify an indicia on the cheek retractor and to mark an image from the plurality of predetermined views based on the identified cheek retractor to indicate which of a plurality of views the image comprises.

51. A non-transitory computer readable storage medium storing a set of instructions executable by a processor of a mobile communication device having a camera, the set of instructions, when executed by the processor, causing the processor to:

the user is guided to take a series of images of the patient's teeth in accordance with a plurality of predetermined views by sequentially performing, for each of the plurality of predetermined views:

displaying on a screen of the mobile communication device an image from the camera and an overlay comprising a tooth contour in one of the predetermined views from the plurality of predetermined views, wherein the overlay is displayed on top of the image from the camera; and

capturing an image when the overlay substantially matches the patient's teeth on the screen; and transmitting the series of images to a remote location.

52. The medium of claim 51, wherein the instructions cause the processor to guide a user to take a series of images of a patient's teeth by automatically capturing the images when the overlay substantially matches the patient's teeth on the screen.

53. The medium of claim 51, wherein the instructions cause the processor to guide the user to take a series of images of the patient's teeth by triggering an indication that the overlay substantially matches the patient's teeth on the screen, wherein the indication is one or more of auditory and visual.

54. The medium of claim 51, wherein the instructions cause the processor to guide a user to take a series of images of a patient's teeth by automatically adjusting the camera to focus the camera in a screen area within an overlay.

55. The medium of claim 51, wherein the instructions cause the processor to guide the user to take a series of images of the patient's teeth by selecting an overlay based on one or more images of the patient's teeth.

56. The medium of claim 51, wherein the instructions cause the processor to guide the user to take a series of images of the patient's teeth by automatically adjusting the light emitted by the camera based on the light level in the screen area within the overlay.

57. The medium of claim 51, wherein the instructions cause the processor to direct the user to take a series of images of the patient's teeth comprising: directing the user to photograph at least one of: front view, buccal view, maxillary view, and mandibular view.

58. The medium of claim 51, wherein the instructions cause the processor to automatically identify each image in the series of images to indicate which view of the plurality of predetermined views each image includes.

59. The medium of claim 51, wherein the instructions cause the processor to automatically determine whether one or more of the series of images was taken using a mirror.

60. The medium of claim 51, wherein the instructions cause the processor to receive an indication of a major dental problem of the patient in the mobile communication device and to aggregate the major dental problem of the patient with the series of images, wherein the instructions further cause the processor to transmit the series of images as the aggregated series of images and the major dental problem of the patient.

61. The medium of claim 51, wherein the instructions cause the processor to instruct a user to retract a cheek of a patient with a cheek retractor.

62. The medium of claim 51, wherein the instructions cause the processor to identify an indicium on the cheek retractor and to mark an image from the plurality of predetermined views based on the identified cheek retractor to indicate which of a plurality of predetermined views the image comprises.

63. The medium of claim 62, wherein the instructions cause the processor to receive an indication from a remote processor regarding whether the patient is a candidate for orthodontic treatment based on the series of images returned to the non-transitory computer readable storage medium.

64. The medium of claim 51, wherein the instructions cause the processor to receive the transmitted series of images and transmit an indication back to the non-transitory computer-readable storage medium of whether the patient is a candidate for orthodontic treatment based on the series of images.

65. A non-transitory computer readable storage medium storing a set of instructions executable by a processor of a mobile communication device having a camera, the set of instructions, when executed by the processor, causing the processor to:

the user is guided to take a series of images of the patient's teeth in a plurality of predetermined views by sequentially performing the following for each of the plurality of predetermined views:

displaying on a screen of a mobile communication device an image from the camera and an overlay comprising a tooth contour in one of predetermined views from a plurality of predetermined views, wherein the overlay is displayed on top of the image from the camera; and

capturing an image when the overlay substantially matches the patient's teeth on the screen; and transmitting the series of images to a remote processor; and

receiving, from the remote processor, an indication of whether the patient is a candidate for orthodontic treatment based on the series of images returned to the non-transitory computer-readable storage medium.

Technical Field

The present disclosure relates generally to methods and apparatus for analyzing dental images, including methods and apparatus for capturing dental images and methods and apparatus for remotely prescreening patients for orthodontic treatment.

Background

In dental and/or orthodontic treatment, a set of 2D facial and dental photographs are often taken. Conventional dental photography uses cameras, for example, digital single-lens reflex (SLR) cameras having a circular flash and a lens having a focal length of 90-100mm as shown in FIGS. 1A-1C. However, SLR cameras are expensive and it may be difficult to align the patient's teeth with the SLR camera. This procedure may also be uncomfortable for the patient. Instead, some doctors and dental technicians may attempt to capture photographs of teeth using a handpiece. However, mobile phones typically have wide-angle camera lenses. If the camera of the mobile phone is held close enough to the patient's teeth to provide an image of the teeth with a high level of detail, the dental photograph may be blurred and have optical distortions. If the camera of the handpiece is too far from the teeth, the dental photographs do not meet orthodontic standards. In general, commonly available cameras, especially mobile phone cameras, may be faster and easier to use.

Therefore, there is a need for new and useful methods and apparatus for obtaining high quality dental images.

Disclosure of Invention

Described herein are methods, apparatus (including devices and systems, such as non-transitory computer-readable storage media and systems including these) for capturing dental images and using these dental images to determine whether a patient is a candidate for orthodontic surgery. In general, the methods and apparatus described herein may obtain images of a patient's teeth for therapeutic use, which may include viewing the patient's teeth, for example, on a screen of a mobile telecommunications device (e.g., a mobile phone or other handheld personal computing device, such as a smart watch, tablet, laptop, etc.).

Any of the methods described herein may include guiding or facilitating taking a predetermined set of images of a patient's teeth from a specified perspective. The designated viewing angle (view) may be used to manually or automatically determine whether the patient is a candidate for a particular orthodontic treatment. In general, it may be necessary or helpful to take the views at an appropriate resolution and focus, especially when the images are automatically analyzed at a later time. Thus, in any of the methods and apparatus described herein, the overlay may be shown on the camera image in preparation for taking a picture of the tooth. The overlay may guide a user (e.g., a dental technician, including a dentist, an orthodontist, a dental assistant, a nurse, etc.) to take the image. In addition, the overlay can be used to help focus and illuminate the tooth during collection of the image. For example, the methods described herein may include displaying an overlay including tooth contours in a predetermined view on a screen (e.g., a screen of a mobile communication device such as a smartphone). The overlay may be displayed on top of the view from the camera, which may display a real-time display from the camera. When the camera is used to image the patient's teeth, the mobile communication device may be moved so that the overlay substantially matches the patient's teeth in the view of the patient's teeth. The method may further include capturing an image of a view of the patient's teeth.

For example, methods of obtaining images of a patient's teeth for therapeutic use are described herein. The method may include viewing a patient's teeth, for example, on a screen of a mobile telecommunications device (e.g., smartphone, mobile phone, etc.) having a camera. The method may further include displaying an overlay on the screen that includes the tooth contour in the predetermined view, wherein the overlay is displayed on an upper layer of the view of the patient's teeth. The method may include moving the mobile telecommunications device relative to the patient's teeth and triggering an indication when the overlay substantially matches the patient's teeth. The method may include capturing an image of a view of a patient's teeth upon a trigger indication.

A method of obtaining an image of a patient's teeth for therapeutic use may include viewing the patient's teeth on a screen of a mobile telecommunications device. The method may further include displaying an overlay on the screen including a crop box and tooth contours in one of a front view, a buccal view, a maxillary view, or a mandibular view, wherein the overlay is displayed on top of the view of the patient's teeth. The method may further comprise moving the mobile telecommunications device such that the overlay substantially matches the patient's teeth in the view of the patient's teeth. The method may include capturing an image of a view of a patient's teeth. The method may further comprise viewing the captured image on the mobile telecommunications device and indicating on a screen of the mobile telecommunications device whether the captured image is out of focus. The method may further include automatically cropping the captured image as indicated by the crop box.

For example, in some embodiments, the overlay may comprise a generic overlay. For another example, in some other embodiments, the overlay may comprise a patient-specific overlay derived from the patient's teeth.

For example, the method may further include automatically triggering an indication when the overlay substantially matches the patient's teeth. For example, the method may further include triggering an indication when the overlay substantially matches the patient's teeth includes estimating an indication of a distance between an edge of the patient's teeth in the view of the patient's teeth and a tooth contour in the overlay. For another example, the method may further include triggering an indication when the overlay substantially matches the patient's teeth includes estimating an indication of a distance between an edge of the patient's teeth at the two or more regions and a tooth contour and comparing the indication to a threshold. For example, the indication may be a visual indication, such as a change in color. In some variations, the indication may be other forms of indication, such as a voice indication.

The method may further include automatically capturing an image of the patient's teeth when the overlay substantially matches the patient's teeth.

The method may further comprise checking the image quality of the captured image and displaying on the screen if the image quality is below an image quality threshold.

The method may also include cropping the captured image based on a cropping contour displayed as part of the overlay. The cutting may be manual or automatic.

Any of these methods may further include evaluating the captured image for the medical treatment by using the image. For example, the method may further include transmitting the captured image to a remote server.

The predetermined view may include a front view, a buccal view, an maxillary view, or a mandibular view. The predetermined view may include a set of tooth images according to orthodontic standards. For example, the method may further include repeating the steps of observing, displaying, moving, and capturing to capture images of the front cheek, upper jaw, and lower jaw of the patient's teeth.

The method may further include imaging the identity of the patient using the mobile telecommunications device and automatically populating a form with user identity information based on the imaged identity.

Any of these methods may further include displaying instructions regarding positioning the patient's teeth on a screen of the mobile telecommunications device prior to displaying the overlay.

Also described herein are means adapted for carrying out any of the methods described herein, including particularly software, firmware and/or hardware adapted for carrying out one or more of these methods. In particular, described herein is a non-transitory computer readable storage medium storing a set of instructions executable by a processor (e.g., a processor of a mobile telecommunications device) that, when executed by the processor, causes the processor to display a real-time image of a patient's teeth on a screen of the mobile telecommunications device, display an overlay including tooth contours in a predetermined view of an upper layer of the image of the patient's teeth, and enable capture of the image of the patient's teeth.

For example, described herein is a non-transitory computer readable storage medium storing a set of instructions executable by a processor of a mobile telecommunications device, which when executed by the processor causes the processor to display a real-time image of a patient's teeth on a screen of the mobile telecommunications device, display an overlay including a tooth gear profile in a predetermined view of an upper layer of the image of the patient's teeth, trigger an indication when the overlay substantially matches the patient's teeth, and when the indication is triggered, enable capture of the image of the patient's teeth.

Also described herein is a non-transitory computer readable storage medium storing a set of instructions executable by a processor of a mobile telecommunication device, which when executed by the processor, causes the processor to display a real-time image of a patient's teeth on a screen of the mobile telecommunication device and an overlay including a crop box and tooth contour lines in one of a front view, a buccal view, a maxilla view or a mandibular view, wherein the overlay is displayed on top of the image of the patient's teeth and is capable of capturing the image of the patient's teeth. A non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to view the captured image and indicate on the screen whether the captured image is out of focus and automatically crop the captured image as indicated by the crop box.

The set of instructions, when executed by the processor, may further cause the processor to display a generic overlay. For another example, a set of instructions may cause the processor to display a patient-specific overlay originating from a patient's teeth.

The set of instructions, when executed by the processor, may further cause the processor to automatically trigger an indication when the overlay substantially matches the patient's teeth. For example, the set of instructions may further cause the processor to estimate an indication of a distance between edges of the patient's teeth in the view of the patient's teeth, and trigger the indication when a tooth contour in the overlay is less than or equal to a threshold. The set of instructions, when executed by the processor, may further cause the processor to estimate an indication of a distance between edges of the patient's teeth at two or more regions in the view of the patient's teeth, and trigger the indication when a tooth contour in the overlay is less than or equal to a threshold. The set of instructions may cause the processor to trigger the indication by displaying a visual indication on the screen. Any suitable visual indication may be displayed, including a color, an intensity (e.g., a color and/or intensity that changes the outline of the tooth overlay, a cropping window, etc.), a text/character indication, or some combination thereof. Alternatively or additionally, the indication may be audible (beeping, tone, etc.) and/or tactile (vibrating, buzzing, etc.).

The set of instructions, when executed by the processor, may further cause the processor to check an image quality of the captured image and display on the screen if the image quality is below an image quality threshold. Quality may automatically determine the focus, illumination (dark/light), etc. of the image, and may alert the user and/or automatically reject or accept the image. The device may further process the image (e.g., sharpen, brighten/darken, etc., including cropping). For example, a non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to automatically crop the captured image based on a crop contour displayed as part of the overlay.

The set of instructions, when executed by the processor, may further cause the processor to transmit the captured image to a remote server. The transmission may be automatic or manual.

The apparatus (e.g., comprising a non-transitory set of instructions) may further cause the processor to display an overlay comprising tooth contours in a predetermined view (e.g., a front view, a buccal view, a maxillary view, or a mandibular view). The apparatus may be configured to take all or part of a set of views. For example, the set of instructions, when executed by the processor, may further cause the processor to repeat the steps of observing, displaying, moving, and capturing to capture anterior cheek, upper jaw, and lower jaw images of the patient's teeth.

In addition to capturing one or more views (e.g., anterior, buccal, maxillary, and mandibular), the devices described herein may be configured to automatically determine patient-specific information about identity and other patient characteristics and associate the information with the captured images. For example, the set of instructions may cause the processor to capture an image of a patient identity (e.g., a driver's license) using the mobile telecommunications device and automatically populate a form with user identity information based on the imaged identity.

In any of these apparatuses, the set of instructions may further cause the processor to display instructions regarding positioning the patient's teeth on a screen of the mobile telecommunications device prior to displaying the overlay.

Any of the methods described herein may be a method of obtaining a series of images of a patient's teeth. These methods may include: displaying a real-time image from a camera on a screen of a mobile telecommunication device having the camera; and sequentially guiding the user to acquire a series of predetermined views of the patient's teeth, for each of the predetermined views: displaying an overlay on the screen superimposed on the live image, the overlay including the tooth gear profile in a predetermined view from the plurality of predetermined views; and triggering capture of an image of the patient's teeth when the overlay substantially matches the image of the patient's teeth in the on-screen display.

For example, a method of obtaining a series of images of a patient's teeth may include: displaying the patient's teeth on a screen of a mobile telecommunication device having a camera; and, the user is guided to sequentially take a series of predetermined views of the patient's teeth, for each predetermined view: displaying an overlay on the screen including tooth contours in a predetermined view, wherein the overlay is displayed on top of a view of the patient's teeth; automatically adjusting a camera to focus in a screen region within an overlay; automatically adjusting light emitted by the camera according to light levels in a screen area within the overlay; triggering capture of an image of the patient's teeth when the overlay substantially matches the patient's teeth in the display of the patient's teeth, wherein the predetermined view includes at least one of each of: front view, buccal view, maxillary view, and mandibular view.

The series of images of the patient's teeth may comprise a set, collection or group. The series may be organized or ordered in a predefined order, for example, based on a predefined view (e.g., perspective). The series of images may be collected or linked together and may include identifying information, including information identifying the respective perspective (e.g., frontal view, buccal view, maxillary view, mandibular view, etc.). In any of these variations, additional information may be included, such as major dental problems of the user and/or patient (e.g., crowding, spacing, smile width, arch width, smile line, horizontal overlay, vertical overbite, cross-bite, bite relationship, etc.). In general, the set of images may refer to a series of predefined views. The predetermined view may refer to a predetermined viewing angle for visualizing the teeth. The viewing angle may refer to a view of the upper and/or lower arch and may include, for example: anterior (e.g., upper anterior and lower anterior, typically with a closed bite), anterior open bite, right buccal (typically with a closed bite), right buccal open bite, left buccal (typically with a closed bite), left buccal open bite, upper jaw (e.g., viewed from the occlusal plane), and lower jaw (e.g., viewed from the occlusal plane). The predetermined view may also include views that include the entire head, such as a side profile of the head, a face view (typically with the mouth closed, not smiling), and a face view with the patient smiling. Dental scopes may be used to capture images of the upper and lower jaws. The systems and methods described herein may automatically determine whether a mirror is used, and may orient the image accordingly.

Any of the methods and apparatus described herein may guide a user through serialization. The method or apparatus may provide audible and/or visual instructions to the user. In particular, as mentioned above, any of these means may comprise an overlay on the display (screen) of the mobile telecommunications device showing contours that may be matched to guide the user in taking the image. The overlay may be displayed as a solid and/or semi-transparent color outline. Overlays may be shown for each predetermined view. The user may view the screen and once the images show that the patient's anatomy substantially matches within the overlay, the images may be captured. Image capture may be manual (e.g., manually triggered to be captured by a user-activated control, such as pressing a button to capture an image) and/or automatic (e.g., detected by the system and automatically triggered to capture an image when an overlay matches the corresponding patient anatomy). In general, capturing or triggering the capture of an image of a patient's teeth (and/or the patient's head) when the overlay substantially matches the image of the patient's teeth in the display may refer to automatic capture/automatic trigger, semi-automatic capture/semi-automatic trigger, or manual capture/manual trigger. Automatic triggering (e.g., automatic capture) may refer to automatic capture of an image, e.g., taking one or more images when the patient's anatomy (e.g., teeth) displayed on the screen matches the overlay on the screen. Semi-automatic triggering (e.g., semi-automatic capturing) may refer to generating a signal, such as an audible sound and/or a visual indication (e.g., flashing light, color change, etc.), when the patient's anatomy (e.g., teeth) displayed on the screen matches the overlay on the screen. Manual triggering (e.g., manual capture) may refer to a user manually taking an image, e.g., taking one or more images when the anatomy (e.g., teeth) of the patient is displayed on a screen matching an overlay.

As described in greater detail herein, automatic or semi-automatic triggering (e.g., automatic or semi-automatic capture of an image) may be accomplished by various well-known image processing techniques. For example, detection of a match between a patient's anatomy (e.g., teeth) and an overlay on the screen may be achieved by edge detection; the edge of the patient's tooth may be compared to the coverage area and if two or more regions (e.g., two opposing regions, etc.) are within a defined distance (e.g., +/-1mm, +/-2mm, +/-3mm, +/-4mm, +/-5mm, +/-6mm, +/-7mm, +/-8mm, +/-10mm, etc., or +/-a corresponding number of image pixels, +/-a certain percentage, such as 1%, 2%, 3%, 5%, 7%, 10% of the screen diameter, etc.). Automatic detection of a match may be determined by machine learning, e.g., training a machine to identify matches of patient anatomy (e.g., teeth) within an overlay to an acceptable match percentage.

Any of these methods may comprise displaying images, in particular real-time images, from a camera of the mobile telecommunications device on a screen of the mobile telecommunications device. Real-time may refer to the current or near simultaneous display of images detected by a camera on, for example, one or more screens of a mobile telecommunications device.

In general, overlays can also be used to improve image quality. In the captured image. For example, any of these methods and apparatus may autofocus imaging only within the area defined by the overlay. For example, any of these methods and apparatus may disable or modify auto-focus of a camera of a mobile telecommunications device (e.g., a mobile phone) and may auto-focus only on an area within the overlay or a sub-area within the overlay (e.g., on anterior teeth, incisors, canines, bicuspids, molars, etc.).

The overlay can also control the illumination (e.g., light) of the image based on the area within all or a portion of the overlay. For example, the device or method may detect and adjust light levels based on light levels within the overlay or sub-regions within the overlay (e.g., on incisors, canines, bicuspids, molars, etc.). The illumination may typically be provided by a mobile telecommunications device which may include a flash or LED light source which may be adjusted for continuous and/or discrete illumination.

In any of the methods and apparatus described herein, images taken for a particular view (e.g., anterior open bite, right buccal open bite, left buccal open bite, upper jaw, lower jaw, etc.) may be labeled with the corresponding view, either manually or automatically. Furthermore, views may be detected and identified by the method or apparatus. In a variation where an overlay of a particular view is provided prior to capturing an image, the view shown in the overlay may determine the label of the resulting image. As described herein, in some variations, automatic detection of the nearest view may be performed on the imaging, and the view (perspective) may be automatically detected. Additionally or alternatively, the mirror image may be detected or identified and the resulting image flipped/rotated and/or marked to indicate that the dental mirror was used to take the image.

In any of the methods and apparatus described herein, the overlay displayed on the image on the screen of the mobile telecommunications device may be automatically selected, for example, by identifying a closest match to one of the predetermined perspectives. The overlay with the closest match may then be used to capture the image of the set of images. Alternatively or additionally, the overlay may be provided first, and then the user may move the camera portion of the mobile telecommunications device to fit the patient's anatomy to the overlay.

As described above, any image of the predetermined viewing angle (view) described herein may be taken using the cheek holder. The means by which the user instructs to take the image may comprise written, pictorial (visual) and/or audible instructions regarding the use of the cheek holder. Any of the methods and devices described herein can automatically detect a cheek holder; this may help to automatically mark and interpret the resulting image. In some variations, the apparatus and/or method may detect one or more markings on the cheek holder and use this information to identify the view, identify a match between the image and the overlay, and the like.

Methods and devices for remotely prescreening patients for orthodontic treatment are also described herein. Any orthodontic treatment, particularly for aligning a patient's teeth, may be attempted. In general, such a method may include acquiring a series of predetermined views of a patient's teeth, and optionally collecting information about the patient, such as one or more major dental problems (e.g., major patient dental problems, such as crowding, spacing, smile width, arch width, smile line, horizontal overlay, vertical overlay, cross-bite, bite relationship, etc.). This additional information may be linked with the series of images and may be used with the series of images to determine whether the patient is a good candidate for orthodontic treatment.

For example, described herein is a method for remotely prescreening patients for orthodontic treatment, the method comprising: guiding a user with a mobile telecommunications device having a camera to take a series of images of a patient's teeth in a plurality of predetermined views; transmitting a series of images from the mobile telecommunications device to a remote location to determine whether the patient is a candidate for orthodontic treatment based on the series of images; displaying on a screen of the mobile telecommunications device an indication of whether the patient is an orthodontic treatment candidate.

In any of these methods, the directing may refer to displaying an overlay including a tooth contour in one of the predetermined views from the plurality of predetermined views on the screen sequentially for each of the predetermined views, wherein the overlay is displayed on an upper layer of the image of the patient's teeth. The overlay may be provided first or may be selected from a set of overlay perspectives that best match the current view imaged by the mobile telecommunications device. Alternatively, the predetermined views may be presented in a fixed order.

As described above, guiding the user may include, for each predetermined view, capturing an image of the patient's teeth when the image of the patient's teeth substantially matches the overlay corresponding to the predetermined view. As described above, the capturing may be manual, automatic, or semi-automatic. For example, any of these methods may include automatically determining when the image of the patient's teeth substantially matches the overlay by detecting edges of the patient's teeth and comparing the detected edges to the overlay.

Any of these methods and apparatus may include automatically adjusting the camera to focus the camera in a screen area within the overlay. The region may include all of the regions within the overlay, or a subset (e.g., corresponding to anterior teeth, posterior teeth, etc.).

Any of these methods and apparatus may include selecting an overlay based on one or more images of a patient's teeth. This may include selecting an overlay corresponding to a particular viewing angle, as described above, and/or it may include customizing the overlay based on the particular anatomy of the patient. For example, the overlay may be selected to match the shape, size, and arrangement of the patient's dentition.

Any of these methods and apparatus may include automatically adjusting light emitted by the camera based on a light level in a screen area within the overlay. The light may be continuous or intermittent (e.g., flashing). Thus, the apparatus or method may first disable the default light sensing of the mobile telecommunications device and may instead use the area (or sub-area of the area) within the overlay to set a light level for adjusting the flash/application light from the mobile telecommunications device.

As described above, any of these methods and apparatus may be configured to capture an image of a patient's teeth when the image of the patient's teeth substantially matches an overlay corresponding to the predetermined view, such as by automatically capturing the image. Similarly, any of these methods and apparatus may capture an image of a patient's teeth when the image substantially matches an overlay corresponding to a predetermined view by semi-automatic capture (e.g., triggering a visual, auditory, or visual and auditory indication that allows a user to take the image). In some variations, multiple images may be taken and averaged or used to select the best image.

Transmitting the series of images from the mobile telecommunications device to the remote location may generally include receiving an indication that the patient is a candidate in the mobile telecommunications device within a fixed time period (e.g., 10 minutes, 15 minutes, 20 minutes, etc.) from transmitting the series of images. Typically, an initial decision that a patient is a good candidate for orthodontic treatment may use the transmitted image set, and may also include major issues. The decision may be made at a remote location (e.g., a remote server, etc.) either manually or automatically. The automatic decision may be based on the amount of movement required to position the teeth in order to address the primary problem and/or the criteria for orthodontic positioning. The methods and devices described herein can provide images with sufficient definition so that individual tooth positions can be determined relative to the dental arch and used to at least roughly approximate the complexity of orthodontic surgery. Situations where the amount of movement and/or type is complex may indicate not a candidate. A case where the amount of movement and/or type is not complex may be indicated as a candidate. Complex tooth movements may include movements greater than a minimum threshold (e.g., greater than 3mm distal/proximal movement, greater than 4mm distal/proximal movement, 5mm distal/proximal movement, greater than 6mm distal/proximal movement, greater than 7mm distal/proximal movement, etc.), and/or rotations greater than a minimum threshold (e.g., greater than 5, 10, 15, 20, 25, 30, 35, 45, etc. degrees), and/or squeezing out one or more teeth greater than a minimum threshold (e.g., greater than 0.5mm, 1mm, 2mm, 3mm, 4mm, etc.).

As described above, generally, any of these methods and apparatus may include automatically identifying each image in a series of images to indicate which view of a plurality of views each image includes (e.g., anterior open bite, right cheek open bite, left cheek open bite, upper jaw, lower jaw, etc.). Any of these methods may further include automatically determining whether one or more of the series of images was taken using a mirror. For example, the image may be automatically inspected to identify reflections (e.g., planes or mirrors), and/or to determine whether the orientation of the teeth within the image is reversed (mirrored) in the image. The mirror image may be inverted to appear as a non-mirror image. Alternatively or additionally, the repeated mirror regions may be cropped from the image.

Any of the methods and apparatus described herein may include receiving, in a mobile telecommunications device, an indication of a patient's primary dental problem and aggregating the patient's primary dental problem with the series of images. Transmitting the series of images may include transmitting the aggregated series of images and a major dental problem of the patient.

As described above, any of the methods and devices described herein can be configured to include instructing a user to retract a cheek of a patient with a cheek retractor. The markings on the buccal retractor can be used to automatically identify the image to indicate which of a plurality of views it includes based on the identified buccal retractor.

Although the terms "user" and "patient" are used herein individually, the user may be a patient. For example, the person taking the image using the methods and apparatus described herein may be a patient. Thus, in any of these methods, the user may be a patient. Alternatively, a separate use (e.g., dentist, orthodontist, dental technician, dental assistant, etc.) may act as a user, taking an image of a patient as described herein.

A method for remotely prescreening patients for orthodontic treatment may include: guiding a user with a mobile telecommunication device having a camera to take a series of images of a patient's teeth sequentially in a plurality of predetermined views, for each predetermined view: displaying an overlay on the screen that includes a tooth contour in one of the predetermined views from the plurality of predetermined views, wherein the overlay is displayed on top of the image of the patient's teeth; capturing an image of the patient's teeth when the overlay substantially matches the patient's teeth in the view of the patient's teeth; transmitting the series of images to a remote location to determine whether the patient is a candidate for orthodontic treatment based on the series of images; displaying on a screen of the mobile telecommunications device an indication of whether the patient is an orthodontic treatment candidate.

A method for remotely prescreening patients for orthodontic treatment may include: guiding a user with a mobile telecommunication device having a camera to take a series of images of the teeth of a patient from a plurality of predetermined views by sequentially displaying an overlay comprising the tooth gear profile in each predetermined view on a screen of the mobile telecommunication device; receiving in a mobile telecommunications device an indication of a patient's primary dental problem; aggregating a series of images and major dental problems in a mobile telecommunications device; transmitting the aggregated series of images and the primary dental problem to a remote location to determine whether the patient is a candidate for orthodontic treatment based on the series of images; displaying on a screen of the mobile telecommunications device an indication of whether the patient is an orthodontic treatment candidate.

Any of the methods (and method steps) described herein may be performed by an apparatus configured to perform the method. For example, described herein are systems for remotely prescreening patients for orthodontic treatment. A system for remotely pre-screening a patient (e.g., remotely from the patient) may include: a non-transitory computer readable storage medium storing a set of instructions executable by a processor of a mobile telecommunication device having a camera, which when executed by the processor, causes the processor to: guiding a user to take a series of images of a patient's teeth in a plurality of predetermined views with a camera; transmitting a series of images from the mobile telecommunications device to a remote location to determine whether the patient is a candidate for orthodontic treatment based on the series of images; displaying on a screen of the mobile telecommunications device an indication of whether the patient is an orthodontic treatment candidate. The non-transitory computer readable storage medium may cause the processor to direct the user to take a series of images of the patient's teeth by: displaying on a screen of the mobile telecommunication device an image from the camera and a tooth profile comprising in one of the predetermined views from the plurality of predetermined views, wherein the overlay is displayed on top of the image from the camera. The non-transitory computer readable storage medium may cause the processor to guide the user to take a series of images of the patient's teeth by automatically adjusting the camera to focus the camera in a screen area within the overlay. The non-transitory computer readable storage medium may cause the processor to guide the user to take a series of images of the patient's teeth by selecting an overlay based on one or more images of the patient's teeth. The non-transitory computer readable storage medium may cause the processor to direct the user to take a series of images of the patient's teeth by automatically adjusting the light emitted by the camera based on the light level in the screen area within the overlay.

The non-transitory computer readable storage medium may cause the processor to direct the user to take a series of images of the patient's teeth by: indicating when an overlay comprising a tooth contour from one of the predetermined views from the plurality of predetermined views is aligned with the view of the patient's tooth from the camera, wherein the overlay is displayed on top of the view from the camera. The non-transitory computer readable storage medium may cause the processor to direct the user to take a series of images of the patient's teeth by: an image of the patient's teeth is automatically taken when an overlay of the tooth gear profile in one of the predefined views comprising the plurality of predetermined views is aligned with the view of the patient's teeth from the camera, wherein the overlay is displayed on top of the view from the camera.

The non-transitory computer readable storage medium may cause the processor to direct the user to take a series of images of the patient's teeth, including directing the user to take at least one of: front view, buccal view, maxillary view, and mandibular view.

Any of these systems may be configured such that the non-transitory computer readable storage medium further causes the processor to receive an indication in the mobile telecommunications device of whether the patient is a candidate within 15 minutes of transmitting the series of images. The non-transitory computer readable storage medium may cause the processor to automatically identify each image in the series of images to indicate which view of the plurality of views each image includes. The non-transitory computer readable storage medium may cause the processor to automatically determine whether one or more of a series of images was taken using the mirror.

The non-transitory computer readable storage medium may further cause the processor to receive, in the mobile telecommunication device, an indication of a patient's primary dental problem and aggregate the patient's primary dental problem with the series of images, wherein the non-transitory computer readable storage medium may be configured to transmit the series of images as an aggregated series of images and the patient's primary dental problem.

The non-transitory computer readable storage medium may further cause the processor to instruct the user to retract a cheek of the patient with the cheek retractor, and/or identify an indicium on the cheek retractor and to mark the image from a plurality of predetermined views based on the identified cheek retractor to indicate which of the plurality of views it includes.

Any of the systems described herein may include a remote processor configured to receive the transmitted series of images and transmit an indication of whether the patient is a candidate for orthodontic treatment back to the non-transitory computer-readable storage medium based on the transmitted series of images.

For example, a system for remotely prescreening patients for orthodontic treatment may include a non-transitory computer-readable storage medium storing a set of instructions executable by a processor of a mobile telecommunications device having a camera, the instructions when executed by the processor cause the processor to: the user is guided to take a series of images of the patient's teeth sequentially in a plurality of predetermined views, for each of which: displaying an image from a camera and an overlay on a screen of the mobile telecommunication device, the overlay comprising a tooth profile in one of the predetermined views from the plurality of predetermined views, wherein the overlay is displayed on top of the image from the camera; capturing an image when the overlay substantially matches the patient's teeth on the screen; and transmits a series of images to a remote location.

A system for remotely prescreening patients for orthodontic treatment may include: a non-transitory computer readable storage medium storing a set of instructions executable by a processor of a mobile telecommunication device having a camera, the instructions when executed by the processor causing the processor to: the user is guided to take a series of images of the patient's teeth sequentially in a plurality of predetermined views, for each of which: displaying on a screen of a mobile telecommunication device an image from the camera and an overlay comprising a tooth contour in one of the predetermined views from the plurality of predetermined views, wherein the overlay is displayed on top of the image from the camera; capturing an image when the overlay substantially matches the patient's teeth on the screen; and transmitting the series of images to a remote location; a remote processor configured to receive the transmitted series of images and transmit an indication of whether the patient is a candidate for orthodontic treatment back to the non-transitory computer-readable storage medium based on the series of images.

Drawings

The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings of which:

fig. 1A-1C show one example of conventional dental photography using a digital SLR camera. Fig. 1A shows an example of a conventional SLR camera that captures an anterior open bite view of a patient's teeth. Fig. 1B shows a front view (closed position) taken with a conventional SLR camera using a pair of buccal retractors held by the patient as shown in fig. 1C.

Fig. 2A shows an example of a screen of an apparatus (such as a system configured to perform a method as described herein) comprising an overlay on the screen of a mobile telecommunication device, the overlay comprising a tooth contour in a predetermined view, shown as a front view of a tooth.

Fig. 2B shows another example of an example of a screen of the device, similar to the screen shown in fig. 2B, showing an overlay and another front view of the teeth. For purposes of illustration, the model is shown as a dental model (e.g., a positive model) of the patient's dentition; the teeth may be directly imaged with or without the use of a retractor.

Fig. 2C shows an example of a screen of an apparatus configured to take an image (e.g., a photograph) of a patient's teeth in a left buccal open bite configuration, including an overlay on the screen showing which teeth should be visible in the view.

Fig. 3A illustrates an example of a patient-specific overlay in accordance with another embodiment of the present disclosure.

FIG. 3B illustrates an example of an indication that is triggered when the overlay substantially matches the patient's teeth. The triggers may be visual (including changing the color of the overlay, displaying or flashing images/icons/colors/symbols on the screen, etc.) and/or audible (sounding a ping, tone, etc.), and/or tactile (e.g., vibrating), etc.

Fig. 4A-4H show 8 specific overlay images of a photograph of the desired type according to orthodontic standards, including an anterior view in fig. 4A, another anterior view in fig. 4B, a maxillary view in fig. 4C, a mandibular view in fig. 4D, a left buccal view in fig. 4E, another left buccal view in fig. 4F, a right buccal view in fig. 4G, and another right buccal view in fig. 4H.

FIG. 5A is an example of a user interface that enables a user to select one of a plurality of overlays for a plurality of dental images in a plurality of predetermined views.

Fig. 5B is an example of an on-screen message for guiding a user to take a dental image.

Fig. 5C is an example of a user interface for taking a side image of a patient.

Fig. 5D is an example of a user interface that captures an image of a patient's face.

FIG. 6A is an example screenshot indicating that a captured image is out of focus after viewing image quality.

Fig. 6B is an example of a poor quality image of a tooth.

FIG. 7A illustrates an exemplary overlay including a crop box and tooth contours in a maxillary view.

FIG. 7B illustrates another exemplary overlay including a crop box and tooth contours in a front view.

FIG. 7C shows an example of an image captured using the overlay of FIG. 7B.

FIG. 8A is an example of a block diagram of a method of obtaining an image of a patient's teeth.

Fig. 8B is an example of a method of remotely prescreening patients for orthodontic treatment using the method of fig. 8A.

Fig. 9 is an example of a set of facial and dental images of a patient captured by using a digital SLR camera.

Fig. 10 is an example of a set of facial and dental images of a patient captured by the methods disclosed herein using a mobile telecommunications device.

11A-11H are examples of a set of dental images captured by the methods disclosed herein using a mobile telecommunications device.

Fig. 12A illustrates a method of focusing within the area of the overlay (illustrated by the dots on the upper incisors), rather than defaulting to autofocus for the camera of the mobile telecommunications device.

Fig. 12B and 12C illustrate a comparison between default autofocus using a mobile telecommunications device (fig. 12B) and a target focus using the systems and methods described herein (e.g., as shown in fig. 12A), assuming that the shaper is focused entirely on the teeth. In the line graph shown (adapted from the photograph), the focus is illustrated by the relative darkness of the lines. The focus can be represented by the darkness of the lines; in fig. 12B, the focus is on the lips and gums, while in fig. 12C, the focus is on the teeth, which will appear more focused than in fig. 12B.

Fig. 13A and 13B show a first example of a comparison between using a default focus and illumination (flash), as shown in fig. 13A, compared to using an area within an overlay (the overlay is not shown in fig. 13A and 13B) to set the focus and illumination levels. In fig. 13A and 13B, the line drawings are adapted according to photographs showing images taken without and with illumination adjusted based on the coverage area, respectively.

Fig. 13C and 13D show another example of a method of setting the focus and the illumination level using an area within the overlay (the overlay is not shown in fig. 13A and 13B).

Fig. 14A shows an example where a patient manually retracts their cheek. Fig. 14B shows the patient retracting their cheek without the assistance of elastic bands or retractors.

Fig. 14C is an example of a buccal retractor that can be used to help retract the cheek. In fig. 14C, the retractor includes one or more markings that can be used to identify the position of the retractor (and thus the patient's teeth) relative to the view.

Figure 15 is an example of a view of a patient using the retractor. In fig. 15, the patient is located at a distance from the camera, and the methods and apparatus described herein may indicate that the camera should be moved closer.

Fig. 16 shows a patient who may use facial recognition to provide information about an image.

Detailed Description

The following description of the various embodiments of the present invention is not intended to limit the invention to these embodiments, but rather to enable any person skilled in the art to make and use the invention.

Described herein are methods and apparatus (including devices, systems, non-transitory computer-readable storage media, etc. that store instructions executable by a processor) for capturing high quality dental images, including obtaining an image or a set of images at predetermined locations of a patient's teeth and/or face for therapeutic use. In particular, described herein are methods and apparatus for remotely prescreening patients for orthodontic treatment that include taking a defined set of images of the patient's teeth at known angles and transmitting them as a set for remote analysis. Although a built-in camera is employed, which is visible in most mobile telecommunication devices, the image must be at a predetermined angle and must be well focused and illuminated enough, as will be described herein. The methods described herein (and the apparatus for performing them) guide the user to take an image of a patient's teeth, for example, by displaying an overlay on the screen of the mobile telecommunications device that includes tooth contours in a predetermined view (or sequence of predetermined views), where the overlay is displayed on top of a view of the patient's teeth (or other body part, such as the face, head, etc.). The user may then move the mobile telecommunications device so that the overlay substantially matches the patient's teeth in the view of the patient's teeth, and capture an image (e.g., manually or automatically by the appliance) of the view of the patient's teeth (and in some cases, the face and head).

Instead of using an expensive and bulky SLR camera, a mobile telecommunication device can be used to capture dental images. Fig. 1A shows imaging of a patient's tooth 103 using a conventional SLR camera 101. The camera is typically held by a user (e.g., dentist, orthodontist, dental technician, etc.) and takes images of the patient's mouth from different locations. The user must manually adjust the patient position and/or the camera position. Fig. 1B shows a typical image of a front view 105 of a patient's teeth taken with an SLR camera. Additionally, one or more retractors 107 or other elements may be used to assist in taking a picture, as shown in FIG. 1C. Any of the methods and devices described herein may include the use of one or more such retractors or other auxiliary devices (including mirrors, probes, etc.), and the devices described herein may prompt the user to place and/or position such devices, including providing guidance for proper positioning.

It may be particularly useful to adapt a conventional handheld consumer electronic device (e.g., a phone (e.g., a smartphone, a smartwatch, a tablet, etc.)) to take one or more preferably a series of tooth images with sufficient clarity (e.g., focus, zoom, resolution, illumination, etc.) so that these images can be used to track the progress of the patient and/or pre-screen the patient for dental or orthodontic surgery. Thus, when taken from the proper orientation and sufficient definition (focus, zoom, resolution, illumination, etc.), such images (and series of images) may be used in one or more of the following: planning a dental/orthodontic operation, determining the feasibility of a dental/orthodontic operation for a particular patient, tracking the progress of a patient in a dental/orthodontic operation, determining the effectiveness of a dental/orthodontic operation, and the like. For example, the methods and devices described herein may be used to specifically track a portion of orthodontic surgery, including one or more stages of treatment (e.g., palatal expansion).

In general, the methods and apparatus may improve upon the prior art by guiding a user through the collection of relevant patient images (e.g., a predetermined set of images), enhancing and/or confirming image quality, correlating patient information, and transmitting the set of images so that they may be used in dental/orthodontic surgery.

For example, the methods and apparatus described herein may use and adapt a user's own handheld electronic device with a camera (e.g., a smartphone) such that the user's device guides the user to take a high quality image of a predetermined sequence of orientations (e.g., at the correct aspect ratio/size, magnification, lighting, focus, etc.). In particular, these devices and methods may include the use of "overlays" on the real-time image of the screen, providing immediate feedback about each desired orientation, which may also be used to adjust the illumination and/or focus, as described herein.

The overlay may include contours of a set of teeth (e.g., perspective contours) that may be used as a guide to aid in placing a camera to capture an image of the patient. The overlay may be based on a generic image of the teeth, or may be customized to the user's teeth or to a patient-specific category (by patient age and/or gender, and/or diagnosis, etc.). The overlay may appear partially transparent, or may be solid, and/or shown in outline.

Fig. 2A shows an example of an overlay 201, which overlay 201 comprises a tooth gear profile in a first predetermined view, here shown as a front view of teeth, on a screen 203 of a mobile telecommunication device. Fig. 2B shows another example of an overlay 201 in another front view of a tooth showing a real-time camera image (observing a tooth model of a patient's tooth) on a screen 203 of a mobile telecommunication device. As shown in fig. 2A and 2B, the overlay on the screen is displayed on top of the view of the patient's tooth model. The overlay on the screen may help guide the user to move the mobile telecommunications device to a desired location such that the overlay matches (or approximately matches, e.g., within 5%, 10%, 15%, 20%, 25%, 30%, etc., or a percentage of the overlay, e.g., > 5%, 10%, 15%, 20%, 25%, 30%, etc., of an outer or inner edge of the overlay) the view of the patient's teeth, in this example, as a model of the patient's teeth. Thus, the user can be guided to take dental photographs through the overlay on the screen, thereby saving time and effort. As will be described in more detail below. The system may be configured to take images manually, automatically, or semi-automatically when the patient's teeth are within the overlay. In FIG. 2A, the tooth is shown too far from the camera and not aligned with the overlay; in fig. 2B, the teeth are too close to the camera, although they are close to an approximate match.

High quality dental images are often required to submit dental cases or to make case assessments. Overlays on the screen of the mobile telecommunications device can improve the quality of dental images by increasing the accuracy of alignment of the camera lens with the patient's teeth. The on-screen overlay with the tooth contour can help the doctor to take high quality dental pictures using the mobile telecommunication device.

Dental photographs captured by a mobile telecommunications device may be automatically uploaded. In general, the methods described herein may improve the efficiency of a user. Taking photographs of teeth by using the methods disclosed herein can be much faster than using a digital camera. Dental images captured by the method may be of higher quality and consistency than a default camera application using only mobile telecommunication devices.

As shown in fig. 2A and 2B, the user may approximately visually match the overlay on the screen with the patient's teeth. The camera of the mobile telecommunication device must be positioned at the correct distance and at the correct angle relative to the teeth in order to capture a high quality image of the teeth. The user can refer to the overlay on the screen to guide the movement of the mobile telecommunication device camera to set the correct view in the view of the patient's teeth and the distance to the patient's teeth. The overlay can help to precisely align the cell phone camera lens with the patient's teeth and position the mobile telecommunication device at the correct distance and at the correct angle to the patient's teeth. For example, the mobile telecommunications device may have to be placed close enough to the teeth rather than too close. Matching the overlay to the patient's teeth can greatly facilitate the alignment process. Once the overlays are matched (and the images focused, e.g., using autofocus, and particularly within all or part of the overlay, as described below), an image of a view of the patient's teeth may be captured.

An overlay having tooth contours in the predetermined view may further provide information such as the desired visible teeth in the predetermined view. For example, fig. 2C shows a buccal view overlay 209 showing the teeth that should be visible in the view. In this example, the overlay may be based on actual patient teeth, or may be generated from a typical patient or synthesized from a patient population to form an average or typical overlay arrangement. Generally, an approximate visual match is required to select the correct distance, correct angle and correct view of the patient's teeth. Also described herein are patient-based overlays of approximated dentitions. In some variations, the overlay may be progressively improved as the patient's teeth are imaged, such that an increasingly precise match may be made between the overlay and the patient's teeth. For example, when the image is focused, and/or when the camera is closer to the patient's teeth, the overlay can be refined to better match the patient's teeth. In some cases, the universal patient overlay may not match the patient's teeth even when aligned (e.g., if the patient has a gap or space in the teeth, etc.). Thus, the method or apparatus may start with a generic overlay and thereafter may refine the overlay in a manner specific to the patient's teeth. For example, as described herein, it may be sufficient to train the system to recognize or classify teeth and select an appropriate overlay, or to accept an existing overlay.

The overlay may be an average (generic) overlay obtained by an artificial model. The average coverage map may not require specific information from the patient. For example, the average coverage map may be obtained from an artificial model that is approximately fit for most patients. The user may be guided to approximately match the average overlay to the patient's teeth. In some other embodiments, the average overlay may include a plurality of overlays of a plurality of sizes and types based on a plurality of artificial models, e.g., overlays for different age groups, overlays for female and male patients, overlays for patients with over-occlusions, under-occlusions, etc. In some variations, a user may manually select an overlay or family of overlays that may fit a patient's teeth, and then may be guided to match the selected overlay to the patient's teeth. In some variations, the method or apparatus may automatically select an overlay or overlay family for the patient. For example, the system can be trained to identify (e.g., using machine learning) images corresponding to a particular type of overlay or family of overlays. The selection may be based on patient information provided by the user (e.g., patient age, gender, etc.) or based on prior dental record information specific to the patient.

In some other embodiments, the overlay may be a patient-specific overlay derived from the patient's teeth. An example of a custom overlay is shown in FIG. 3A. In some variations, an apparatus (e.g., software/application software) may image the user's teeth, or may receive data derived from scans (including digital scans) of the user's teeth, and may generate an overlay, such as the contour lines shown in fig. 2A-2C. For example, the patient-specific overlay 301 may be obtained from a 3D scan or 3D impression of a particular patient's particular tooth model and/or patient's teeth. This may provide a best match for locating a handheld electronic device (e.g., a smartphone) when taking an image.

The alignment may be done manually by the user, semi-automatically or automatically. For example, any of the methods and devices described herein can be used to indicate (e.g., visually and/or audibly and/or tactilely) an image of a tooth aligned in a frame with an overlay. For example, fig. 3A illustrates an example of automatically detecting alignment in real-time using a patient's teeth and overlay. In fig. 3A, an overlay (which may be displayed in a first color, e.g., red) shows a line showing a gear profile in a first desired predetermined view (e.g., front) when a patient's teeth are not aligned with the overlay. Once the device (e.g., software/application executing on the device) determines that the teeth are substantially aligned with (e.g., substantially match) the overlay, as shown in fig. 3B, the screen may display an indication, for example, in fig. 3B, by changing the color of the overlay from a first color to a second color (e.g., green, blue, etc.) and/or automatically capturing an image. Note that the method or apparatus may automatically determine the alignment (match or approximate match) between the overlay and the patient's teeth in a variety of ways. For example, the method or apparatus may determine alignment when a threshold amount/percentage of edges of a patient's teeth in the image are within a predetermined minimum distance of the overlay. For example, alignment may be indicated when greater than 50%, greater than 60%, greater than 70%, greater than 80%, greater than 90%, etc. of the edge of the tooth (e.g., the outer periphery of the patient's tooth) is within about 1mm, about 0.5mm, about 0.1mm, about 0.05mm, etc. The edges of the teeth may be determined by any suitable technique (e.g., edge detection image processing, etc.).

As described above, in general, the overlay may be generic, categorical (e.g., suitable for all patients having one or more reference-related features), or patient-specific. The overlay may be a treatment-specific overlay. The patient's teeth may vary. For example, a patient's teeth may change over time, including treatment. The overlay can be modified to more accurately match the patient's teeth based on predicted or actual changes to the patient's teeth as they are treated (e.g., sequentially aligned treatment). Patient-specific coverage or treatment-specific coverage may allow the user to learn about treatment progress in real time.

During the imaging process, the user may view the patient's teeth, for example on a screen of a mobile telecommunications device. The user may further display an overlay including tooth contours in a predetermined view on the screen, wherein the overlay is displayed on top of the view of the patient's teeth. The coverage map may be an average coverage map, or a patient specific coverage map, or a treatment specific coverage map. The user may move the mobile telecommunications device relative to the patient's teeth such that the overlay substantially matches the patient's teeth in the view of the patient's teeth. The user may then capture an image of a view of the patient's teeth.

As described above, the method may further include estimating the quality of contour matching between the contour lines of the overlay and the patient's teeth to assist in taking high quality photographs. When the patient's teeth are in the desired position and at the correct angle, the overlay substantially matches the patient's teeth in the view of the patient's teeth. The methods and apparatus described herein may help prevent accidental confusion of views or angles, and may include real-time reaction to images on the screen of the mobile telecommunications device. Interactive contour matching estimation can capture high quality dental images. The method may include interactively estimating contour matches between contours of the overlay and the patient's teeth, for example in real time, by using the indications. In some embodiments, the method and apparatus may automatically detect a match between the patient's teeth and the overlay on the screen to confirm the position of the teeth.

Thus, the methods and devices disclosed herein may trigger an indication when the overlay substantially matches the patient's teeth. When the indication is triggered, the method or apparatus may automatically capture (or enhance the manual capture) of an image of the patient's teeth view, as shown in fig. 3B. The user can move the mobile telecommunications device relative to the patient's teeth to match the screen overlay to the patient's teeth. If the overlay nearly matches the tooth, an indication may be triggered indicating the time at which the image was captured, e.g., for progress tracking. Alternatively, in some other embodiments, the mobile telecommunications device may automatically capture an image of the teeth when the indication is triggered. In some variations, manually or automatically triggering the camera to capture an image may result in multiple images being captured in rapid succession. The focus, illumination and/or exposure time may be adjusted between these images and the best image may be used or the images may be averaged or combed.

The devices (e.g., systems) described herein may also provide guidance by indicating when an image is too far from or too close to the camera. For example, the camera output may be analyzed to determine an approximate distance to the patient's mouth and compare that distance to the expected distance to acquire the best image. For example, in any of these variations, image processing may be used to identify the patient's face (e.g., facial recognition), mouth (e.g., feature recognition using machine learning), or other features, such as nose, eyes, etc., to determine the distance from the camera and/or the area to be focused on. Any of the methods and devices described herein may include the use of tooth identification (with or without overlays). For example, the apparatus may be configured to automatically detect one or more teeth and trigger an alarm to capture an image (or automatically capture an image) when the identified one or more teeth are in a desired orientation, size, and/or position on the screen.

As described above, the indication may be triggered by estimating an indication of a distance between an edge of a patient's tooth in the patient's tooth view and a tooth contour line in the overlay. As another example, the indication may be triggered by estimating an indication of the distance between the edge of the patient's tooth and the tooth contour at two or more regions and comparing the indication to a threshold. As another example, the indication may be triggered by an indication of an estimated average deviation of the contour line from the contour of the patient's teeth. The indication may be triggered by various ways to estimate the match between the outline of the overlay and the patient's teeth, and is not limited to the examples shown herein. Generally, in any of the examples described herein, the captured image may be associated with an indication of a predetermined view and/or user identification information and/or date information (including a label, tag, etc.). For example, an image such as that shown in 3B may be taken and marked as "front".

The indication may be a visual indication, such as a color change as shown in FIG. 3B. When the user moves the mobile telecommunication device so that the overlay almost matches the teeth, the outline of the overlay may change color to green. For example, it may not match exactly, but every tooth on the contour may coincide with every real tooth on the screen. When the color of the contour changes from red to green, it indicates that the overlay almost matches the tooth, which is the time when the image was taken.

In some variations, the apparatus and methods may include providing one or more visual or auditory cues to the user when aligning the images. For example, one or more arrows may be displayed on the screen to indicate that the user should move the mobile telecommunication device in a particular direction (or a particular amount) to align the patient's teeth with the overlay.

For another example, the indication may be a sound (e.g., tone or voice) that is triggered when the overlay matches the patient's teeth. For example, the cell phone may generate a beep and indicate that the overlay matches the tooth, and is the time at which the image was taken. The method may include automatically capturing an image of the patient's teeth upon triggering the indication and/or allowing the user to manually take one or more images.

When the overlay substantially matches the patient's teeth, the automatic capture of an image of the patient's teeth may be triggered by the device. For example, a shutter of a camera of the mobile telecommunication device may be triggered by an internal indication.

In general, the methods and apparatus described herein are configured to capture a series of predetermined specific views of the teeth. Thus, the device may be application software ("app") that guides the user to take a particular sequence of images. The specific image may be a set having a particular clinical and/or therapeutic significance, such as frontal/anterior (mouth closed), frontal/anterior (mouth open), left cheek (mouth closed/open), right cheek (mouth closed/open), palate, chin, side profile, face, and the like.

Fig. 4A-4H show 8 specific overlay images of a photograph of the desired type according to orthodontic standards, including an anterior view in fig. 4A, another anterior view in fig. 4B, a maxillary view in fig. 4C, a mandibular view in fig. 4D, a left buccal view in fig. 4E, another left buccal view in fig. 4F, a right buccal view in fig. 4G, and another right buccal view in fig. 4H.

For example, the method of obtaining a dental image may include displaying an overlay on a screen that includes tooth contours in one of a front view, a buccal view, a maxillary view, or a mandibular view, wherein the overlay is displayed on top of a view of the patient's teeth. The method and apparatus may guide the user through all or a partial set of these images. For example, FIG. 5A shows an example of a user interface to enable a user to select one of a plurality of overlays for a plurality of dental images in a plurality of predetermined views, or to display to the user which images it will lead the user through. The predetermined view may include a set of tooth images according to known orthodontic standards.

In fig. 5A, eight views (including three exterior/head or facial views) are included as part of the predetermined set, including a side profile view 501 of the patient's head, a front view 503 of the patient's head, a front view 505 of the patient's head where the patient is smiling, a view of the upper jaw (as viewed from the occlusal plane) 507, a view of the lower jaw (as viewed from the occlusal plane) 509, a right cheek view 511 of the jaw being closed, a front view 513 of the jaw being closed, a left cheek view 515 of the jaw being closed, a right cheek view of the jaw being open, a front view of the jaw being open, and a left cheek view of the jaw being open. The device may guide the user through capturing each of these views, and may allow the user to select the images to be captured (e.g., in any order) or may require a predetermined order to be used. In addition, as shown in FIG. 5B, the device may provide positioning or other guidance to the user when capturing the image. Figure 5B is an example of an on-screen message for guiding a user to take an image of a tooth with a retractor. The image may be a pop-up message/reminder, which may optionally be disabled, for example, for experienced users. In fig. 5B, message 525 indicates that a buccal retractor should be used (e.g., for intra-oral views such as up, down, anterior, buccal views). In some variations, the device may automatically detect the cheek retractor and may use this information to orient the camera, and/or identify the patient's teeth, as described below.

Fig. 5C is an example of a user interface of an apparatus that assists a user in taking a side image of a patient. In fig. 5C, the user interface displays an overlay of the side outline of the patient's head 531 (on the screen of the mobile telecommunications device). In fig. 5D, the user interface shows an overlay 533 of a facial image of a general patient. The overlay may include a feature, such as an ear-nose, mouth, etc. In any of these methods and apparatus, a facial recognition tool (software tool) may be used to help identify and/or confirm the position of the patient's head and/or teeth.

The method may include displaying a plurality of overlays for each of a plurality of dental images in a plurality of predetermined views on a user interface. As described above, the user may select one of the overlays. For example, there may be three different facial images and 8 different dental images, as shown in fig. 5A. For each desired view of the dental image, there may be an open position image and a closed position image.

The method may also allow the user to take more images of their choice. For example, the method may have a user interface to allow the user to input his or her selected angle and distance, thereby enabling the user to freely capture custom images. In some embodiments, the method may also enable the user to take pictures in several movements to restore the 3D structure of the teeth.

Any of these methods and apparatus may further include viewing the captured image on the mobile telecommunications device to confirm image quality and/or automatically accept/reject the image. For example, the method or apparatus may be configured to check the image quality of the captured image and display on the screen if the image quality is below an image quality threshold. Fig. 6A is an example of a user interface with a notification window 606, the notification window 606 indicating that a captured image is out of focus after viewing image quality. Fig. 6B is an example of an image with poor tooth quality. In some other embodiments, the method may include automatically filtering out poor quality images, such as blurriness, wrong viewing angle, and the like. For example, when the image is too dark or some of all teeth are blocked, as shown in fig. 6B, the image may be excluded and the user is asked to take a substitute image.

In fig. 6B, the device also indicates that a maxillary image should be taken using the dental mirror 609. The apparatus may be configured to automatically detect when a dental mirror is present, for example by detecting reflections or symmetries within the image field. In fig. 6B, mirror 611 includes an excessive amount of reflected light.

The method may include evaluating image quality and alerting a physician whether an image may be rejected. For example, if the image is too dark, blurred or out of focus, a warning message may be displayed on the screen to alert the doctor to take the picture again. The quality of the image can be evaluated and evaluated by various methods. For example, the method may include evaluating a focus state of the image. As another example, the method may include analyzing the image using a library. For yet another example, the method may include analyzing a brightness of the image. In some embodiments, the method may further comprise using an image recognition algorithm to ensure that all of the desired teeth are visible.

FIG. 7A illustrates an exemplary overlay including a crop box and tooth contours in a maxillary view. FIG. 7B illustrates another exemplary overlay including a crop box and tooth contours in a front view. In some embodiments, the method may further include displaying an overlay on the screen including the crop box and tooth contour lines in one of a front view, a buccal view, a maxillary view, or a mandibular view, wherein the overlay is displayed on an upper layer of the patient's teeth. The method may further include automatically cropping the captured image as indicated by the crop box.

The doctor typically has to wait one or more days to crop and edit the dental image before uploading the image, and the method described herein may further include displaying an overlay having a crop box and a dental gear profile in a predetermined view, and automatically cropping the captured image. FIG. 7A illustrates a crop box 702 in a maxillary view of teeth. FIG. 7B illustrates another crop box 704 in a front view of a tooth. By matching the overlay, the method can automatically crop the image to obtain the correct dental view. After taking the picture, the doctor can still re-crop and edit the picture at a later time.

FIG. 8A is an example of a block diagram of a method of obtaining an image of a patient's teeth. A user (e.g., a dental professional, such as a doctor, orthodontist, etc.) may obtain rapid feedback of treatment progress and/or an assessment of treatment suitability. As shown in fig. 8A, the method may include (optionally) authorizing using the patient ID in step 801. For example, the method may include imaging the identity of the patient using a mobile telecommunications device. The method may further include automatically populating the form with user identification information based on the imaged identification.

In general, a method for obtaining a series of images of a patient's teeth may include viewing the patient's teeth on a screen of a mobile telecommunications device (in real time), as shown at step 803. The method may further include displaying an overlay including the predetermined view and (optionally, a crop box) on the screen. The overlay of the predetermined view may be, for example: front view (jaw open or closed), cheek view (left, right and jaw open/closed), maxilla view or mandible view, etc. As shown in step 805, an overlay may be displayed on top of a view of the patient's teeth. As described herein, a method or apparatus performing the method may be configured to include a selection step for determining which overlay to use (e.g., by automatically identifying teeth/faces displayed in a screen and/or by a user manually selecting from a menu, etc.). For example, the method may further comprise enabling a user to select a predetermined view. The user interface may display to the user a predetermined view and a plurality of overlays for the plurality of predetermined photo views (e.g., front (open/close), left cheek (open/close), right cheek (open/close), upper jaw (upper jaw), lower jaw (lower jaw), etc.). The user may select one of the plurality of overlays to capture a corresponding dental image. For each predefined view, an overlay with outline lines is shown on the screen of the mobile telecommunication device. The method may further include moving the mobile telecommunications device such that the overlay substantially matches the patient's teeth in the view of the patient's teeth, as shown in step 806. For example, the method may further comprise displaying instructions regarding positioning the patient's teeth on a screen of the mobile telecommunications device prior to displaying the overlay. Optionally, the method may include using the coverage area to adjust focus 811, illumination, exposure 813, and the like, as described herein.

In some embodiments, the method may further comprise triggering an indication when the overlay substantially matches the patient's teeth, as shown in (optional) step 807. For example, if the overlays do not match, the overlays will be displayed using red. If the overlay matches the tooth, the color of the contour line changes to green. The method may further include capturing an image of a view of the patient's teeth, as shown in step 808. Steps 803 to 808 may be performed for each predetermined photo view 815. The user can take several pictures for each view to obtain a more accurate estimate of the progress of the treatment. The method may further include repeating the steps of viewing, displaying, moving, and capturing to capture anterior, buccal, maxillary, and mandibular images of the patient's teeth. In addition, the device may check the image to ensure that the quality is sufficiently high (e.g., focus, etc.); if not, the steps of the view may be repeated.

The method can also include transmitting the captured images to a remote server and/or evaluating the captured images for medical treatment using the collected set of images 809 as in step 810. The captured dental images may be transferred to a server portion for more accurate treatment progress estimation and/or pre-screening of patients. For example, the captured dental images may be used for case evaluation before starting aligner treatment to evaluate whether the patient is a treatment candidate, as illustrated in fig. 8B. When the doctor starts the treatment, the captured dental images can also be used to set case attributes, e.g., the captured images can be used to fill in a prescription form. The captured dental images may further enable the physician to track the treatment process and give them real-time cues or feedback of the treatment process.

In fig. 8B, the method of remotely prescreening patients for orthodontic treatment may incorporate any of the steps of fig. 8A. In fig. 8B, the method first comprises guiding the user with the mobile telecommunication device having a camera through sequentially taking a series of images 851 of the patient's teeth in a plurality of predetermined views for each predetermined view. As described above, one of the steps described above with respect to fig. 8A may be used, including but not limited to: an overlay including a tooth contour in one of the predetermined views from the plurality of predetermined views is displayed on the screen, wherein the overlay is displayed on the upper layer 855 of the image of the patient's tooth. Any of these methods may further include capturing an image 857 of the patient's teeth when the overlay substantially matches the patient's teeth in the view of the patient's teeth. Once the complete set of images (and any other information, including photo location, image tags, date of series, etc. for each image) is collected, they can be aggregated into a single set. This information may then be transmitted (e.g., including a series of images) to a remote location to determine 859 whether the patient is a candidate for orthodontic treatment based on the series of images. Thereafter, the method can display an indication 861 on the screen of the mobile telecommunications device whether the patient is a candidate for orthodontic treatment.

Fig. 9 is an example of a set of facial and dental images of a patient captured by using a digital SLR camera. In fig. 9, the image includes 8 images: side profile, frontal, smiling frontal, upper jaw, lower jaw, right cheek (closed jaw), anterior (closed jaw), and left cheek (closed jaw). Fig. 10 is an example of a set of facial and dental images from these same views captured by the method disclosed herein using a mobile telecommunications device. Fig. 11A-11H are examples of a set of dental images captured by the method disclosed herein by using a mobile telecommunication device, showing the front (fig. 11A), front open (fig. 11B), upper (fig. 11C), lower (fig. 11D), right cheek closed (fig. 11E), right cheek open (fig. 11F), left cheek closed (fig. 11G) and left cheek open (fig. 11H). As shown in fig. 9-11H, the methods disclosed herein may enable a user to capture high quality tooth images in multiple predetermined views to meet orthodontic standards.

Orientation(Focus)

Any of the methods and devices described herein may include tooth-specific focusing. In general, a camera of a mobile telecommunications device may be configured such that the camera is automatically focused on one or more teeth. In variations where the system is configured to detect a patient's teeth within an image, the device may then automatically focus on the teeth. Alternatively or additionally, the apparatus or method may be configured to use the overlay or a region within the overlay to focus on the patient's teeth. For example, the apparatus (e.g., a non-transitory computer-readable storage medium storing a set of instructions executable by a processor of a mobile telecommunications device having a camera) may be configured to focus on a subset of an area within an overlay on a display and to automatically focus within the area or sub-area. Within this region or sub-region, the camera may be controlled to perform any suitable type of autofocus, including but not limited to: contrast detection autofocus, phase detection autofocus, and laser autofocus.

For example, the apparatus or method may first disable the native autofocus of the camera, which may be the default in a particular region (e.g., the center region), motion detection and/or object (e.g., face) recognition, or some variation of these. If used with an overlay and a method of matching the overlay to a patient's teeth, the local autofocus may instead focus on the lips, buccal retractor, tongue, etc., rather than on the teeth or a portion of the teeth. Rather, by limiting autofocus to areas localized to the overlay or a portion of the overlay, the apparatus and method may instead focus appropriately on the tooth.

For example, fig. 12A shows an example of a screen 1201, the screen 1201 having an overlay 1203 shown layered on any image from a camera displayed on the screen. In fig. 12A, the focal region (point) within the overlay is shown as a point 1205 within the overlay of one of the superior incisors. Thus, in this example, the focus on the screen is shown as the front (incisor) tooth when the tooth is approximately aligned with the overlay. Fig. 12 shows an example of an overlay of a front (open mouth) view. Other views may have a focus within an overlay centered on a different tooth or more than one tooth. For example, the front view may have a focal point within the overlay that is located above the bicuspid or visible molar and/or canine teeth. A user (e.g., a doctor, an orthodontist, a technician, an assistant, etc.) may align the patient's teeth with the overlay and take an image and may control the mobile telecommunications device such that the area within the overlay is automatically controlled to focus within the area of the overlay. In the exemplary images shown in fig. 12A and 12C, the focus is limited to only one of the central incisors within the overlay. Fig. 12B and 12C provide a comparison between images taken with a patient's teeth, which are typically aligned within an overlay for the front (closed) image 1211, where the camera for the mobile telecommunication device is set to its default autofocus technique (on the left as shown in fig. 12B) as compared to the above-described target autofocus, where the autofocus is limited by the apparatus to a region 1213 (shown in fig. 12C, on the right) corresponding to the incisor within the overlay.

Adapted illumination mode

Any of the devices and methods described herein may also be configured to automatically and accurately adjust the illumination and/or exposure time so that the tooth is optimally illuminated for imaging so that the details may be clear. As described above, for autofocus within an image, the illumination (illumination) can be similarly adjusted by using the overlay or a sub-region of the overlay to set the intensity of the applied illumination (e.g., flash).

In general, a camera of a mobile communication device may include a light source that provides illumination when capturing an image. The camera may have one or more illumination modes for operation including, for example, burst flashes of light (pulses or flashes of light associated with image capture), a flashlight (continuously illuminated), or no flash/no illumination (e.g., no illumination provided). Any of the methods and apparatus described herein may improve the captured image of the patient's teeth by automatically selecting and controlling a particular illumination pattern for each image captured (e.g., each predetermined view). For example, the apparatus may be configured, inter alia, by adjusting or controlling the illumination pattern so that no flash light is used when capturing a face image (e.g., a side profile face image, a frontal face image with or without a smile, etc.); when a bite image is taken (e.g., upper/upper and lower/lower bites), the light emission mode may be set to burst flashing; and flashlight illumination may be automatically selected when taking an intraoral picture (e.g., front view, cheek view, etc.). The intensity of the flash may also be adjusted. For example, the intensity of the applied light may be adjusted based on the light level detected from the image area within the overlay on the screen. In some variations, any additional lighting may be selected for use based on the light intensity within the coverage area first; if the light level is below a threshold within all or a portion of the overlay (e.g. within the lower 5%, 10%, 15%, 20%, 25%, 30%, etc. of the dynamic range of the light intensity of the camera of the mobile telecommunications device), the illumination mode may be selected based on the type of predetermined view of the overlay. For example, if the overlay shows an in-exit view (e.g., front mouth open, cheek mouth open, etc.), the illumination mode may be set to flashlight, and in some cases the brightness level of the flashlight may be adjusted based on the light level detected within the overlay. If the overlay corresponds to a bite view (e.g., upper jaw, lower jaw), the illumination mode may be set to burst flashing, and in some cases, the brightness level of the flashlight may be adjusted based on the light level detected within the overlay.

Figures 13A-13B and 13C-13D show a comparison between dental images taken guided by overlays (not visible on the images shown). Fig. 13A shows a maxillary occlusion view taken with a dental scope using default focus and no flash. For comparison, fig. 13B shows the same tooth with focus limited to a sub-region of an overlay (not shown) and flash suddenly lit up as a burst flash. In fig. 13B, additional details of the teeth 1307 in the upper jaw are visible as seen with the dental mirror 1305. Fig. 13C and 13D show a similar example in which the mandible 1309 is imaged without limiting focus to the coverage area (fig. 13C) and without flashing (e.g., flashing in an "off" mode). For comparison, a predetermined view of the same teeth is shown in fig. 13D, with the focus adjusted to the area within the overlay of the lower teeth and the flash automatically adjusted to on (flashlight mode). The image is also captured using a dental scope.

Thus, any of the methods and apparatus described herein may automatically adjust the illumination provided by turning on or off the automatic flashing and/or by setting a light level or allowing a user to adjust a light level. For example, the method of the device may switch between automatically flashing, no flashing/lighting, a user adjusted light level ("flashlight mode"), or an automatically adjusted light level based on one or more of user preferences and the image to be captured. For example, when taking a side profile or facial image, the device may be configured to cause the flash to turn on, e.g., default to the camera's auto-flash function if a flash is present. The flash may be turned off when taking intraoral images (e.g., front view, buccal view, maxillary view, mandibular view, etc.), and instead the device may be adjusted to use a "flashlight" mode where the light is continuously on, particularly during imaging. As described above, the level of light may be automatically set, or may be adjusted by the user. For example, when taking an intraoral image of a tooth, the flashlight function of the camera light may be set to a level near about 10-50% of the peak intensity (e.g., between 15-40%, between 20-30%, etc.). In some variations, the flashlight intensity may be adjusted in real-time based on image quality, and in particular, based on the light level of the area within the overlay (e.g., centered on or near the molars). Alternatively or additionally, the user may manually adjust the light intensity in the flashlight mode, for example, by adjusting a slider on the screen of the device or by one or more buttons on the side of the device.

Similarly, the exposure time of the camera may be adjusted based on the amount of light within the full or partial coverage region of the imaging field. The image exposure and/or depth of field range of the control image may be guided by the area within the overlay. For example, the apparatus (e.g., a non-transitory computer-readable storage medium storing a set of instructions executable by a processor of a mobile telecommunications device having a camera) may control the camera such that any auto-exposure function of the camera is modified to base the exposure on a target point or region within the overlay. The depth of field may also be adjusted by the device.

As described above, the exposure time may be automatically set based on the area within the overlay. The area within the overlay used to set the exposure may be different from the area used to set the focus or light intensity. In addition, these regions may be different for different views. For example, in the anterior image, the focus may be set using the area on the anterior (e.g., incisor) teeth within the overlay, as shown in fig. 12A, while the area for setting the illumination (e.g., light intensity) may be set based on more anterior teeth (e.g., molars, premolars, etc.), and the area for setting the exposure may be the same as the area for setting the focus, the same as the area for setting the illumination, or may be different from any of these. In some variations, all three regions may be the same.

Buccal retractor detection

Any of the methods and devices (e.g., systems) described herein can include a buccal retractor and/or automated detection of a buccal retractor. For example, the devices and methods described herein may avoid having the user take images of a predetermined view of the patient without using a cheek retractor, particularly when the predetermined view benefits from using a cheek retractor. Figure 14A shows an example of a right cheek view of a patient's right cheek portion manually retracted (using a finger). Such images may be useful, and in fact a plurality of such images may be taken and combined (by image stitching) together, for example by pulling or manually retracting the cheek from different angles and combining the different images to display as much of the cheek view as possible, including the upper and lower teeth. However, in some variations, it may be beneficial to use a mechanical buccal retractor, such as the one shown in fig. 14C. This may allow more teeth to be visible rather than manually retracting the cheeks (as shown in fig. 14A) or simply using facial muscles to pull the lips and cheeks as shown in fig. 14B.

As described above (e.g., using an alarm window, etc.), the devices and methods described herein can alert the user to use the cheek retractor prior to taking an intraoral picture. However, the user (and possibly the patient) may choose not to use the buccal retractor, resulting in an image of the intended view within the mouth that is particularly non-optimal, and may be more difficult to process. In some variations, the device or method may automatically detect the presence of a cheek retractor and may alert the user when no cheek retractor is detected. For example, machine learning can be used in an exercise device to automatically inspect a cheek retractor. Alternatively or additionally, the buccal retractor may comprise one or more markings on the buccal retractor in the visible region, which can be easily detected. Indicia (e.g., dots, barcodes, QR codes, text, patterns, icons, etc.) may be detected by the device and may also be used to help automatically identify the location of the patient and thus which predetermined view should be used or is being taken (e.g., which predetermined view overlay).

In the exemplary device shown in fig. 14C, cheek retractor 1408 includes a plurality (six in this example) of markings 1406, 1406', 1406 "' on the outer surface that will be visible when the patient is imaged while wearing the cheek retractor. The indicia may be on the outer surface and the upper surface, which when worn will be above the patient's lips. The illustrated integral buccal retractor has a left side with a patient handle 1409 connected to a right side with a second patient handle 1411 by a flexible bridging region 1413. The patient facing side includes a channel or recess for holding the lips and cheeks, and the bridging region can apply a force to separate the left and right sides, holding the cheeks open, exposing the teeth. The buccal retractor may be colored (e.g., may include colored dots) for identifying the orientation of the patient's teeth when worn. In the example shown in fig. 14C, two of the markings 1406, 1406 'have different orientation markings, shown as horizontal lines on one 1406 and vertical lines on the other 1406' (the lines may be formed by semicircular flat edges forming the markings). These orientation marks may be formed in the device or they may be printed on the device. In some variations, the indicia may be a different color and/or shape, and thus may provide orientation and/or distance information. For example, the markers may be used to determine the distance between the patient (or retractor device) and the camera. As described herein, the method and/or apparatus may determine an estimate of the distance from the size of an individual marker and/or the spacing between different markers in the resulting image. In general, it may be desirable to take any intra-oral image with the camera about 6 to about 16 inches from the patient's mouth (e.g., between about 7 and 14 inches, between about 8 and 12 inches, etc.). As described above, if the distance is outside the predetermined range, the device may instruct the user to adjust the location (e.g., closer or farther, etc.). Similarly, the device may center the image using the marker such that the tooth (or a portion of the tooth) is centered within the image.

Markings on the retractor may also be used to assist in automatically cropping the image.

Any of the apparatus and methods described herein may also make an estimate between the patient and the camera of the mobile telecommunications device. For example, any of these methods and apparatus may perform face detection from an image to identify a patient's face; once identified, the size and location of the face (and/or any landmarks from the patient's face, such as eyes, nose, ears, lips, etc.) and an approximate distance to the camera may be determined. This information can be used to guide the user in positioning the camera; for example, the user is instructed to be closer or farther from the patient in order to take an image as described above. For example, in fig. 15, box 1505 on the screen identifies the automatically detected face/head of the patient; the face is identified using face recognition software provided by the developer. Fig. 16 shows another example of face recognition. In both examples, the method and apparatus may determine that the camera is too far away by comparing the size of the identified face (box region) to the size of the field. In both cases, instructions (audio, visual, text, etc.) may be provided to the user to move the camera closer. In fig. 15, the image to be taken is a front view of the teeth, and the user may be instructed to move closer to focus on the teeth. In fig. 16, the view may be an image of the patient's face (not smiling) and may indicate that the user is close. The distance to the patient may be estimated in any suitable manner. For example, as mentioned, the distance may be derived from the size of a rectangle made around the patient's face or mouth (e.g., if it is too small, the camera is too far away). The size of the "box" identifying the face may be absolute (e.g., must be above a set value) or a percentage of the image field size.

As described above, using other identified features, the distance of the camera may be approximate, including the eye separation distance (pupil separation distance, etc.) when these features are visible. Any of these distance estimates may be stored with the image for later use, for example, in estimating size or projecting three-dimensional features.

Continuous imaging (photographing) using teeth is also described herein. For example, rather than taking a single image at a time, the apparatus or method is configured to take a general picture of the patient by using a continuous shooting mode. A series of fast images may be taken while moving the mobile device. The movement may be guided by the device and may be from left to right, top to bottom, etc. From the user's perspective, it may be similar to taking a video, but a series of images (still images) may be extracted by the device. For example, the device may automatically view the image and match (or approximately match) the view to a predetermined view, e.g., using an overlay as described above. The device may select only those images that are of sufficiently high quality. For example, blurred, dark, or non-optimally positioned photographs may be automatically rejected. Similarly, multiple photographs may be combined (by stitching, averaging, etc.).

In some variations, the user may be a patient, and the apparatus may be configured to allow the user to make successive "selfie" shots in this manner. For example, the continuous shooting mode may allow a patient to take their smile photo with only a camera (e.g., a back camera). The apparatus may use face and/or smile detection to guide the patient and indicate whether the camera is well positioned. In variations where the screen is not facing the user, the apparatus may be configured to provide a user detectable output (e.g. a flash, a sound, etc.) indicating that the patient should begin moving the device around their head/mouth and may indicate that they should move the camera closer or farther, right/left, up/down, etc. For example, an indication such as a light (e.g., a flashing light) or a sound (voice or tone, etc.) may be used as the indication. For example, a flashing light and/or sound may be used to indicate to the patient when to start moving the mobile device, and the apparatus may start taking pictures in a continuous mode (also referred to as a burst mode) and moving the mobile device in the indicated direction.

As described above, any of these variations may include automatic detection of teeth, for example, by machine learning. Automatic detection of the patient's teeth may improve picture quality. In some variations, machine learning (e.g., a machine learning framework provided by Apple with iOS 11) may be used to detect the presence of teeth when taking a picture and further guide the user. For example, the user may be alerted when the teeth are not visible, or automatically select which of the predetermined views to overuse, to indicate whether the angle is correct, to indicate that the user is too close or too far from the patient, and so forth.

In general, in addition to the images captured by the device, additional information may be extracted from the sequence of images taken in the above-described continuous shot, including the orientation and position of the camera relative to the patient. For example, position information may be extracted from the time series, including the relative distance and/or angle of the camera with respect to the patient's mouth. Additional information may also be used, such as a motion sensor (e.g., accelerometer, etc.) in the mobile telecommunications device. The image may be provided with this additional sensor information (e.g., accelerometer, angle, etc.) information. Such information may help guide the user, e.g., indicate that the user is closer/farther, moves more slowly, etc., as well as help calculate dimensional information (e.g., size, three-dimensional position, and/or surface or volume information, etc.).

Pre-screening for orthodontic treatment

As noted above, in general, the devices and methods described herein may be used to remotely pre-screen patients for orthodontic treatment. For example, the methods described herein may include and/or may be used to determine whether a patient benefits from orthodontic corrective surgery to orthodontically move and correct the patient's teeth using, for example, a series of dental aligners. These methods and apparatus may be part of a case evaluation tool. The user (dental professional or in some cases a potential patient) may be guided to take a set of predetermined views as described above and these views may be transmitted (e.g. uploaded) from the mobile telecommunications device to a remote location. At a remote location, which may include a server, the images may be processed manually, automatically, or semi-automatically to determine whether the patient is a candidate for orthodontic surgery.

The series of predetermined views described herein may be used to determine whether a patient is a good candidate by identifying (manually or automatically) the amount and degree of tooth movement required to straighten teeth. A patient requiring too much tooth movement may be indicated as not a candidate. A patient in need of surgical treatment (e.g., a patient in need of palatal expansion, etc.) may be indicated as not a candidate. In some variations, a patient requiring tooth extraction and/or reduced interproximal tooth reduction may be indicated as not a candidate, at least for some orthodontic treatments. In variations, the apparatus or method may instead indicate which type of orthodontic treatment is optimal, rather than simply determining whether the patient is or is not a candidate for a particular orthodontic treatment.

As described above, these methods may be used to pre-screen any type of orthodontic treatment, including, for example, straightening teeth using one or a series of temporary aligners (e.g., which may be replaced periodically, e.g., once per week). The type of orthodontic treatment may be limited to relatively easy orthodontic corrective surgery, such as orthodontic treatment using aligners that may require less than x months (e.g., 1 month or less, 2 months or less, 3 months or less, 4 months or less, etc.) to complete.

Any of these methods and apparatus may be configured to collect information about the patient before, during, or after capturing a series of images, as described above. In addition to or instead of patient identification information, the apparatus may also include information regarding the patient and/or the user's primary orthodontic concerns about the patient's teeth (e.g., tooth crowding, tooth spacing, smile width/arch width, smile line, horizontal overlay, vertical overlay, cross-bite, bite relationship, etc.). The device may include a menu of these questions and may allow the user (dental professional and/or patient) to select one or more of them or enter their own questions. One or more major dental problems may be added to a set or series of images from a predetermined view. For example, a major dental problem may be attached or combined to an image or series of images and transmitted to a remote site and used to determine whether the patient is a good candidate for a particular orthodontic procedure.

Generally, the images can be used to quantify and/or model the position of the patient's teeth. The position and orientation of the patient's teeth relative to the arch or relative to other teeth in the arch may provide an estimate of the motion or procedure required to correct (e.g., align) the patient's teeth, or in some variations, provide a progress in the treatment being performed.

The methods described herein may also include monitoring a patient undergoing orthodontic treatment. For example, the step of guiding the user with the same or a different mobile telecommunication device with a camera to take a series of images of the patient's teeth from a plurality of predetermined views (e.g. by sequentially displaying an overlay comprising the tooth gear profile in each predetermined view on the screen of the mobile telecommunication device) may be repeated during orthodontic treatment in order to monitor the treatment. Images taken before treatment can be compared with images taken during or after treatment.

In any of the methods of the apparatus described herein, the images may be uploaded to a remote server or storage facility, or they may be maintained locally to the mobile telecommunications device. When maintained locally on the mobile device, any copies sent remotely for analysis may be corrupted within a predetermined amount of time (e.g., after analysis is complete) so that no additional copies are sent. The image and any accompanying information may generally be encrypted.

As described above, any of the methods and apparatus described herein may be configured to automatically detect a mirror, such as a dental mirror, used to take any image. For example, the device may be configured to recognize that the image is a reflection, or to recognize a mark on a mirror. The reflection may be determined by identifying discontinuities (e.g., lines) at the edges of the mirror and/or mirror/inverted images of portions of the image (e.g., teeth). When a mirror is detected, the device may display a mirror image icon (or may indicate on the mirror icon). In some variations, the resulting image may be inverted (mirrored), such that the image has the same orientation as if the mirror were not used.

The images (e.g., a predetermined series of images) may be used to supplement additional information (e.g., scans, 3D models, etc.) of the patient's teeth. Images taken as described herein may provide information about the shape, position, and/or orientation of the patient's teeth and gums, including information about the patient's roots. Thus, this information may be used in conjunction with other images or models, including 3D models (e.g., digital models) of the patient's teeth, and may be combined with or may supplement this information.

Various embodiments of the present disclosure also disclose a non-transitory computer readable storage medium storing a set of instructions executable by a processor of a mobile telecommunication device, which when executed by the processor, enable the processor to display a real-time image of a patient's teeth on a screen of the mobile telecommunication device, display an overlay including tooth contours in a predetermined view of an upper layer of the image of the patient's teeth, and capture an image of the patient's teeth. For example, a non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to display a generic overlay. As another example, a non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to display a patient-specific overlay derived from a patient's teeth.

For example, a non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to automatically trigger an indication when the overlay substantially matches the patient's teeth. For example, a non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to estimate an indication of a distance between edges of teeth of the patient in the view of the teeth of the patient, and trigger the indication when a tooth profile line in the overlay is less than or equal to a threshold. For example, a non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to estimate an indication of a distance between edges of the patient's teeth at two or more regions in the view of the patient's teeth, and trigger the indication when a tooth contour in the overlay is less than or equal to a threshold. For example, a non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to trigger the indication by displaying a visual indication on the screen.

In general, various embodiments of the invention also disclose a non-transitory computer readable storage medium storing a set of instructions executable by a processor of a mobile telecommunication device, which when executed by the processor, cause the processor to display a real-time image of a patient's teeth on a screen of the mobile telecommunication device and an overlay including a crop box and a tooth contour line in one of a front view, a cheek view, a maxilla view or a mandible view, wherein the overlay is displayed on top of the image of the patient's teeth and is capable of capturing the image of the patient's teeth. A non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to view the captured image and indicate on the screen whether the captured image is out of focus and automatically crop the captured image as indicated by the crop box.

For example, a non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to check an image quality of the captured image and display on the screen if the image quality is below an image quality threshold. For example, a non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to automatically crop the captured image based on a crop contour displayed as part of the overlay. For example, a non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to transmit the captured image to a remote server.

For example, a non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to display an overlay comprising tooth contours in a predetermined view (e.g., a front view, a buccal view, a maxillary view, or a mandibular view). For example, a non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to repeat the steps of observing, displaying, moving, and capturing to capture anterior, oral, mandibular, and mandibular images of the patient's teeth. For example, a non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to capture an image of a patient identity using the mobile telecommunications device and automatically populate a form with user identity information with the imaging-based identity. For example, a non-transitory computer readable storage medium, wherein the set of instructions, when executed by the processor, may further cause the processor to display instructions regarding positioning a patient's teeth on a screen of the mobile telecommunication device prior to displaying the overlay.

The systems, devices, and methods of the preferred embodiments and variations thereof may be embodied and/or carried out at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions. The instructions are preferably executed by a computer-executable component that is preferably integrated with a system that includes a computing device configured with software. The computer readable medium may be stored on any suitable computer readable medium, such as RAM, ROM, flash memory, EEPROM, an optical device (e.g., CD or DVD), a hard disk drive, a floppy disk drive, or any suitable device. The computer-executable components are preferably general-purpose or special-purpose processors, but any suitable special-purpose hardware or hardware/firmware combination may alternatively or additionally execute instructions.

Examples of the invention

In one example, described herein is a photo upload mobile application (control software) that can be installed on a mobile device, such as a smartphone, and can control a smartphone camera to take dental images of a patient in a particular manner. The particular mode may include a particular sequence of photographs, as well as controlling image quality and/or imaging characteristics to more accurately plan or track a treatment process, such as a series of dental aligners.

For example, an application program ("app") may be configured to require login (username, password) to be used. The login may be done using an alternative login (e.g., fingerprint, password, etc.). Upon login, the application may display a list of patients and/or may allow for the addition of patients. The user may, for example, select or add patients from a displayed selectable list. Adding the patient may include entering identifying information about the patient (e.g., first name, last name, date of birth, location and gender, such as country, city, state, zip code, etc.). Thereafter, the application may direct the doctor or clinician to take a predetermined series of images, such as the images shown in FIG. 5A, and described above.

For existing patients, the application may allow the user to view pictures that have already been taken and/or to begin taking new pictures. Thus, any of the devices (including applications) described herein may allow a user to take a requested series of images (which may depend on patient identity, treatment plan, etc.) and may include taking additional images (e.g., 3 or more, 4 or more, 5 or more, 6 or more, 7 or more, 8 or more, 9 or more, 10 or more, 11 or more, 12 or more, etc.). For each patient, photographs may be taken at different times showing the progress of the treatment. For example, a series of photographs (such as those shown in fig. 5A) may be taken prior to treatment and taken one or more times during and/or after treatment. Additional photographs may also be taken at any of these times. When the user is taking the series of pictures, the application may guide the user to take a picture of the patient from the predetermined series, and may display one or more contour lines (including a typical set of teeth, head/face, etc. as described above) on the screen. The application program may also guide the user in applying auxiliary devices, such as retractors and/or dental mirrors. Once the image is captured, the application may be manually pre-cropped and/or cropped by the user. The application may also examine the content of the image to confirm focus, subject (angle, etc.) lighting, etc. The application may prompt the user to re-capture and/or save the image.

The images/photographs taken may be uploaded by the device to a remote server or site, along with patient and/or physician identifying information and/or information about when taken, and stored for later retrieval and/or transmission by the physician to a third party. The doctor may approve or view the image using an application or using another computer that accesses a remote site. The images may be uploaded separately or they may be uploaded as a composite of multiple separate images.

The physician may also review and/or attach comments on or accompanying the image. In any of these devices, the application may also provide access to patient case management software, including modeling and 3D imaging/rendering of the patient's teeth.

The application may also include instructions (e.g., frequently asked questions section, tutorial, etc.).

In some variations, a physician may mark or otherwise indicate that a particular patient or case is being processed by a remote server, for example, for clinical evaluation, and/or to prepare a series of aligners. As described above, the application may be used on any mobile device having or communicating with a camera, such as a smartphone, a smartwatch, a tablet, and the like.

Although the examples described herein are described specifically for mobile telecommunications devices (such as smart phones, tablets, etc.), in some variations, the methods and apparatus implementing them may be performed with devices including displays and processors that are not limited to mobile telecommunications devices. For example, the methods and apparatus may be configured for use with a Virtual Reality (VR)/Augmented Reality (AR) headset or any other imaging device that may transmit an image (e.g., a photograph) directly to a computer via a direct connection (e.g., a cable, a dedicated wireless connection, etc.), and/or may save the image to a removable or removable memory (e.g., an SD card); the image data may then be uploaded to a remote server through another device. Thus, any of the methods and apparatus described herein that describe or relate to a mobile telecommunications device may be performed with an imaging device.

For example, a method for remotely prescreening patients for orthodontic treatment may include: guiding a user with an imaging device having a camera to take a series of images of a patient's teeth in a plurality of predetermined views, the series of images being transmitted from the imaging device to a remote location to determine whether the patient is a candidate for orthodontic treatment based on the series of images; and displaying on the screen an indication of whether the patient is a candidate for orthodontic treatment.

Similarly, a method for remotely prescreening patients for orthodontic treatment may include: using an imaging device having a camera, a user is guided to take a series of images of a patient's teeth from a plurality of predetermined views by: sequentially displaying an overlay including the tooth gear profile in each of the predetermined views on a screen of the imaging device; receiving, in an imaging device, an indication of a major dental problem of a patient; aggregating a series of images and major dental problems in an imaging device; transmitting the aggregated series of images and the primary dental problem to a remote location to determine whether the patient is a candidate for orthodontic treatment based on the series of images; and displaying an indication of whether the patient is an orthodontic treatment candidate on a screen of the imaging device or a device in communication with the imaging device.

As another example, a system for remotely prescreening patients for orthodontic treatment may include: a non-transitory computer readable storage medium storing a set of instructions executable by a processor of an imaging device having a camera, the instructions when executed by the processor cause the processor to: guiding a user to take a series of images of a patient's teeth in a plurality of predetermined views with a camera; transmitting a series of images from the imaging device to a remote location to determine whether the patient is a candidate for orthodontic treatment based on the series of images; displaying, on a screen of the imaging device or a device in communication with the imaging device, an indication of whether the patient is an orthodontic treatment candidate.

When a feature or element is referred to herein as being "on" another feature or element, it can be directly on the other feature or element or intervening features and/or elements may also be present. In contrast, when a feature or element is referred to as being "directly on" another feature or element, there are no intervening features or elements present. It will also be understood that when a feature or element is referred to as being "connected," "attached," or "coupled" to another feature or element, it can be directly connected, attached, or coupled to the other feature or element or intervening features or elements may be present. In contrast, when a feature or element is referred to as being "directly connected," "directly attached" or "directly coupled" to another feature or element, there are no intervening features or elements present. Although described or illustrated with respect to one embodiment, features and elements so described or illustrated may be applied to other embodiments. Those skilled in the art will also appreciate that references to a structure or feature that is disposed "adjacent" another feature may have portions that overlap or underlie the adjacent feature.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. For example, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as "/".

Spatially relative terms, such as "below," "over," "above," and the like, may be used herein for ease of description to describe one element or feature's relationship to another element or feature as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is inverted, elements described as "below" or "beneath" other elements or features would then be oriented "above" the other elements or features. Thus, the exemplary term "below" can include both an orientation of above and below. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Similarly, the terms "upward," "downward," "vertical," "horizontal," and the like are used herein for explanatory purposes only, unless explicitly stated otherwise.

Although the terms "first" and "second" may be used herein to describe various features/elements (including steps), these features/elements should not be limited by these terms unless context dictates otherwise. These terms may be used to distinguish one feature/element from another feature/element. Thus, a first feature/element discussed below could be termed a second feature/element, and similarly, a second feature/element discussed below could be termed a first feature/element, without departing from the teachings of the present invention.

As used in this specification and claims, including as used in the examples and unless otherwise expressly specified, all numbers may be changed to begin with the word "about" or "approximately", even if the term does not expressly appear. The phrases "about" or "approximately" may be used when describing magnitude and/or position to indicate that the described value and/or position is within a reasonable range of expected values and/or positions. For example, a numerical value may be +/-0.1% of the value (or range of values), +/-1% of the value (or range of values), +/-2% of the value (or range of values), +/-5% of the value (or range of values), +/-10% of the value (or range of values), and the like. Any numerical range recited herein is intended to include all sub-ranges subsumed therein.

Although various illustrative embodiments have been described above, any of numerous variations may be made to the various embodiments without departing from the scope of the invention as described by the claims. For example, the order in which the various described method steps are performed may often be changed in alternative embodiments, and one or more method steps may be skipped altogether in other alternative embodiments. Optional features of various device and system embodiments may be included in some embodiments and not in others. Accordingly, the foregoing description is provided primarily for the purpose of illustration and should not be construed as limiting the scope of the invention as set forth in the claims.

The examples and illustrations contained herein show by way of illustration, and not by way of limitation, specific embodiments in which the subject matter may be practiced. As described above, other embodiments may be utilized and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Such embodiments of the inventive subject matter may be referred to herein, individually or collectively, by the term "invention" merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or inventive concept disclosed in fact, if more than one. Thus, although specific embodiments have been illustrated and described herein, any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.