阅读说明：本技术 用于监测用户眼部健康的电子设备及其操作方法
(Electronic device for monitoring eye health of user and operation method thereof
) 是由 克里希纳·基索·加
艾希什·库玛·辛格
马哈玛德拉菲·雷曼萨布·马尼雅
维萨赫·蓬内卡图·奇拉 于 2019-03-26 设计创作，主要内容包括：本文的实施方式公开了用于监测用户的眼部健康的方法和系统,该方法包括确定用户的电子设备周围的环境光。在确定出环境光低于预定阈值的情况下,该方法包括以连续方式和以预配置间隔中的至少一种方式确定显示的至少一个内容的流明输出。基于所确定的流明输出和至少一个用户简档,该方法包括估计用户的瞳孔大小的变化。此外,该方法包括基于所估计的用户瞳孔大小的变化为用户生成眼部健康指数。此外,该方法包括基于用于查看至少一个内容的眼部健康指数将至少一个内容标记为安全的。(Embodiments herein disclose methods and systems for monitoring eye health of a user, the method including determining ambient light around an electronic device of the user. In the event that it is determined that the ambient light is below the predetermined threshold, the method includes determining a lumen output of the at least one content displayed in at least one of a continuous manner and at preconfigured intervals. Based on the determined lumen output and the at least one user profile, the method includes estimating a change in pupil size of the user. Further, the method includes generating an eye health index for the user based on the estimated change in the user's pupil size. Further, the method includes marking the at least one content as safe based on an eye health index for viewing the at least one content.)
1. A method of operating an electronic device (100) for monitoring eye health of a user of the electronic device (100), the method being performed by one or more processors of the electronic device (100), the method comprising:
identifying an ambient light level around the electronic device (100);
identifying, at preconfigured intervals, a lumen output of at least one content to be displayed on a display (104b) of the electronic device (100) based on identifying that the ambient light level is below a first predetermined threshold;
Identifying a change in pupil size of the user based on the lumen output and at least one user profile; and
providing an eye health index to the user based on the identified pupil size change of the user.
2. The method of claim 1, further comprising:
controlling a brightness setting of the display (104b) in accordance with the eye health index of the user.
3. The method of claim 1, further comprising:
based on identifying that the ambient light level is below a second predetermined threshold, displaying at least one label corresponding to each of the at least one content based on the eye health index, wherein the at least one label includes at least one of a security label, an unsafe label, a predetermined range label.
4. An electronic device (100) for monitoring eye health of a user, the electronic device comprising:
a display (104 b); and
one or more processors configured to:
identifying an ambient light level around the electronic device (100);
identifying, at preconfigured intervals, a lumen output of at least one content to be displayed on the display (104b) based on identifying that the ambient light level is below a first predetermined threshold;
Identifying a change in pupil size of the user based on the lumen output and at least one user profile; and
providing an eye health index to the user based on the identified pupil size change of the user.
5. The electronic device (100) of claim 4, wherein the one or more processors are further configured to:
identifying a brightness of the at least one content to be displayed, and a backlight level; and
identifying a lumen output of the at least one content to be displayed based on a relationship between the brightness of the at least one content and the backlight level.
6. The electronic device (100) of claim 5, wherein a lumen output of the at least one content to be displayed is indicative of a total light emitted from the display (104b) and a corresponding brightness change.
7. The electronic device (100) of claim 5, wherein a relationship between the brightness of the at least one content to be displayed and the backlight level is identified as being based on at least a portion of a machine-learned lumen model.
8. The electronic device (100) of claim 4, wherein the pupil size change of the user is identified using a pupil size estimation model identified based on a Holland equation.
9. The electronic device (100) of claim 4, wherein the eye health index of the user is identified using an eye fatigue model identified based on hierarchical regression modeling.
10. The electronic device (100) of claim 9, wherein the eye fatigue model identifies a relationship between an eye health index and at least one of: the identified pupil size change, the at least one user profile, a lumen output of content to be displayed, or at least one activity of the user.
11. The electronic device (100) of claim 4, wherein the one or more processors are further configured to:
controlling a brightness setting of the display (104b) based on the eye health index of the user.
12. The electronic device (100) of claim 4, wherein the one or more processors are further configured to:
based on identifying that an ambient light level is below a second predetermined threshold, displaying at least one tag corresponding to each of the at least one content based on the eye health index for the user, wherein the at least one tag includes at least one of a safe tag, an unsafe tag, a predetermined range tag.
13. The electronic device (100) of claim 4, wherein the one or more processors are further configured to:
displaying, on the display (104b), at least one recommendation related to a display setting based on the eye health index of the user.
14. The electronic device (100) of claim 4, wherein the one or more processors are further configured to:
identifying a new eye health index based on identifying a change in the at least one content to be displayed; and
displaying, on the display (104b), at least one content according to the new eye health index.
15. The electronic device (100) of claim 4, wherein the preconfigured intervals are changeable.
Technical Field
The present disclosure relates to eye health monitoring, and more particularly to monitoring eye health of a user based on background display radiation exposure.
Background
In conventional approaches, the display settings (brightness levels) of the electronic device may be controlled (automatically or manually) based only on ambient light to reduce user eye fatigue.
Disclosure of Invention
Technical problem
Currently, with the proliferation of electronic display devices (e.g., mobile phones, smart phones, tablets, computers, laptops, internet of things (IoT) devices, wearable computing devices, etc.), the time spent by users viewing content has increased significantly. This can lead to asthenopia, also known as asthenopia, ocular weakness or computer vision complications. This may be caused by various reasons, such as prolonged use of the electronic device, fatigue due to viewing in low light environments, exposure to bright light/glare, prolonged activity requiring focus and concentration, continuous gaze in dark environments, and the like. Symptoms may be dry eye, difficulty focusing, irritating eye pain, headache, slow focusing, excess tears, increased light sensitivity, etc.
Technical scheme
It is a primary object of embodiments herein to disclose methods and systems for generating an eye health index for a user based on total light emitted from an electronic device and a change in pupil size of the user, wherein the eye health index is indicative of a health state of the user's eyes.
It is another object of embodiments herein to disclose a method for determining total light emitted by a display (display radiation) of an electronic device using a lumen model.
It is another object of embodiments herein to disclose a method for estimating a pupil size change of a user based on display radiation and a user profile.
It is another object of embodiments herein to disclose a method of controlling configuration/setting of a display based on an eye health index for displaying content.
It is another object of embodiments herein to disclose a method of marking content as safe for viewing based on an eye health index.
It is another object of embodiments herein to disclose a method of providing recommendations to a user based on an eye health index.
Advantageous effects
Accordingly, embodiments herein provide methods and systems for monitoring the ocular health of a user. The method herein includes determining ambient light surrounding an electronic device being used by a user. When the determined ambient light is below the predetermined threshold, the method includes determining a lumen output of at least one content displayed by the electronic device in at least one of a continuous manner and a preconfigured interval. Further, the method includes estimating a change in pupil size of the user based on the lumen output and the at least one user profile. Based on the estimated change in pupil size of the user, the method includes generating an eye health index for the user.
Accordingly, embodiments herein provide an electronic device for monitoring eye health of a user. The electronic device includes a memory, a display module, and an eye health monitoring engine. The eye health monitoring engine is configured to determine ambient light around an electronic device being used by a user. When the determined ambient light is below a predetermined threshold, the eye health monitoring engine is configured to determine a lumen output of at least one content to be displayed by the electronic device at least one of continuously and at predetermined intervals. The eye health monitoring engine is further configured to estimate a change in pupil size of the user based on the lumen output and the at least one user profile. Based on the estimated change in the user's pupil size, the eye health monitoring engine generates an eye health index for the user.
These and other aspects of the exemplary embodiments herein will be better understood when considered in conjunction with the following description and the accompanying drawings. It should be understood, however, that the following description, while indicating exemplary embodiments and numerous specific details thereof, is given by way of illustration and not of limitation. Many changes and modifications may be made within the scope of the exemplary embodiments herein without departing from the spirit of the present application, and the exemplary embodiments herein include all such modifications.
Drawings
Embodiments herein are illustrated in the accompanying drawings, in which like reference numerals refer to corresponding parts throughout the various views. Embodiments herein will be better understood by the following description with reference to the accompanying drawings, in which:
FIG. 1 is a block diagram illustrating various modules of an electronic device for monitoring the health of a user's eyes according to embodiments disclosed herein;
FIG. 2 is a block diagram illustrating various units of an eye health monitoring engine for generating an eye health index, according to embodiments disclosed herein;
FIG. 3 is a flow chart illustrating a method for monitoring the health of a user's eyes according to embodiments disclosed herein;
FIGS. 4A and 4B are exemplary diagrams illustrating generation of an eye health index for a user according to embodiments disclosed herein;
FIG. 5 is an exemplary diagram illustrating displaying content based on an eye health index according to embodiments disclosed herein;
FIG. 6 is an exemplary diagram illustrating indicia based on the content of an eye health index according to embodiments disclosed herein;
FIG. 7 is an exemplary diagram illustrating providing recommendations to a user based on eye health indices in accordance with embodiments disclosed herein;
FIGS. 8A and 8B are example diagrams illustrating monitoring of a user's eye health according to embodiments disclosed herein;
9A, 9B, 9C, and 9D are example diagrams illustrating recordings of activities of a user for monitoring the health of the user's eyes, according to embodiments disclosed herein;
FIG. 10 illustrates an exemplary use case in which eye health status, advice, and recommendations are displayed on an application resident on an electronic device, in accordance with embodiments disclosed herein;
FIG. 11 illustrates an exemplary use case for providing a vision protection mode in accordance with embodiments disclosed herein;
FIG. 12 illustrates an example use case for enabling a soothing User Interface (UI) based on an eye health index, according to embodiments disclosed herein;
fig. 13 illustrates an example use case for providing alerts related to use of a Virtual Reality (VR) device and limiting brightness in the VR device and implementing a soothing UI based on an eye health index, in accordance with embodiments disclosed herein;
14A and 14B are exemplary diagrams illustrating controlling brightness settings on an electronic device based on an eye health index according to embodiments disclosed herein;
15A and 15B are example diagrams illustrating controlling glare emitted from an electronic device based on an eye health index according to embodiments disclosed herein;
FIG. 16 is an exemplary use case illustrating an eye health index based child care mode according to embodiments disclosed herein;
FIG. 17 is a flow chart illustrating an example use case for adapting content color based on eye health index according to embodiments disclosed herein;
FIG. 18 is a flowchart illustrating an example use case for providing suggestions to a user based on eye health indices, according to embodiments disclosed herein;
FIG. 19 is an exemplary diagram illustrating providing recommendations to a user based on an eye health index, according to embodiments disclosed herein; and
FIG. 20 is an example use case illustrating improvement in smart screen readability according to embodiments disclosed herein.
Detailed Description
The exemplary embodiments herein and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques are omitted so as to not unnecessarily obscure the embodiments herein. The description herein is intended merely to facilitate an understanding of ways in which the exemplary embodiments herein may be practiced and to further enable those of skill in the art to practice the exemplary embodiments herein. Accordingly, the disclosure should not be construed as limiting the scope of the exemplary embodiments herein.
Embodiments herein disclose methods and systems for monitoring the health of a user's eyes based on total light emitted from a display of an electronic device. Referring now to the drawings, and more particularly to the exemplary embodiments shown in fig. 1-20, wherein like reference numbers represent corresponding features throughout.
FIG. 1 is a block diagram illustrating various modules of an electronic device 100 for monitoring the health of a user's eyes according to embodiments disclosed herein. The electronic device 100 herein refers to a digital device including at least one display device for displaying content. Examples of the electronic device 100 may be, but are not limited to, a mobile phone, a smartphone, a tablet, a Personal Digital Assistant (PDA), a laptop, a computer, a wearable device, an IoT (internet of things) device, a wearable computing device, a vehicle infotainment system, a medical device, a camera, a Television (TV), a Virtual Reality (VR) device, a vehicle display, and so forth. Content herein refers to at least one of images, videos, animations, text, applications, icons, etc.
Electronic device 100 includes an eye health monitoring engine 102, at least one display module 104, a memory 106, a tagging module 108, and a recommendation module 110. In an embodiment, the ocular health monitoring engine 102 may include at least one of a single processor, multiple processors, multiple homogeneous cores, multiple heterogeneous cores, multiple Central Processing Units (CPUs) of different kinds, and the like. The display module 104 includes a display processor 104a and a display panel/display 104 b. In an embodiment, display 104b may include at least one display device/visual interface suitable for electronic device 100 and capable of displaying content to a user. In an embodiment, the display 104b may include at least one of a Liquid Crystal Display (LCD), a Light Emitting Diode (LED), an organic electroluminescent diode (OLED), a Cathode Ray Tube (CRT), and the like. In an embodiment, the tagging module 108 may be implemented as at least one of a single processor, multiple processors, multiple homogeneous cores, multiple heterogeneous cores, multiple Central Processing Units (CPUs) of different kinds, and the like. In an embodiment, the recommendation module 110 may be implemented as at least one of a single processor, multiple processors, multiple homogeneous cores, multiple heterogeneous cores, multiple Central Processing Units (CPUs) of different kinds, and the like. In an embodiment, the ocular health monitoring engine 102, the tagging module 108, and the recommendation module 110 may be implemented using a single hardware device or multiple hardware devices, e.g., a single processor, multiple processors, multiple homogeneous cores, multiple heterogeneous cores, and multiple CPUs. Alternatively, at least two of the ocular health monitoring engine 102, the tagging module 108, and the recommendation module 110 may be implemented using separate one or more hardware devices, e.g., a single processor, multiple processors, multiple homogeneous cores, multiple heterogeneous cores, and multiple CPUs.
The electronic device 100 may also access at least one of a database (not shown) and the memory 106 to obtain content to be displayed. The electronic device 100 may also connect to a server (not shown) using at least one of the internet, a wired network (local area network (LAN), ethernet, etc.), a wireless network (Wi-Fi network, cellular network, Wi-Fi hotspot, bluetooth, Zigbee, etc.) for accessing content to be displayed to the user. Electronic device 100 may monitor the health of the user's eyes in response to user interaction with electronic device 100 or at least one of user-provided commands, actions, gestures, and the like. In an embodiment, a device such as a server (not shown) may be implemented for monitoring the health of the user's eyes by collecting information from the electronic device 100. In another embodiment, the server may be at least one of a remote server, a cloud server, and the like.
Electronic device 100 may also be coupled with a device, such as, but not limited to, at least one camera, iris scanner, sleep tracking device, etc., for monitoring user interaction with electronic device 100, user activity/sleep patterns, etc. Further, the monitored interactions of the user with the electronic device 100 and the activities of the user may be provided to the ocular health monitoring engine 102.
Eye health monitoring engine 102 may be configured to generate an eye health index when ambient light surrounding electronic device 100 is below a predetermined threshold. To generate the eye health index, the eye health monitoring engine 102 identifies and interprets light emitted from the display 104b (display radiation) of the electronic device 100 over a predetermined period of time (which may be a long period of time) in different environments including day, night, low light, time of day, age of the user, health of the user, and so forth. Eye health monitoring engine 102 uses the lumen model to identify the total light emitted from display 104b of electronic device 100. Based on all of the identified emitted light and user profile data, the ocular health monitoring engine 102 uses a pupil size estimation model to estimate the change in pupil size (pupil dilation change) of the user. Based on the estimated change in pupil size of the user, the user profile, and the user's activities, the ocular health monitoring engine 102 generates an ocular health index for the user using an eyestrain model. The eye health index indicates a health state of the user's eyes.
In an embodiment, the ocular health monitoring engine 102 may estimate an expected eye strain (ocular health index) due to a difference between previously displayed content and new content to be displayed.
The display processor 104a may be configured to control the configuration/setting of the display 104b based on the eye health index. In an embodiment, the display processor 104a may increase/decrease the brightness level of the display 104 b. In another implementation, the display processor 104a may enable a soothing User Interface (UI)/lighting/color scheme for displaying content. Thus, the strain on the eyes of the user is reduced. The display 104b may be configured to display content on the display 104b according to a configuration set by the display processor 104 a.
The tagging module 108 may be configured to assign tags to content for viewing. The marking module 108 scans the content (available/stored on the electronic device(s) 100) and feeds the content to the eye health monitoring engine 102 to generate an eye health index for the scanned content using the lumen model, the pupil size estimation model, and the eye fatigue model. The labeling module 108 analyzes the eye health index generated for each scanned content (content to be displayed) and assigns a label to each scanned content for viewing. The tags may be, but are not limited to, security tags, non-security tags, predefined control range tags, and the like. The security tag indicates that the content can be safely viewed in low light. Unsafe labels indicate the possibility of increasing eye strain when viewing content in low light. The predefined control range tags indicate requirements for modification of the configuration of the display 104b to enable a user to view content in low light in a non/minimal eyestrain manner.
The recommendation module 110 may be configured to provide recommendations to the user based on the eye health index of the user. In an embodiment, the recommendation module 110 recommends at least one of these, but not limited to media, text, applications, settings/configuration changes, other parameters/content, and the like, based on the eye health index of the user. In another embodiment, the recommendation module 110 recommends at least one of these, but not limited to font size, color theme, etc., based on the eye health index of the user. Thus, the advice provided based on the eye health index may reduce stress on the user's eyes. In another embodiment, the recommendation module 110 suggests a device/clinic based ophthalmic exam/visit ophthalmologist to the user based on the user's eye health index.
Memory 106 may be configured to store content, user profiles, information regarding display radiation emitted from electronic device 100, as well as pupil size changes, eye health indices of the user, and the like. Memory 106 may include one or more computer-readable storage media. The memory 106 may include non-volatile storage elements. Examples of non-volatile storage elements may include magnetic hard disks, optical disks, floppy disks, flash memory, or forms of electrically programmable memory (EPROM) or Electrically Erasable and Programmable (EEPROM) memory. Further, in some examples, memory 106 may be considered a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or propagated signal. However, the term "non-transitory" should not be construed to mean that the memory 106 is not removable. In some examples, the memory 106 may be configured to store a greater amount of information than memory. In some examples, a non-transitory storage medium may store data that may change over time (e.g., in Random Access Memory (RAM) or a cache).
Fig. 1 shows exemplary units of an electronic device 100, but it should be understood that other embodiments are not limited thereto. In other embodiments, electronic device 100 may include a fewer or greater number of units. Furthermore, the labels or names of the elements are for illustrative purposes only and do not limit the scope of the embodiments herein. One or more units may be combined together to perform the same or substantially similar functions in the electronic device 100.
Fig. 2 is a block diagram illustrating various units of the ocular health monitoring engine 102 for generating an ocular health index for a user according to embodiments disclosed herein. Eye health monitoring engine 102 includes a processing unit 202, a lumen output estimation unit 204, a pupil size estimation unit 206, and an eye health measurement unit 208.
The processing unit 202 may be configured to determine the ambient light (ambient lux level/ambient light level). The ambient light includes at least one of low/dark light, eye light, night light, ambient light, and the like. In an embodiment, the processing unit 202 may determine the ambient light using an ambient sensor. In the event that it is determined that the ambient light is below the predetermined threshold (low light condition), the processing unit 202 calculates histogram data of the content to be displayed. In an embodiment, the processing unit 202 performs periodic sampling of content to calculate histogram data for the content. The processing unit 202 also identifies one or more display characteristics of the content (e.g., color temperature, bit depth, pixel color distribution, etc.), the backlight of the display 104b, and the display panel profile (resolution, panel type, etc.). The processing unit 202 provides the histogram data, the display characteristics of the content, the backlight and the display panel profile to the lumen output estimation unit 204.
The lumen output estimation unit 204 may be configured to calculate the total light emitted from the display 104b of the electronic device 100 (the brightness (lumen) output of the display radiation/content) and the corresponding brightness variation. Embodiments herein use the terms "display radiation," "luminance output of content," "total light emitted by content," "total brightness," etc. interchangeably and refer to the total light emitted from display 104 b. In an embodiment, the lumen output estimation unit 204 uses a lumen model to calculate the total light emitted from the display 104 b. The lumen output estimation unit 204 uses the histogram data, the display characteristics of the content, the backlight and the display panel profile as inputs to the lumen model. The lumen model calculates the total light/total brightness emitted from the display 104 b. In an embodiment, the total light emitted from display 104b is due to a combination of content (including display characteristics of the content) and backlight level (brightness level) on display 104 b.
To estimate the total light emitted from the display 104b, the lumen model establishes a relationship between the sub-pixels of the content and the corresponding brightness at a given backlight. The sub-pixels may constitute pixels of the content, wherein the sub-pixels herein refer to at least one of red, green, and blue. Each sub-pixel may have 256 levels of color. Zero means black and 255 means a fully saturated pure color. Initially, the lumen model analyzes 64 colors out of 1600 ten thousand colors (4 levels per sub-pixel, and a level can be obtained by dividing the total luminance range of a sub-pixel into 3 equal parts). Without the red sub-pixel, the lumen model observes the luminance behavior of green and blue (sub-pixels) as linear. The luminance behavior of the green and blue sub-pixels can be expressed as:
[ mathematical diagram 1]
L(O，G，B)＝1.002*L(G)+0.748*L(B)
Where L is the luminance function, G and B are the green and blue sub-pixel levels, and the above equation can be obtained from a linear regression of the luminance behavior.
Once the linear relationship of pixel brightness with green and blue sub-pixel brightness is obtained, the lumen model adds the red sub-pixel to the mathematical fig. 1 and observes the same linear relationship/brightness behavior. The luminance behavior with the addition of a red sub-pixel can be expressed as:
[ mathematical FIG. 2]
L(R，G，B)＝1.243*L(R)+0.9462*L(O，G，B)＝1.234*L(R)+0.948*L(G)+0.708*L(B)
Furthermore, the lumen model observes a similar linear relationship or luminance behavior without the green sub-pixel as the combination of the red and blue sub-pixels. The luminance behavior of the red and blue sub-pixels can be expressed as:
[ mathematical FIG. 3]
L(R，O，B)＝1.177*L(R)+1.182*L(B)
The same linear relationship/luminance behavior is observed for the lumen model even with the addition of the green sub-pixel. The luminance behavior with the addition of a green sub-pixel can be expressed as:
[ mathematical FIG. 5]
L(R，G，B)＝0.688*L(R)+0.772*L(G)＝0.913*L(G)+1.078*L(R，G，B)＝1.269*L(R)+0.913*L(G)+1.274*L(B)
Similarly, the same linear relationship or luminance behavior is observed for the lumen model for the combination of red and green without the blue sub-pixel. The luminance behavior can be expressed as:
[ mathematics figure 6]
L(R，G，B)＝2.515*L(B)+1.185*L(R，G，O)＝0.816*L(R)+0.932*L(G)+2.515*L(B)
From the mathematical figures 1-6, the lumen model infers that there is a linear relationship between the pixel brightness and the brightness of the individual sub-pixels of the content. Thus, the lumen model applies a linear regression, with the luminance of each sub-pixel as the predictor variable and the luminance of the corresponding pixel as the output variable. The equation of the luminance of each pixel obtained by applying linear regression can be expressed as:
[ mathematical FIG. 7]
L(R，G，B)＝1.011*L(R)+0.896*L(G)+2.332*L(B)
In an embodiment, the lumen model may use machine learning based linear regression to find the best fit line for the output luminance of a pixel for a given luminance value of a sub-pixel.
Furthermore, the pixels present on the content may constitute the total light emitted by the display 104 b. The lumen model calculates the average luminance value of the pixels present on the content as the total luminance of the content.
The luminance of each pixel can be expressed as:
[ mathematical FIG. 8]
L(RGB)＝1.072*L(R)+0.9148*L(G)+2.3828*L(B)
The pixel luminance with the red sub-pixel can be expressed as:
[ mathematics figure 9]
L(R)＝0.0007*R2-0.0223*R+0.5378
The pixel luminance with the green sub-pixel can be expressed as:
[ mathematical FIG. 10]
L(G)＝0.0035*G2-0.1112*G+2.1628
The luminance of the blue pixel can be expressed as:
[ mathematical FIG. 11]
L(B)＝0.0002*B2-0.0121*B+0.4412
The brightness of the content may be expressed as:
[ mathematical FIG. 12]
Wherein p1, p2, p3, …, pn may be pixels existing in the content, and n1, n2, n3, …, nn may represent the number of these pixels, respectively.
The total light emitted from the display 104b depends on the brightness of the content and the backlight. The total light emitted from the display 104b calculated based on the brightness and backlight of the content may be expressed as:
[ mathematical FIG. 13]
Where Crk, CGk, and Cbk represent the counts of red, green, and blue subpixels at level k, and L (rk), L (gk), and L (bk) represent the luminance values of the red, green, and blue subpixels at backlight level k.
Further, the lumen model converts the brightness of the content from various backlight levels of the electronic device 100 to the current backlight level. For example, consider that the determined backlight level may be 160. The lumen model converts the brightness of the content from the backlight level 160 to the current backlight level of the electronic device 100. The lumen model specifies the relationship between the brightness of the content at different backlight levels. The formulaic relationship between the brightness of the content at different backlight levels can be expressed as:
[ mathematical FIG. 14]
[ mathematical FIG. 15]
Y＝314.9x-1.134
Where x may be the current backlight brightness level of the electronic device 100 and Y may be a division factor for determining the desired content brightness for the backlight brightness level x.
Further, the total light emitted from the display 104b may be denoted as L (content)/Y. Thus, the lumen model uses a combination of the brightness of the content and the backlight to determine the total light emitted from the display 104 b.
The pupil size estimation unit 206 may be configured to estimate a change in pupil size of the user. In an embodiment, the pupil size estimation unit 206 estimates a change in the pupil size (pupil dilation variance) of the user using a pupil size estimation model/hierarchical regression model. Thus pupil size depends on accommodation brightness, accommodation field area (visual field area), observer/user age, monocular stimulation, binocular stimulation, etc. The pupil size estimation unit 206 provides the determined display radiance and the user profile as input variables of a pupil size estimation model. The user profile may include information such as, but not limited to, the user's age, the user's eye profile, iris scan data, and the like. In an embodiment, iris scan data may be collected using a device such as an iris scanner, camera, or the like. In embodiments, the user profile may also include information about the user such as, but not limited to, mental activity, emotional arousal, contrast, recognition, attention, and the like.
Upon receiving the input variables, the pupil size estimation model uses the Holladay (holla) formula as a reference for representing the relationship between the input variables, and estimates the pupil diameter of the estimated light emitted from the display 104 b. The pupil diameter estimated using the charla formula can be expressed as:
[ mathematical FIG. 16]
D(in mm)＝7*Expo(-0.1007*L^0.4)
Where D represents the estimated pupil diameter and L represents the luminance of the content.
In another embodiment, the pupil size estimation model may consider the adaptation field area (field of view area) to estimate the pupil diameter. The pupil diameter estimated using the Holland equation based on the adaptive field area can be expressed as:
[ mathematical FIG. 17]
Where D denotes the estimated pupil diameter, L denotes the luminance of the content, and a denotes the adaptation field area in m2 units.
Once the pupil diameter for the determined total light emitted from display 104b is estimated, the pupil size estimation model calculates the difference between the estimated pupil diameter and the pupil diameter obtained from the user profile. Based on the calculated difference, the pupil size estimation model estimates a change in the pupil size of the user.
The eye health measurement unit 208 may be configured for generating an eye health index of the user. In an embodiment, the eye health measurement unit 208 uses an eye fatigue model to generate an eye health index for the user. The eye health measurement unit 208 supplies the determined pupil size change, the user profile, the activity data of the user, the display radiation emitted by the electronic apparatus 100, and the like as input variables to the eye fatigue model. The user's activity data includes information such as, but not limited to, user sleep patterns, electronic device interaction data, and the like. In an embodiment, an infrared sensor/front camera of electronic device 100 may be used to determine user interaction with electronic device 100. In another embodiment, the display processor 104a may display a pop-up message to check the user's activities by the electronic device 100. In another implementation, the display processor 104a can record gestures (touches, swipes, taps, clicks, drags, etc.) performed by the user on the electronic device 100. In an embodiment, the sensor device may be used to record a sleep pattern of the user.
The eyestrain model performs a subjective evaluation of the input variables for an estimated change in the pupil size of the user. Based on the subjective evaluation of the input variables, the eyestrain model generates an eye health index (eye health state/eyestrain data) representing the health state of the user's eyes. In an embodiment, the eye fatigue model generates an eye health index over the past 24 hours that represents eye fatigue data over the past 24 hours. In another embodiment, the eyestrain model generates an overall eye health index that represents eyestrain data from the beginning of using electronic device 100. In another embodiment, the ocular health index may vary from a level 0 (LVL0) to a level 10 (LVL 10). LVL0 represents minimal eyestrain and LVL10 represents maximal eyestrain.
Fig. 2 shows exemplary elements of the ocular health monitoring engine 102, but it should be understood that other embodiments are not so limited. In other embodiments, the ocular health monitoring engine 102 may include a fewer or greater number of units. Furthermore, the labels or names of the elements are for illustration purposes only and are not intended to limit the scope of the embodiments herein. One or more cells may be combined together to perform the same or substantially similar functions in ocular health monitoring engine 102.
Fig. 3 is a flow chart illustrating a method for monitoring eye health of a user according to embodiments disclosed herein.
In step 302, the method includes determining, by the eye health monitoring engine 102, a level of ambient light surrounding the electronic device 102 being used by the user. The eye health monitoring engine 102 may use an environmental sensor to determine the ambient light level.
In step 304, the method includes determining, by the eye health monitoring engine 102, a lumen output of the displayed content (the display radiation emitted from the electronic device 100) at least one of continuously and at preconfigured intervals when the determined ambient light is below a predetermined threshold. In an embodiment, eye health monitoring engine 102 may identify a lumen output of at least one content displayed on a display of electronic device (100) at a consistent preconfigured interval. In another embodiment, the eye health monitoring engine 102 identifies that the interval of lumen output may be varied. For example, the interval may be preconfigured as a function of time. In another example, the interval may be preconfigured according to a state of the electronic device, such as a battery state or a device temperature. In the event that it is determined that the ambient light is below the predetermined threshold, the eye health monitoring engine 102 determines histogram data of the content to be displayed, display characteristics of the content, the backlight of the display 104b, a display panel profile, and the like. Eye health monitoring engine 102 provides the determined histogram data, display characteristics of the content, backlight and display panel profiles as input variables for the lumen model. The lumen model calculates a lumen output of the content to be displayed for the determined ambient light. The lumen model calculates a lumen output based on the brightness of the content and the backlight. To calculate the brightness of the content, the lumen model interprets the pixels and sub-pixels of the content and establishes a relationship between the sub-pixel (at least one of red, green and blue) level and its corresponding brightness at the determined backlight. Further, the lumen model establishes a relationship between the combined luminance of two sub-pixels (at least one of red, green and blue) of the content. Thereafter, the lumen model considers the third sub-pixel (at least one of red, green and blue) and builds the relationship of the pixel intensities as a function of the individual sub-pixels. The lumen model determines an average of the luminance of all pixels present in the content to determine the luminance of the content. Further, the lumen model establishes a relationship between the backlight level and the brightness of the content to determine the lumen output of the content/total light emitted from the display 104 b. In an embodiment, the lumen model applies a machine learning based linear regression to the input variables to establish the relationship of the luminance of the pixels as a function of the individual sub-pixels and the relationship between the backlight level and the content luminance. Thus, the lumen output of the content/the total light emitted from the display 104b may be calculated based on the RGB (sub-pixel) level and the backlight level of the content.
In step 306, the method includes estimating, by the eye health monitoring engine 102, a change in pupil size of the user using the lumen output and the user profile. Eye health monitoring engine 102 provides the calculated lumen output of the content and the user profile to a pupil estimation model for estimating changes in pupil size of the user. In an embodiment, the pupil estimation model uses the Holladay formula for estimating the change in the user's pupil size based on the change in the user's pupil size and the lumen output of the content/total light emitted from the display 104 b.
In step 308, the method includes generating, by the eye health monitoring engine 102, an eye health index for the user based on the estimated change in the user's pupil size. The eye health monitoring engine 102 uses the eyestrain model to generate an eye health index based on the user's changes in pupil size, the user profile, the lumen output, and the user's activities.
The various actions, modules, steps, etc. in method and flowchart 300 may be performed in the order presented, in a different order, or concurrently. Moreover, in some embodiments, some of the acts, modules, steps may be omitted, added, modified, skipped, or the like, without departing from the scope of the present invention.
Fig. 4A and 4B are exemplary diagrams illustrating generation of an eye health index for a user according to embodiments disclosed herein. The processing unit 202 of the eye health monitoring engine 102 calculates histogram data of the content to be displayed. Based on the histogram data, the lumen output estimation unit 204 of the eye health monitoring engine 102 determines the total light emitted from the display 104b (lumen output of content) using a lumen model. The pupil size estimation unit 206 of the eye health monitoring engine 102 uses the pupil size estimation model, the lumen output of the content, and the user profile to estimate the change in pupil size of the user. In an embodiment, the pupil size estimation model may be automatically corrected by acquiring periodic inputs from a device coupled to the electronic device 100, such as an iris scanner or a camera. Based on the estimated change in the pupil size of the user, the eye health measurement unit 208 of the eye health monitoring engine 102 generates an eye health index using the eye fatigue model. In examples herein, the eye health index may be displayed in a needle shape on a semi-circular chart.
Fig. 5 is an exemplary diagram illustrating content displayed based on an eye health index according to embodiments disclosed herein. Embodiments herein enable the eye health monitoring engine 102 to determine the lumen output of content based on histogram data in low light levels. For example, the lumen output of the content to be displayed may be LUX 10. Further, eye health monitoring engine 102 estimates a change in pupil size of the user based on the lumen output of the content. Based on the estimated change in the user's pupil size, the eye health monitoring engine 102 estimates an eye health index for the user. The display processor 104a adjusts the brightness level of the display 104b based on the eye health index. For example, consider that the eye health index shifts due to a pupil size change of 0.7mm (from 5.7mm to 5 mm). Based on the eye health index, display processor 104a performs a dark-to-light transition of content by increasing the brightness of display 104b so that the lumen output of the content can be increased to LUX 20. Thus, the transition of content from dark to light may reduce fatigue (or headaches) of the user's eyes, and may provide a soothing effect to the user's eyes.
Fig. 6 is an example diagram illustrating marking content as safe for viewing based on an eye health index according to embodiments disclosed herein. Embodiments herein scan and analyze video/content stored on memory 106 that may be available for viewing. The eye health monitoring engine 102 estimates the pupil dilation variance (change in pupil size of the user) of the content to be viewed over the period by applying a lumen model and a pupil size estimation model. Based on the estimated pupil dilation variance, the eye health monitoring engine 102 generates an eye health index using an eye fatigue model. The tagging module 108 may tag the content as safe/unsafe for low light viewing based on the eye health index. For example, as shown in FIG. 6, a security tag is assigned to a video for viewing the video in low light conditions.
Fig. 7 is an exemplary diagram illustrating providing recommendations to a user based on an eye health index according to embodiments disclosed herein. Embodiments herein enable the eye health monitoring engine 102 to generate an eye health index for a user based on display radiation emitted from the electronic device 100. Based on the generated eye health index, the recommendation module 110 recommends at least one of content/video/channel, application, configuration/setting changes, font size/color theme, brightness, etc. to reduce user eye fatigue. Further, the recommendation module 110 may provide recommendations to the user for device/clinic based eye testing, visiting an ophthalmologist, and the like.
Fig. 8A and 8B are exemplary diagrams illustrating monitoring eye health of a user according to embodiments disclosed herein.
Fig. 8A shows an example graph of eye health index generation. As shown in fig. 8A, the eye health monitoring engine 102 performs periodic sampling of the content to be displayed and calculates histogram data of the content to be displayed. Eye health monitoring engine 102 calculates the lumen output (display radiation/total display light) of the content using a lumen model. The lumen model receives information such as, but not limited to, histogram data, ambient light around the electronic device 100, display panel profiles, etc. to calculate lumen output. Based on the received information, the lumen model determines a sub-pixel light intensity response and applies linear regression to the determined sub-pixel light intensity response to predict a linear model. The linear model predicts the pixel light response aggregate/lumen output.
In addition, the ocular health monitoring engine 102 uses a pupil size estimation model to estimate changes in the user's pupil size. The pupil size estimation model estimates the change in the user's pupil size based on the lumen output and the user profile (user age, iris scan data, eye health profile, etc.). Based on the estimated change in the user's pupil size, the ocular health monitoring engine 102 uses an eye fatigue model to generate an ocular health index. The eye fatigue model uses the estimated change in user pupil size, the lumen output, the user profile, and the user activity to calculate an eye health index. In an embodiment, the eye fatigue model may be corrected using at least one of the user's activities (sleep mode, input/feedback from devices such as iris scanners, cameras, etc.) to generate an eye health index. In examples herein, the eye health index may include a health state of the user's eyes, such as, but not limited to, a general state, a transitional state, an alarm state, a severe state, and the like. Based on the asthenopia data/ocular health index over a period of time, the ocular health monitoring engine 102 may provide the user with a binocular health index metric (ocular health index of the last 24 hours and overall ocular health index). Further, the ocular health monitoring engine 102 may provide the user with instantaneous eye strain data.
Fig. 8B illustrates the ocular health monitoring engine 102 for monitoring the ocular health of a user. As shown in fig. 8B, the eye health monitoring engine 102 initially performs periodic sampling of the display content histogram, the backlight level of the display, pupil size, VR usage, blink rate, display brightness, and content histogram (in low light situations), among others.
The eye health monitoring engine 102 then statistically converts the sampled data, including determining display radiation, pupil and blink variations, emitted from the display panel 104 b. The eye health monitoring engine 102 may use the display panel profile, histogram, and backlight data to calculate the brightness change. The eye health monitoring engine 102 calculates pupil dilation and constriction changes. Further, the ocular health monitoring engine 102 models ocular fatigue using a hierarchical regression model, which may include a self-correcting technique with a processed sampled data set. Then, the eye health monitoring engine 102 predicts the eye health (eye health index) of the user in consideration of the user profile.
Fig. 9A, 9B, 9C, and 9D are example diagrams illustrating user activity records for monitoring the health of a user's eyes according to embodiments disclosed herein. In an exemplary embodiment, a front camera of the electronic device 102 may be used to determine whether the user is viewing the display (recording the user's activities). When no user activity is detected, as shown in fig. 9A, embodiments herein turn off the electronic device 102. In another embodiment, as shown in FIG. 9B, an infrared sensor may be used to detect the presence of a user in the vicinity of the electronic device 100. In another embodiment, a sleep tracking device as shown in fig. 9C and 9D may be used to track the sleep patterns of a user. In addition, a pop-up message may be displayed to the user to check the user's liveness of the electronic device 100. Further, the last gesture (touch, drag, click, tap, etc.) performed on the electronic device 100 may be recorded. Accordingly, embodiments herein use at least one of a camera, an infrared sensor, a sleep tracking device, etc. of the electronic device 100 to determine the activity of the user. The user's activity may be, but is not limited to, a period of time that the user is viewing content displayed on electronic device 100, use of electronic device 100 by the user in a dark or non-lit state at night, use of electronic device 100 by the user holding electronic device 100 too close to the eyes, and so forth. Thus, the recorded user activity may be used to predict the health state of the user's eyes.
Fig. 10 illustrates an example use case in which eye health status, advice, and recommendations are displayed on an application program resident on the electronic device 100, according to embodiments disclosed herein. Embodiments herein enable the ocular health monitoring engine 102 to predict ocular health status based on sampled ambient light levels around the electronic device 100, display radiation profiles, user profiles, and the like. Based on the predicted ocular health status, recommendation module 110 may display ocular health status, suggestions/hints, recommendations to the user. Therefore, eye fatigue can be reduced.
Fig. 11 is a block diagram illustrating an example use case for providing a vision protection mode according to embodiments disclosed herein. Embodiments herein provide a vision protection mode to enable at least one feature present on the electronic device 100, such as, but not limited to, a content tagging feature, a soothing UI feature, a child care feature, a color adaptation feature, and the like, based on an eye health index. When the content tagging feature is enabled, the tagging module 108 may assign tags for secure viewing to content present on the electronic device. The label may be displayed in the application based on ambient light (e.g., night/day/ambient light, etc.). The label may be a dynamic label having a color for displaying the expected fatigue strength. When the soothing UI feature is enabled, the display processor 104a may enable a soothing UI based on the eye health index, which further provides relief to the user's eyes. When the child care feature is enabled, the display processor 104a may adjust the display control to reduce the impact on eye health by monitoring the child exposed to the display radiation. When the color adaptation feature is enabled, the display processor 104a recommends text colors and background colors to the user to reduce fatigue of the user's eyes.
Fig. 12 illustrates an example use case for enabling a soothing UI based on an eye health index according to embodiments disclosed herein. Embodiments herein provide ambient light, content to be displayed, and a pre-generated eye health index as inputs to the eye health monitoring engine 102. The eye health monitoring engine 102 compares the ambient light to a predetermined threshold for display content. When the ambient light is greater than the predetermined threshold, the ocular health monitoring engine 102 provides the content to the display 104b for display without any modification.
When the ambient light level is less than the predetermined threshold, the eye health monitoring engine 102 instructs the display processor 104a to generate a new frame of lumen output similar to the previous frame using the lumen model applied to the received input. As shown in fig. 12, based on the eye health index, the relaxing UI can perform a content transition from dark to light (the brightness may change to LUX 20). Thus, the optical adaptation of the eye that may occur due to the change of the content/display can be eliminated. Furthermore, the enablement of the soothing UI provides relief to the eyes of the user and reduces headaches/fatigue.
Fig. 13 illustrates an example use case for providing alerts related to use of a Virtual Reality (VR) device 100 (electronic device 100) and masking brightness in the VR device and enabling a soothing UI based on an eye health index, according to embodiments disclosed herein. Embodiments herein enable the ocular health monitoring engine 102 to monitor eye strain caused by continuous use of the VR device 100. In an embodiment, based on the monitored eye strain, the recommendation module 110 may display a warning to the user. In another embodiment, based on the monitored eye strain, the display processor 104a may limit the brightness in the VR device 100. In another embodiment, based on the monitored eye strain, the display processor 104a may enable a soothing UI to eliminate light adaptation of the eyes due to changes in content.
Fig. 14A and 14B are exemplary diagrams illustrating controlling a brightness setting of the electronic device 100 based on an eye health index according to embodiments disclosed herein. The eye health monitoring engine 102 adjusts the brightness of the display panel 104b based on the eye health index, as shown in fig. 14A. Further, the eye health monitoring engine 102 determines the maximum brightness upper limit based on the eye health index, as shown in fig. 14B. The upper brightness limit may be applicable to manual and automatic brightness settings. Therefore, determining the maximum brightness upper limit and adjusting the brightness may prevent further deterioration of the eye health of the user.
Fig. 15A and 15B are example diagrams illustrating controlling glare emitted from the electronic device 100 based on an eye health index according to embodiments disclosed herein. Sudden turn-on of the electronic device 100 under low light conditions may generate glare that stimulates the user. In this case, the display processor 104a may adjust the display light based on the health status of the user's eyes. Consider a scenario as shown in fig. 15A, in which the eye health status of the user is a general status. The display module 104 may display the content without adjusting the display settings (so the brightness increase time is negligible). Thus, the electronic device 100 can be switched directly from the OFF state to the ON state without any modification.
Consider another scenario, as shown in fig. 15B, where the eye health status of the user is a severe status. The calculated increase in brightness for severe eye health status may be 10 seconds. Thus, the display module adjusts the display light rather than directly switching the electronic device 100 from the OFF state to the ON state. Accordingly, glare generated by the sudden opening of the electronic device 100 may be controlled, which further reduces eye fatigue of the user.
Fig. 16 illustrates an exemplary use case of an eye health index based child care mode according to embodiments disclosed herein. The eye health monitoring engine 102 may estimate the age of the user. In an embodiment, the eye health monitoring engine 102 may estimate the age of the user using input from at least one of an iris scanner, a camera, or the like. The eye health monitoring engine 102 compares the estimated age to a predetermined threshold. Upon determining that the estimated age is below the predetermined threshold, the eye health monitoring engine 102 identifies the user as a child and monitors exposure of the child to display radiation. Based on the monitored exposure of the child to the display radiation, the ocular health monitoring engine 102 generates an ocular health index. Based on the eye health index, the display processor 104a may adjust the display control to reduce the eye health impact. The display control may be at least one of content whiteness control, linear age dependent brightness control, dynamic brightness reduction per 5% eye health degradation, and the like.
FIG. 17 illustrates an exemplary use case for adjusting content color based on eye health index according to embodiments disclosed herein. The display processor 104a may transform the color of text and background (present in the content) according to the eye health index. In the examples herein, based on the eye health index, the display processor 104a may transform the color of the text and background, as shown in fig. 17, for placement on the display 104. In an embodiment, the display processor 104a may convert the color of the text to red. The insensitivity of rod cells (rod) to red light leads to the use of red light in certain special cases; for example in a submarine control room, in a research laboratory, an aircraft or during naked eye astronomy, since no eye accommodation needs to be changed.
FIG. 18 illustrates an example use case for providing suggestions to a user based on eye health indices in accordance with embodiments disclosed herein. Upon determining that the user's eye health index is transitioning, recommendation module 110 may provide recommendations such as, but not limited to, a recommendation to activate a 20-20 eye rule (20 seconds off every 20 minutes and focus the eyes on something at least 20 feet away), a recommendation to decrease the brightness by at least a certain amount (e.g., 10%), a recommendation to wash the eyes after 2 hours, and the like.
Upon determining that the user's eye health index is being alerted, recommendation module 110 may provide a recommendation, such as, but not limited to, a recommendation for activating a 20-20 eye rule, a recommendation for reducing brightness by at least 20%, a recommendation for washing the eyes after 1 hour, and the like. Further, upon determining that the user's eye health index is being alerted, the display processor 104a may enable a soothing UI feature and a smart screen readability feature (as shown in fig. 20) and the like on the electronic device 100.
Upon determining that the user's eye health index is severe, recommendation module 110 may provide recommendations, such as, but not limited to, a recommendation for activating a 20-20 eye rule, a recommendation for reducing brightness by at least 35%, a recommendation for washing the eyes after 30 minutes, and the like. Further, upon determining that the eye health index of the user is severe, the display module 104 may enable a soothing UI feature and a smart screen readability feature on the electronic device 100 (as shown in fig. 20), enable a content color adaptation feature (as shown in fig. 17), and the like.
Fig. 19 is an exemplary diagram illustrating providing suggestions to a user based on an eye health index according to embodiments disclosed herein. As shown in fig. 19, ocular health monitoring engine 102 scans the content (stored in memory or accessed from a server/database, one or more electronic devices, etc.) and estimates the expected eye strain. Based on the estimated expected eye strain and the current eye health state, the recommendation module 110 may recommend appropriate videos, applications, settings (brightness, wallpaper, etc.), etc. to the user.
FIG. 20 is an example user scenario illustrating improvement in smart screen readability according to embodiments disclosed herein. If the automatic brightness mode on the electronic device 100 is turned off, the user may be faced with the difficulty of strong sunlight when changing brightness. Thus, the eye health monitoring engine 102 collects data from the environmental sensors and calculates the total display light. Based on the total display light, the eye health monitoring engine 102 checks whether the screen can be read by the current display settings. Upon determining that the screen is unreadable by the current display setting, the eye health monitoring engine 102 instructs the display processor 104b to arrange the brightness change icon on the display, which further displays the brightness change icon with high contrast. Therefore, power consumption can be reduced and convenience of manual brightness setting by a user can be protected.
Embodiments disclosed herein may be implemented by at least one software program running on at least one hardware device and performing network management functions to control elements. The elements shown in fig. 1 and 2 may be at least one of a hardware device or a combination of a hardware device and a software module.
Embodiments disclosed herein describe methods and systems for monitoring the ocular health of a user. It will thus be appreciated that the scope of protection is extended to a program which, in addition to computer readable means having messages therein, also comprises program code means for performing one or more steps of the method when the program is run on a server or a mobile device or any suitable programmable device. In a preferred embodiment, the method is implemented by or together with a software program written in another programming language, for example very high speed integrated circuit hardware description language (VHDL), or by one or more VHDL or several software modules executed on at least one hardware device. The hardware device may be any type of portable device that can be programmed. The apparatus may further comprise means which may be, for example, a hardware device such as an ASIC, or a combination of hardware and software devices such as ASICs and FPGAs, or at least one microprocessor and at least one memory having software modules located therein. The method embodiments described herein may be implemented partly in hardware and partly in software. Alternatively, the present invention may be implemented in different hardware devices, for example, using multiple CPUs.
The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and/or adapt for various applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Thus, while the embodiments herein have been described in terms of embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modification within the spirit and scope of the embodiments described herein.
