阅读说明：本技术 控制系统和渲染管道 (Control system and rendering pipeline ) 是由 A·M·伊什 A·J·格罗斯 B·颜 E·C·N·唐 J·R·巴特尔 W·C·韦尔奇三世 于 2020-01-06 设计创作，主要内容包括：一种系统,其包含：渲染引擎,其用以渲染场显示器的场纹理和可操纵中央凹显示器的中央凹纹理；以及合成器,其包含用以根据所述场纹理产生用于所述场显示器的帧的场合成器和用以根据所述中央凹纹理产生用于所述中央凹显示器的帧的中央凹合成器。所述系统进一步包含合成管理器,所述合成管理器被设计成对所呈现的包含所述场显示器和所述中央凹显示器中的一个或多个数据的内容进行排序和选择。(A system, comprising: a rendering engine to render field textures of the field display and foveal textures of the steerable foveal display; and a compositor comprising an original compositor to generate frames for the field display from the field texture and a foveal compositor to generate frames for the foveal display from the foveal texture. The system further includes a composition manager designed to order and select content presented including one or more data in the field display and the foveated display.)

1. A system, comprising:

a foveated display to locate a steerable display image having a first Pixel Per Degree (PPD) resolution and having a first field of view;

a field display to provide a field display image having a second resolution, the field display image having a PPD lower than the first resolution and a second field of view greater than the first field of view;

a rendering engine to render a field texture of the field display and a foveal texture of the foveal display;

a combiner including an original combiner to generate frames for the field display according to the field texture and a foveal combiner to generate frames for the foveal display according to the foveal texture;

a composition manager designed to order and select content presented that contains one or more data in the field display and the foveated display.

2. The system of claim 1, further comprising:

the composition manager is to trigger a cutout from the field display when the foveal display is positioned to fill the cutout.

3. The system of claim 2, wherein the cut-out from the field display is populated with one or more data from:

current foveal texture data;

previous foveal texture data; and

field texture data.

4. The system of claim 3, wherein the field texture data is resampled field texture data.

5. The system of claim 1, wherein the rendering engine and the compositor are asynchronous.

6. The system of claim 5, wherein the foveated synthesizer and the field synthesizer are asynchronous.

7. The system of claim 1, wherein the rendering engine and the compositor are synchronized.

8. The system of claim 1, further comprising:

upon determining that the foveal display should be repositioned, the foveal display is quickly stopped before moving the foveal display and is re-enabled after securing the foveal display to allow quick, small repositioning of the foveal display.

9. The system of claim 8, further comprising:

once it is determined that the foveal display should be repositioned, it is determined whether movement should be deferred, and the movement is deferred.

10. The system of claim 9, wherein when the movement is postponed, the foveal texture from a previous frame is resampled for display.

11. The system of claim 1, further comprising:

a metadata system that attaches metadata to the frame data generated by the compositor, the metadata for rendering a foveal image.

12. The system of claim 8, wherein the metadata includes pose, gaze, and/or position measurements, and the compositor is further used between a latest pose, gaze, and/or position and the metadata to crop the foveal texture.

13. The system of claim 1, further comprising:

a digital correction system to provide distortion correction for an optical stack of a steerable display.

14. A system, comprising:

a rendering engine to render a field texture and a foveal texture at a first time base;

a synthesizer including an field generator to generate frames for a field display according to the field texture, the field generator having a second time base different from the first time base; and

a combiner including a foveated combiner to generate frames for a foveated display positioned in a foveated region of a user and displayed in a cut-out region of the field display.

15. The system of claim 14, further comprising:

a foveated state machine to track a state of the foveated display and manipulable elements associated with the foveated display, the foveated state machine providing data to the field generator to cut or fill in foveated display areas.

16. The system of claim 15, wherein the population of the foveal area is based on a current foveal texture if the mirror has moved, and the population is based on a previous foveal texture if the movement of the mirror has been postponed.

17. The system of claim 14, wherein movement of the foveal display is based on gaze predictions due to content, regions of interest, high definition content regions in a scene, and/or system operations based on eye movement classification.

18. The system of claim 14, wherein the rendering engine over-renders the size of the foveal region to allow small mirror movements without re-rendering.

19. The system of claim 14, further comprising:

a composition manager to delay movement of mirrors of the foveated display to compensate for display delays in the system.

20. A system, comprising:

a foveated display to locate steerable display images having a first display level and having a first field of view;

a field display to provide a field display image having a second display level lower than the first display level and a second field of view greater than the first field of view;

a rendering engine to render a field texture of the field display and a foveal texture of the foveal display;

a combiner including an original combiner to generate frames for the field display according to the field texture and a foveal combiner to generate frames for the foveal display according to the foveal texture;

a composition manager designed to order and select content presented that contains one or more data in the field display and the foveated display.

Technical Field

The present invention relates to displays, and more particularly, to steerable displays.

Background

Near-eye displays have competing requirements for displaying images at high resolution over a large field of view (FOV). For many applications in virtual and augmented reality, the field of view should be greater than 90 degrees, and ideally, the binocular field of view would extend over 180 degrees. At the same time, the resolution of the display should match the resolution of the human visual system so that little or no pixelation is perceived in the virtual image. Combining these two requirements in a single system presents a number of challenges.

In addition, due to timing issues, it is difficult to design an optical system that can project a wide FOV image to a user with a sufficiently high resolution over the entire field of view.

Drawings

The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

FIG. 1 is an overview image of one embodiment of a system in which a rendering engine may be implemented.

FIG. 2 is a block diagram of one embodiment of a rendering engine.

FIG. 3 is a block diagram of one embodiment of a rendering engine and compositor system.

FIG. 4A is a flow diagram of one embodiment of a rendering engine and compositor system.

Figure 4B is a logic diagram of one embodiment of a foveal manager.

Fig. 5A-5B are flow diagrams of one embodiment of eye tracking and mirror control in a system.

FIG. 6 is a timing diagram showing an embodiment of a timing architecture.

FIG. 7 shows one embodiment of a timing diagram for a synchronization system.

FIG. 8 is a block diagram of a computer system that may be used with a rendering engine.

Detailed Description

A control system and rendering pipeline for a system including a point of regard display is disclosed. In one embodiment, the foveal display is a steerable display that can be positioned to display smaller, higher resolution images, which are generally positioned to match the fovea of the user's retina. The human eye perceives a higher resolution at the fovea and the ability to perceive resolution drops rapidly away from the area. Thus, aligning a smaller field of view (FOV) high resolution display to the foveal region, while the lower resolution display covers the remainder of the field of view provides perception of the high resolution display within the entire FOV. Although the term "point of regard display" or "foveated display" is used in this application, and in one embodiment the display is directed at the user's foveated region, in other embodiments the disclosed steerable display may be positioned elsewhere within the user's field of view.

In one embodiment, the foveated display is paired with a field display that provides a lower resolution display within a wider field of view. In one embodiment, the area corresponding to the display area (location and size) of the foveated display is cut out of the field display and the edge between the foveated display and the field display is blended.

One embodiment of implementing such a point-of-regard image is described in us patent No. 10,514,546 issued on 24.12.2019, which is incorporated herein by reference.

To achieve such a display, the system must match the image displayed by the field display with the image displayed by the steerable display in position and in brightness and hue. If there is a mismatch perceived by the user, it may be considered as a blur, flicker, or other optical artifact that degrades the user experience. In one embodiment, the control system and rendering pipeline of the present invention are capable of creating such synchronized presentations. In one embodiment, the rendering of the field display and the foveated display may be asynchronous. In one embodiment, the displays may have different frame rates. However, the timing of the elements is such that the user can seamlessly perceive the presentation. In this context, the term "presentation" means the perception by a viewer of a visual image or series of images displayed to a user, which may be displayed via a combination of a field display and a foveal display.

A system architecture that is capable of presenting high resolution images to a user over a wide field of view while reducing rendering, data rate, and panel requirements would enable new applications for augmented reality and virtual reality systems.

The following detailed description of embodiments of the invention refers to the accompanying drawings, in which like references indicate similar elements, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It is to be understood by those of ordinary skill in the art that other embodiments may be utilized and that logical, mechanical, electrical, functional and other changes may be made without departing from the scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims.

Fig. 1 shows an arrangement of a hybrid display comprising a field display and a foveated display. In one embodiment, the field displays 110, 115 may be separate field displays for the right eye 115 and the left eye 110. In another embodiment, the field displays 110, 115 may be a single display spanning both eyes.

The field displays 110, 115 have cutouts 130, 135 and the foveated displays 120, 125 are inserted into the cutouts. In one embodiment, the foveal displays 120, 125 may be manipulated and positioned based on the user's line of sight. In one embodiment, the foveal displays 120, 125 are steerable and their location is based on scene content, environmental elements, events, and/or other factors.

There is a mixing 140, 145 between the field displays 110, 115 and the foveated displays 120, 125. The blends 140, 145 ensure that the transition between the field displays 110, 115 and the foveated displays 120, 125 is smooth and unobtrusive. In one embodiment, blending may include blending based on transparency masks (alpha masks), dithering, temporal multisampling antialiasing, spatial antialiasing, blurring, and adjustment of brightness, color, and so forth.

In one embodiment, the mixing zones 140, 145 are dynamic in size and/or shape. In one embodiment, the size and/or shape of the foveal regions 120, 125 may be adjustable. In one embodiment, the size and/or shape of the foveal regions 120, 125 may vary due to a variety of factors including scene content, user behavior, the photodynamic characteristics of the system, eye tracking performance throughout the viewing range, and the like.

FIG. 2 illustrates one embodiment of exemplary optical systems 210, 280 and associated processing system 240. In one embodiment, the processing system may be implemented in a computer system that includes a processor. In one embodiment, the processing system 240 may be part of a display system. In another embodiment, the processing system 240 may be a separate system that may be local or remote to the optical system.

In one embodiment, the display 210, 280 may be implemented in a wearable device, such as a head-mounted display.

The steerable display image is presented to the eyes of the user through the right-eye steerable display 220 and the left-eye steerable display 230. In one embodiment, the steerable displays 220, 230 direct the steerable display images primarily to the center of the field of view of the user's eyes. In another embodiment, the steerable display may place the foveal image in a location outside the center of the user's eye field of view. In one embodiment, the position may be fixed relative to the user's head position, fixed relative to one or more environmental elements, fixed relative to a line of sight, and the like. In one embodiment, the foveal image location may be based on scene content, environmental elements, events, and/or other factors. In one embodiment, the images may be directed to different locations, as described below. In one embodiment, the steerable display image is a high resolution image with a limited field of view.

In one embodiment, the steerable display image is a variable resolution image. In one embodiment, the variable resolution is highest at the center of the display. In one embodiment, the change in resolution corresponds to a change in the maximum resolution perceived by the user's eye, which decreases as the user's eye moves farther away from the center of the fovea. In one embodiment, the variable resolution image has discrete resolution regions, with the first highest resolution region being at the center and the one or more lower resolution regions being further from the center. In one embodiment, the lower resolution region is concentric with the highest resolution region.

In one embodiment, the steerable display may be a multi-focus display. In a multi-focus display, in one embodiment, the sequence of images at different focal lengths is displayed quickly enough that the user perceives a single multi-focus image. In another embodiment, in a multi-focal display, the image is projected at multiple focal lengths simultaneously. In either case, the user perceives the image with multiple focal lengths as a single multi-focal image.

In one embodiment, left-eye display 230 repeats the elements described below for right-eye display 220. An image is created for the right eye using the first display element 222. In one embodiment, the display element is a Digital Micromirror Device (DMD). In one embodiment, display element 222 is a scanning micro-mirror device. In one embodiment, the display element 222 is a scanning fiber optic device. In one embodiment, the display element 222 is an Organic Light Emitting Diode (OLED). In one embodiment, the display element 222 is a Liquid Crystal On Silicon (LCOS) panel. In one embodiment, the display element 222 is a Liquid Crystal Display (LCD) panel. In one embodiment, the display elements 222 are micro light emitting diode (μ LED) panels. In one embodiment, the display element is a scanning laser system. Other types of display elements currently in use or developed in the future may be used with the present system. In one embodiment, left eye display 230 matches a right eye display.

In one embodiment, display elements 222 may include a combination of such elements, or alternative elements that may be used to generate a high resolution image for display to a user. Some embodiments of display elements are discussed in more detail in fig. 3 below.

In one embodiment, the first display element 222 is located in a near-eye device such as glasses or goggles. In one embodiment, for a multi-focus manipulable display 210, the display 220 may also include a focus adjuster 223 that adjusts the focal length for one or more image portions in the multi-focus display as the images are displayed. Such adjustments may be time-based or location-based. In one embodiment, the OPLE system described in U.S. patent 10,187,634 may be used to implement a focus adjuster 223 for an adjustable multi-focus display.

In one embodiment, the focus and field of view of the steerable display is set using an intermediate optical element 224. The intermediate optical element 224 may include, but is not limited to, lenses, mirrors, holographic optical elements, and Diffractive Optical Elements (DOEs).

In one embodiment, the size and/or shape of the field of view may be adjustable. In one embodiment, the size and/or shape of the foveal region may be adjustable. In one embodiment, the size and/or shape of the foveal region may vary due to a variety of factors including scene content, user behavior, the photodynamic characteristics of the system, eye tracking performance throughout the viewing range, and the like.

In one embodiment, the focus of the virtual image displayed by display 220 is set to infinity for a single focus display. In another embodiment, the focal point of the virtual image is set closer than infinity. In one embodiment, the focus of the virtual image may be changed. In one embodiment, the virtual image is multifocal and has two or more focal lengths perceived at the same time, as described above.

In one embodiment, the steerable display image is directed primarily to the center of the field of view of the user's eyes. As described above, the system may alternatively be directed to different locations. The positioning of the steerable display images may be based on eye tracking data, scene content, environmental elements, events, and the like, as will be discussed below.

In one embodiment, the field of view (FOV) of the steerable display image is greater than 1 degree. In one embodiment, the FOV of the steerable display image is between 1 degree and 20 degrees. In one embodiment, the range of movement is +/-20 degrees and the field of view is 20 degrees, resulting in a total display range of 60 degrees.

In one embodiment, the image data generated for the steerable display may be larger than the display size of the image to account for inaccuracies in eye tracking, to derive the area required to blend the foveal display with the field display such that the blending is not perceptible to the user, and/or to account for the time required to reposition the steerable display for various types of eye movements. In one embodiment, the size of the field of view of the steerable display is greater than 5 degrees. In one embodiment, the system is designed to account for time delays in the display system and changes in the user's gaze location that may occur between frame updates, and to account for time delays in the eye tracking system and the amount by which the user's gaze vector may change between reporting updates from the eye tracker. Timing to achieve these aspects will be discussed in more detail below.

In one embodiment, the system further includes a field display 280, or a pair of field displays, one for each eye, to display a field display image having a field of view of 20 to 220 degrees. In one embodiment, the field display is a lower resolution display. In one embodiment, a lower resolution display refers to a display having a lower pixel density, e.g., fewer pixels per viewable angle (PPD). In this context, the term "PPD" refers to the average pixel per degree in the display. In some embodiments, the steerable display may have a variable number of pixels per degree. In this embodiment, the PPD of the steerable display external to the steerable display may be lower than the PPD of the field display. However, in one embodiment, the average pixels per degree in the display of the steerable display are higher than the field display. In some embodiments, the total number of pixels in the field display may be larger than the steerable display because the field display is larger.

In one embodiment, the resolution of the field display may not be lower than the resolution of the steerable display. In one embodiment, the difference between the steerable display and the field display may include one or more of the following in addition to or instead of the lower resolution: lower color depth, smaller color gamut, lower color resolution, lower contrast, lack of multi-focus aspects, different sharpness, or any other display aspect that reduces the perceived quality of the image on the field display compared to a steerable display. The term "low-level" refers to the difference between the steerable display and the field display, i.e. the perceived quality of the image projected by the field display is lower than the perceived quality of the image projected by the steerable display.

In one embodiment, a set of one or more positioning elements 226 is used to position the steerable display image. In one embodiment, the positioning element 226 comprises a steerable mirror. In one embodiment, the positioning element 226 may include one or more of a curved mirror, a Fresnel reflector, a diffractive element. In one embodiment, the diffractive element is a surface relief grating. In one embodiment, the diffractive element is a volume hologram.

Projection assembly 227 projects an image from right-eye display 220.

In one embodiment, the display 220 also includes a combiner 228 to project the image to the eyes of the user. In one embodiment, combiner 228 comprises a waveguide. In one embodiment, the waveguide is a multilayer waveguide. The combiner 228 may include other elements that enable the steerable display to reach the user's eyes. In one embodiment, the combiner 228 combines the steerable image from the projection assembly 227 and the field display image 285 using a fully or partially transparent optical system to present a blended image to the user. In one embodiment, combiner 228 is a partially transparent system that includes a waveguide. In one embodiment, this partially transparent system comprises a partial mirror that may be flat or have optical power. In one embodiment, this partially transparent system includes a diffractive optical element. In one embodiment, the partially transparent system comprises a holographic optical element. In one embodiment, this image is presented to the user through a direct-view optical system. In one embodiment, this partially transparent system contains inclusions that reflect or scatter light.

A similar set of elements exists for the left eye manipulable display 230. In one embodiment, the right eye steerable display 220 and the left eye steerable display 230 are matched. In another embodiment, the steerable display may include different elements. In one embodiment, the steerable display is adjustable to the eyes of the user such that it is perceptually matched; that is, the user thinks they are a match. In one embodiment, right display 220 and left display 230 create stereoscopic vision to create or enhance the illusion of depth in the image, as is conventionally known.

In one embodiment, the eye tracker 242 tracks a gaze vector of the user, such as where the eye is looking. In one embodiment, the eye tracker is a camera-based eye tracking system 242. In one embodiment, the camera-based eye tracking system 242 includes holographic optical elements. In one embodiment, eye tracking system 242 is an infrared scanning laser with a receiving sensor. In one embodiment, infrared scanning laser eye tracking system 242 includes holographic optical elements. In one embodiment, eye tracking system 242 is an optical flow sensor. Other eye tracking mechanisms may be used for eye tracker 242. The position calculator 245 determines the center of the user's field of view based on data from the eye tracker 242. In one embodiment, the center of the field of view for each eye of the user is determined.

In one embodiment, the adjustable positioning elements 226, 236 of the steerable display 210 are used to adjust the position of the right eye steerable display 220 and the left eye steerable display 230 based on data from the eye tracker 242 to position an image to be directed primarily toward the center of the field of view of the user's eyes. In one embodiment, the adjustable position elements 226, 236 are used to adjust the right eye steerable display 220 and the left eye steerable display 230 to position the eye range (eye box) or exit pupil towards the center of the field of view of the user's eyes.

In one embodiment, the position of the steerable display image is adjusted by changing the angle of one of the position elements 226, a mirror. In one embodiment, the angle of the mirror is changed using electromagnetic force. In one embodiment, electrostatic forces are used to change the angle of the mirror. In one embodiment, piezoelectric forces are used to change the angle of the mirror. In one embodiment, the location element 226 is an image source or display element 222 that moves to position an image. In another embodiment, rather than using a mirror, a computer-generated hologram implemented on a spatial light modulator is used to manipulate the foveal image. Although the term "manipulating" is used, in one embodiment, no element is physically moved. Instead, the reflective characteristics of the positional element 226 or one of the elements in the display are adjusted to change the position of the output of the display 220, 230. In one embodiment, this is accomplished by using a Spatial Light Modulator (SLM).

In one embodiment, additional display 280 communicates with processing system 240 via communication logic 270, 290. In one embodiment, the additional display includes a field display 285. In one embodiment, the additional display includes separate field displays 285 for the right and left eyes. In one embodiment, the additional display 280 may include a single field display 285 for viewing by both eyes of the user.

In one embodiment, the additional display 280 may have a variable resolution. In one embodiment, the resolution decreases away from the center of the field display 285, corresponding to a decrease in the maximum resolution perceived by the eye.

In one embodiment, the additional display 280 is a micro LED (light emitting diode) or a micro LED array. In one embodiment, the additional display 280 is an OLED (organic LED) array. In one embodiment, the additional display 280 uses a scanning laser. In one embodiment, the additional display 280 is an LCD (liquid crystal display) panel. In one embodiment, the additional display 280 is an LCOS (liquid crystal on silicon) display. In one embodiment, the additional display is a DLP (digital light processing) display.

In one embodiment, the synchronization signal generator 292 is used to synchronize the display of the independently steerable display 210 with the display of the field display 285 when the field display 285 is a separate system. This will be discussed in more detail below.

In one embodiment, synchronization signal generator 292 is a signal that synchronizes the presentation of the display. In one embodiment, the synchronization signal generator 292 is used to synchronize the adjustable mirror or other positioning element 226, 236 of the steerable display 210 with the field display 285. In one embodiment, the synchronization signal generator 292 controls a fast shut down mechanism to temporarily disable the light source that is part of the projection assembly 227, 237 of the steerable display. This synchronizes the displays 210, 285. This timing will be described in more detail below.

In one embodiment, the additional display 280 includes a mixer system 294 to mix the edges of the steerable display image with the field display image to ensure that the transition is smooth and imperceptible to the user.

In one embodiment, the field display image is presented to the user using a fully or partially transparent optical system. In one embodiment, the combiner 228 combines images from the steerable display and the field display for presentation to a user. In one embodiment, combiner 228 is a partially transparent system that includes a waveguide optical system. In one embodiment, this partially transparent system comprises a partial mirror that may be flat or have optical power. In one embodiment, this partially transparent system includes a diffractive optical element. In one embodiment, this image is presented to the user through a direct-view optical system. In one embodiment, this partially transparent system contains inclusions that reflect or scatter light.

In one embodiment of the additional display 280, the display subsystem is used to display images in the monocular viewing area, referred to as the peripheral display 288. In one embodiment, the peripheral display 288 may be two separate displays for the right and left eyes. In another embodiment, there may be only a peripheral display 288 on one side.

In one embodiment, the peripheral display 288 is one or more discrete LEDs. In one embodiment, the peripheral display 288 functions as an indicator, warning mechanism, or feedback mechanism, rather than being a complete display. The peripheral display images may be used to attract the attention of the user, or to convey information subtly. For example, the peripheral display 288 may be used to indicate to the user that he or she should be aware of real-world events that may not otherwise be noticed when using the display system 200. In one embodiment, the use of such a peripheral display 288 is optional and may be set by the user.

In one embodiment, the additional display subsystem 280 has no intermediate optical elements to manipulate the FOV or focus of the image. In another embodiment, this subsystem 280 includes intermediate optical elements. In one embodiment, the intermediate optical element may comprise a microlens array.

Image data displayed by the steerable display 210 and the field display 285 is generated by the processing system 240. In one embodiment, the processing system 240 is distributed, with various subsystems performing different aspects of the overall system processing. In one embodiment, the processing system 240 may include multiple processors.

In one embodiment, the processing system 240 includes an eye tracker 242. In one embodiment, the eye tracker 242 tracks a gaze vector of the user, such as where the eye is looking. In one embodiment, the eye tracking system is a camera-based eye tracking system 242. Alternatively, the eye tracker 242 may be based on an infrared laser. The position calculator 245 determines the center of the user's field of view based on data from the eye tracker 242.

In one embodiment, the position calculator 245 additionally uses data from the slip detection system 263. In one embodiment, the slip detection 263 detects movement of the headset/goggles on the user's head relative to the user's body to identify slips or other movements that deviate the true position of the user's eyes from the calculated position. In one embodiment, the position calculator 245 may compensate for such slippage by adjusting the calculated position used by the system to position the steerable display. In one embodiment, the slide detection also serves to ensure that the display is adjusted to provide parallax for the multi-focus display steerable display 210.

In one embodiment, the processing system 240 further includes a position validator 247 that validates the positioning of the position elements 226, 236 to ensure that the manipulatable display 220, 230 is in its intended position. In one embodiment, the position elements 226, 236 contain a local sensing and control mechanism that sets the foveal display to a calculated position and then holds the foveal display in that position until a new target position is set. In one embodiment, the position of the steerable element is sensed using optical sensing. In one embodiment, the position of the steerable element is sensed using a magnetic position element. In one embodiment, the sensing mechanism may be a camera. In one embodiment, the sensing mechanism may be a transmission. The sensing mechanism may be another type of sensor that can determine the position of the optical element. Other mechanisms may be used to locally sense the position of the steerable element.

In one embodiment, the foveal position verifier 247 verifies that the actual position of the foveal display matches the position of the generated steerable image display by obtaining real position data from the local sensing and control mechanisms of the position elements 226 and 236 and comparing the real position data to the target position from the positioner 265. In one embodiment, if the foveal location verifier 247 reports that the actual location of the manipulatable display is not within the accuracy threshold of the target location, the processing system 240 may alter the display to provide the appropriate image data. This is discussed in more detail below.

In one embodiment, the eye movement classifier 260 may use data from the eye tracker 242 to predict where the user's gaze vector will move based on identifying the direction and type of eye movement. The positioner 265 may use this data to move the manipulatable display 220, 230 based on the next position of the user's gaze vector. In one embodiment, the positioner 265 may utilize user data such as eye movement classification and eye movement tracking to predictively position the displays 220, 230. In one embodiment, the positioner 265 may use the classified eye movements to time the movement of the display to minimize the user's perception of the movement. In one embodiment, the locator 265 may additionally use data regarding upcoming data in a frame to be displayed to identify the optimal location of the display 220, 230.

In one embodiment, the positioner 265 may position the displays 220, 230 at locations not indicated by the line of sight vector. For example, where the displayed frame data has only a small amount of relevant data (e.g., a butterfly illuminated on an otherwise dark screen), or the frame is intended to have the viewer move his line of sight to see a particular location on the display. Other environmental or contextual reasons for the location may be used. In another embodiment, the positioner 265 may adjust the position of the steerable display to ensure that the foveated display and the field display are well mixed, or otherwise reduce or eliminate visual artifacts. In another embodiment, the positioner 265 may reposition the manipulatable display 210 to match the display image to a scene element or other external feature.

The processing system 240 may further include a cut logic 250. The cut-out logic 250 defines the position of the manipulatable display 220, 230 and provides display position information to the associated field display 285. This data is used by the field display 285 to generate a field display image containing a cut-out in the field display corresponding to the portion of the field display replaced by the steerable display. Alternatively, the image may be "cut out," i.e., the entire image may be rendered, and the cut out portion of the image may be discarded prior to display.

Having a cut-out ensures that there is no interference between the steerable display image and the field display image. In one embodiment, when a cut is present, the mixer logic 255 mixes the edges of the cut with the edges of the steerable image positioned within the cut region to ensure smooth transitions.

In another embodiment, the steerable display 210 may be used to display a sprite, i.e., a brighter element overlaid on the field image. In this case, in one embodiment, neither notch logic 250 nor mixer logic 255 is used. In one embodiment, the cut-out logic 250 and the mixer logic 255 may be selectively activated as desired.

In one embodiment, the display elements 222 and 232 are not shown when the system moves the steerable display 210. In one embodiment, the display is not shown by extinguishing the light source of the manipulable display to prevent the user from perceiving movement of the display 220, 230 and associated visual artifacts, such as blurring, color separation, brightness flicker, etc. In one embodiment, the right display 220 and the left display 230 may be controlled asynchronously for timed movement.

In one embodiment not shown, when the steerable display 210 is not shown, the processing system 240 fills the cutout holes in the field display using the field display logic 256 to prevent the user from perceiving artifacts or holes in the display. In one embodiment, the padding data is based on data originally generated for the field display and the cut-out. In one embodiment, the fill data is based on data originally generated for the manipulatable display. In one embodiment, the filler data is a mix of data generated for the field display 285 and the steerable display 210. In one embodiment, the field display fill logic 256 uses various filtering techniques including blurring, antialiasing, dithering, transparency blending (alpha blend), etc. to adjust the fill data so that the user does not perceive that the kerf aperture is temporarily filled by a different display source.

In one embodiment, the system may utilize an asynchronous steerable display 210 with an independent asynchronous additional display 280. In this case, in one embodiment, the scheduler and synchronization logic 272 synchronizes the displays. In one embodiment, the separate additional display 280 is synchronized with an adjustable mirror or other positioning element of the steerable display 210. This synchronizes the display. The additional display 280 may receive positioning data. This process is discussed in more detail below. In another embodiment, the additional display 280 and the manipulatable display 210 may be synchronized. In such a configuration, the scheduler and synchronization logic 272 may be eliminated or its logic may be simplified.

In one embodiment, the processing system 240 may include an optical distortion correction system 275. The optical distortion correction system 275 may adjust the image to add increasing distortion from the center to the edges of the image. In one embodiment, this intentional distortion distorts the pixel correction. In one embodiment, the perceived size of the optical distortion correction system 275 increases as one moves from the center to the edge of the image. This change in perceived resolution reduces the amount of processing required because fewer pixels are required to cover the same angular area of the steerable display image. In one embodiment, the optical distortion correction system 275 is applied to a steerable display image. In one embodiment, such optical distortions may contribute to the mixing between the steerable display 210 and the additional display 280. In another embodiment, the steerable display 210 containing the optical distortion correction system 275 may be used without a field display. The steerable display also provides a simpler optical design and saves processing of the blending. In one embodiment, such variable resolution highly distorted images have a large ratio between the pixel sizes of the center and the edges. The total FOV of such a display may be quite large (e.g., 180 degrees).

In one embodiment, the optical distortion correction system 275 applies corrections to the field display and the foveated display. In one embodiment, for a field display, the optical correction creates a mask of the foveal region that is cut out and blended or filled with the foveal display according to a time sequence, as discussed below. In one embodiment, optical distortion correction system 275 is applied to radial distortion correction for field optics and color correction.

In one embodiment, optical distortion correction system 275 blends the edges of the foveal region, applies radial distortion correction to the field optics, applies a homography transform to correct the mirror position, applies radial distortion correction to the foveal optics, and applies color correction. In one embodiment, it is unique to apply separate radial distortion corrections to the field optics and the foveal optics. In one embodiment, radial distortion correction for field optics uses a look-up table or spline fitting to reconcentrate distortion correction based on mirror (or other position element) position. In one embodiment, the homography transformation that corrects the mirror position uses a look-up table or spline fitting coefficients from a calibration step. In one embodiment, the homography correction is done over RGB colors.

In one embodiment, roll-off logic 277 provides roll-off at the edge of the display. In one embodiment, the roll-off may include a resolution roll-off (decreasing the resolution towards the edge of the display area). In one embodiment, the resolution roll-off may be achieved by magnification of the optical distortion correction system 275. In one embodiment, the roll-off comprises a brightness and/or contrast roll-off (reducing brightness and/or contrast towards the edge). Such roll-off is designed to reduce the steepness of the edges of the display. In one embodiment, the roll-off may be designed to roll off to "none," i.e., gradually decrease from full brightness/contrast to gray or black or ambient color. In one embodiment, the steerable display 210 may use roll-off logic 277 when there is no associated field display. In one embodiment, when there is a field display in the system, roll-off logic 277 may be part of field display 285.

In one embodiment, when the manipulatable display 210 is a multi-focal display, the focal point selector 267 identifies multiple focal lengths for display and generates appropriate display data for each focal length. In one embodiment, in a multi-focus display, a set of time-sequential images are displayed at different focal lengths. A set of consecutive images is displayed at a certain speed and thus the user sees it as a single image with multiple focal lengths.

In one embodiment, the digital correction system 252 corrects the generated image data to accommodate any distortion due to the multi-focus display. In one embodiment, the digital correction system 252 further adjusts the size of each image of the multi-focus display so that the blending between the steerable display 210 and the field display 285 remains smooth.

FIG. 3 is a block diagram of one embodiment of a controller and rendering pipeline. In one embodiment, the user pose detector 315 may include an external camera, one or more sensors such as gyroscopes, motion detectors, and other sensors. In one embodiment, one or more cameras and/or sensors on the headset may be used for the user pose detector 315. This is called exhaustive tracking. Alternatively or additionally, cameras and/or sensors that are not part of the headgear may be used for the user pose detector 315.

In one embodiment, the user pose detector 315 may include a "relative" change detector and/or an absolute pose detector. The "relative" change detector detects changes in pose over time, while the absolute position measurement places the user in 3D space. Data from the user pose detector 315 is passed to the interrupt generator 320, as well as the field display renderer 330 and the metadata system 390.

Interrupt generator 320 generates interrupts that are used by scheduler 385 to control field display renderer 330 and foveated display renderer 340. The interrupt generator 320 receives data from the pose detector 315 and from the gaze detector 325. Incoming display (scene) data 322 is passed to the renderers 330, 340. A line of sight detector 325 is also coupled to the interrupt generator 320 so that it can be used by the scheduler 385 to control the foveal display renderer 340 because the foveal display data may need to be changed if the user's line of sight moves. Other triggers may also be used by scheduler 385 to adjust foveated display renderer 340.

Data from the line of sight detector 325 is also provided to the composition manager 380. The composition manager 380 manages the combination of the foveated display and the field display to present a seamless image to the user. In one embodiment, the composition manager 380 uses data from the line of sight detector 325 to enable the positioning of the cut-outs in the field display to correspond to the position of the foveated display. The renderers 330, 340 render textures. In one embodiment, the foveal display renderer 340 renders multiple depths for a multi-focus foveal display.

The line of sight detector 325 also provides data to a position detector and actuator 350 that controls movement of a steerable mirror or other steerable element, which enables positioning of the foveated display 365. The actuator 350 moves the element to properly position the foveal display 365. The foveal verifier 355 verifies that the actual position of the foveal display 365 matches the expected position of the rendered image.

In one embodiment, for a foveated display based on a DLP/LCOS/laser scanner system, the fast shut-off element 370 can turn off the illumination (LED or laser or other illumination source) of the foveated display 365 if the foveated display 365 element is not properly positioned. This stops displaying or extinguishes the foveated display 365, leaving only the field display 360.

In one embodiment, for an emissive display (e.g., a uLED or OLED), the quick turn off element 370 can extinguish the entire foveal display 365. In one embodiment, stopping the display of the foveated display 365 includes sending blank image data to the display 365.

In one embodiment, the quick-close 370 also sends data to the composition manager 380 that closes the cut-out of the field display, leaving the field display intact, while the high-resolution foveated display 365 is not available. In another embodiment, the cut-out may be filled with padding data. In one embodiment, the fill data for the cutouts in the field display 360 may be based on image data rendered for the field display, the foveated display, or a combination of the field display and the foveated display.

In one embodiment, the composition manager 380 selects which data is valid for display in the cut-out in the field display 360 based on information from the foveal verifier 355. When the foveal display is correctly positioned, the current foveal texture from the foveal display renderer 340 is used.

In one embodiment, the re-projected field display data may be used when there is no current foveal texture data for the location. In some cases, previous foveal texture data may be used. The composition manager 380 ensures that the cuts are filled with data when they exist based on the accuracy and availability of the data from the various sources.

Data from the renderers 330, 340 enters respective compositors 335, 345. In one embodiment, the synthesizers 335, 345 may be asynchronous. In one embodiment, the renderers 330, 340 may operate on separate clocks and at different frequencies than the synthesizers 335, 345.

The field display 360 displays right and left eye field images, while the foveal display 365 displays right and left eye foveal images. In one embodiment, the foveal image and/or the field image may have multiple focal planes or depths, as described above. In such an embodiment, the frame data generated by the synthesizers 335, 345 may include a sequence of image elements associated with multiple focal lengths. The sequence may be a time sequence. In another embodiment, the focal length may be based on location rather than time. In one embodiment, the foveal image and/or the field image may have variable focus.

In one embodiment, synchronizer 375 ensures that asynchronously generated frame data from the field display and the foveated display are properly aligned. In one embodiment, synchronizer 375 uses time stamps from the graphics processing unit to coordinate the various time domains. In one embodiment, synchronizer 375 synchronizes with the device in the system having the largest latency to drive rendering decisions.

In one embodiment, the metadata system 390 may generate and attach metadata to the rendering data, composition data, and/or frame data. This metadata may be used in various ways. In one embodiment, the metadata may be used to post update or re-project just before the display is turned on. The metadata may be used to account for changes in line of sight, pose, or mirror position from rendering the frame to be displayed. Thus, in one embodiment, the metadata system 390 receives data from various systems, including the user pose detector 315, the gaze detector 325, the position detector and actuator 350, and the foveated display renderer 340.

As described above, in one embodiment, the compositors 335, 345 run at a different update rate than the renderers 330, 340. The foveal display renderer 340 over-renders the foveal/field area and the metadata system 390 marks the area with metadata, representing the line of sight, pose, and/or mirror position associated with the rendered content.

When foveal data is to be displayed, foveal display compositor 345 compares the metadata to the latest pose, gaze, and/or position measurements, and uses the differences to select an appropriate portion of the over-rendered texture and send it to foveal display 365.

In one embodiment, the metadata system 390 may be used to indicate one or more regions of interest. In one embodiment, some points in the scene are rated as areas of greater interest than other points in the scene. These regions of interest may be used to limit the effective point of regard to the content marked in the metadata.

In one embodiment, the metadata system 390 may be used to mark several regions of a scene that are points of regard.

In one embodiment, the metadata system 390 may be used to tag frames with upcoming information when playing video content. This can be used to let the system know if any changes have occurred to the pipeline.

In one embodiment, the metadata may contain the manipulation information of the manipulatable display, so that the manipulation information may be sent down to the mirror control system in the same time domain as the frame information itself.

In one embodiment, the metadata may also be used for micro-correction of the mirror.

In one embodiment, the metadata system 390 may be used in a portion of a local relative eye tracker that tracks eye movement and uses its historical data tracked since rendering frames to send data in new mirror positions and/or crop frames as a late update.

In one embodiment, the metadata system 390 may provide late update priority for some objects in the scene, for example, if an object needs to be pinned to a real world object, it takes precedence over the rest of the scene.

FIG. 4A is a flow diagram of one embodiment of a rendering engine and compositor system. The system includes a rendering engine 410 and a direct mode compositor 450.

In one embodiment, the rendering engine 410 has its own time base, while the compositor 450 has a separate time base. In one embodiment, the synthesizer 450 has separate time bases for the two displays, the field display time base (field vertical sync) and the foveated display time base (foveated vertical sync). This means that in one embodiment, each element may run at different speeds and refresh rates using a separate time base. In one embodiment, data and displays having separate time bases may be synchronized by using a synchronization block that maintains presentation synchronization for the various displays. Presentation synchronization in this context means that the display content from both displays is considered synchronized by the user.

In one embodiment, this also enables the rendering engine 410 and the compositor system 450 to run at different rates. In one embodiment, the compositor 450 is faster than the rendering engine. In one embodiment, the rendering engine 410 operates at a variable number of frames per second and the compositor 450 operates at two different frame rates, one for the field display and the other for the foveated display.

In another embodiment, the compositor and rendering system are synchronized and run in the same time base. In one embodiment, all displays are synchronized and run in the same time base. In one embodiment, the displays run on the same clock to ensure that they remain synchronized. In one embodiment, the displays may operate at the same update rate. In another embodiment, the display update rate of one display may be an integer multiple of the rate of the other display. In one embodiment, the display update rate of one display may be asynchronous to the update rate of another display. In one embodiment, the field display and the foveated display are each part of a single larger display.

In one embodiment, the rendering engine 410 processes data continuously. When new display data is received for rendering, the rendering engine 410 is driven by the rendering engine time base loop, the rendering texture, and optionally the depth of the multi-focus display.

In one embodiment, the interrupt drives the stereoscopic projection matrix in the rendering engine 410 upon detecting a new pose, receiving an interrupt. The pose defines the position of the user in three-dimensional space, including position and orientation. In one embodiment, pose has six degrees of freedom. Whenever a new pose is detected, an interrupt is sent to the stereoscopic projection matrix of the rendering engine 410.

In one embodiment, the interrupt triggers an update of a projection matrix defining the field display. The left and right displays of the field display are rendered as textures.

In one embodiment, the update to the stereoscopic projection matrix also triggers an update to the foveal viewport. A foveated viewport is a region of a field display that is considered a "foveated region" that is rendered at higher resolution by a separate display, the foveated display, as described above. When the foveal viewport is updated, it triggers re-rendering of the left and right image/texture and depth data of the foveal display. The foveal left/right data and field left/right data and the viewport data are passed to a direct mode synthesizer 450.

In some embodiments, various other processes may trigger updates to the foveated viewport, including changes to displayed data, operating system level changes, changes to scene content, environmental elements, events, and/or other triggering events. In one embodiment, any such trigger updates the rendering of the foveal data.

In addition to being triggered by the new pose, the rendering engine 410 may also trigger an update of the foveal viewport based on data from the foveal state machine 455. The foveal state machine 455 is used to trigger the rendering engine 410 and compositor 450 when the user's gaze vector changes to ensure that the foveal display is properly positioned and displays a high resolution image in the correct location.

In one embodiment, scheduler 440 controls the various elements and triggers the actual rendering by rendering initiation. In one embodiment, scheduler 440 receives data from the initiator and the foveal composition initiator.

In one embodiment, the compositor 450 is also triggered by an interrupt when a new pose is detected. In one embodiment, the field display is interrupted when new pose data is received, when new texture data is received, or when there is an offset from the field display vertical sync. The compositor 450 receives texture and foveal viewport data from the renderer 410.

In one embodiment, the field display time base is a field vertical sync, which in one embodiment has a period of 11 milliseconds (90 Hz). In one embodiment, the range is 60Hz to 180 Hz. In one embodiment, when a new pose is detected or new rendering data is received from rendering engine 410, the rendered scene is re-projected based on the current pose.

In one embodiment, the foveal manager 455 is used to drive the foveal display to update the position and content of the foveal display as the gaze vector changes. The foveal manager 455 also provides the line of sight vector data to the scheduler, and thus to the field display compositor, so that the field display compositor knows the location of the foveal display. The foveal manager 455 further provides this data to the scheduler 440.

This enables the field display combiner to identify the appropriate region of the field display for the cutout of the foveated display. In one embodiment, the field display data is cut off when the foveated display is properly positioned and ready for display, and the field display data is filled when the foveated display is not properly positioned or does not have the appropriate data for display. This data used to locate the cut in the field display is based on data from the foveal manager 455.

In one embodiment, lens correction is applied. In one embodiment, lens correction achieves simplification of optics by utilizing software correction in the rendering pipeline to correct for distortion. In one embodiment, the system may use a dynamic distortion map. In one embodiment, the correction may include corrections for radial distortion, pincushion barrel distortion, chromatic aberration, and other distortions. In one embodiment, the field frames are packed subsequently.

In one embodiment, the field display swap chain is then updated. In one embodiment, the switch chain is a physical buffer comprising a front buffer and a back buffer. In one embodiment, the scene builder writes to the back buffer and the front buffer sends to the display. In one embodiment, the local producer writes directly to the pre-buffer during the blanking period. In another embodiment, the swap chain is a virtual frame buffer for the display. In another embodiment, another type of buffer may be used, or the rendered image may be projected directly.

In one embodiment, simultaneously, but not synchronously, the foveal display receives an interrupt and the foveal synthesis initiator triggers the synthesizer. In one embodiment, the foveated display uses foveated vertical synchronization, which may be asynchronous with the field vertical synchronization driving the field display. In a system with multiple asynchronous subsystems, the system can determine which subsystem(s) will drive global synchronization. In one embodiment, the display with the largest latency in the system drives the rendering decision, and the resulting foveated display repositioning/updating. In another embodiment, the field display compositor and the foveated display compositor may be synchronized, e.g., the foveated vertical synchronization may be the same as the field vertical synchronization. In another embodiment, the field display and the foveated display may be two portions of a single display. In this case, the field display and the foveated display may share the same vertical synchronization (Vsync) signal. In one embodiment, a single display may have two portions, with the field and foveal portions of the display being optically isolated from each other. In one embodiment, the optical isolation may be a black region.

The foveal manager 455 determines whether the foveal rendering received from the rendering engine 410 is valid.

Foveal rendering, which in one embodiment includes foveal RGB, viewport, and texture received from the rendering engine 410, is effective when there is no change or only a small difference between the position of the rendered foveal data and the current foveal position. Foveal rendering is effective if the area rendered in the current foveal data completely or almost completely overlaps with the actual position of the foveal display.

In one embodiment, the foveal region is over-rendered by the rendering engine 410. This over-rendering may be helpful when the compositor is running at a higher frequency than the renderer, and may be used to reproduce small mirror movements. In one embodiment, this provides some flexibility for inaccurate mirror movement or eye tracking. In one embodiment, the over-rendering may be up to 50% of the foveal region. In one embodiment, the over-rendering may be asymmetric, such as predictive over-rendering, based on expected eye movement. In one embodiment, the overdraw area may be used to allow the compositor to compose a new scene without having to re-render the scene for small movements of the mirror.

In one embodiment, if foveal rendering is valid, then the fast display path is used.

If foveal rendering is valid and mirror movement does not need to be delayed to properly position the foveal display in time to display data, then the current foveal texture is sampled. In one embodiment, the system applies distortion correction and packs the foveal frame.

In one embodiment, the distortion correction provides software correction for distortion caused by a steerable mirror or display element. This achieves simplification of the optical device. In one embodiment, software correction of distortion may correct for differences in distortion based on movement of a mirror or other steerable display element. In one embodiment, the distortion changes as the mirror/element moves. By pre-warping the data prior to packing the frame, the system can use software distortion correction. In one embodiment, the system may provide different corrections for different portions of the image based on mirror/element positioning.

In one embodiment, the system may use a dynamic distortion map based on the position of the mirror/element. By creating a dynamic distortion map, the system can apply distortion correction based on the known position of the mirror/element. In one embodiment, the dynamic distortion map may be pre-computed and applied based on a look-up table. In another embodiment, the dynamic distortion map may be continuously evaluated according to polynomial splines or other mathematical models.

There may be multiple sources of distortion for the foveated display. In one embodiment, there is an optical assembly between the display panel and the manipulating element. The optical element introduces distortion independent of the manipulation position. This type of distortion may be referred to as projector distortion. In one embodiment, this distortion is radial. In one embodiment, this distortion may be described by a 2D interpolation polynomial or another 2D function. The position of this distortion is fixed relative to the foveal field of view. In one embodiment, software distortion correction may also be used to correct projector distortion.

The variation of the steering position introduces a transformation that can be described by a homography matrix and depends on the steering position. This type of distortion may be referred to as a homography transform. In one embodiment, software distortion correction may also be used to correct homography transform distortion.

In one embodiment, there is an optical element between the steering element and the user's eye. These elements also introduce distortion. This distortion is fixed relative to the total steerable range of the system and does not move with the foveated field of view as the system is steered. This type of distortion may be referred to as scan distortion. In one embodiment, the scan distortion is radial. In one embodiment, this distortion may be described by a 2D interpolation polynomial or another 2D function.

These sources of distortion can be corrected using their own corrections. In one embodiment, the distortion is mathematically corrected by converting coordinates on the display panel to coordinates on the intended image. In one embodiment, there are all three sources of distortion, and the correction for projector distortion is applied first, followed by the correction for homography transform distortion. The homography transform also corrects for any rotational misalignment of the foveated display panel. In one embodiment, the correction for scan distortion is applied last.

In one embodiment, both projector distortion and scan distortion may be further decomposed into a plurality of radial distortions. This may be done for computational efficiency or to account for misalignment between optical elements. Being able to apply these distortion corrections allows the design constraints of the optical system to be relaxed. In one embodiment, the total distortion of the system at the edge of the foveal field of view may be up to 50%, and the rotational misalignment of the panels may be up to 180 degrees.

In one embodiment, there are separate image corrections for different focal planes. In one embodiment, the image correction includes distortion correction, color correction, brightness/vignetting correction, temporal filtering, blurring, and dithering. In one embodiment, all of these corrections are applied to a given focal plane at once. In another embodiment, one or more of these corrections are applied separately or independently.

In one embodiment, if the foveal display is multi-focal, the data alternates between multiple focal planes of the foveal display.

Data is packed for the exchange chain.

The foveal swap chain or another buffer is then updated.

Returning to the determination of whether the foveal manager is valid for rendering, in one embodiment, if foveal rendering is valid but mirror movement is deferred, then the previous foveal texture is sampled. In one embodiment, mirror movement may be delayed because the mirror cannot reach the desired position and cannot be fixed in the desired position within the time frame required to properly display the foveal image. In one embodiment, mirror movement may be deferred due to missing the time limit of the event triggering the move command.

Since the mirror movement is postponed, the updated rendering of the new position based on the projection of the mirror is inaccurate. Thus, based on the delayed movement of the mirror, the previous foveal texture is sampled and the cut position of the field display is matched to the actual position of the cut.

Mirror deferral may occur when foveal rendering is effective for various reasons, including but not limited to:

a pipe delay in which the mirror movement is postponed to coincide with the time at which the updated foveal image will be displayed to the user;

the mirror is not yet ready for a new position movement;

moving the mirror at this time may reduce the user's visual experience;

content in the scene is used to indicate mirror position, and the content of interest is not positionally aligned with the most recently rendered foveal frame;

the user's gaze has returned to the previous location;

detecting an erroneous gaze movement;

a large saccade is detected and the system is preparing for a large mirror movement, or changing the size of a mirror movement;

the host system detects that it has missed the time limit to send a moving mirror message down to the mirror;

indicating that new pose data for mirrors that should not be moved has arrived;

and the like.

When rendering is active, but mirror movement is deferred, the sampled previous foveal texture will be passed on to distortion correction.

If the foveal manager determines that foveal rendering is not valid, it means that foveal rendering data cannot be used and the image should be re-rendered in the correct position. In one embodiment, this is referred to as a slow path because the rendering engine must re-render the foveal image. In one embodiment, the foveal display may resample from one or both of the re-projected field texture and the foveal texture to display a foveal image. In one embodiment, if the actual foveal region does not overlap with the rendered data, the system may turn off the foveal display and utilize only the field display. In one embodiment, the fast LED turn-off technique discussed above is used.

As described above, the synthesized field display data and the foveated display data are both packed into one frame. In one embodiment, the right and left image data is packed into a single display frame. In one embodiment, the frame packing format may vary. In one embodiment, frame packing may pack the left image and the right image into a single display frame. In one embodiment, all four data sets (left and right data sets for a field display and left and right data sets for a foveated display) may be packed into a single frame. In one embodiment, each data set (left and right data sets for a field display and left and right data sets for a foveated display) is packed into its own separate frame.

In one embodiment, the system may use custom interleaving of frame packing to account for the relative sizes of the field display and the foveated display. In one embodiment, the foveated display may be 720x720 and the field display is 1280x 720. Frame packing may be packed as 2000x720 frames, covering both. A Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), System On Chip (SOC), or other processor may be used within the display controller to unpack frame data for the display.

In one embodiment, the field display and the foveal display for a single eye are each part of a single display. In this case, the field display image and the foveated display image may be packed into a single larger frame, where the larger frame is the size of the display panel. For example, the entire display for one eye may be a 1280x720 panel, and the field display may be packed into the 640x720 section on the left side of the panel, while the foveal display may be packed into the 640x720 section on the right side of the panel. In another embodiment, the field display and the foveated display are different resolutions. In another embodiment, the combination of the field display and the foveated display does not use up the entire display panel. In this embodiment, the unused pixels may be black in one embodiment, or optically blocked in the system, allowing optical separation between the field display and the foveated display. In one embodiment, unused portions of the display panel may be physically blocked. This may provide an improved contrast ratio.

Some frame packing methods that may be used include:

stereoscopic left/right color of field (into one display frame);

the foveated stereoscopic left/right color (in one display frame) at time T0 for focal plane 0, and sent alternately to the foveated display; and

the stereoscopic left/right color of the fovea at time T1 for focal plane 1 (in one display frame) is sent to the foveal display alternately.

In another embodiment, the packing level may be different. For example, in one embodiment, all rendered textures (color + depth) are packed and sent together to one device with multiple displays (essentially synthesized on the device, containing the depth to focal plane conversion). In another embodiment, all synthesized textures are packed into a single frame and sent to one device (which, after unpacking, renders the field and focal planes of T0 and T1 in turn). In another embodiment, all synthesized textures of the above single time T0 or T1 are sent together to one device and host sequence. In another embodiment, all foveal textures (for multiple planes) are packed together rather than sent in time sequence, but the field textures are sent independently and the device orders the planes. In one embodiment, the textures for all focal planes to the left/right of the volume are sent independently (substantially without packing). In one embodiment, the packed frame data may contain only a subset of the entire frame. In one embodiment, the frame data may be compressed.

FIG. 4B is a block diagram of one embodiment of the elements of the foveal manager 455. The foveal region processor 460 utilizes system data 495 that describes the system and mirror capabilities and time delays, and state data 490 that contains the current mirror position and time until vertical synchronization, as well as other data that describes the current state of the system. In one embodiment, the foveal region processor 460 receives position and vector data from the predictive pose logic 465, scene metadata 475 for the upcoming scene, and gaze vectors and eye movement classifications from the predictive gaze system 467. The foveal area processor 460 uses this data to calculate the foveal viewport 480 and mirror control output 485.

In one embodiment, foveal area processor 460 utilizes real-time inputs to generate real-time outputs. However, the foveal area processor 460 may utilize analysis based on various data inputs to determine where the foveal display should be directed. In one embodiment, the system uses machine learning to calculate the optimal position and data for the foveal display. Foveal region processor 460 uses advanced filtering to eliminate non-productive movement based on available data, including state data, system data, and upcoming scene data/predicted gaze and pose data. In one embodiment, the advanced filtering is specific to a particular use case. In one embodiment, the advanced filtering is specific to the user. In one embodiment, the foveal region processor may determine the content of the display and the location of the display to reduce artifacts/optimize visual quality. The foveal region processor may also take into account other factors in determining the foveal viewport 480 and associated mirror control output 485, such as reducing mirror movement, reducing eye strain, and reducing power consumption.

In one embodiment, the foveal area processor 460 receives a request for XYZ in the future, such as displaying foveal area data at a particular location. Foveal region processor 460 determines what action it will take, for example, at time a 'at X' Y 'Z', based on the above inputs. This data is fed back to the rendering engine to render the X 'Y' Z 'data for availability to the foveated display at time A'.

Fig. 5A-5B are flow diagrams of one embodiment of eye tracking and mirror control in a system. Fig. 5A shows one embodiment of a host CPU and eye tracker flow, while fig. 5B shows one embodiment of a corresponding headset microcontroller unit (MCU) and mirror control flow.

The process begins at block 505 when new eye tracking data is received. At block 510, the data is filtered, and a new line of sight vector is calculated.

In one embodiment, the data is then analyzed. In one embodiment, the eye movement is classified. In one embodiment, the classification may include movements such as panning and smooth tracking. In one embodiment, the elapsed time since the last mirror was moved is also analyzed, and the system determines if moving the mirror would degrade the user experience.

At block 515, the system then decides whether to move the mirror based on whether moving the mirror would degrade/improve the user experience. If it is determined that the mirror is not to be moved, the process ends at block 520. The mirror does not move and no new frame to be displayed is generated.

If it is determined at block 515 to move the mirror, the process at block 522 determines whether the movement of the mirror should be postponed. As described above, mirror movement may be postponed due to data limitations, physical limitations, or other reasons. If the process determines to defer movement at block 522, then at block 524 mirror movement is deferred for a desired period of time. In one embodiment, the mirror movement may be postponed for one frame. After the delay ends, the process returns to block 525. If the move is not to be deferred, then at block 522 the process proceeds directly to block 525.

At block 525, the process sends the new foveal region data to the renderer. The renderer renders the latest foveal region at block 530. In one embodiment, the rendering is performed asynchronously to the mirror movement.

When the movement instruction is received, and in one embodiment in parallel with sending new foveal area data to the renderer, at block 535 the system sends new mirror coordinates to the head-mounted device. In one embodiment, the data sent to the head mounted device includes a move mirror command with destination coordinates. In one embodiment, the data further includes mirror movement delay data specifying a length of the mirror delay. The mirror delay is the time until the mirror movement is initiated. In one embodiment, the mirror delay also includes the time until the foveal display should be stopped. This delay may be referred to as a mobile delay time.

In one embodiment, the position and mirror delay (if any) are calculated based on eye movement classification and gaze data. This data is sent to the headset MCU and mirror controls at block 537 along with the new mirror coordinates. In one embodiment, the foveal display is extinguished during movement of the mirror.

Because the mirror movement causes the foveated display to temporarily lack usability in one embodiment, the hole in the field display at the old position for the foveated display is filled at block 540. In one embodiment, the system looks up the number of field display frames to fill based on the superposition of the magnitude of the mirror movement and the display presentation time. In one embodiment, the length of time that the mirror is moving and settling time is calculated using a look-up table. In one embodiment, a mirror stabilization signal indicating that the movement of the mirror is within the stabilization threshold, and thus that the foveal display is available for use, may be sent back from the head-mounted MCU (see block 585) to end this cycle.

In one embodiment, the data sent to the head mount MCU and mirror controls includes stabilization criteria that define when the mirror reaches its destination and the foveal display can safely display without degrading the user experience. In one embodiment, this stability criterion includes position accuracy data and the amount of time the mirror must be within the accuracy limits of the target destination.

In one embodiment, the stability criteria are dynamically set. The stability criteria may be dynamically determined based on the magnitude of the steering movement, the magnitude of gaze changes reported by the gaze detector, the content in the scene, and the like.

In one embodiment, when the time length to fill a hole in a field display is 0 frames, while the field display never fills a hole in a field display, then a bypass path is used at block 542, proceeding directly to block 555.

In one embodiment, the padding data may be rendered field display data, rendered foveal data, foveal data from a previous frame, reprocessed field display data, or a combination of the above types of data.

In one embodiment, at block 545, the padding is with progressive luminance padding and non-padding. In one embodiment, at block 545, the padding may use one or more filtering methods, such as antialiasing, spatial dithering, temporal dithering, blurring, and the like. Various filtering methods to fill field displays aim to prevent visual artifacts during the filling process, such as flashing caused by sudden changes in brightness of the short persistence display. In one embodiment, progressive blending is utilized to minimize visual artifacts at the end of saccade masking roll-off.

A short persistence display turns on only a small portion of a frame, for example 2 milliseconds of an 11 millisecond frame; this is to prevent the head mounted display from shaking. When the field display is filled immediately after the foveal frame is displayed, the shock may produce unwanted glints because the glints from the short-persistence field display are added to the previously displayed foveal frame, and the eye thinks this combination is too bright. In one embodiment, this flashing can be mitigated by reducing the brightness of the field display fill for the first frame after the foveated display is turned off. In one embodiment, only half the luminance is filled for the field displays in the first frame. In one embodiment, the luminance of the field display may be 10% to 90% for the first frame. In one embodiment, this means that the user perceives a uniform brightness during the transition. In one embodiment, the field display brightness gradually increases over several frames as the cuts are filled.

At block 550, in one embodiment, the process determines whether sufficient time has elapsed for the mirror to move. In one embodiment, this determination is made based on a look-up table that indicates the amount of time moved. If sufficient time has not elapsed, the process continues to wait. In one embodiment, during standby, the field display is shown without a cutout as described above.

Once sufficient time has elapsed, at block 555, in one embodiment, the process determines whether the generated foveal image is correct for the mirror destination. If not, in one embodiment, the process uses data from the previous foveal frame. In another embodiment, field data may be used. In one embodiment, what data is used depends on the most accurate data available for the actual position of the foveal mirror to ensure that there is no position mismatch and thus an error.

Once the foveated image is determined to be correct, at block 560, a foveated hole is cut out of the field display at the new location. In one embodiment, an "enable foveal display" command is sent to the headset at block 565. In one embodiment, these two parts, blocks 560 and 565, are asynchronous. The timing diagram of fig. 6 provides more detail.

Turning to fig. 5B, at block 570, the headset MCU receives the move mirror command sent by the host CPU and eye tracker at block 537.

At block 571, the process determines whether the mirror movement should be postponed, as determined at block 537. If postponed, the mirror movement and display of the foveal display are delayed based on the data provided.

At block 573, in one embodiment, the foveated display is not shown. In one embodiment, the foveated display is not shown because the illumination source of the foveated display is disabled. In effect, deactivating the foveal display illumination source turns off the foveal display. In one embodiment, the foveated display may be blanked and not displayed by sending a blank (black) frame. In one embodiment, instead of disabling the foveal display illumination source or sending a black frame, the system may stop displaying the foveal display in another manner.

In one embodiment, at block 575, the movement is delayed long enough to ensure that the foveal display is not displayed. In one embodiment, the delay is 1.2 milliseconds. In one embodiment, the delay is less than 4 milliseconds. In one embodiment, there is no delay, as the delay time in sending the "move mirror command" is sufficient to ensure that the illumination source is off.

Additionally, in one embodiment, the system looks up the mirror movement control coefficients for this mirror movement at block 577. These parameters are based on the starting position, the scale of movement and the direction of movement. The direction is relevant in one embodiment as it may affect the return spring force of the movement. In one embodiment, a PID (proportional, integral, derivative) control loop is used. At block 580, the process updates the control loop coefficients and sets a new position target in the control loop. Mirror movement then begins.

At block 583, a move is performed. In one embodiment, the movements used are fast movements and braking, known as bang-bang control. In one embodiment, a brief hard start initiates movement to overcome the noise floor, and then the control algorithm takes over the movement to properly position the mirror. In one embodiment, at the end of the movement, the mirror may have residual vibration.

At block 585, the process determines whether the mirror has stabilized at the new target destination. In one embodiment, the mirror is considered stable when it is no longer moving or vibrating. In one embodiment, a mirror is considered stable when the remaining mirrors move and inaccuracies with respect to final position do not degrade the visual experience of a typical viewer. In one embodiment, a movement of 1 to 5 arc minutes is considered stable. As discussed above, in one embodiment, the threshold for stabilizing the mirror is dynamic. If the mirror is stable, at block 587, the process determines whether an "enable foveal display" command has been received. This command is received from the host CPU and eye tracker, which is sent at block 565 discussed above.

If the mirror is stable, the illumination source for the foveated display is enabled at block 580. At this point, the user sees the field display of the correctly positioned foveated display in the cutout of the field display.

FIGS. 5A and 5B are flow diagrams of steering control in one embodiment of a steerable foveal display. Those skilled in the art will understand how to extend the same system to other similar embodiments. As mentioned above, in one embodiment, the steerable element is not a mirror. In one embodiment, the host CPU and eye tracker are integrated with the headset MCU and mirror control.

Fig. 6 and 7 are timing diagrams showing embodiments of timing architectures.

FIG. 6 illustrates one embodiment of a timing diagram showing the relative timing of actions taken by the eye tracker system, the field display, the foveated display frame buffer, and the foveated rendering window. Further, commands and user views to a Head Mounted Display (HMD) are also shown.

It should be noted that while the relative timing of the commands is based on the relationship between these systems, in one embodiment, the field display and the foveated display are on separate clocks and are not synchronized.

Fig. 6 shows a period of 60 milliseconds after reporting a new gaze vector (from the eye tracker system) for smooth tracking type eye movement. In this illustration, the mirror movement takes 9 milliseconds, during which field frame 0 is displayed and the perforations are filled, and the foveal frame is not shown. It can be seen that at time 21 milliseconds, the foveal LED is off, while at time 32 milliseconds, the foveal LED is turned back on after the mirror movement is complete and the mirror is reported to have stabilized.

Fig. 6 also shows the duty cycle of the field display. In one embodiment, the field display lights up for 2 milliseconds within 11 milliseconds (e.g., the field display is 9 milliseconds dark). Duty cycles of this type are known in the art of displays. In one embodiment, the foveal display is displayed continuously without such duty cycles. In another embodiment, the foveated display may also illuminate only a portion of the frame in the same manner that the field display is illuminated.

In one embodiment, the foveated display has multiple depths, for example it is a multi-focus display. In one embodiment, for such a multi-focus display, the depths are displayed sequentially, e.g., for frame 0, the first frame portion is depth a and the second frame portion is depth B. It should be noted that although only two frame depths are shown, a multi-focus display may include more than two frame depths. In one embodiment of a multi-focus display, frame 0 displays depth a and frame 1 displays depth B.

Unlike smooth tracking, the magnitude of saccadic motion varies greatly. Thus, motion may extend over multiple frames. This means that there may be multiple field frames without a foveated display. Those skilled in the art will appreciate the extension of the frame display timing shown in fig. 6 for longer movements.

FIG. 7 shows one embodiment of a timing diagram for a synchronization system. In one embodiment, the system is synchronized and all subsystems run on the same time base, e.g., on a system-on-chip where a single integrated controller controls all elements of the system. In one embodiment, the foveated display has an off period per frame. In one embodiment, the manipulable element is moved during the off period of the foveated display so that the visual experience of a typical viewer is not degraded when the manipulable element is moved. In one embodiment, the field display and the foveated display are each part of a single larger display, as discussed above.

In fig. 7, the field display and the foveated display are refreshed synchronously. Further, the foveated display has a blanking period. When the steerable element of the foveated display needs to be moved, it moves during this blanking period. In the example of fig. 7, the new line of sight position is reported at 22 milliseconds. The steerable display foveal display and field display are updated at 11 milliseconds and then a new gaze location is displayed at 33 milliseconds for one frame. In this example, both the field display and the foveated display have a frame rate of 90 Hz.

The synchronization displays may also operate with the field display and the foveated display at integer multiples of each other.

FIG. 8 is a block diagram of a computer system that may be used with a rendering engine. However, it will be apparent to those of ordinary skill in the art that other alternative systems of various system architectures may also be used.

The data processing system depicted in FIG. 8 includes a bus or other internal communication means 840 for communicating information, and a processing unit 810 coupled to bus 840 for processing information. The processing unit 810 may be a Central Processing Unit (CPU), a Digital Signal Processor (DSP), or other type of processing unit 810.

In one embodiment, the system further includes Random Access Memory (RAM) or other volatile storage 820 (referred to as memory), coupled to bus 840 for storing information and instructions to be executed by processor 810. Main memory 820 also may be used for storing temporary variables or other intermediate information during execution of instructions by processing unit 810.

In one embodiment, the system further includes a Read Only Memory (ROM)850 and/or a static storage device 850 coupled to bus 840 for storing static information and instructions for processor 810. In one embodiment, the system also includes a data storage device 830, such as a magnetic or optical disk and its corresponding disk drive, or flash memory or other memory capable of storing data when power is not supplied to the system. In one embodiment, data storage device 830 is coupled to bus 840 for storing information and instructions.

The system may further be coupled to an output device 870, such as a Cathode Ray Tube (CRT) or Liquid Crystal Display (LCD) coupled to bus 840 via bus 860, for outputting information. The output device 870 can be a visual output device, an audio output device, and/or a tactile output device (e.g., a vibration, etc.).

An input device 875 may be coupled to the bus 860. Input device 875 may be an alphanumeric input device such as a keyboard including alphanumeric and other keys for enabling a user to communicate information and command selections to processing unit 810. Additional user input devices 880 may be further included. Such user input devices 880 are cursor control devices 880, such as a mouse, trackball, stylus, cursor direction keys, or touch screen, which may be coupled to bus 840 via bus 860 for communicating direction information and command selections to processing unit 810 and for controlling movement on display device 870.

Another device that may optionally be coupled to computer system 800 is a network device 885 for accessing other nodes of the distributed system via a network. The communication device 885 may comprise any of a number of commercially available networking peripheral devices, such as those used for coupling to ethernet, token ring, internet or wide area networks, personal area networks, wireless networks, or other methods of accessing other devices. The communications device 885 may further be a zero modem connection, or any other mechanism that provides a connection between the computer system 800 and the outside world.

It should be noted that any or all of the components of this system and associated hardware shown in FIG. 8 may be used in various embodiments of the present invention.

Those of ordinary skill in the art will appreciate that a particular machine embodying the present invention may be configured in a variety of ways depending on the particular implementation. Control logic or software implementing the present invention may be stored in main memory 820, mass storage device 830, or other storage medium that is accessible locally or remotely to processor 810.

It will be apparent to those of ordinary skill in the art that the systems, methods, and processes described herein can be implemented as software stored in main memory 820 or read only memory 850 and executed by processor 810. This control logic or software may also reside on an article of manufacture comprising a computer readable medium having computer readable program code embodied therein and accessible by the mass storage device 830 for causing the processor 810 to operate in accordance with the methods and teachings herein.

The invention may also be embodied in a handheld or portable device containing a subset of the computer hardware components described above. For example, a handheld device may be configured to contain only the bus 840, the processor 810, and the memories 850 and/or 820.

The handheld device may be configured to include a set of buttons or input signaling components that the user may utilize to select from a set of available options. These devices may be considered input device # 1875 or input device # 2880. The handheld device may also be configured to include an output device 870, such as a Liquid Crystal Display (LCD) or matrix of display elements, for displaying information to a user of the handheld device. Such handheld devices may be implemented using conventional methods. The practice of the present invention with such devices will be apparent to one of ordinary skill in the art in view of the disclosure of the invention provided herein.

The invention may also be embodied in a special purpose device that contains a subset of the computer hardware components described above, such as a kiosk or a vehicle. For example, an apparatus may include processing unit 810, data storage device 830, bus 840, and memory 820, and have no input/output mechanisms, or only basic communication mechanisms, such as a small touch screen that allows a user to communicate with the device in a basic manner. Generally, the more specialized the device, the fewer components required for the device to function. In some devices, communication with the user may be through a touch-based screen or similar mechanism. In one embodiment, the device may not provide any direct input/output signals, but may be configured and accessed through a website or other network-based connection through the network device 885.

Those of ordinary skill in the art will appreciate that any configuration of a particular machine implemented as a computer system may be used in accordance with a particular embodiment. The control logic or software implementing the present invention may be stored on any machine-readable medium that is locally or remotely accessible to the processor 810. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, a machine-readable medium includes read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash-memory devices, or other storage media that can be used for temporary or permanent data storage. In one embodiment, the control logic may be implemented as a transmittable data, such as electrical, optical, acoustical or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.).

In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.