Temporal supersampling for point of gaze rendering systems
阅读说明:本技术 用于注视点渲染系统的时间超级采样 (Temporal supersampling for point of gaze rendering systems ) 是由 A.扬 C.霍 J.R.斯塔福德 于 2018-06-07 设计创作,主要内容包括:提供用于使用时间超级采样来提高与注视点渲染视图的周边区相关联的显示分辨率的方法和系统。提供一种用于使得能够从低分辨率采样区重构较高分辨率像素以获得片段数据的方法。所述方法包括用于从GPU的光栅器接收片段并且用于在多个先前帧上对具有所述低分辨率采样区的所述片段应用时间超级采样以获得多个颜色值的操作。所述方法还包括用于基于经由所述时间超级采样获得的所述多个颜色值来在缓冲器中重构多个高分辨率像素的操作。此外,所述方法包括用于发送所述多个高分辨率像素以进行显示的操作。(Methods and systems are provided for using temporal super sampling to improve display resolution associated with peripheral regions of a gaze point rendered view. A method is provided for enabling higher resolution pixels to be reconstructed from a low resolution sampling region to obtain fragment data. The method includes operations for receiving a fragment from a rasterizer of a GPU and for applying temporal supersampling on a plurality of previous frames to the fragment having the low resolution sample region to obtain a plurality of color values. The method also includes an operation for reconstructing a plurality of high resolution pixels in a buffer based on the plurality of color values obtained via the temporal supersampling. Further, the method includes an operation for transmitting the plurality of high resolution pixels for display.)
1. A method for rendering higher resolution pixels from a low resolution sampling region, comprising:
receiving a fragment from a rasterizer;
applying temporal supersampling to the segment having the low resolution sample region over a plurality of previous frames to obtain a plurality of color values;
reconstructing a plurality of high resolution pixels in a buffer based on the plurality of color values obtained via the temporal supersampling; and
sending the plurality of high resolution pixels from the buffer for presentation on a display.
2. The method of claim 1, wherein the temporal supersampling comprises sampling at a location within the low resolution sample region of each of the plurality of previous frames.
3. The method of claim 1, wherein the temporal supersampling comprises sampling locations of the low resolution sampling regions as determined by pixel re-projection.
4. The method of claim 1, wherein the temporal supersampling comprises sampling positions of the low resolution sampling region as determined by dithering.
5. The method of claim 1, wherein a number of the plurality of high resolution pixels is greater than a number of the plurality of previous frames.
6. The method of claim 1, wherein the reconstruction of the plurality of high resolution pixels further comprises a blending of the plurality of color values.
7. The method of claim 1, wherein the plurality of high resolution pixels are associated with a native resolution of the display or a resolution greater than the native resolution of the display.
8. The method of claim 1, wherein the display is associated with a Head Mounted Display (HMD).
9. The method of claim 1, wherein the low resolution sampling regions are associated with a peripheral region of a point of regard rendered view.
10. A graphics system, comprising:
a Graphics Processing Unit (GPU) to apply temporal supersampling to a plurality of previous frames including a low resolution sample region, wherein the temporal supersampling obtains a plurality of color values;
a frame buffer to store the plurality of previous frames rendered by the GPU; and
a display buffer, wherein a plurality of high resolution pixels are reconstructed based on the plurality of color values obtained via the temporal supersampling of a previous frame;
wherein the plurality of high resolution pixels are configured for presentation on a display.
11. The graphics system of claim 10, wherein the temporal supersampling comprises sampling at a location within the low resolution sample area of each of the previous frames to obtain the plurality of color values.
12. The graphics system of claim 10, wherein the temporal supersampling defines sampling locations of the low resolution sampling regions as determined by pixel reprojection.
13. The graphics system of claim 10, wherein the temporal supersampling defines sampling locations of the low resolution sampling regions as determined by dithering.
14. The graphics system of claim 10, wherein a number of the plurality of high resolution pixels is greater than a number of the plurality of previous frames.
15. The graphics system of claim 10, wherein the reconstruction of the plurality of high resolutions further comprises a blending of the plurality of color values.
16. The graphics system of claim 10, wherein the plurality of high resolution pixels are associated with a native resolution of the display or a resolution greater than the native resolution of the display.
17. The graphics system of claim 10, wherein the display is associated with a Head Mounted Display (HMD).
18. The graphics system of claim 10, wherein the low resolution sampling region is associated with a peripheral region of a point of regard rendered view.
19. A non-transitory computer-readable medium storing a computer program executable by a processor-based system, comprising:
program instructions for receiving a fragment from a rasterizer, the fragment being associated with a low resolution sample region;
program instructions for applying temporal supersampling on a plurality of previous frames to the segment to obtain a plurality of color values;
program instructions for reconstructing, in a buffer, a plurality of high resolution pixels associated with the low resolution sample region, the plurality of high resolution pixels being based on the plurality of color values obtained via the temporal supersampling; and
program instructions for sending the plurality of high resolution pixels from the buffer for presentation on a display.
20. The non-transitory computer readable medium of claim 19, wherein the temporal supersampling defines a sampling location of the low resolution sampling region on the plurality of previous frames as determined by pixel reprojection or as determined by dithering, and wherein the plurality of high resolution pixels are associated with an original resolution of the display or a resolution greater than the original resolution of the display.
Technical Field
The present disclosure relates generally to a point-of-gaze rendered view for Virtual Reality (VR) content provided by a Head Mounted Display (HMD), and more particularly to methods and systems for generating higher resolution pixels in certain regions within the point-of-gaze rendered view using temporal super sampling.
Background
Virtual Reality (VR) presented through Head Mounted Displays (HMD) is increasingly becoming a way for consumers to interact with various types of content. As VR applications used to generate VR content become rendered with higher and higher resolution images and higher complexity, the computational, network, and memory costs required to support these VR scenes also increase. For example, as image resolution is increased, the associated graphics pipeline needs to perform more and more operations associated with generating pixel data from geometry data generated by a VR application. Also, the amount of memory required to store the geometry and pixel data required to run a VR application may increase proportionally. Further, if the VR application is executed on a computing system that communicates with the HMD over a network connection (e.g., wired or wireless), the amount of data that would otherwise need to be sent over the network connection increases.
Thus, bottlenecks often occur when executing VR applications with computing and graphics requirements. Bottlenecks can result in reduced frame rate (frames per second), increased delay or lag, reduced resolution, and increased aliasing, all of which are detrimental to the overall user experience. Some attempts to reduce the computational, memory, and network costs associated with executing VR applications have resulted in VR scenes with lower resolution, pixelation, visual artifacts, etc., which can negatively impact the VR experience.
It is in this context that various embodiments have emerged.
Disclosure of Invention
Embodiments of the present disclosure provide methods and systems for enabling reconstruction of higher resolution pixels for display in an undersampled region of a VR scene by using temporal supersampling. In one embodiment, a method for reconstructing higher resolution pixels from a low resolution sampling region is provided. The method provides operations for receiving a fragment from a rasterizer. The method also includes an operation for applying temporal supersampling on a plurality of previous frames to the segment having the low resolution sample region to obtain a plurality of color values. According to certain embodiments, the method may further include an operation for reconstructing a plurality of high resolution pixels in a buffer based on the plurality of color values obtained via the temporal supersampling. Further, the method includes an operation for sending the plurality of high resolution pixels from the buffer for presentation on a display. Thus, the provided methods are capable of rendering higher resolution images that are sent for display without the large and sometimes expensive memory usage typically associated with rendering high resolution images. Thus, the method provides a solution to the technical problem of being able to increase the image resolution associated with VR scenes while maintaining low memory usage.
In another embodiment, a graphics system includes a Graphics Processing Unit (GPU) to apply temporal supersampling to a plurality of previous frames including a low resolution sample region, wherein the temporal supersampling obtains a plurality of color values.
In another embodiment, a non-transitory computer readable medium storing a computer program executable by a processor-based system includes program instructions for receiving a fragment from a rasterizer, the fragment being associated with a low resolution sample region. The implementation also includes program instructions to apply temporal supersampling to the segment over a plurality of previous frames to obtain a plurality of color values. Program instructions are also provided in an embodiment for reconstructing, in a buffer, a plurality of high resolution pixels associated with the low resolution sampling region, the plurality of high resolution pixels being based on the plurality of color values obtained via the temporal supersampling. Additionally, implementations provide program instructions for sending the plurality of high resolution pixels from the buffer for presentation on a display.
Other aspects of the disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the disclosure.
Drawings
The disclosure may best be understood by reference to the following description taken in conjunction with the accompanying drawings, in which:
fig. 1A and 1B illustrate presenting Virtual Reality (VR) content with two resolutions to a Head Mounted Display (HMD) user, in accordance with various embodiments.
Fig. 2A and 2B illustrate presenting VR content to an HMD user, the VR content having a foveal area, a mid foveal area, and a peripheral area, in accordance with certain embodiments.
Fig. 3A-3H illustrate various embodiments of point-of-regard rendering views.
Fig. 4 illustrates a multi-resolution display or screen defined by a gaze-point rendered view and an unfolded view of associated relative pixel sizes, according to some embodiments.
Fig. 5 illustrates a screen defined by a gaze point rendered view having a foveal region, an intermediate foveal region, and a peripheral region, and a conceptual scheme for reconstructing higher resolution pixels in a low resolution peripheral region, according to some embodiments.
Fig. 6 illustrates a conceptual scheme for reconstructing a set of higher resolution pixels from a low resolution sample region using temporal super sampling and pixel re-projection over multiple frames stored in a buffer, according to various embodiments.
Fig. 7 illustrates a conceptual scheme for outputting high resolution pixels using high resolution sample regions according to one embodiment.
Fig. 8 illustrates a conceptual scheme for outputting low-resolution pixels using low-resolution sample regions according to one embodiment.
Fig. 9 illustrates a conceptual scheme for outputting high resolution pixels for a static object using a low resolution sampling region by temporal super sampling according to one embodiment.
Fig. 10 illustrates a conceptual scheme for outputting high resolution pixels for a dynamic object using low resolution sample regions by temporal super sampling according to one embodiment.
Fig. 11 illustrates a conceptual model for generating higher resolution pixels from low resolution pixels for sampling by utilizing temporal super sampling with a regular sampling pattern.
Fig. 12 illustrates a conceptual model for generating higher resolution pixels from low resolution pixels for sampling by utilizing temporal supersampling with a quasi-random sampling pattern.
Fig. 13A illustrates an embodiment of reconstructing a set of 16 high resolution pixels from a low resolution sampling region used during temporal super sampling over 16 frames.
Fig. 13B illustrates an embodiment of reconstructing a set of 16 high resolution pixels from a low resolution sample region used during temporal super sampling over a number of frames less than the number of high resolution pixels.
Fig. 14 illustrates the overall flow of a method that enables reconstruction of higher resolution pixels from a low resolution sample region using color values obtained by temporal supersampling over multiple previous frames.
Fig. 15 illustrates an additional embodiment of a Head Mounted Display (HMD) that may be used with the proposed method and/or system.
FIG. 16 is a diagram of a
Detailed Description
The following embodiments describe methods, computer programs, and devices for increasing the final display resolution of a region within a VR scene associated with a lower resolution sampling region by time super-sampling the low resolution sampling region. It will be apparent, however, to one skilled in the art, that the present disclosure may be practiced without some or all of these specific details. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present disclosure.
The Virtual Reality (VR) environment provided by HMDs is an increasingly popular medium for consumers to interact with content and for content creators to deliver content to consumers. Furthermore, as VR scenes become more complex and displayed at higher resolutions, computational, memory, and network costs also increase. Thus, improvements to current methods of computer graphics rendering and anti-aliasing of VR scenes displayed via HMDs would be beneficial in terms of computing, memory and network resources, as well as the VR experience for the end user.
One way to reduce the computational, memory, and network costs (and associated delays) of a particular VR scene described herein is to display the VR scene using a point-of-gaze rendering view. According to one embodiment, point-of-regard rendering may define regions within the display that are displayed at a higher resolution, quality, level of detail, sharpness, frame rate, etc. than other regions. According to these and other embodiments, the region with higher resolution (or higher quality, level of detail, sharpness, frame rate) may be referred to as a fovea or fovea, and is generally related to the location at which the user is gazing or gazing. In addition, regions that do not have a higher level of resolution may be referred to as peripheral regions or perimeter regions, and may generally be associated with regions at which the user is not gazing. Thus, the point-of-regard rendering view and/or system represents one such solution to the technical problem that can reduce the computational, memory, and network costs associated with rendering a VR scene without negatively impacting the user's experience.
For regions rendered at a lower resolution (e.g., peripheral regions), the amount of pixel and/or fragment data that needs to be stored in memory to render the low resolution regions is correspondingly reduced. For example, if the resolution of a given region within a scene is reduced by a factor of 4, the amount of memory required to store pixel data for each video frame of the region within the scene will be proportionally reduced by a factor of 4. According to some embodiments, regions rendered at a lower resolution (e.g., peripheral regions) may also be referred to as undersampled regions because these regions are sampled at a lower frequency.
As mentioned above, reducing the amount of memory used to render each video frame of a given scene would be beneficial for VR systems because typically the speed of memory cannot be kept synchronized with the speed of a processor, such as a Central Processing Unit (CPU) or a Graphics Processing Unit (GPU). Therefore, reducing the resolution associated with the peripheral region within a point-of-regard rendered view by reducing memory usage and by maintaining consistent memory access would be an improvement over existing VR systems. For example, one of the improvements resulting from the point-of-regard rendering system described herein may include an improvement in the delay or lag associated with rendering an interactive VR scene, which may be noted by currently average HMD users.
However, as the resolution of a given computer-generated scene decreases, the frequency and extent of low-resolution associated artifacts may increase in the form of jagged edges or lines ("jaggies"), pixelation, and other visual artifacts. Even if these low resolution regions are limited to only peripheral regions within the point-of-regard rendered view (e.g., the user's peripheral vision), the HMD user may still be able to identify certain types of aliasing due to the reduction in resolution in these regions. It is known in the relevant art that although foveal vision is generally not easily resolved by human peripheral vision, it is sensitive to detecting certain types of visual inconsistencies or patterns. For example, if the resolution is reduced low enough, the user's peripheral vision will be able to detect the presence or appearance of pixilated areas, jagged edges, flicker, and other forms of aliasing or graphics artifacts. Thus, there is a need to both maintain low memory usage by rendering relatively lower resolution in the peripheral region of the display, and to reduce aliasing associated with low resolution regions within the point-of-regard rendering system and/or view.
The systems, methods, and devices described herein enable a point of regard rendering system and/or view to maintain a reduction in memory usage associated with low resolution regions while reducing the degree of pixelation and aliasing of those low resolution regions. In one embodiment, a system or method uses temporal supersampling on a low resolution sampling region to sample at different locations within a low resolution pixel over a specified number of past frames to create a higher resolution pixel for display. Temporal supersampling records a plurality of pixel values sampled from a plurality of temporally separated frames. It should be noted that according to some embodiments, a single buffer (e.g., within video RAM) may be used to accumulate these pixel values over time. These embodiments would have the advantage of not requiring multiple buffers (frames) of data to be maintained. Thus, the use of temporal super-sampling for low resolution sampled regions (e.g., undersampled regions or peripheral regions) provides a technical solution that can be implemented to address the pixelation and aliasing problems associated with low resolution regions without requiring substantial increases in, for example, memory usage.
For some embodiments, the central recessed area may be fixed or static relative to the display. In such embodiments, the central recessed area may be located toward the center of the screen or display. In other embodiments, the foveal region may be dynamically positioned relative to the screen or display. For example, in some embodiments, a foveal region may be defined to move within a display or screen in a predetermined manner or by software programming. In other embodiments, the dynamic foveal region may track or follow a user's point of gaze (POG) or gaze direction. Thus, the area within the display corresponding to the user's POG is rendered at a higher quality, level of detail, and/or clarity than areas further away from the user's POG without compromising the user's visual experience.
In some embodiments, for locations where no foveal region is present, the peripheral region will be defined within the screen or display by the point of regard rendering. For example, if the central recessed region is positioned toward the center of the display, the peripheral region should occupy the remainder of the display (or at least a portion thereof) toward the periphery of the display. If the foveal region is to be moved to a different area of the display, the peripheral region should fill the rest of the display where no foveal region is currently present.
FIGS. 1A and 1B illustrate a two-resolution R1And R2To present Virtual Reality (VR) content to the
According to the embodiment shown in fig. 1A and 1B, the gazing
Also shown in fig. 1A is a
Thus, according to certain embodiments, the
According to some embodiments, for a given VR scene, the resolution R1May be higher than R2. In these embodiments, the
Point-of-regard rendering exploits the phenomenon of focus by providing a configuration, format and paradigm of rendering, post-rendering and/or graphics processing for display, wherein one or more zones (e.g., foveal zones) are defined by a higher level of resolution, a higher level of detail, a higher level of texture and/or a higher level of clarity than other zones. According to some embodiments, the foveal region is made to correspond to a region of the display that the user is currently viewing or predicting to be viewing. In other embodiments, the central foveal region may be placed in a static manner in the central region of the display that the user will spend a significant amount of time looking at. Also, as mentioned previously, the point of regard rendering may define a peripheral region of the display corresponding to where the user is not gazing or is predicted not to be gazed.
Embodiments contemplated herein enable the use of a point of regard rendering display configuration to take advantage of focused physiological phenomena by rendering and/or displaying higher quality (e.g., resolution, level of detail (LOD), clarity, frame rate) content within a display region associated with a field of view in focus of a user (e.g., a center of gaze and a surrounding field of view projected onto a fovea of a user). Additionally, embodiments contemplated herein are capable of displaying content with lower quality in regions of the display that are not associated with the center of the user's gaze (e.g., the user's peripheral field of view). Thus, only a portion of a given scene may be rendered and/or processed for display at high quality or high resolution under point-of-regard rendering, as compared to rendering the entire display or screen at full quality or full resolution.
One of the technical advantages of point-of-regard rendering is the ability to reduce the computational and video transmission costs associated with rendering and delivering a given scene at full quality (e.g., high resolution, sharpness, level of detail, frame rate, etc.) for the entire display (e.g., each pixel on the display). In the case of both wired systems (e.g., high definition multimedia interface (HMD) and/or displayport implementations) and wireless systems, there is a video transmission cost. By rendering a portion (e.g., 20 to 50%, 5 to 75%, 25 to 40%) of the entire display at high resolution and/or quality, computing resources (e.g., GPU, CPU, cloud computing resources) and video transmission resources (e.g., transmitting data to and from the HMD, from the computing device, and/or from the combined HMD/computing device to a remote server) may be reduced and allocated for other uses.
According to another embodiment, the point-of-regard rendering method and/or system may enable a reduction in the amount of data required to display a given scene on the HMD even if the GPU associated with the HMD/computing device computes full resolution video frames of the scene. For example, if the GPU is associated with a computing device wirelessly connected to the HMD, the gaze point rendering methods and/or systems described herein may enable a reduction in the amount of wireless data transmitted from the computing device to the HMD for rendering certain regions of the scene.
According to the embodiment shown in FIG. 1A, the
In one embodiment, it is contemplated that the
According to another embodiment, it is contemplated that the central recessed
According to some embodiments, the resolution R2Can be prepared from less than R1Any resolution of (2). As a non-limiting example, R2Can be distinguished byAnd (3) rate characterization: 320x240(240p), 640x360(nHD, 360p), 960x540(qHD, 540p), 1280x720(HD, 720p), 1600x900(HD +), etc. It is contemplated that R, according to various embodiments, is1And R2May change during the entire VR scene and/or between different VR scenes. Furthermore, the resolutions discussed are intended to be examples only, and are not limiting of various other resolutions that may be implemented in various embodiments, whether standardized or not.
Fig. 1B illustrates the
While certain embodiments have been shown with a dynamic foveal region that tracks the user's gaze direction, other embodiments may include a fixed foveal region that does not track the user's gaze direction.
Fig. 2A shows the
According to the embodiment shown in fig. 2A, the central
According to one embodiment, the
It is also contemplated that the central
Fig. 2B shows that the
As described above, the foveal region 212 may be made to correspond to approximately 5 to 75% of the HMD user's 101 field of view, or to 5 to 75% of the total displayable space within the
Fig. 3A-3H illustrate various embodiments of point-of-regard rendering views. For example, FIG. 3A illustrates a point-of-regard rendered display having a foveal region characterized by a circular boundary. Fig. 3B illustrates a gaze point rendered view having a foveal region characterized by an ellipse, an oblong or an oval, which may be used with the methods and systems described herein. Additionally, fig. 3C illustrates an embodiment of a point of regard rendering configuration, where the foveal region is shown as a rectangular shape with rounded corners.
Fig. 3D and 3E illustrate embodiments of a point-of-regard rendering view with a circular foveal region. Fig. 3D additionally shows a central foveal region that also has a circular shape, located outside the foveal region between the foveal region and the peripheral region. Further, fig. 3E illustrates two intermediate foveal regions arranged in a nested manner. It is contemplated that, in general, any number of intermediate foveates may be utilized in various embodiments, with each successive intermediate foveates that are increasingly distant from the foveates having progressively lower quality (e.g., resolution, sharpness, level of detail, frame rate, refresh rate) associated therewith. It is also contemplated that although the middle region is shown as having a similar shape to a given foveal region within the point-of-regard rendered display, in other embodiments such similarity is not required. For example, the intermediate regions of fig. 3D and 3E may be characterized by shapes other than circles.
Fig. 3F illustrates an embodiment of a point-of-regard rendered view and/or display having a dynamic foveal region bounded by a frame. In these and other embodiments, the foveal region may track the user's gaze such that the foveal region is shown within an area of the display and/or view that is consistent with the HMD user's gaze direction, so long as the user's gaze remains within a certain area characterized by the bounding box. Thus, the foveal region may track the user's gaze upward until the gaze moves outside of the bounding box. According to some embodiments, the foveal region may still attempt to track the gaze outside the bounding box by shifting to a location within the bounding box that is determined to be closer to the gaze than other locations. Of course, the geometries and shapes shown in fig. 3A-3F are intended to be exemplary and not limiting. For example, any number of other shapes or boundaries (including squares, trapezoids, diamonds, and other polygons) may be used to define the fovea and/or the mid-fovea in accordance with the methods and systems described herein.
In general, each of the embodiments shown in fig. 3A-3E may have a foveal region that is ' fixed ' relative to the display and/or view or that dynamically tracks the user's gaze when viewing the respective gaze point rendering view and/or display. For example, for some types of VR content, it may be that: it is expected that the HMD user will directly view most VR sessions. Thus, certain embodiments may render views and/or displays using a fixed point of regard relative to the display and/or view of the VR environment.
Fig. 3G illustrates a VR scene 300 generated using point-of-gaze rendering according to the methods and systems described herein. The point-of-regard rendering produces a foveal area 302 and a plurality of intermediate foveal areas 304-310. In fig. 3G, the number of intermediate foveates 304-310 is arbitrary, with the resolution of each intermediate foveated region gradually decreasing as the display of the intermediate foveated region becomes farther from the foveated region. For example, the central foveal area 306 may include anywhere between 1 to 100 additional central foveal areas.
FIG. 3H depicts various exemplary relationships between the resolution of the display region and the distance of the region from the foveal region or gaze point. For example,
Fig. 4 illustrates a display or screen defined by a gaze-point rendered
An expanded
According to some embodiments, even though the
Further depicted by fig. 4 is a
Fig. 5 shows a representative display 500 defined by a gaze point rendered view having a foveal area and a peripheral area. The foveal region is shown as an array comprising an array of representative pixels 502 that may correspond to the original pixels of the display 500. The peripheral region is shown to include, for example, a low resolution pixel array 504 that may be associated with 4 original pixels, respectively. According to the illustrated embodiment, each of the high-resolution pixels 506a, 506b, 506c, and 506d of frame N is extracted from corresponding pixel data 508a, 508b, 508c, and 508d stored in a time buffer 510 of frame N.
Certain aspects of the embodiments described herein allow "low resolution" pixels to be rendered at a higher resolution by extracting pixels associated with lower resolution pixels from low resolution pixel data stored in a temporal buffer. For example, low resolution pixels 512 are shown to include original pixels 514a, 514b, 514c, and 514d, each of which is extracted from low resolution pixel values stored for a different frame. In particular, the original pixel 514a is extracted from the pixel data 516a, which is the pixel value of the low-resolution pixel 512 at frame N, obtained by dithering and sampling at the upper left corner of the low-resolution pixel.
Original pixel 514b is shown as being extracted from pixel data 516b, which includes the same pixel value for low-resolution pixel 512, but from a previous frame (N-1) and a different dither position (e.g., upper right corner). In addition, original pixel 514c is extracted from pixel data 516c, which includes pixel values obtained from frame N-2 and from a different dither position (e.g., lower left corner). In addition, original pixel 514b is shown as being extracted from pixel data 516d, which includes the sampled pixel value for frame N-3 at the lower right dither position.
Thus, a higher resolution display output for the low resolution pixels 512 may be achieved by extracting the original pixels 514a through 514d for the previous frames that have been correspondingly dithered based on the pixel data 516a through 516d stored in the time buffer without having to increase the number of pixel values required to store each frame. For example, for low resolution pixel 512, only one color value is stored for each of frames N, N-1, N-2, and N-3. In contrast, 4 pixel values are stored for the group of high resolution pixels 506 a-506 d of frame N alone (there are still approximately 4 pixel values for the pixels of each of frames N-1, N-2, and N-3, and so on).
That is, the number of pixel values required per frame is 4 pixel values for the 4 groups of high resolution pixels 506a to 506 d. In contrast, the group of 4 pixels 514a to 514d associated with the low resolution pixel 512, although having "the same resolution" as the group of high resolution pixels 506a to 506d, only needs to store one pixel value or color per frame in the temporal buffer. Accordingly, embodiments described herein enable higher resolution pixels to be displayed and/or constructed in low resolution pixel regions without increasing the number of pixel values stored in the time buffer 510 per frame (e.g., without increasing memory usage). According to some embodiments, the process of extracting the original pixels 514 a-514 d from the time-defined low resolution pixel data stored in the time buffer may utilize temporal super sampling to sample different locations of the low resolution pixels.
Fig. 6 illustrates a conceptual scheme for creating a set of higher resolution pixels from
According to various embodiments, a sampled
According to some embodiments, the projection of the sampled pixel values from sampled
In one embodiment, temporal antialiasing reprojection may be used to perform the reprojection shown in FIG. 6. For example, in some embodiments, the following exemplary equation may be used:
world position (1) screen position (inverse projection of current view)
Previous screen position (previous view projection) world position (2)
According to some implementations, equations (1) through (3) above may be used to sample a previous frame stored in a time buffer. For example, equation (1) may map the current pixel back to world space. Equation (2) projects the position onto the previous frame using the camera (view projection matrix) of the previous frame, and equation (3) converts the previous screen position to uv coordinates that can be used to sample the previous frame in the time buffer. Thus, the associated graphics pipeline will know the location at which to sample the previous frame (e.g., frames N-1, N-2, N-3, etc.). For example, the dashed lines shown in fig. 6 may represent re-projections using derived uv coordinates for determining the location at which each previous frame is to be sampled. In particular, re-projection 620 may result in a change in sample position (e.g., 0.5 pixels in x) between
Fig. 7 illustrates a conceptual scheme for outputting a set of four high-
Also shown in fig. 7 is a sampled
Fig. 8 shows a conceptual scheme for rendering an object 802 using low-resolution pixels 804 over multiple frames 800. The object 802 is shown as being static over multiple frames 800. The low resolution pixel 804 is shown to be about 4 times the size of the high resolution pixel in fig. 8. For example, if each of the high-
During sampling 806, the low-resolution pixels 804 are shown sampled at the center of the low-resolution pixels 804 at each frame and result in the storage of 1 color value 808 per frame. The sampled color value 810 (e.g., black) of the entire low-resolution pixel 804 is shown as being derived from the sample 806 of the object 802 using the low-resolution pixel 804. The output color/rendered image 812 of the plurality of frames 800 is shown reflecting the sampled color values 810. For example, the output color/rendered image 812 is shown as including a black type of 'large pixel'. According to some embodiments, the output color/rendered image 812 may be mapped to a peripheral region of the HMD display associated with a lower resolution.
Fig. 9 shows a conceptual scheme of using
For example,
It should be noted that in accordance with the illustrated embodiment, the
Also shown in FIG. 9 are the sample colors and
According to the illustrated embodiment, the color output/rendered
A
According to some embodiments, the sample locations shown in the
It should be noted that the rendering using
Thus, embodiments described herein achieve the technical benefit of improving the functionality of VR systems by increasing resolution without increasing memory usage (e.g., the number of color values stored per pixel per frame). The increase in resolution also reduces aliasing associated with low resolution regions of the point of regard rendering system, which may improve the overall quality of experience for the viewer. For example, if the rendering process shown in fig. 9 is performed on the entire region composed of a plurality of low-
Fig. 10 illustrates a conceptual scheme of a rendering process using temporal
The resulting rendered
FIG. 11 illustrates a conceptual model for generating higher resolution pixels associated with a rendered
In accordance with the illustrated embodiment,
According to the illustrated embodiment, each of the rendered images 1116-1124 is shown to be the result of a
In some embodiments, the average of the
According to the illustrated embodiment,
Also shown in FIG. 11 are rendered
According to the illustrated embodiment, rendered
Fig. 12 illustrates a conceptual model for generating higher resolution pixels associated with a rendered
Further, the sampling positions are shown to be unevenly distributed in the 4 quadrants of the low resolution pixel 1204. For example, the top-right sub-pixel is shown as being sampled at 3 locations of a window of 8 frames, while the bottom-left sub-pixel is shown as being sampled at only 1 location. There are quantitative algorithms for sampling within the low resolution pixels 1204, some of which may minimize the occurrence of clustering or uneven distribution of sample locations. Thus, the sampling patterns and/or algorithms illustrated herein are illustrative and not restrictive, as there are any number of supersampling patterns that may be used in conjunction with the embodiments described herein without departing from the spirit or scope of the embodiments. Furthermore, although the temporal supersampling embodiment for constructing a high resolution rendered image is shown based on one current frame and 7 previous frames, there may be any number of frames from which a high resolution rendered image may be constructed using temporal supersampling and re-projection, in accordance with various embodiments. Further, although the rendered image is shown as being constructed from pixel values associated with the current frame, other embodiments may be such that the most recent rendered image does not have to be mapped from pixel values associated with the most recent frame.
Fig. 13A illustrates an implementation of reconstructing a set of 16
Fig. 13B illustrates an implementation in which a set of 16
It should be noted that where a pixel shader is mentioned, for some implementations it is also meant to refer to a compute shader. Further, although exemplary sampling patterns are shown, for example, in fig. 13A and 13B, it should be noted that any number of sampling patterns may be implemented with the embodiments described herein.
Fig. 14 illustrates the overall flow of a method that enables reconstruction of higher resolution pixels from a low resolution sample region using color values obtained by temporal supersampling over multiple previous frames. The embodied method includes an
The method shown in fig. 14 then proceeds to
Fig. 15 illustrates an additional embodiment of an HMD 1500 that may be used with the proposed method and/or system. HMD 1500 includes hardware such as gaze detector 1502, processor 1504, battery 1506, virtual reality generator 1508, buttons, sensors, switch 1510, sound positioning device 1512, display 1514, and memory 1516. HMD 1500 is also shown to include a location module 1528 that includes magnetometer 1518, accelerometer 1520, gyroscope 1522, GPS 1524, and compass 1526. Also included on HMD 1500 are speaker 1530, microphone 1532, LED 1534, object for visual recognition 1536, IR light 1538, front camera 1540, rear camera 1542, gaze tracking camera 1544, USB 1546, permanent storage 1548, vibro-tactile feedback device 1550, communications link 1552, Wi-Fi 1554, ultrasound communication device 1556, bluetooth 1558, and Photodiode (PSD) array 1560. In some embodiments, HMD 1500 may also include one or more CPUs 1562, one or more GPUs 1564, and a video memory 1566.
FIG. 16 is a diagram of a
The
According to some implementations, the GPU1612 is also shown implementing a
The GPU1612 is also shown executing a pixel shader 1618 (also referred to as a fragment shader) that is used to obtain color values for pixels to be displayed. According to some embodiments, temporal supersampling as described herein may be performed with the aid of a pixel shader, for example, by accessing frame buffer/temporal buffer 1622. Further, according to some embodiments,
Although the method operations are described in a particular order, it should be understood that other internal management operations may be performed between the operations, or the operations may be adjusted so that they occur at slightly different times, or may be distributed in a system that allows processing operations to occur at various intervals associated with processing.
One or more embodiments may also be fabricated as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data which can thereafter be read by a computer system. Examples of the computer readable medium include hard disk drives, Network Attached Storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-R, CD-RWs, magnetic tape, and other optical and non-optical data storage devices. The computer readable medium can include computer readable tangible media distributed over a network coupled computer system so that the computer readable code is stored and executed in a distributed fashion.
Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the embodiments are not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.