Apparatus and method for providing content
阅读说明:本技术 提供内容的设备和方法 (Apparatus and method for providing content ) 是由 耶夫亨尼·雅克辛 奥列克桑德尔·拉多姆斯基 奥列克桑德尔·舒尔 于 2018-07-25 设计创作,主要内容包括:提供了一种设备和方法。该方法包括:基于移动信息识别所述设备的视线方向。该方法还包括:识别在第一直线与第二直线之间形成的角度,该第一直线从显示器上的预定位置开始并且对应于所识别的视线方向,该第二直线从该预定位置开始并且对应于遵循从多个帧获得的移动轨迹的方向。该方法还包括:基于所识别的角度和相对于设备的视场而确定的阈值来确定是否开启用于提供与所识别的视线方向相对应的帧的调整模式。该方法额外包括在所述多个帧中识别第一帧和第二帧。该方法还包括显示该第一帧,然后显示该第二帧。(An apparatus and method are provided. The method comprises the following steps: identifying a gaze direction of the device based on the movement information. The method further comprises the following steps: an angle formed between a first straight line starting from a predetermined position on the display and corresponding to the identified line-of-sight direction and a second straight line starting from the predetermined position and corresponding to a direction following a movement trajectory obtained from a plurality of frames is identified. The method further comprises the following steps: determining whether to turn on an adjustment mode for providing a frame corresponding to the identified gaze direction based on the identified angle and a threshold determined relative to a field of view of the device. The method additionally includes identifying a first frame and a second frame in the plurality of frames. The method also includes displaying the first frame and then displaying the second frame.)
1. A method of a device, the method comprising:
identifying a gaze direction of the device based on movement information generated by measuring movement of the device;
identifying an angle formed between a first straight line starting from a predetermined position on a display included in the apparatus and corresponding to the identified line-of-sight direction and a second straight line starting from the predetermined position and corresponding to a direction following a movement trajectory obtained from a plurality of frames;
determining whether to turn on an adjustment mode for providing a frame corresponding to the identified gaze direction based on the identified angle and a threshold determined relative to a field of view of the device;
in the adjustment mode, identifying a first frame and a second frame of the plurality of frames based on the identified gaze direction; and
the first frame is displayed and then the second frame is displayed.
2. The method of claim 1, wherein determining whether to turn on the adjustment mode is based on a comparison between the identified angle and the threshold.
3. The method of claim 1, wherein the movement trajectories obtained from the plurality of frames are generated by:
extracting one or more keypoints from each of the plurality of frames displayed at different times;
generating movement information of one or more keypoints in each of the plurality of frames as a function of time based on the location value of each of the one or more keypoints;
determining a direction vector of each of the plurality of frames according to a time from a predetermined reference point using the movement information; and
connecting each of the determined direction vectors.
4. The method of claim 1, wherein the predetermined location on the display comprises a location corresponding to a center of a horizontal axis of the display, and the line of sight direction of the device corresponds to a direction perpendicular to a center of a screen of the display.
5. The method of claim 1, wherein determining the first frame comprises: determining a frame provided at a time when the adjustment mode is turned on as the first frame.
6. The method of claim 3, wherein determining the second frame comprises:
determining a first value indicating an inter-frame distance based on a second direction vector of an object included in each of the first frame and any third frame among the plurality of frames;
determining a second value using the identified angle and the movement distance of the device;
determining a number of frames to scroll based on the first value and the second value; and
determining the second frame by scrolling as many frames as the number of frames to be scrolled in the plurality of frames based on the first frame.
7. The method of claim 1, wherein the device comprises a Head Mounted Display (HMD) device.
8. The method of claim 1, wherein the plurality of frames comprises frames for providing a 360 degree image.
9. The method of claim 1, further comprising: inserting a reinforcement frame for screen switching between the first frame and the second frame.
10. The method of claim 1, further comprising: in response to determining not to turn on the adjustment mode, displaying the plurality of frames according to predetermined time information.
11. A device configured to perform one of the methods of claims 1-10.
Technical Field
The present disclosure relates to an apparatus and method for providing content.
Background
Virtual Reality (VR) is created by using computer-like artificial techniques, referring to the technology itself or a specific environment that resembles reality but is not real. The environment or situation created by a VR may stimulate the user's five senses and may give the user a realistic spatiotemporal experience. Further, the user can interact with objects implemented in the VR by performing controls or executing commands using the actual device, as well as be immersed completely in the VR. In the sense that a VR can interact with a user and create a user experience, a VR can be distinguished from a unilaterally implemented simulation.
The user may move while using the VR device, and the VR device may provide VR content that reflects the user's movement.
Disclosure of Invention
Technical problem
There is a need to provide VR content that reflects a change in viewing direction according to movement of a user viewing the VR content.
Solution to the problem
The present disclosure provides an apparatus and method for providing VR content reflecting a change in a viewing direction according to a user's movement when the user viewing the VR content freely travels or moves.
An apparatus according to an embodiment may include: a display; a gesture sensor configured to measure movement of the device and output movement information related to the measured movement; a memory; a controller configured to: identifying a gaze direction of the device based on the movement information; identifying an angle formed between a straight line starting from a predetermined position on the display and corresponding to the identified line-of-sight direction and a straight line starting from the predetermined position and corresponding to a direction following a movement trajectory obtained from a plurality of frames stored in the memory; determining to turn on an adjustment mode for providing a frame corresponding to the identified gaze direction based on the identified angle and a threshold determined relative to a field of view of the device; in the adjustment mode, identifying a first frame and a second frame of the plurality of frames based on the identified gaze direction; and control the display to display the first frame and then the second frame.
A method of an apparatus according to an embodiment, comprising: identifying a gaze direction of the device based on movement information generated by measuring movement of the device; identifying an angle formed between a straight line starting from a predetermined position on a display included in the apparatus and corresponding to the identified line-of-sight direction and a straight line starting from the predetermined position and corresponding to a direction following a movement trajectory obtained from a plurality of frames; determining whether to turn on an adjustment mode for providing a frame corresponding to the identified gaze direction based on the identified angle and a threshold determined relative to the device field of view; in the adjustment mode, identifying a first frame and a second frame of the plurality of frames based on the identified gaze direction; the first frame is displayed and then the second frame is displayed.
According to the present disclosure, a VR device may provide a frame to a user that reflects changes in a gaze direction of the VR device according to a user's motion or movement.
The user may move while using the VR device, and may find an object located on a moving trajectory within the virtual reality content range while moving the head left and right. Further, the user may observe the found object by moving his/her head in multiple directions (such as tilting his/her head or moving his/her head to the right or left). When the user moves his/her head in a plurality of directions, the VR content provides the user with an object image corresponding to the moved viewpoint so that the user can observe the object in a plurality of directions. The user may also observe a rear view of the object while moving. Accordingly, the VR device may provide more realistic content and may draw a new degree of attention that enables a user to control VR content, so that the user may subjectively observe an object as in the real world. The VR device provides VR content corresponding to the visual direction even if the visual direction of the VR device is beyond the user's perspective. Thus, the field of view (FOV) that the VR device can provide can be increased.
The VR device identifies movement of the user and provides frames corresponding to the identified movement. For example, when the user tilts his/her head backward, the VR device may provide a second frame to the user after the first frame, where the second frame has been provided at a second viewpoint that is a viewpoint before a time as long as a moving distance generated when the user tilts his/her head backward relative to the first frame provided to the user at the first viewpoint.
Further, when the user leans his/her head forward, the VR device may provide a third frame to the user after the first frame, wherein the third frame is a frame provided at a third viewpoint that is a viewpoint before a time as long as a moving distance generated when the user leans his/her head forward, relative to the first frame provided to the user at the first viewpoint. Thus, the user can see the past or future map of the object by tilting his/her head. Such functionality enables the VR device to provide VR content with increased degrees of freedom (DOF) to a user.
Before adopting the following inventive manner, it may be advantageous to set forth definitions of certain words and phrases used in this patent document: the terms "include" and "comprise," as well as derivatives thereof, mean inclusion without limitation; the term "or" is inclusive, meaning and/or; the phrases "associated with ….. and" associated therewith, "and derivatives thereof, may mean including, included within, interconnected with … …, containing, contained within … …, connected to or with … …, coupled to or with … …, communicable with … …, cooperative with … …, interleaved, juxtaposed, proximate to, bound to or with … …, having a characteristic of … …, and the like; the term "controller" refers to any device, system or part thereof that controls at least one operation, such a device may be implemented in hardware, firmware or software, or some combination of at least two of the same. It should be noted that the functionality associated with any particular controller may be centralized or distributed, whether locally or remotely.
Further, the various functions described below may be implemented or supported by one or more computer programs, each formed from computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof adapted for implementation in suitable computer readable program code. The phrase "computer readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer readable medium" includes any type of medium capable of being accessed by a computer, such as Read Only Memory (ROM), Random Access Memory (RAM), a hard disk drive, a Compact Disc (CD), a Digital Video Disc (DVD), or any other type of memory. A "non-transitory" computer-readable medium does not include a wired, wireless, optical, or other communication link that transmits transitory electrical or other signals. Non-transitory computer-readable media include media capable of permanently storing data as well as media capable of storing data and thereafter overwriting the data, such as rewritable optical disks or erasable memory devices.
Definitions for certain words and phrases are provided throughout this patent document, those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.
Drawings
For a more complete understanding of the present disclosure and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, wherein like reference numbers represent like parts:
fig. 1 shows a flow chart of an operational procedure of a VR device according to an embodiment of the present disclosure;
fig. 2A and 2B illustrate diagrams of virtual spherical space concepts and virtual reality content concepts provided by a VR device in accordance with embodiments of the disclosure;
fig. 3 illustrates a method of determining an angular deviation of a gaze direction of a VR device in accordance with an embodiment of the disclosure;
fig. 4 shows a flow chart for measuring an angular deviation of a gaze direction of a VR device, in accordance with an embodiment of the disclosure;
fig. 5 shows a diagram of a plurality of frames of a VR device according to an embodiment of the disclosure;
fig. 6 shows a diagram of a movement trajectory of a frame of a VR device according to an embodiment of the disclosure;
fig. 7A to 7C illustrate a relationship between an angular deviation of a line-of-sight direction of a VR device and a field of view of the VR device according to an embodiment of the present disclosure;
fig. 8 illustrates a diagram of an appearance of an object shown to a user based on a gaze direction of a VR device, in accordance with an embodiment of the present disclosure;
fig. 9 shows a diagram explaining a method of determining a number of frames to scroll in a VR device, according to an embodiment of the present disclosure;
fig. 10 shows a diagram explaining the concept of switching to a past frame or a future frame in a VR device according to an embodiment of the present disclosure;
fig. 11 shows a diagram of a method for inserting a reinforcement frame between a selected plurality of frames in a VR device, according to an embodiment of the disclosure;
fig. 12 shows a graph of a relationship between buffer size and camera relative speed in a VR device according to an embodiment of the disclosure;
13A and 13B illustrate a method for measuring a movement angle of an object between arbitrary frames in a VR device in accordance with an embodiment of the present disclosure;
fig. 14 shows a diagram explaining a method of determining a number of frames to scroll in a VR device, according to an embodiment of the present disclosure;
fig. 15 shows a diagram providing an example of a configuration of a VR device according to an embodiment of the present disclosure; and
fig. 16 shows a diagram providing an example of a configuration of a base station according to an embodiment of the present disclosure.
Detailed Description
Fig. 1 through 16, discussed below, and the various embodiments used to describe the principles of the present disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably arranged system or device.
Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the following description of the present disclosure, a detailed description of known configurations or functions incorporated herein will be omitted when it is determined that the detailed description may make the subject matter of the present disclosure unclear. Terms described below are defined in consideration of functions in the embodiments, and meanings of the terms may be changed according to intentions, conventions, and the like of a user or an operator. Therefore, the definition of the terms should be determined based on the contents of the entire specification.
Although the terms first, second, etc. may be used to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Therefore, within the technical scope of the present disclosure, the first element to be mentioned hereinafter may be the second element.
First, referring to fig. 1, an operation process of a Virtual Reality (VR) device will be described according to an embodiment.
Fig. 1 shows a flow diagram of an operational process of a VR device according to an embodiment of the disclosure.
A VR device according to an embodiment may be an electronic device such as a terminal, a smartphone, a vehicle, glasses, goggles, or a Head Mounted Display (HMD) worn on a user's head. In the following, it is assumed that the VR device is an HMD.
The VR device includes a gesture sensor. The VR device may detect a change in a posture (position) of the VR device by using a posture sensor. At
Movement information of the VR device provided to the VR device by the gesture sensor may be represented as a deviation. The deviation refers to a degree of movement of the VR device expressed based on the reference position. In particular, the deviation from the rotational movement may be measured as an angular deviation indicating the degree of rotation about the reference axis as an angle. The angular deviation may include an angle of vertical movement or an angle of horizontal movement relative to the reference axis. When the direction of gravity is the reference axis and the reference axis corresponds to the z-axis, the vertical movement relative to the reference axis may be expressed as a pitch value rotating about the x-axis and/or a roll value rotating about the y-axis. Further, the horizontal movement relative to the reference axis may be expressed as a yaw value rotating about the z-axis.
The posture sensor may include one or more of an acceleration sensor (accelerometer) that measures acceleration, a gyroscope that measures angular velocity, and a magnetometer that is a geomagnetic sensor. As an example of the movement information measurement, a roll value or a pitch value may be measured via an accelerometer or a gyroscope, and a yaw value may be measured via a gyroscope or a magnetometer. However, this is an example of a method of obtaining movement information of the VR device, and the type of the posture sensor or the measurement method is not limited thereto as long as a method can measure the movement of the VR.
The VR device may render multiple frames. A frame may comprise at least one 360 degree (°) image. One frame may include at least one piece of photographing time information corresponding to one 360-degree image. The photographing time information may include time stamp information. Further, one frame may further include information on auditory, olfactory, or tactile sensation that changes according to a position or time, and a visual image that changes according to a position or time.
The 360 degree image may be an image of a 360 degree scene surrounding a camera used to capture the image. Alternatively, the 360 degree image may be an image obtained by merging images captured in a plurality of directions such that a scene around a camera used to capture the image is oriented similar to the 360 degree scene. In addition, the 360 degree image may be an image actually captured by the camera, or may be a computer graphic image formed similarly to the image actually captured by the camera. The camera for capturing the 360 degree image may or may not be included in the VR device. That is, the VR device does not necessarily include a camera for capturing 360 degree images.
Fig. 2A and 2B illustrate diagrams of virtual spherical space concepts and VR content concepts provided by a VR device according to embodiments of the disclosure.
The 360 degree image included in the frame may be an image of a scene photographed at 360 degrees within a radius from the camera where the camera can capture the image. That is, the 360 degree image may be an image of the interior surface of the virtual spherical space. Fig. 2A is a diagram illustrating a concept of a virtual spherical space provided by a VR device according to an embodiment. Referring to fig. 2A, when the
A 360 degree image may be captured as the camera moves. Alternatively, the 360 degree image may be a computer graphics image formed similarly to an image captured while the camera is moving. When a trajectory generated along the movement of the camera during image capture is referred to as a camera movement trajectory, the camera may capture images of a scene within the view angle of the camera at arbitrary time intervals while moving along the camera movement trajectory. Each scene taken at arbitrary time intervals may be used to display one 360 degree image.
Fig. 2B is a diagram illustrating a VR content concept provided by a VR device according to an embodiment. Referring to fig. 2A and 2B, a plurality of 360-degree images, or a plurality of virtual spherical spaces sp, sp1, sp2,. and spN (in which a plurality of 360-degree images are provided to the user 1), captured according to an arbitrary time interval and included in the
The plurality of 360 degree images may include images captured during travel, sports, driving, and engaged in extreme sports, among others. The plurality of 360-degree images may include images of various landscapes, objects, creatures, places, buildings, etc., to be provided to the user, and may include images of objects photographed in various directions. For example, the plurality of 360 degree images may include images of the object taken in multiple directions at arbitrary time intervals while circling around the object.
The photographing time information may include information on a time at which each of the plurality of 360-degree images is captured. The photographing time information may include information on a specific time at which each of the plurality of 360-degree images is photographed, or may include information on a front-rear relationship of the plurality of 360-degree images. Accordingly, when a plurality of 360-degree images are reproduced in an order corresponding to the photographing time information for each 360-degree image, the user can view a scene, which is photographed while the camera is moving, based on the camera movement trajectory.
The VR device may provide VR content to the user. VR content may include a plurality of frames. The VR content may be a plurality of frames reproduced in time series, or may be a plurality of frames selectively arranged and reproduced. Since the user views images that have been taken while viewing VR content, the user may experience a particular environment or situation that is not real. Depending on how similar the VR content is to the real world, whether the VR content is able to provide an immersive and realistic experience for the user may vary. Hereinafter, a process of selecting a frame to be reproduced by the
Referring back to fig. 1, in operation 200, the VR device determines an angular deviation β of the gaze direction based on movement information measured by the posture sensor. This will be described with reference to fig. 3.
Fig. 3 illustrates a method for determining an angular deviation of a gaze direction of a VR device, in accordance with an embodiment of the disclosure.
Referring to fig. 3, the angular deviation β of the line-of-sight direction represents the difference between the line-of-sight direction dv of the
The line-of-sight direction dv of the
The frame movement trajectory dm may refer to a camera movement trajectory along which a plurality of 360-degree images included in a plurality of frames are captured, or may refer to a movement trajectory estimated from a plurality of frames. That is, when reproducing a plurality of 360-degree images included in the
In the embodiment, the definition of the line-of-sight direction dv and the definition of the frame movement trajectory dm of the
Hereinafter, a process of determining the frame movement trajectory dm by the VR device to determine the angular deviation β of the line of sight direction will be described in detail.
Fig. 4 shows a flow chart for measuring an angular deviation of a gaze direction of a VR device, according to an embodiment of the disclosure.
Fig. 5 illustrates a diagram of multiple frames of a VR device according to an embodiment of the disclosure.
Referring to fig. 4 and 5, in
The key features may represent features (g, h, i, j, g ', h', i ', j', g ", h", i ", or j") of each of a plurality of 360 degree images (360
Methods of converting image data to coordinates that do not change according to scale may be used to extract key features. First, the scale space of the 360 degree image, expressed as a function of L (x, y, σ), can be computed. Here, σ may represent a scale parameter. L (x, y, σ) can be calculated by a convolution operation (mathematical formula 1) of the 360-degree image I (x, y) and the variable-scale gaussian G (x, y, σ).
[ mathematical formula 1]
L(x,y,σ)=G(x,y,σ)*I(x,y)
Here, the variable-ratio gaussian G (x, y, σ) may satisfy
[ mathematical formula 2]
A gaussian difference function (equation 3) can be calculated based on the difference between two adjacent scale spaces, where the proportional variables are σ, k σ, respectively.
[ mathematical formula 3]
In addition, the difference of gaussians function can provide a gaussian proportional normalized laplacian of gaussians operator
An approximation of (d). According to the experimental results, can useTo obtain the most stable key features of the image relative to other functions.D and
the relationship between the heat diffusion coefficient and the thermal diffusion coefficient can be obtained by the following equations 4 to 7, including a thermal diffusion equation (equation 4).[ mathematical formula 4]
[ math figure 5]
[ mathematical formula 6]
[ math figure 7]
D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y)
In order to determine local maxima and local minima in an image using a gaussian difference function (hereinafter referred to as a gaussian difference function image), arbitrarily selected sample points in the gaussian difference function image and their surrounding regions may be compared. Specifically, the sample point can be compared with eight pixels adjacent to the sample point in a portion of three rows and three columns around the sample point. Further, in each of the adjacent two gaussian difference function images, nine pixels in the portion of three rows and three columns can be compared with the sample points. That is, a sample point may be compared to adjacent pixels in a gaussian difference function image, and the sample point may be compared to a corresponding pixel in the adjacent gaussian difference function image. In the above case, the sampling point may be compared with 26 pixels. A sample point may be selected only if it is greater than all 26 pixels or less than all 26 pixels. The selected sample points may be candidates for keypoints. The keypoints may indicate the location of key features on the image.
In order to remove values that are not suitable for indicating the characteristics of an image due to values corresponding to too low a contrast or indicating edges, and to select a keypoint from the keypoint candidates that can reliably represent the characteristics of the image, the taylor series of the gaussian difference function D (x, y, σ) may be extended to the second term (mathematical expression 8) for the scale space.
[ mathematical formula 8]
Here, x ═ (x, y, σ)T.
The local threshold (local maximum or local minimum) may be obtained based on determining that the derivative of equation 8 with respect to x is 0
[ mathematical formula 9]
When the equation 9 is substituted into the equation 8, it can be calculated
The function value (equation 10).[ mathematical formula 10]
The value of
After 16 rows and 16 columns of pixels are selected around the selected keypoint, the image inside it can be gaussian blurred, and then the gradient magnitude m (x, y) and direction 8(x, y) of each point can be determined (equations 11 and 12). Here, L may be a data value of a gaussian blur image, and a gradient magnitude (equation 11) and a direction (equation 12) may be determined using a difference between L (x, y).
[ mathematical formula 11]
[ mathematical formula 12]
θ(x,y)=tan-1((L(x,y+1)-L(x,y-1))/(L(x+1,y)-L(x-1,y)))
A histogram may be used to represent the gradient magnitude of 16 rows and 16 columns of pixels. The highest point of the histogram may be the direction of the keypoint corresponding to the highest point of the histogram. In addition, for example, a point whose height is about 80% or more of the height of the highest point of the histogram may also be the direction of the keypoint. Thus, in the case of a keypoint having a plurality of peaks of similar size, a plurality of directions can be determined for one keypoint.
Based on the values obtained by determining the gradient magnitude and direction of the keypoint and its surrounding parts, keypoint descriptors indicating features of the shape around the keypoint can be determined. The 16 rows and 16 columns of pixels, which determine the magnitude and direction of the gradient at each point, may be multiplied by a gaussian weighted window function. Then, for example, 16 rows and 16 columns of pixels may be represented by 4 rows and 4 columns of keypoint descriptors. The components of the individual pixels of 16 rows and 16 columns can be summarized into keypoint descriptors of 4 rows and 4 columns. Each block of 4 rows and 4 columns of keypoint descriptors may include eight arrows indicating eight directions representing 360 degrees. Each arrow may be the sum of gradients around the respective direction. Thus, the feature vector of the keypoint descriptor may comprise 128 elements, i.e. eight arrows for each of the 16 (4X4) blocks. The determined plurality of keypoint descriptors of the image may be included in a key feature of the image that does not change with the rotational transformation of the image.
The process of extracting key features of an image has been described as an example, but the process of extracting key features is not limited thereto. Key features may be extracted using the Speeded Up Robust Features (SURF) algorithm or the oriented FAST and rotated brief (orb) algorithm.
In
[ mathematical formula 13]
In the transformation matrix M12In (equation 13), r may represent a rotation value and t may represent a translation value. q may represent a quaternion. However, in one embodiment, it may be assumed that a quaternion is not required and q is set to [0, 0, 0, 1]. In particular, the matrix R for rotation values (equation 14) may be a matrix of rotation vectors θ (radians) in the direction of the right-hand rule with respect to the z-axis (equation 15).
[ mathematical formula 14]
[ mathematical formula 15]
Alternatively, the matrix R for the rotation value may be a matrix (equation 16) of a direction rotation vector θ (radian) in right-hand rule with respect to the x-axis, may be a matrix (equation 17) of a direction rotation vector θ (radian) in right-hand rule with respect to the y-axis, and may be a matrix (equation 18) for scale transformation.
[ mathematical formula 16]
[ mathematical formula 17]
[ mathematical formula 18]
Further, the matrix R for the rotation values may be the matrix AXY(equation 19), the matrix AXYVector rotation in right hand direction with respect to x-axis
(radians) and rotates the vector θ (radians) relative to the y-axis. The matrix R for the rotation values may be the matrix AXZ(equation 20), the matrix AXZVector rotation in the direction of the right hand rule with respect to the z-axis(radians) and rotates the vector ψ (radians) with respect to the z-axis. The matrix R for the rotation values may be the matrix AYZ(equation 21), the matrix AYZVector θ (radians) is rotated in the direction of the right hand rule relative to the y-axis and vector ψ (radians) is rotated relative to the z-axis. The matrix R for the rotation values may be a matrix A (equation 22) that rotates the vector in the direction of the right-hand rule with respect to the x-axis(radians), the vector θ (radians) is rotated in a right-hand regular direction relative to the y-axis and the vector ψ (radians) is rotated relative to the z-axis.[ math figure 19]
[ mathematical formula 20]
[ mathematical formula 21]
[ mathematical formula 22]
Hereinafter, determining the transformation matrix M shown in mathematical formula 13 will be described12The method of (1).
First, a method for determining a two-dimensional transformation matrix of M12 will be described.
In a pair of key features that match each other, when a keypoint of a key feature of a first frame is represented by (u, v) and a keypoint of a key feature of a second frame is represented by (x, y), a two-dimensional transformation matrix that is rotated by (x, y) θ (radian) in a counterclockwise direction with respect to an origin (0, 0) may be represented as mathematical expression 23. Here, txAnd tyMay correspond to a value associated with a translation of (x, y).
[ mathematical formula 23]
Transformation refers to rotating an initial position using a rotation matrix and using a translation vectorThe initial position is translated. When the key point of the key feature of the first frame is F1The key point of the key feature of the second frame is F2When the rotation matrix is R, the translation vector is T, and the transformation matrix including the rotation matrix and the translation vector is M, the following mathematical formula 24 is satisfied.
[ mathematical formula 24]
F1=RF2+T=MA
By assigning 1 to the diagonal and 0 to all other positions, the matrix can be expanded to the required size. That is, the following equation 25 is satisfied.
[ mathematical formula 25]
Determining a two-dimensional transformation matrix M12The method of (3) will be described below.
The transformation matrix including only the two-dimensional rotation component can be expressed as the following mathematical expression 26.
[ mathematical formula 26]
The transformation matrix containing only two-dimensional translation components can be expressed as the following mathematical formula 27.
[ mathematical formula 27]
When the matrices shown in math figures 26 and 27 are multiplied, a two-dimensional transformation matrix including both rotation components and translation components as shown in math figure 28 can be determined.
[ mathematical formula 28]
The result shown in equation 29 can be obtained using equation 23 and equation 24.
[ mathematical formula 29]
Equation 29 may be represented as equation 30.
[ mathematical formula 30]
In equation 30, a two-dimensional transformation matrix can be represented similarly to equation 31 below.
[ mathematical formula 31]
Next, a description will be given of a method for determining the three-dimensional transformation matrix M12The method of (1).
The transformation matrix containing only the three-dimensional rotation component can be expressed as the following mathematical expression 32.
[ mathematical formula 32]
The transformation matrix including only the three-dimensional translation component may be represented as the following mathematical formula 33.
[ mathematical formula 33]
When the matrices shown in equations 32 and 33 are multiplied, a three-dimensional transformation matrix including both the rotation component and the translation component shown in equation 34 can be determined.
[ mathematical formula 34]
The result shown in equation 35 can be obtained using equation 23 and equation 24.
[ math figure 35]
Here, (u, v, w) is the keypoint of the key feature of the first frame, (y, y, z) is the keypoint of the key feature of the second frame, tx、tyAnd tzIs a value related to the translation of (y, y, z).
Equation 35 can be expressed as equation 36 below.
[ mathematical formula 36]
In equation 36, a three-dimensional transformation matrix M12May be equation 37.
[ mathematical formula 37]
In the step of converting the key points (u, v) of the key features of the first frame and the key points (x, y) of the key features of the second frame into three-dimensional coordinates (u, v, w) and (y, y, z) by using an algorithm, a depth map may be used.
Although the determination of the transformation matrix M by using key features has been described above by various examples12(equation 13), but the method may not be limited thereto as long as the method can express the transformation between frames.
The VR device may be based on the determined transformation matrix M12(equation 13) to determine a direction vector of each of the plurality of frames, and the frame movement trajectory dm may be determined by connecting the plurality of direction vectors.
When the direction vector of the initial direction of the first frame is a first direction vector V1 and the direction vector of the second frame is a second direction vector V2, the VR deviceCan be implemented by using a transformation matrix M12(equation 13) determines a second direction vector V2 (equation 39). The first direction vector V1 may include V as an x component1xV as the y component1yAnd V as the z component1z. The first direction vector (V1) may be the start point of the frame movement trajectory dm and may be an arbitrary reference point. The first direction vector V1 may be set to V1x=0、V1y0 and V1z0, and may be set to any point. By combining the first direction vector V1 with the transformation matrix M12The multiplication determines the second direction vector V2 (math 39).
[ mathematical formula 38]
[ math figure 39]
Fig. 6 illustrates a diagram of a movement trajectory of a frame of a VR device according to an embodiment of the disclosure.
Referring to fig. 6, when the first direction vector V1 and the second direction vector V2 are determined, a frame movement trajectory dm may be determined. The frame movement trajectory dm may be determined by connecting the end point of the first direction vector V1 and the start point of the second direction vector V2. The frame movement trajectory dm may be determined by sequentially connecting direction vectors of respective frames of a plurality of frames including the first frame and the second frame.
Referring back to fig. 3 and 4, in
Referring back to fig. 1 and 7A-7C, at operation 300 the VR device compares the previously determined angular deviation of the gaze direction, β, to a half value (i.e., α/2) of the field of view (FOV) α of the VR device.
Fig. 7A to 7C illustrate a relationship between an angular deviation of a line-of-sight direction of a VR device and a field of view of the VR device according to an embodiment of the present disclosure.
The field of view α of the
Referring to fig. 7A, when the
An example of fig. 7C will be described with reference to fig. 8.
Fig. 8 illustrates a diagram of an appearance of an object shown to a user based on a gaze direction of a VR device, according to an embodiment of the present disclosure.
Referring to fig. 8, the
Referring back to fig. 1, 7A to 7C, and 8, in operation 400, the adjustment mode of the visual object representation may start from a time when the head of the
Hereinafter, a method for selecting a frame reflecting the gaze direction of the VR device will be described in detail.
When the head movement value of the
if(β>=a/2)&HL!=0than Start_Frame_detected();
Referring back to fig. 1, 7A through 7C, when the
When the first selected frame is determined, the
Fig. 9 shows a diagram for explaining a method of determining the number of frames to scroll in a VR device according to an embodiment of the present disclosure.
Referring to fig. 9, when two random frames are represented such as Va ═ V (V), respectivelyax,Vay,Vaz) And Vb ═ Vbx,Vby,Vbz) Can calculate the distance V between two random frames using equation 40L. Distance V between two random framesLCan be estimated as the actual distance of movement of the camera that has captured the image for two random frames. Here, Va may be the first selection frame.
[ mathematical formula 40]
Further, when the user is at H1 (H) based on the position of the user's head in the two random frames, respectively1x,H1y,H1z) And H2 (H)2x,H2y,H2z) Then, the moving distance H of the head of the user in two random frames can be calculated using equation 41L. Here, the moving distance H of the head of the userLMay be the same distance as the VR device is moved.
[ math figure 41]
The moving distance H of the head of the user between two frames can be obtainedLProjected at a distance VLDistance H' aboveL(equation 42). As shown in math figure 42, the moving distance H of the user's head can be usedLDetermining the distance H' from the angular deviation omega of the viewing directionL. And can be used forThe projection distance H' between two framesLAnd a distance VLThe ratio applies an integer function to determine the frame count C value (equation 43). Here, the frame count C value may indicate a number of frames to scroll through the selected first selected frame to determine a second selected frame. Having determined the value of frame count C, a second selected frame may be determined by scrolling through as many frames as the number of frame counts C in the first selected frame.
[ mathematical formula 42]
H`L=HL·cos(ω)
[ math figure 43]
Although the method for determining the second selection frame has been described by way of example, the method for determining the second selection frame is not limited thereto. For example, the calculation may be performed in two dimensions, and when two random frames are respectively expressed as Va ═ V (V), for exampleax,Vay) And Vb ═ Vbx,Vby) When the direction vector is zero, the distance V between two random framesLMay be calculated using equation 44. Here, Va may be the first selection frame.
[ math figure 44]
Further, when the user is respectively H1 (H) based on his/her head position in the two random frames1x,H1y) And H2 (H)2x,H2y) In two random frames, the moving distance H of the user's headLCan be calculated using equation 45.
[ mathematical formula 45]
Further, as described above, the moving distance H of the head of the user between two frames can be obtainedLProjected at a distanceVLDistance H' aboveL(math 42) and may be calculated by comparing a projected distance H 'between two frames'LAnd a distance VLThe ratio applies an integer function to determine the frame count C value (equation 43). When the frame count C value is determined, the second selected frame may be determined by scrolling as many frames as the number of frame counts C in the first selected frame.
Referring back to fig. 1, in operation 600, when a second selection frame reflecting a line-of-sight direction of the device is determined, the VR device switches the first selection frame that has been provided to the user to the second selection frame. That is, the VR device may provide the user with a frame reflecting a change in the gaze direction of the VR device according to the movement of the user's head. Taking fig. 8 as an example, when the movement of the
The user may travel while using the VR device and may find an object located on a movement trajectory within the VR content while moving his/her head left and right. Further, the user may observe the found object by moving his/her head in multiple directions (such as tilting his/her head or moving his/her head to the right or left). When the user moves his/her head in a plurality of directions, the VR content provides the user with an object image corresponding to the moved viewpoint, so that the user can observe the object in a plurality of directions. The user may also observe the rear view of the object while moving. Accordingly, the VR device may provide more realistic content and may draw a new degree of attention that enables a user to control VR content, so that the user may subjectively observe an object as in the real world. Even if the line of sight direction of the VR device is not outside the user's perspective, the VR device provides VR content corresponding to the line of sight direction, so that the FOV of the field of view provided by the VR device can be increased.
Fig. 10 shows a diagram explaining a concept of switching to a past frame or a future frame in a VR device according to an embodiment of the present disclosure. Fig. 10 shows an example of representing a frame as a virtual spherical space sp that can be provided to the
The VR device recognizes the movement of the user's head and provides a corresponding frame. Thus, as shown in fig. 10, when the
Further, as shown in fig. 10, when the
Fig. 11 shows a diagram of a method for inserting a reinforcement frame between selected frames in a VR device, according to an embodiment of the disclosure.
The VR device may determine a first select frame cf1 and a second select frame cf2, and may then insert a first boost frame sf1 between the first select frame cf1 and the second
Fig. 12 shows a graph showing a relationship between a buffer size and a camera relative speed in a VR device according to an embodiment of the present disclosure.
The VR device having the determined plurality of selected frames may determine a video buffer size to perform online streaming. Caching may be performed on the VR device (i.e., in the client device) for calculations in real-time. As shown in fig. 12, the video buffer size of multiple 360 degree images of the camera capture VR device may be adjusted according to the relative camera speed for objects being photographed by the camera. Here, the relative camera speed may refer to a speed at which an object moves in a plurality of frames. Referring to the example of fig. 12, the video buffer size may be 100Kb when the relative camera speed is 10Km/h, and 1Mb when the relative camera speed is 1 Km/h. That is, when the camera stops or moves slowly, the video buffer size may be determined to be large, and when the camera moves quickly, the video buffer size may be determined to be small. The relative camera speed and video buffer size may be inversely related. For example, when capturing video using a camera in a running vehicle, the relative speed between the camera and the object may be 20km/h with a video buffer size of 50 kb. Further, for example, when capturing video using a stop motion camera, the relative speed between the camera and the object may be 2km/h with a video buffer size of 800 kb.
Fig. 13A and 13B illustrate a method of measuring a movement angle of an object between any frames in a VR device according to an embodiment of the present disclosure.
Fig. 14 shows a diagram explaining a method of determining the number of frames to scroll in a VR device according to an embodiment of the present disclosure.
Referring to fig. 13A and 13B, the
[ mathematical formula 46]
Δγ=γ2-γ1
Referring to fig. 14, the positions of
In addition, depth maps for two random frames may be determined. The depth map may be determined using one or more of a stereo reconstruction method and a simultaneous localization and mapping (SLAM) method. However, the method for obtaining the depth map is not limited thereto. The distance M from the
The moving distance H of the head of the
[ math figure 47]
[ math figure 48]
When determining the second selection frame reflecting the movement of the user's head, the VR device may switch the first selection frame that has been provided to the user to the second selection frame.
Fig. 15 shows a diagram providing an example of a configuration of a
Fig. 16 shows a diagram providing an example of a configuration of a base station according to an embodiment of the present disclosure.
The
It should be noted that the method illustrations, system configurations, device configurations, etc. shown in fig. 1-16 are not intended to limit the scope of the present disclosure. That is, it should not be construed that all configurations or operations described in fig. 1 to 16 should be understood as essential elements of the embodiments, and the embodiments including only some elements may be implemented within a range not departing from the gist of the present disclosure.
This may be achieved by providing a storage device storing the corresponding program code to any unit of a base station or VR device of the communication system. That is, the base station or the controller of the VR device may perform the above-described operations by reading and executing program codes stored in the storage device in the form of a processor or a Central Processing Unit (CPU).
The various element units, modules, etc. of a base station or VR device described in this specification can operate using hardware circuitry (e.g., complementary metal oxide semiconductor-based logic circuitry), firmware, software, and/or a combination of hardware and firmware and/or software embedded in a machine-readable medium. By way of example, various electrical configurations and methods may be performed using circuits such as transistors, logic gates, and application specific integrated circuits.
While the disclosure has been shown and described with reference to certain embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the disclosure. Accordingly, the scope of the present disclosure should not be limited to the above-described embodiments, but should be defined by the appended claims and equivalents thereof.
- 上一篇:一种医用注射器针头装配设备
- 下一篇:用于视频编码的设备和方法