Method and arrangement for calibrating a head mounted display
阅读说明：本技术 用于校准头戴式显示器的方法和布置 (Method and arrangement for calibrating a head mounted display ) 是由 米卡埃尔·罗泽尔 于 2019-05-28 设计创作，主要内容包括：本公开涉及用于校准头戴式显示器(HMD)的相机的方法。所述方法包括在HMD的透镜的前面设置校准目标。校准目标和透镜中的每一个基本上在对应二维平面中延伸。所述方法还包括确定校准目标的横向位置。横向位置与校准目标在二维平面中的位置相关。所述方法还包括确定透镜的横向位置。横向位置与透镜在二维平面中的位置相关。所述方法还包括基于所确定的校准目标的横向位置并且基于所确定的透镜的横向位置而确定校准目标未对准。所述方法还包括执行HMD的硬件校准。硬件校准针对校准目标未对准而被调整。本公开还涉及一种布置和一种计算机程序产品。(the present disclosure relates to a method for calibrating a camera of a Head Mounted Display (HMD). The method includes placing a calibration target in front of a lens of the HMD. Each of the calibration target and the lens extends substantially in a corresponding two-dimensional plane. The method also includes determining a lateral position of the calibration target. The lateral position is related to the position of the calibration target in the two-dimensional plane. The method also includes determining a lateral position of the lens. The lateral position is related to the position of the lens in the two-dimensional plane. The method also includes determining a calibration target misalignment based on the determined lateral position of the calibration target and based on the determined lateral position of the lens. The method also includes performing a hardware calibration of the HMD. The hardware calibration is adjusted for calibration target misalignment. The disclosure also relates to an arrangement and a computer program product.)
1. A method for calibrating a camera (240) of a Head Mounted Display (HMD) (201), the method comprising the steps of:
-providing a calibration target (210) in front of a lens (230) of the Head Mounted Display (HMD) (201), wherein each of the calibration target (210) and the lens (230) each extends substantially in a corresponding two-dimensional plane;
-determining a lateral position of the calibration target (210), wherein the lateral position of the calibration target (210) is related to the position of the calibration target (210) in the two-dimensional plane;
-determining a lateral position of the lens (230), wherein the lateral position of the lens (230) is related to the position of the lens (230) in the two-dimensional plane;
-determining a calibration target misalignment (280) based on the determined lateral position of the calibration target (210) and based on the determined lateral position of the lens (230); and
-performing a hardware calibration of the camera (240) of the Head Mounted Display (HMD) (201), wherein the hardware calibration is adjusted for the calibration target misalignment (280).
2. The method of claim 1, wherein the step of determining the lateral position of the calibration target (210) comprises: determining a lateral position of a center point of the calibration target (210).
3. The method of any of claims 1-2, wherein the step of determining the lateral position of the calibration target (210) comprises: illuminating the calibration target (210) with first light from a first light source (260) from a side of the calibration target (210) opposite a side of the lens (230) such that the first light from the first light source (260) is transmitted through the calibration target (210) to the lens (230).
4. The method of claim 3, wherein the step of determining the lateral position of the calibration target (210) further comprises: detecting the first light with the camera (240) arranged on a side of the lens (230) opposite the calibration target (210).
5. The method according to any of claims 1 to 4, wherein the step of determining the lateral position of the lens (230) comprises: the lateral position of the lens center (235) is determined.
6. the method according to any of claims 1 to 5, wherein the step of determining the lateral position of the lens (230) comprises: determining the lateral position of the lens (230) at a number of calibration distances (290) between the lens (230) and the calibration target (210), wherein the calibration distance (290) is a distance between the lens (230) and the calibration target (210) perpendicular to a lateral extension of the lens (230) and a lateral extension of the calibration target (210), and wherein the lateral position of the lens (230) is determined based on the determined lateral position of the lens (230) at a number of calibration distances (290) between the lens (230) and the calibration target (210).
7. The method according to any of claims 1 to 6, wherein the step of determining the lateral position of the lens (230) comprises: illuminating the calibration target (210) with a second light from a second light source (270), wherein the second light source (270) is arranged between the lens (230) and the calibration target (210).
8. The method of claim 7, wherein the step of determining the lateral position of the lens (230) further comprises: detecting a reflection of the second light with the camera (240) arranged on a side of the lens (230) opposite the calibration target (210).
9. The method of claim 8, wherein the step of determining the lateral position of the lens (230) further comprises: a color block (297) in the reflection of the second light is determined.
10. the method of claim 9, wherein the step of determining the lateral position of the lens (230) further comprises: determining a lens center (235) based on intersections between the determined longitudinally extending lines of color patches (297).
11. The method according to any of claims 1 to 10, wherein the step of setting a calibration target (210) in front of a lens (230) of the Head Mounted Display (HMD) (201) comprises: mounting the calibration target (210) in a calibration mount (220) and fixing the calibration mount (220) relative to the Head Mounted Display (HMD) (201).
12. An arrangement (200) for calibrating a camera (240) of a Head Mounted Display (HMD) (201), the arrangement (200) comprising:
-the Head Mounted Display (HMD) (201), the Head Mounted Display (HMD) (201) comprising the camera (240) and comprising at least one lens (230), wherein the lens (230) extends substantially in a two-dimensional plane;
-a calibration target (210), the calibration target (210) being arranged in front of the lens (230), wherein the calibration target (210) extends substantially in a two-dimensional plane;
-a calibration processor (250), the calibration processor (250) being arranged to
i. Determining a lateral position of the calibration target (210), wherein the lateral position of the calibration target (210) is related to a position of the calibration target (210) in the two-dimensional plane;
Determining a lateral position of the lens (230), wherein the lateral position of the lens (230) is related to the position of the lens (230) in the two-dimensional plane;
determining a calibration target misalignment (280) based on the determined lateral position of the calibration target (210) and based on the determined lateral position of the lens (230); and
Performing a hardware calibration of the camera (240) of the Head Mounted Display (HMD) (201), wherein the hardware calibration is adjusted for the calibration target misalignment (280).
13. The arrangement (200) according to claim 12, the arrangement (200) further comprising a calibration mount (220), wherein the calibration mount (220) comprises the calibration target (210), and wherein the calibration mount (220) is fixed relative to the Head Mounted Display (HMD) (201).
14. The arrangement (200) according to any one of claims 12-13, wherein the Head Mounted Display (HMD) (201) is a Virtual Reality (VR) Head Mounted Display (HMD) (100).
15. The arrangement (200) according to any of claims 12 to 14, the arrangement (200) further comprising a first light source (260), the first light source (260) being arranged to illuminate the calibration target (210) with first light from a side of the calibration target (210) opposite to a side of the lens (230) such that the first light from the first light source (260) is transmitted through the calibration target (210) to the lens (230).
16. The arrangement (200) according to any one of claims 12 to 15, the arrangement (200) further comprising a second light source (270), the second light source (270) being arranged to illuminate the calibration target (210) with a second light, wherein the second light source (270) is arranged between the lens (230) and the calibration target (210).
17. The arrangement (200) according to claim 15 or 16, wherein the first light source (260) and/or the second light source (270) comprises at least one light emitting diode (110) and 119).
18. The arrangement (200) according to any of claims 12 to 17, wherein the camera (240) is arranged on a side of the lens (230) opposite to the calibration target (210).
19. An arrangement (200) according to any of claims 12 to 18, wherein the lens (230) is a fresnel lens.
20. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the steps of:
-determining a lateral position of a calibration target (210) in front of a lens (230) of a Head Mounted Display (HMD) (201), wherein the calibration target (210) and the lens (230) each extend substantially in a corresponding two-dimensional plane, and wherein the lateral position of the calibration target (210) is related to a position of the calibration target (210) in the two-dimensional plane;
-determining a lateral position of the lens (230), wherein the lateral position of the lens (230) is related to the position of the lens (230) in the two-dimensional plane;
-determining a calibration target misalignment (280) based on the determined lateral position of the calibration target (210) and based on the determined lateral position of the lens (230); and
-performing a hardware calibration of a camera (240) of the Head Mounted Display (HMD) (201), wherein the hardware calibration is adjusted for the calibration target misalignment (280).
Technical Field
the present disclosure relates to methods and arrangements for calibrating a Head Mounted Display (HMD). The disclosure also relates to a computer program product.
Background
head Mounted Displays (HMDs) require calibration. This is due to the fact that: production tolerances will result in a manufacturing process that does not produce a perfectly aligned part by one hundred percent. The calibration criteria may take into account production tolerances and may ensure that good performance and a good user experience of the HMD is provided. However, the calibration process tends to be time consuming and resource consuming. Thus, the calibration process may increase production costs.
Furthermore, it may be difficult to fix the calibration target relative to the HMD. This is due to the design of many HMDs, particularly HMDs with round plastic and high tolerance joints.
disclosure of Invention
It is an object of the present disclosure to provide methods, arrangements and computer program products that mitigate at least some of the above mentioned characteristics.
It is a further object of the present disclosure to provide alternative methods, alternative arrangements and alternative computer program products for calibrating a Head Mounted Display (HMD).
According to one aspect, at least some of the objects are achieved by a method for calibrating a camera of a Head Mounted Display (HMD). The method includes placing a calibration target in front of a lens of the HMD. The calibration target extends substantially in a corresponding two-dimensional plane. The lenses extend substantially in corresponding two-dimensional planes. The method also includes determining a lateral position of the calibration target. The lateral position is related to the position of the calibration target in the two-dimensional plane. The method also includes determining a lateral position of the lens. The lateral position is related to the position of the lens in the two-dimensional plane. The method also includes determining a calibration target misalignment based on the determined lateral position of the calibration target and based on the determined lateral position of the lens. The method also includes performing a hardware calibration of a camera of the HMD. The hardware calibration is adjusted for calibration target misalignment.
This enables good calibration of the camera of the HMD. By determining the lateral position of the calibration target and the lens, the specific values of these two characteristics can be taken into account during hardware calibration. Therefore, production tolerances can be better addressed. Furthermore, this allows for better hardware calibration, e.g. with respect to intrinsic and extrinsic parameters of the camera.
In one example, the step of determining the lateral position of the calibration target comprises determining the lateral position of a center point of the calibration target. This enables an easy adaptive algorithm.
In one example, the step of determining the lateral position of the calibration target includes illuminating the calibration target with the first light from the first light source from a side of the calibration target opposite a side of the lens such that the first light from the first light source is transmitted through the calibration target to the lens. This enables an easy determination of the lateral position of the calibration target.
In one example, the step of determining the lateral position of the calibration target further comprises detecting the first light with a camera disposed on a side of the lens opposite the calibration target. This allows the lateral position to be determined using a camera in the HMD, thus reducing the need for additional equipment for calibration.
In one example, the step of determining the lateral position of the lens comprises determining the lateral position of the center of the lens. This enables an easy adaptive algorithm.
In one example, the step of determining the lateral position of the lens comprises determining the lateral position of the lens at a number of calibration distances between the lens and the calibration target. The calibration distance is the distance between the lens and the calibration target perpendicular to the lateral extension of the lens and the lateral extension of the calibration target. The lateral position of the lens is determined based on the lateral position of the lens determined at a number of calibration distances between the lens and the calibration target. This enables a more reliable calibration algorithm.
In one example, the step of determining the lateral position of the lens comprises illuminating the calibration target with a second light from a second light source, wherein the second light source is preferably arranged between the lens and the calibration target. This enables an easy determination of the lateral position of the lens. Furthermore, the light source in the HMD may be used to determine the lateral position, thus reducing the need for additional equipment for calibration.
In one example, the step of determining the lateral position of the lens further comprises detecting the reflection of the second light with a camera arranged on a side of the lens opposite the calibration target. This allows the lateral position to be determined using a camera in the HMD, thus reducing the need for additional equipment for calibration.
In one example, the step of determining the lateral position of the lens further comprises determining a color block in the reflection of the second light. This enables an easy determination of the lateral position of the lens, especially in the case where a fresnel lens is used in the HMD. In one example, the color block is caused by stray light transmitted through the lens due to reflection from the calibration target.
In one example, the step of determining the lateral position of the lens further comprises determining a lens center based on the determined intersections between the longitudinally extending lines of the plurality of color patches. This enables an easy determination of the lateral position of the lens, especially in the case where a fresnel lens is used in the HMD.
In one example, the step of disposing the calibration target in front of the lens of the HMD includes mounting the calibration target in the calibration mount and fixing the calibration mount relative to the HMD. This enables more accurate hardware calibration.
According to one aspect, at least some of the objectives are achieved by an arrangement for calibrating a camera of a Head Mounted Display (HMD). The arrangement includes an HMD. The HMD includes at least one lens. The lens extends substantially in a two-dimensional plane. The arrangement includes a calibration target. The calibration target is arranged in front of the lens. The calibration target extends substantially in a two-dimensional plane. The arrangement further comprises a calibration processor. The calibration processor is arranged to determine a lateral position of the calibration target. The lateral position of the calibration target is related to the position of the calibration target in the two-dimensional plane. The calibration processor is further arranged to determine a lateral position of the lens. The lateral position of the lens is related to the position of the lens in the two-dimensional plane. The calibration processor is further arranged to determine a calibration target misalignment based on the determined lateral position of the calibration target and based on the determined lateral position of the lens. The calibration processor is further arranged to perform a hardware calibration of a camera of the HMD, wherein the hardware calibration is adjusted for calibration target misalignment.
in one embodiment, the arrangement further comprises a calibration mount. The calibration mount includes a calibration target. The calibration mount may be fixed or movable relative to the HMD.
in one embodiment, the HMD is a Virtual Reality (VR) HMD.
In one embodiment, the arrangement further comprises a first light source. The first light source is arranged to illuminate the calibration target with first light from a side of the calibration target opposite a side of the lens such that the first light from the first light source is transmitted through the calibration target to the lens.
In one embodiment, the arrangement further comprises a second light source. The second light source is arranged to illuminate the calibration target with the second light, wherein the second light source is preferably arranged between the lens and the calibration target.
In one embodiment, the first light source or the second light source comprises at least one light emitting diode.
In one embodiment, the camera is arranged on a side of the lens opposite the calibration target.
In some embodiments, the lens is a fresnel lens.
According to one aspect, at least some of the objects are achieved by a computer program product, wherein the computer program product comprises instructions which, when executed by a computer, cause the computer to perform the steps of: a lateral position of a calibration target in front of a lens of the HMD is determined. The calibration target and the lens each extend substantially in a two-dimensional plane. The lateral position of the calibration target is related to the position of the calibration target in the two-dimensional plane. The computer program product further comprises instructions which, when executed by a computer, cause the computer to perform the steps of: determining a lateral position of the lens; and determining a calibration target misalignment based on the determined lateral position of the calibration target and based on the determined lateral position of the lens. The lateral position of the lens is related to the position of the lens in the two-dimensional plane. The computer program product further comprises instructions which, when executed by the computer, cause the computer to perform the steps of: a hardware calibration of the HMD is performed. The hardware calibration is adjusted for calibration target misalignment.
The arrangement and the computer program product provide corresponding advantages as described in relation to the method. Furthermore, it should be emphasized that features disclosed in relation to the method may be readily applied to the arrangement and vice versa. The same features apply to the computer program product.
drawings
for a more detailed understanding of the present invention, and the objects and advantages thereof, reference is made to the detailed description which should be read in conjunction with the accompanying drawings. Like reference symbols in the various drawings indicate like elements. In the drawings, there is shown in the drawings,
Fig. 1a depicts a schematic diagram of components of an HMD in use;
fig. 1b depicts some aspects of the components of the HMD in use;
Fig. 1c depicts an embodiment of a component of an HMD;
Fig. 2a depicts a schematic diagram of an arrangement according to the present disclosure;
Fig. 2b depicts a schematic of an arrangement according to the present disclosure.
FIG. 3 depicts a schematic example of a calibration target;
Fig. 4 a-4 b depict schematic examples of two lenses in an HMD;
FIG. 5a depicts an example of a transmission calibration image;
FIG. 5b depicts an example of a reflectance calibration image; and
Fig. 6 depicts a flow chart of a method according to the present disclosure.
Detailed Description
Fig. 1a and 1b depict schematic views of selected components of an embodiment of an HMD in use. The depicted HMD may be a Virtual Reality (VR) HMD 100. VR HMD 100 may include an eye tracking system. It should be understood that the present disclosure is equally applicable to HMDs without VR functionality.
Referring first to fig. 1a, in addition to a VR HMD 100, a user's eyes 102 and head 104 are shown. The VR portion of the VR HMD 100 shown includes two VR displays 105 and two VR lenses 130, one VR display 105 and one VR lens 130 for each eye 102. The VR display 105 is positioned in front of the eye 102 and the VR lens 130 is positioned between the eye 102 and the VR display 105. Instead of two VR displays 105, two regions of a single VR display may be used. The eye tracking portion of the VR HMD 100 includes two so-called hot mirrors 135 and two cameras 120. To capture images of the eye 102 for eye tracking, a hot mirror 135 is disposed between the VR display 105 and the VR lens 130. Further, an illuminator (not shown) is arranged on or in the VR HMD 100 such that the illuminating light is directed to the eye 102. The reflection of the illumination light from the eye 102 toward the hot mirror 135 will be reflected toward the camera 120, where in the camera 120 the illumination light is detected to produce an image of the eye. For example, the type of hot mirrors 135 may be such that they will reflect light in the infrared band but be transparent to light in the visible band. The illuminator (not shown) used will then produce illuminating light in the infrared band, and the camera 120 will contain an image sensor capable of detecting light in the infrared band.
fig. 1b shows a side view of selected components of the VR HMD 100. Illumination light from an illuminator (not shown) toward the eye 102 will be reflected back toward the hot mirror 135 and through the VR lens 130 and toward the camera 120, where the illumination light is detected to produce an image of the eye 120.
The previous figures depict the possibility of an intended use of the HMD. However, it should be understood that some aspects of the present disclosure may be applied in connection with the manufacture of HMDs, i.e., in front of the intended use of the HMD. On the other hand, it should also be emphasized that the present disclosure is not limited to this manufacturing process, and may be equally applied at any time of the HMD's useful life.
Fig. 1c shows an exploded view of selected components of VR HMD 100. Selected components for one eye are shown including an illuminator cover 124, an illuminator in the form of light emitting diodes LEDs 110-119, a camera 120 including an image sensor, a VR lens 130, a lens cup or lens tube 126, a hot mirror 135, a VR display 105, and an electronics board 128. FIG. 1c shows an example arrangement of an illuminator in the form of LEDs 110-119, where the LEDs 110 and 119 are arranged along the periphery of the VR lens 130 to create a pattern when the eye 102 is illuminated. Illumination light from the LEDs 110 to 119 reflected from the eye and hot mirror 135 is detected in the camera 120 to produce an image of the eye.
The above are merely examples of selected components of a VR HMD to better understand how particular embodiments may be implemented in practice. However, the present disclosure is by no means limited to this example, but can be used in principle for any kind of HMD.
Fig. 2a depicts a schematic diagram of an arrangement 200 according to the present disclosure. The arrangement 200 is an arrangement for calibrating a Head Mounted Display (HMD) 201. The HMD 201 may be a VR HMD 100 as described with reference to fig. 1a to 1 c.
The arrangement 200 comprises an HMD 201. The HMD 201 includes at least one lens 230. The at least one lens 230 may be a VR lens 130. In one example, HMD 201 includes at least two lenses 230, one lens 230 for each eye. In one example, at least one lens 230 is a fresnel lens. The at least one lens 230 extends substantially in a two-dimensional plane.
In the case of two lenses 230 (one for the left eye and one for the right eye) the two-dimensional plane in which the two lenses 230 extend substantially may be spanned by corresponding vectors x r, y r or x l, y l, respectively, where subscripts r and l denote the right and left lenses, respectively, the vector perpendicular to the two vectors x and y will be denoted z. in the case of left and right lenses, and the vectors perpendicular to the corresponding right and left vectors x and y will be denoted z r and z l, respectively.
The lateral position of the lens 230 is related to the position of the lens in a two-dimensional plane in which the lens extends substantially (i.e., in a plane spanned by the vectors x and y). The lens 230 includes a lens center 235. The lens center 235 may be the geometric center of the lens. However, any other definition of the lens center 235 is possible as long as the lens center represents a specific predetermined point of the lens. It should be understood that lens center 235 generally does not coincide with the intersection of vectors x, y, and z. This condition may occur occasionally. However, typically, when the lens 230 is mounted in the HMD 201, the lens center 235 will not be perfectly aligned with the predetermined coordinate system.
the arrangement 200 further comprises a calibration target 210. The calibration target 210 is arranged in front of the lens. In a preferred embodiment, the term "in front of the lens" means that the calibration target is arranged at a different position in the z-direction compared to the lens. The calibration target 210 extends substantially in a two-dimensional plane. In one example, the calibration target 210 extends in substantially the same two-dimensional plane as the at least one lens 230. In the case where the arrangement 200 includes more than one lens 230, more than one calibration target 210 may be provided. In one example, two calibration targets 210 are provided, one for the right lens 230 and one for the left lens 230. Alternatively, only one calibration target 210 is provided for more than one lens 230. In this case, calibration of more than one lens 230 may then be performed.
Preferably, the two-dimensional plane in which the at least one lens 230 extends and the two-dimensional plane in which the calibration target 210 extends are substantially parallel, i.e. they preferably substantially coincide at z-0.
The calibration target 210 has a lateral position. The lateral position of the calibration target 210 is related to the position of the calibration target in a two-dimensional plane (i.e., in the x and y directions) in which the calibration target extends substantially.
the lateral position of the lens 230 and the calibration target 210 is typically two-dimensional, i.e., typically has a component in the x-direction and a component in the y-direction.
the arrangement 200 may include a calibration mount 220. The calibration mount 220 may include a calibration target 210. In one example, the calibration mount 220 is movable relative to the HMD 201. In one example, the calibration target 210 is arranged to be movable substantially along the z-direction at the calibration mount 220. In one example, the calibration target 210 is arranged to be substantially fixed along the x and y directions at the calibration mount 220. This allows the calibration target 210 to be moved to different calibration distances relative to the lens 230. The calibration distance is the distance between the lens and the calibration target perpendicular to the line of lateral extension of the lens and the line of lateral extension of the calibration target (i.e. in the z-direction).
The arrangement 200 further comprises a calibration processor 250. The calibration processor 250 may include electronic circuitry. The calibration processor 250 is arranged to determine the lateral position of the calibration target 210. The calibration processor 250 is further arranged to determine the lateral position of the lens 230. The calibration processor 250 is further arranged to determine a calibration target misalignment based on the determined lateral position of the calibration target 210 and based on the determined lateral position of the lens 230. The calibration processor is further arranged to perform a hardware calibration of the HMD 201, wherein the hardware calibration is adjusted for calibration target misalignment. This is further described with reference to fig. 6.
The arrangement 200 may comprise a first light source 260. The first light source 260 is arranged to illuminate the calibration target 210 with the first light from a side of the calibration target 210 opposite to a side of the lens 230. This illumination is arranged in a manner such that the first light from the first light source 260 is transmitted through the calibration target 210 to the lens 230. In one example, this is achieved by arranging the first light source 260 at a side of the calibration target 210 opposite to the side of the lens 230, as illustrated in fig. 2 b. However, this is not a requirement. The first light source 260 may in principle be arranged at any position. As an example, the first light may be directed with a mirror, an optical fiber, or the like, such that the calibration target 210 is illuminated with the first light from a side of the calibration target 210 opposite a side of the lens 230. The first light source 260 may include one or more Light Emitting Diodes (LEDs).
the arrangement 200 may comprise a second light source 270. The second light source 270 is arranged to illuminate the calibration target 210 with the second light from a side of the lens 230, wherein the second light source is preferably arranged between the lens and the calibration target. In one example, this is achieved by arranging the second light source 270 at a side of the lens 230 opposite to a side of the calibration target 210. However, this is not a requirement. The second light source 270 may in principle be arranged at any position. As an example, the second light may be directed with a mirror, an optical fiber, or the like, such that the calibration target 210 is illuminated with the second light from a side of the lens 230 opposite to the side of the calibration target 210. The second light source 270 may include one or more LEDs.
in one embodiment, the second light source 270 is the LEDs 110 to 119 described with reference to FIG. 1 c. Thus, the components of the HMD 201 may be used for calibration, thus minimizing the need for possible additional elements during calibration.
In one example, the arrangement 200 includes a camera 240. The camera 240 is preferably disposed on a side of the lens 230 opposite the calibration target 210. The camera 240 may be arranged to detect the first light transmitted through the calibration target 210. The camera 240 may be arranged to detect the second light reflected from the calibration target 210. In one example, the camera 240 may be the camera 120 described with reference to fig. 1 a-1 c. The camera 240 may be part of the HMD 201. This reduces the need for additional components needed to calibrate the HMD 201.
In one example, the calibration processor 250 is arranged to switch the first light source 260 and/or the second light source 270 on/off. As an example, the calibration processor 250 may be arranged to control the first light source 260 and the second light source 270 in such a way that at most only the first light or only the second light is emitted during calibration. Thus, it is ensured that the camera will only receive light from light transmitted through the calibration target 210 or only light reflected from the calibration target 210. However, this is not a requirement. In principle, other solutions are possible. In one example, one or more controllable filters (not shown) are disposed between the calibration target 210 and the camera 240. In situations where the characteristics (e.g., polarization or wavelength used) of the first and second light are different, the optical filter may be adjusted and controlled to pass substantially only the first light or only the second light. The calibration processor 250 may be arranged to control the camera 240. The calibration processor 250 may be arranged to perform control of the camera 240 and the first and second light sources 260, 20 via the corresponding communication channels L240, L260, L270. The communication channels L240, L260, L270 may be wired or non-wired. The communication channels L240, L260, L270 may be combined into a common communication channel, e.g. a common bus. The calibration processor 250 may be arranged to receive information from the camera 240, e.g. an image taken by the camera when the first light or the second light is present.
Examples of images taken by the camera when only the first light or only the second light is detected, respectively, can be seen in fig. 5a and 5 b. This will be described in more detail with reference to fig. 5a and 5 b.
Fig. 2b depicts a schematic diagram of an arrangement 200 according to the present disclosure. The elements correspond to the elements already described with reference to fig. 2 a. In fig. 2b, the HMD 201 comprises two lenses 230. In addition, calibration distance 290 and calibration target misalignment 280 are shown. It should be emphasized that in the example of fig. 2b, the lateral position of the left lens 230 is zero in the x-direction. However, this is often not the case. Furthermore, it should be emphasized that the example depicted in fig. 2b is merely a two-dimensional sketch. Therefore, in general, there is also a misalignment in the y direction. In one example, the calibration distance 290 may be varied such that, for example, an image may be formed by the calibration target at several calibration distances.
Fig. 3 depicts a schematic example of a calibration target 210. The illustrated example is a side view of the calibration target 210. When implemented in the arrangement 200 of fig. 2b, the depicted example will be in the x and y directions. The calibration target 210 may include several rows with many small openings 299 in each of the rows. So as not to unduly complicate the drawing, only one small opening is depicted with a reference numeral. In the example shown, the calibration target 210 includes nine rows with nine small openings 299. The small opening 299 is depicted by a black dot. The small opening 299 is preferably arranged to let light pass through it. In the example shown, cross 298 is disposed in the middle of calibration target 210. Cruciform 298 may extend into some of small openings 299. Cross 298 is preferably arranged to allow light to pass therethrough. The light passing through the cross 298 and the small opening 299 is preferably at least the first light. The remaining part of the calibration target 210 is preferably arranged to be non-transparent to light, preferably in particular to the first light.
Many different examples of calibration targets are possible. In one example, the calibration target 210 includes at least one element that allows a reference point to be determined at the calibration target. In the example shown, the middle cross 298 allows a reference point to be determined at the calibration target 210.
Fig. 4 a-4 b depict schematic examples of two lenses 230 in HMD 201. Fig. 4a and 4b are a schematic and a top view, respectively, as indicated by the coordinate system. The coordinate system is consistent throughout the different figures of the present disclosure. Fig. 4a depicts ten light sources 430 at the outer periphery of each lens 230. These ten light sources 430 (reference numerals are given only on the left lens so as not to unduly complicate the figure) may for example be the ten LEDs 110 to 119 described with reference to fig. 1 c. Ten light sources 430 may constitute the second light source 270. However, it should be emphasized that any other number of light sources may constitute the second light source 270.
Fig. 5a depicts an example of a transmission calibration image. The transmitted calibration image corresponds to an image taken by the camera 240 as the first light is transmitted through the calibration target 210 and the lens 230 to the camera 240. In the example shown, the transmitted calibration image originates from a calibration target 210 similar to the calibration target 210 depicted in FIG. 3. The size of the cross and the many small openings are slightly different.
Fig. 5b depicts an example of a reflectance calibration image. The reflected calibration image corresponds to an image taken by the camera 240 when light from the second light source 270 is reflected on the calibration target and then passes through the lens 230 to the camera 240. The reflectance calibration image shows a number of color patches 297, only one of which is shown with a reference numeral. In fig. 5b, ten color blocks 297 with relatively large intensity can be seen. Furthermore, many color patches with relatively small intensities can be seen. Similar patterns are particularly noticeable in the case where the lens 230 is a fresnel lens. The patches typically extend largely in one direction and only a small amount in the vertical direction. The direction in which the color patches extend substantially will be indicated hereinafter as the longitudinal axis of the color patches. It should be noted that the longitudinal axes of the different patches are typically not parallel. A portion of the approximate longitudinal axis of the color block 297 is depicted by a black line 296. The longitudinal axis of the color block does not extend beyond the color block even if represented as a longitudinal extension. As can be seen from fig. 5b, the longitudinal axes of the different patches substantially coincide at a certain point of the figure.
Fig. 6 depicts a flow chart of an example of a method 600 according to the present disclosure. Method 600 is a method for calibrating a camera of an HMD. The HMD may be any of the HMD 201 or VR HMD 100 described above.
The method 600 begins at step 610. Step 610 includes placing a calibration target in front of a lens of the HMD. The calibration target may be any of the calibration targets described above, for example with reference to fig. 2a and 3. The calibration target and the lens each extend substantially in a two-dimensional plane. The lens may be a fresnel lens. Step 610 may include installing a calibration target in a calibration mount. The calibration mount may be a calibration mount in which a calibration target is fixed. The calibration mount may be a calibration mount that allows the calibration target to move in one direction (e.g., a direction perpendicular to a two-dimensional plane in which the calibration target substantially extends). This facilitates moving the calibration target to different calibration distances 290 as described with reference to fig. 2 b. Step 610 may include fixing the calibration mount relative to the HMD. The method continues at step 620.
Step 620 includes determining the lateral position of the calibration target. The lateral position is related to the position of the calibration target in a two-dimensional plane in which the calibration target extends substantially. Step 620 may include determining a lateral position of a center point of the calibration target. An example of a central point is a cross of a calibration target as described with reference to figure 3. However, the center point may have any form. It is also not necessary to have the point at which the lateral position is determined at the center of the calibration target. In principle, the lateral position of any point at the calibration target can be determined. However, in this case, it is advantageous to know the relative position of the determined point on the calibration target.
Step 620 may include illuminating the calibration target with the first light from the first light source. The illumination is performed from the side of the calibration target opposite to the side of the lens. The illumination is performed in a manner such that first light from the first light source is transmitted through the calibration target to the lens. Step 620 may include detecting the first light with a camera disposed on a side of the lens opposite the calibration target. As an example, a camera may take a picture of the first light. An example of such a picture is given in fig. 5 a. From the detected light, the lateral position of the calibration target can be determined. As an example, from a picture of a camera, a lateral position of a calibration target may be determined. As an example, a displacement of the calibration target in a two-dimensional plane (in which the calibration target mainly extends) will move the cross shape in fig. 5a in the corresponding direction. Thus, by detecting the location of the cross in fig. 5a, the lateral position of the calibration target can be determined. The method continues at step 630.
Step 630 includes determining the lateral position of the lens. The lateral position of the lens is related to the position of the lens in a two-dimensional plane in which the lens mainly extends. Step 630 may include determining the lateral position of the lens center. However, the lateral position of the lens center is not necessarily determined. In principle, any position of the lens in the two-dimensional plane in which it extends can be determined. Advantageously, the relation between the determined point and the lens is known. However, since calibration is typically performed in connection with manufacturing, and the already assembled lens is typically known, the relation between the determined point and the lens is also typically known.
Step 630 may include determining the lateral position of the lens at several calibration distances between the lens and the calibration target. The calibration distance is the distance between the lens and the calibration target perpendicular to the lateral extension of the lens and the lateral extension of the calibration target. This can advantageously be carried out in a situation in which the calibration target is mounted on the calibration mount. The lateral position of the lens may be determined based on the lateral position of the lens determined at a number of calibration distances between the lens and the calibration target. As an example, the lateral position of the lens may be determined as a statistical average or a weighted statistical average of the lateral positions of the lens determined at a number of calibration distances between the lens and the calibration target. The weighting may be based on a specific calibration distance. Basing the determination of the lateral position of the lens on the determination of the lateral position of the lens at several calibration distances may improve the accuracy of the results and/or the confidence of the results. However, in principle, it is sufficient to determine the lateral position of the lens at only one calibration distance.
Step 630 may include illuminating the calibration target with a second light from a second light source. The second light source is preferably arranged between the lens and the calibration target. Step 630 may include detecting a reflection of the second light with a camera disposed on a side of the lens opposite the calibration target. As an example, a camera may detect an image of the reflection. An example of an image of the detected reflection is shown in fig. 5 b. From the detected reflections, e.g. from images of the detected reflections, the lateral position of the lens may be determined.
Step 630 may include determining color blocks in the reflection of the second light. An example of a color patch can be seen in fig. 5 b. The determination of color patches may be performed via image analysis (e.g., computer vision). Step 630 may include determining the longitudinal extension of the patch. This may also be performed via image analysis. In one example, this is performed via a flood fill (floodfill) algorithm. In one example, a line is fitted to the longitudinal extension of the color patch. The lens center may be determined based on the intersection between the longitudinally extending lines of the determined color patches. This may also be performed via image analysis. It should be emphasized that the longitudinal extensions do not necessarily all intersect at one point. In principle, the different intersections may have a certain offset to some extent. As an example, the center point of the intersection may be determined as the lens center. However, other ways of determining the lens center based on the intersection may also be used. As an example, an optimization algorithm may be used for the fitted straight line. Color blocks are particularly noticeable at the fresnel lens. However, color lumps may also occur for other lens designs. In addition, other lens designs may provide other ways of determining the lateral position of the lens.
However, it should be emphasized that steps 620 and 630 are independent of each other. One determines the lateral position of the calibration target and one determines the lateral position of the lens. Thus, neither perfect alignment of the lens in the HMD is assumed, nor is the calibration target located at a specific position. Not relying on either of these two assumptions increases the accuracy and flexibility of the method. The method continues at step 640.
Step 640 includes determining a calibration target misalignment based on the determined lateral position of the calibration target and based on the determined lateral position of the lens. Calibration target misalignment may be a one-dimensional quantity, e.g., a distance to an initial expected location of the calibration target. However, typically, the calibration target misalignment is a two-dimensional or three-dimensional vector. This vector may, for example, describe the displacement relative to the expected original position of the calibration target. The method continues at step 650.
Step 650 includes performing a hardware calibration of a camera of the HMD. The hardware calibration is adjusted for calibration target misalignment. The hardware calibration may include calibrating intrinsic parameters and/or extrinsic parameters of the camera. The hardware calibration may include calibrating lens distortion. These parameters may then be stored in a memory (e.g., a memory of the HMD). When operating the HMD, software operating the HMD may then adjust for extrinsic and/or intrinsic parameters of the camera. The software may further adjust for the lateral position of the lens determined in step 630. As an example, the term "adjusting" may include compensating an image captured by a camera for calibration target misalignment and/or lateral position of a lens. After step 650, the method 600 ends.
Method 600 may be performed for one or several lenses of the HMD (e.g., for a lens intended for a user's left eye and a lens intended for a user's right eye). The method 600 may be performed by any of the elements described with reference to fig. 1-5 b. Further, the method 600 may include any feature or characteristics attributed to any of these elements. In contrast, the arrangement 200 or elements thereof may comprise features as described with reference to fig. 6 and/or be arranged to perform any of the method steps as described with reference to fig. 6.
According to aspects of the disclosure, a computer program product may be provided, tangibly embodied in a non-transitory computer-readable medium. The computer program product may include instructions that, when executed by a computer, cause the computer to perform some or all of the steps described with respect to the method 600. A non-transitory computer-readable medium may include instructions that, when executed by a computer, cause the computer to perform some or all of the steps described with respect to method 600.
The computer program product may be executed in whole or in part, for example, by the calibration processor 250 described with reference to fig. 2 a. The non-transitory computer readable medium may be executed, for example, in whole or in part, by the calibration processor 250 described with reference to fig. 2 a.