阅读说明：本技术 用于改进盲人用户或视力受损用户的环境感知的医疗装置 (Medical device for improving the environmental perception of blind or visually impaired users ) 是由 皮埃尔·布里安德 于 2019-11-15 设计创作，主要内容包括：本发明涉及一种用于改进盲人用户或视力受损用户的环境感知的装置,包括：-一组机械致动器(3),其预期与用户的皮肤接触,-至少一个数码相机(6),其被设计成获取面向所述用户的环境的当前数字图像,-处理电路(2),其连接到所述相机以从所述所获取数字图像接收像素信号并且将所述像素信号的至少一个部分转换成控制信号,所述控制信号中的每一个向这组致动器中的一个机械致动器供电,-眼睛跟踪模块,用于跟踪(7)所述用户的每只眼睛以识别所述用户的注视方向。所述处理电路随后在由所述相机(6)拍摄的所述环境中选择作为所述注视方向的函数的所获取的当前图像的区域并且将所述区域的所述像素信号转换成控制信号,所述控制信号中的每一个向所述组(3)中的一个致动器供电以刺激所述用户的皮肤。(The invention relates to a device for improving the perception of the environment of a blind or visually impaired user, comprising: -a set of mechanical actuators (3) intended to be in contact with the skin of a user, -at least one digital camera (6) designed to acquire a current digital image facing the environment of the user, -processing circuitry (2) connected to the camera to receive pixel signals from the acquired digital image and to convert at least one portion of the pixel signals into control signals, each of the control signals powering one of the mechanical actuators of the set of actuators, -an eye tracking module for tracking (7) each eye of the user to identify the gaze direction of the user. The processing circuit then selects a region of the acquired current image as a function of the gaze direction in the environment captured by the camera (6) and converts the pixel signals of the region into control signals, each of which supplies one actuator of the group (3) to stimulate the skin of the user.)

1. An apparatus for improving the perception of the environment of a blind or visually impaired user, the apparatus comprising:

at least one set of mechanical actuators intended to be in contact with the skin of the user,

-at least one digital camera arranged for acquiring a current digital image of the environment facing the user,

-processing circuitry connected to the camera to receive signals from pixels of the acquired digital image and to convert at least part of the pixel signals into control signals, each control signal powering one mechanical actuator of the set of actuators,

characterized in that the apparatus further comprises at least one eye tracking module for at least one eye of the user to identify a gaze direction of the user,

and wherein the processing circuitry is arranged to select a region of the acquired current image in the environment captured by the camera that depends on the gaze direction, and to convert signals from pixels of the region into control signals, each control signal powering one actuator of the group to stimulate the skin of the user.

2. An apparatus according to claim 1, wherein the processing circuitry is arranged to select a limited area of the current image within the acquired current image that depends on the gaze direction, and to convert signals from pixels of the limited area into control signals, each control signal powering one actuator of the group to stimulate the user's skin.

3. The apparatus according to any one of claims 1 and 2, characterized in that it comprises:

two sets of mechanical actuators intended to come into contact with the skin of the user, an

-two eye tracking modules for respective eyes of the user, for defining two regions in a current image and converting signals from pixels of each of the two regions into two respective sets of control signals powering the two respective sets of actuators.

4. The apparatus according to one of the preceding claims, characterized in that the camera and the eye tracking module are mounted at least on a common mechanical support, the eye tracking module being oriented towards one eye of the user, the camera being oriented to acquire a current digital image facing the environment of the user.

5. Device according to one of the preceding claims, characterized in that at least one connection between the processing circuit and the actuator is a wireless link for transmitting control signals of the actuator.

6. Device according to one of the preceding claims, characterized in that each actuator is arranged for stimulating the skin of the user according to an intensity as a function of the received control signal, and that the control signals are each a function of the color intensity of a pixel.

7. The apparatus of claim 6, wherein each actuator moves while vibrating to stimulate the user's skin by cyclic vibration, the frequency of the vibration of an actuator being determined by the color type of a pixel.

8. The apparatus according to one of the preceding claims, wherein the image area has a general shape corresponding to a field of view shape of the eye tracked by the eye tracking module, and wherein the actuators of the set of actuators are distributed according to a two-dimensional shape corresponding to the field of view shape.

9. A method implemented by a processing circuit of an apparatus according to one of the preceding claims, the method comprising:

-receiving signals from pixels of the acquired digital image,

-receiving measurements by eye tracking of a user's gaze direction,

-determining a boundary in the acquired image of a region corresponding to the direction of the gaze and selecting a signal from pixels of the region,

-converting signals from pixels of the area into control signals, each control signal intended to power one actuator of the set of mechanical actuators, the number of pixels in the area corresponding to the number of actuators comprised by the set of mechanical actuators,

-transmitting respective control signals to the actuators in the group.

10. A computer program comprising instructions for implementing the method according to claim 9 when the instructions are executed by processing circuitry of an apparatus for improving context awareness.

Technical Field

The invention relates to improving the perception of the environment, particularly for blind or visually impaired users. The term "blind user" is understood to mean a person suffering from prolonged blindness or blindness caused only by the environment, for example (in complete darkness for military or other uses) for the purpose of tactile use or as a supplement to virtual or augmented reality or other uses.

Background

Retinal implants are known from the prior art. These are devices surgically implanted behind one eye of the blind that electrically stimulate neurons going to the brain by reproducing radio-carried images from video sensors mounted on glasses. However, the cost and implementation complexity of these devices limit their use.

In parallel to these devices, portable devices for the blind are also known from the prior art, which are able to convert video images taken from an environment facing the individual into "tactile images" reproduced on the skin or mucous membranes of the individual. The currently used version comprises a front camera connected to an electrostimulation unit (low intensity) located in this case on the user's tongue. While this reproduction is fairly basic, the blind is still useful.

One major drawback of these devices is that their performance is greatly reduced when the object at which the camera is aimed is moving. Often, users must turn their chest, neck, head, or other direction toward the source of the sound. Eventually, such systems can eventually leave the user tired of the device.

Disclosure of Invention

The present invention will improve this situation.

For this purpose, the invention provides a device that improves the spatial perception of the environment by measuring the movement of the blind's eyes and by closed-loop controlling the image reproduced on the skin in accordance with this measurement.

The object of the present invention is therefore an apparatus for improving the environmental perception of a blind or visually impaired user, said apparatus comprising:

at least one set of mechanical actuators intended to be in contact with the skin of the user,

at least one digital camera arranged for acquiring a current digital image of the user-facing environment,

-processing circuitry connected to the camera to receive signals from the pixels of the acquired digital image and to convert at least part of the pixel signals into control signals, each control signal powering one mechanical actuator from the set of actuators.

In particular, the apparatus further comprises at least one eye tracking module for at least one eye of the user in order to identify the gaze direction of the user. Furthermore, the processing circuitry is arranged for selecting an area of the acquired current image in the environment captured by the camera, which area depends on the gaze direction, and for converting signals from pixels of the area into control signals, each control signal powering one actuator of the group to stimulate the skin of the user.

The term "mechanical actuator" is generally understood to mean an element that can be arranged, for example, in a matrix and each element is capable of moving upon receiving an electrical pulse. These may therefore be piezoelectric elements, which are designed, for example, with:

the amplitude of the displacement is a function of the electrical signal received by the piezoelectric element, and/or

The frequency of the displacement is a function of the electrical signal received by the piezoelectric element.

Alternatively, these may be piezoelectric elements in combination with micro-electromechanical (or "MEMS") type systems.

As another variant, the emission of focused ultrasound waves distributed on a planar matrix may be used to stimulate the skin of the user. As a further variant, an electrode is provided which emits an electrical pulse that is perceived as a tiny shock (the intensity of which may be characteristic of the image grey level and have a repetition frequency characteristic of a particular color, as will be demonstrated later by way of example embodiments).

The "skin" of a user is understood above to refer both to the skin of the user (on the body or elsewhere) and to the mucous membrane of e.g. the palate, tongue or elsewhere, and also corresponds to the general medical term "epidermis". Thus, the actuator may take the form of an electrode that handles micro-currents and is, for example, housed in the intra-buccal artificial palate (e.g., the portion that engages the artificial palate), but does not prevent the user from speaking, eating, or drinking.

Furthermore, the term "digital camera" is understood to mean the following fact: the image acquired from the environment comprises pixels, typically arranged in a matrix, and an electrical signal may be assigned to each pixel that powers the actuator. For example, for a monochrome ("black and white") image, the intensity of the electrical signal powering the actuator may be proportional to the gray level of the pixel (e.g., from dark black gray to light white gray). Depending on the strength of this electrical signal, the mechanical actuator may be displaced with a larger or smaller amplitude and/or frequency.

Thus, the device in the sense of the present invention allows reproducing the attention of a user focused on a specific area of the environment by means of an eye tracking measurement of the user and thus by means of the gaze of the user. For example, if the user has heard a sound in a particular direction, the user may look towards this direction, which the apparatus achieves by rendering a "tactile image" of the selected area of the environment in this direction.

For this purpose, the invention must overcome the prejudice that the blind person loses eye-tracking ability. However, for those people who are at least recently blind (and who are assumed to retain their eyes), it is counter-intuitive to observe that the eyes of the blind are still moving (despite their blindness).

The object of the present invention is therefore an apparatus in the form of an equipment unit controlled by a software application and intended for the purpose of reducing in particular the effects of obstacles, in this case blindness for blind users. In this regard, the device of the present invention may be considered a medical device.

In one embodiment, the processing circuitry may be arranged for selecting a limited area of the current image depending on the gaze direction in the acquired current image, and for converting signals from pixels of said limited area into control signals, each control signal powering one actuator from the set to stimulate the skin of the user.

In a variant, the processing circuitry may be arranged to rotate a moving camera (not necessarily wide-angle) so as to follow the gaze direction of the user. However, it may be advantageous to implement a fixed (wide-angle) camera and to select by computer the relevant area in the image thus acquired at a wide angle depending on the gaze direction.

However, the processing circuitry may also be arranged for rotating the moving (at least in azimuth) wide-angle camera as well, to track eye movements converging towards the nose of the user (which corresponds to searching for "myopia"). Furthermore, these cameras are capable of auto-focusing (with a blur detection module), which allows the user to obtain a spatial perception of the environment in both near space ("near vision") and far space ("far vision"). Outside the focusing situation and in particular for "far vision", the cameras can be adjusted by default in an orientation relative to each other while conforming as closely as possible to the physiological orientation of the ipsilateral eyes of the user (approximately 15 ° transverse to the sagittal plane, hence approximately 30 ° orientation of one camera relative to the other).

In one possible embodiment, the apparatus further comprises:

two sets of mechanical actuators intended to come into contact with the skin of the user, an

Two eye tracking modules for the respective eyes of the user, defining two regions in the current image and converting the signals from the pixels of each of the two regions into two respective sets of control signals powering two respective sets of actuators.

By applying eye tracking to each eye of the user, the device may also provide perception of the three-dimensional space in front of the user's gaze by means of a pair of mechanical actuator sets. Indeed, thanks to this binocular coordination capability, the device may provide the user with a perception of the three-dimensional space being viewed, both for eye tracking and for a set of mechanical actuators. As will be demonstrated below with reference to fig. 1, the actuator groups are distinct and spatially separated from each other, but the human brain is still naturally configured for reproducing a single three-dimensional image starting from two separate two-dimensional images (as it is usually done for both eyes of a user).

Thus, for a person who is blind but here assumed to still have the ability to move both eyes, the user may receive two "tactile images" on his skin, which images may be mutually attested to for 3D perception.

However, this is a complicated embodiment. A monocular tracking module may be provided (for economic reasons, or otherwise if the user has only one eye to measure pupil movement) that may have two cameras to capture the user-facing environment (possibly with an angle of about 30 ° between each camera axis to conform to the usual physiological orientation of the eye's optical axis). The only possible uncertainty using monocular tracking is discrimination:

-a situation where the user is viewing a nearby image and the eyes converge generally towards the nose, an

The case where the user is looking away, but to one side, and towards the nose.

In this case, the camera can "autofocus" and thus distinguish between each of these two cases.

In one embodiment of the invention, the camera and the eye tracking module are mounted at least on a common mechanical support, the eye tracking module being oriented towards one eye of the user (typically the same side of the user), the camera being oriented to acquire a current digital image of the environment facing the user. Thus, the cameras may be oriented along the physiological axis of the ipsilateral eye of the user (or each camera oriented along the axis of one eye.

Thus, the distance between the camera and the eye tracking module remains constant over time on the stand, allowing the user's gaze location to be correlated to what the camera is taking at any given time.

Advantageously, the camera is of the wide-angle type with a focal length between 10mm and 35 mm.

In one embodiment of the invention, the apparatus comprises a wireless link for transmitting the control signal to the actuator for at least one connection between the processing circuit and the actuator.

The wireless link provides the following advantages: for example, connecting an actuator (which may stimulate the individual's body or back, as described below with reference to fig. 1) to the rest of the device in a compact manner; and creating a link between a mechanical mount and a mechanical actuator, for example, that groups the eye tracking module, the camera, and the processing circuitry, etc.

Since the skin of the user's hand is particularly sensitive, in one variant, mechanical actuators integrated into the glove may be provided, for example a set of mechanical actuators advantageously placed on the user's palm in the form of respective patches. For example, a glove with a mechanical actuator patch is provided for each hand, each hand corresponding to one image of the environment to be perceived (and therefore with a possible three-dimensional reconstruction). The camera, eye tracking module and processing circuitry may be mounted on a common cradle and wirelessly connected to the patch of the glove.

In one variation where the mechanical actuator may stimulate a portion of the user's head (e.g., the forehead as illustrated in fig. 5), the assemblies (camera, eye tracking module, etc., and actuator) may be mounted to a common support, for example in the form of a headset worn on the forehead and found over the user's eyes.

In one embodiment, the actuators are arranged for stimulating the skin of the user in dependence on the intensity as a function of the received control signals, and the control signals are each a function of the color intensity of the pixels.

"color intensity of a pixel" is understood to mean the luminance of a pixel of a given color.

For example, for a "black and white" image, each actuator may be activated according to the gray level of the pixel going from white to black. As a variant, a set of three actuators may be provided for each pixel of the color image, each actuator having an intensity or vibration frequency that varies with the "red, blue, green" color level of this pixel.

For example, for "rendering" of a color image, the brightness of the pixel (e.g., from dark to light) may be converted into the intensity of the actuator movement (thus pressing harder against the user's skin), whereas the vibration frequency of each actuator may be determined by the type of color (red or blue or green) assigned to that actuator. For example, an actuator reproducing a blue level vibrates faster than an actuator reproducing a red level (green or yellow is represented by an actuator having an intermediate vibration frequency).

Thus, in embodiments where each actuator is movable when vibrating to stimulate the user's skin by cyclic vibrations, the vibration frequency of the actuator may depend on the type of pixel color.

In one embodiment, the image area may have a general shape corresponding to a field of view shape of the eye tracked by the eye tracking module, and the actuators of the set of actuators may then be arranged to be distributed according to a two-dimensional shape corresponding to the field of view shape.

It will therefore be appreciated that in the apparatus, a "one to one" correspondence with the set of actuators is provided for the pixels in the region. Thus, the number of the plurality of actuators in the group may correspond to the number of pixels in the area. Spatially, a correspondence of pixels in the area to the positions of the respective actuators of the set of actuators may also be provided. However, this is an exemplary embodiment. For example, the actuator may be associated with a group of four pixels or more of the image (e.g., having an average gray level). Similarly, the actuators may be irregularly spaced within the set of actuators (as the set of actuators may extend over more or less sensitive areas of the body, more sensitive areas generally being able to receive more actuators).

The device may generally comprise processing circuitry, in particular for driving the actuators based on pixels of the acquired image area. This processing circuit then implements a method, another subject of the invention, and comprising the following steps:

-receiving signals from pixels of the acquired digital image,

-receiving measurements by eye tracking of a user's gaze direction,

-determining a boundary corresponding to the gaze direction in the acquired image of the region and selecting a signal from pixels of the region,

-converting signals from pixels of the area into control signals, each control signal intended to power one actuator of the set of mechanical actuators, the number of pixels in the area corresponding to the number of actuators comprised by the set of mechanical actuators,

-transmitting respective control signals to the actuators in the group.

The processing circuitry of the apparatus may generally comprise (as shown in fig. 4) a processor and a memory storing instructions of a computer program to implement this method when the instructions are executed by the processor. The invention also aims at such a computer program (the general algorithm of which can be illustrated by fig. 3 described below).

Drawings

Further advantages and features of the present invention will become apparent upon reading the description of exemplary embodiments presented below and upon examination of the accompanying drawings, in which:

FIG. 1 shows a schematic view of a

Fig. 1 shows an individual wearing a medical device for improving environmental perception according to the subject matter of the present invention according to a first exemplary embodiment;

FIG. 2

Figure 2 is a perspective view of a part of the device that is the subject of the invention with reference to figure 1;

FIG. 3

FIG. 3 is a flow chart of a general algorithm corresponding to the method of the subject invention with reference to FIG. 1;

FIG. 4

FIG. 4 schematically illustrates a processing circuit of the apparatus shown in particular in FIG. 1;

FIG. 5

FIG. 5 shows an individual wearing a medical device for improving environmental perception according to a second exemplary embodiment;

FIG. 6

Fig. 6 shows an individual 5 observing a frontal environment, i.e. a wearer of the device in the sense of the present invention;

FIG. 7

FIG. 7 schematically illustrates one of the mechanical actuator groups;

Detailed Description

Fig. 1 shows an arrangement for improving the perception of the environment in a space, typically in front of a user. The device is worn by a blind or amblyopic user 5, and comprises a mechanical stand 1 integrating a wide-angle camera 6 and an eye tracking module 7. More precisely, in the case when the user's eyes are still moving and when eye tracking is then possible on each eye, the apparatus may comprise two wide-angle cameras 6 (as shown in fig. 1), one for each photographing the environment in front of each eye of the user; and two eye tracking modules 7, each for tracking the movement of one eye.

Two cameras (in particular wide-angle cameras) are typically mounted in fixed corresponding positions on the stand 1, and selected areas in the wide-angle images partially overlap to provide a stereoscopic effect, allowing 3D perception. For this purpose, the device also comprises two spatially separated sets of mechanical actuators 3, as shown in fig. 1.

The camera 6 and the tracking module 7 are connected via a wired link to the processing circuitry 2 which drives the two sets of mechanical actuators 3.

In the example shown, the mechanical support 1 is placed on the head of the user 5, while the two sets of mechanical actuators 3 are located on the body of the user 5. The movement of the eyes of the user 5 is measured in real time by the eye tracking module 7. Further, by way of example here, a region in the acquired image corresponding to the current gaze direction of the user is selected. Alternatively, the cameras (not necessarily wide-angle) may be mounted on respective pivot axes with the cameras oriented in the user's current gaze direction. In yet another alternative, a single wide-angle camera may be provided, but with eye tracking for each eye, and two regions are selected in the environment of the shot, again providing a stereoscopic effect through partial overlap of the two regions.

In embodiments where the environment is captured by one or more "wide-angle" type cameras, the focal length of such cameras is selected so as to cover what the field of view of the user 5 would correspond in the acquired image (if the user is physically sound).

The movement of the eyes of the user 5 is measured while the wide-angle camera 6 photographs the environment. More precisely, as shown in fig. 6, the measurements carried out by the eye tracking module 7 are analyzed by the processing circuit 2 in order to select, on the basis of the measurements of the gaze direction D (for each eye), an appropriate pixel region Z of the current image of the environment E acquired only by the wide-angle camera 6.

The measurements carried out can be sent to the processing circuit 2 via a wireless link. In fact, the wireless link in this application allows the user 5 to be less constrained and to have greater freedom of movement, the processing circuitry 2 and the mechanical actuator 3 being located on the body (or generally the back) of the user 5.

The processing circuit 2 processes the received data and sends instructions to the mechanical actuators AM of the set 3.

The mechanical actuator 3, which is located on the chest, stimulates the skin of the user 5 according to commands received by the processing circuit 2. The mechanical actuator "reproduces" on the skin of the user the image improved after processing the data in the processing circuit 2.

In practice, the processing circuit 2 converts the pixel intensity (e.g. grey level) of each pixel into a mechanical intensity of the stimulus through the actuator. A particular vibration frequency may be associated with each color of the pixel (e.g., a low frequency for red and a high frequency for blue). For example, increased vibration frequencies may also be assigned to various colors from red to purple in the colors of the visible spectrum.

Fig. 2 shows a perspective view of a machine support 1 having a face 9 and two branches 8 held on the face 9, for example rigidly attached to it mechanically via an articulated link (not shown). Two eye tracking modules 7 are positioned towards the eyes, typically for taking the pupils and tracking their movement (typically by shape recognition implemented by a computer module in the images thus acquired), and from this determine the gaze direction of the user. On the other hand, the two wide-angle cameras 7 are located in the direction of the environment (along the physiological axis of the individual's eyes).

Fig. 3 shows a flow chart outlining possible steps of the method implemented by the processing circuit 2. The method may include acquiring a current image by the camera 6 at step S1. At step S2, eye tracking is performed in order to determine the current gaze direction of the user (based on measurements made by eye tracking by one or more eye tracking modules 7 of the gaze direction of the user 5). Thus correlating the field of view with the gaze orientation. At step S3, depending on this current gaze direction, a region Z corresponding to gaze direction D is selected in the acquired images of environment E (as described above with reference to fig. 6). More specifically, the signals from the pixels of this zone Z are selected to be converted at step S4 into control signals sp for the mechanical actuators AM (e.g. the group of actuators 3 shown in fig. 7). These may be piezoelectric elements or MEMS as described above, controlled by an electrical signal sp, in order to drive these elements when vibrating as follows:

at a vibration amplitude as a function of pixel intensity (e.g. grey level), an

At a vibration frequency determined by the color type (e.g. three frequency levels for blue, green or red).

These control signals are then sent to the mechanical actuator AM at step S5.

It goes without saying that these steps S1 to S5 are carried out for each eye (and therefore each current camera image) and are continuously performed in real time for continuously acquired images.

The two vibration images of the two mechanical actuators 3 on the skin are intended to be as close as possible to those images that the retina might capture if the individual is physically sound. This embodiment allows re-teaching the surrounding space to the individual 5.

Fig. 4 schematically outputs the apparatus comprising the wide angle camera 6 and the eye tracking module 7 and the processing circuitry 2, which here comprises:

an input interface IN for receiving signals from the devices 6 (pixels of the current image) and 7 (measurement of the gaze direction),

a memory MEM for storing, at least temporarily, data corresponding to these signals and instructions of a computer program in the sense of the present invention for implementing the method described above with reference to fig. 3,

a processor PROC able to cooperate with the input interface IN and the memory MEM to read and execute the instructions of the computer program and thus to deliver the control signals for the actuator AM via

An output interface OUT connected to the group of actuators 3.

Fig. 5 shows an apparatus worn on the head by a user 5 for improving the spatial perception of the environment. The device comprises a mechanical support 1 extending on either side of the head, and two mechanical actuators 3 located on the forehead of an individual 5. The machine support 1 comprises a face 9 and two branches 8 held on the face, as illustrated in fig. 2. In contrast to the arrangement illustrated in fig. 1, the processing circuit 2 is integrated into the mechanical support 1. The mechanical support 1 takes the form of a headset allowing the user 5 complete freedom. Here, a short wired link between the mechanical support 1 and the mechanical actuator 3 may be provided in this embodiment. This is because the device does not restrict the movement of the user 5 and the entire device is on the user's head.

It goes without saying that the invention is not limited to the exemplary embodiments described above, and that the invention can be extended to other variants.

Thus, for example, color-specific vibration frequencies have been described above. As a variant, for black and white images, the vibration frequency can be chosen to vary with the grey level of each pixel.

Previously, the distribution of the actuators in the group of actuators was defined according to a two-dimensional shape corresponding to the shape of the field of view assumed by the selected image area, in particular the number of actuators in each group corresponding to the number of pixels in the selected image area. However, as a variant, it is possible to gradually activate the group of actuators in increasing numbers (starting from the centre of the group of actuators up to the whole total actuators in the group) as the user becomes accustomed to the device. As another variation, again as the user becomes accustomed to the device, the vibration (or other stimulus) amplitudes of the N actuators may be averaged out of the set of N actuators, and then the perceived definition may be subsequently refined by reducing N. It goes without saying that, in order to accelerate this adaptation, it is also possible to correct (by computer) and thus reduce the initial ocular aberrations, for example to attenuate saccadic movements.