阅读说明：本技术 一种适用于视觉障碍人士的智能眼镜系统 (Intelligent glasses system suitable for people with visual impairment ) 是由 杨乾 黎少旭 邵洋 宋云龙 于 2020-12-31 设计创作，主要内容包括：本发明公开一种适用于视觉障碍人士的智能眼镜系统,包括设置在眼镜内部的中央处理单元,以及与中央处理单元连接的三射摄像头、激光雷达、语音播报装置；所述三射摄像头包括主摄像头、红外成像摄像头、景深灰度摄像头,所述中央处理单元通过景深灰度摄像头、红外成像摄像头,主摄像头采集数据结合进行三维立体建模；使用主摄像头和红外成像摄像头进行三维几何信息判断,并使用激光雷达测距,获得深度距离；所述中央处理单元将上述信息转化为语音,通过语音播报装置提示视觉障碍人士。本发明帮助视觉障碍人士解决生活中的问题,如文字阅读播报、前方物体位置播报、外出导航语音指引、避障提示等,以震动、语音的提示方式提醒视力障碍人士。(The invention discloses an intelligent glasses system suitable for people with visual impairment, which comprises a central processing unit arranged in glasses, and a three-shot camera, a laser radar and a voice broadcasting device which are connected with the central processing unit; the three-shot camera comprises a main camera, an infrared imaging camera and a depth-of-field gray camera, and the central processing unit performs three-dimensional modeling by combining data collected by the main camera through the depth-of-field gray camera and the infrared imaging camera; judging three-dimensional geometric information by using a main camera and an infrared imaging camera, and ranging by using a laser radar to obtain a depth distance; the central processing unit converts the information into voice and prompts the visually impaired people through the voice broadcasting device. The invention helps the visually impaired people to solve the problems in life, such as text reading and broadcasting, front object position broadcasting, outgoing navigation voice guidance, obstacle avoidance prompting and the like, and reminds the visually impaired people in a vibration and voice prompting mode.)

1. An intelligent glasses system suitable for people with visual impairment is characterized by comprising a central processing unit arranged in glasses, and a three-shot camera, a laser radar and a voice broadcasting device which are connected with the central processing unit; the three-shot camera comprises a main camera, an infrared imaging camera and a depth-of-field gray camera, and the central processing unit performs three-dimensional modeling by combining data collected by the main camera through the depth-of-field gray camera and the infrared imaging camera; the central processing unit judges three-dimensional geometric information by using the main camera and the infrared imaging camera, and obtains a depth distance by using a laser radar for ranging; the central processing unit converts the information into voice and prompts the visually impaired people through the voice broadcasting device.

2. The system of claim 1, wherein the three-dimensional stereo modeling comprises stereo imaging with a depth-of-field grayscale camera, creating stereo model data; filling the three-dimensional object in layers by using an infrared imaging camera, and reducing the three-dimensional appearance of the object; identifying a target for the object shape details by using a main camera; and calculating the actual state of the object by background operation on the data, and modeling.

3. The smart eyewear system of claim 1, wherein the determination of the three-dimensional geometric information using the main camera and the infrared imaging camera comprises: taking the main camera and the infrared imaging camera as binocular cameras; calibrating the binocular cameras to obtain internal and external parameters and homography matrixes of the two cameras; correcting the original image according to the calibration result, wherein the two corrected images are positioned on the same plane and are parallel to each other; matching pixel points of the two corrected images; and calculating the depth of each pixel according to the matching result, thereby obtaining a depth map.

4. The intelligent glasses system for the visually impaired according to claim 1 or 3, wherein the glasses are further provided with an infrared ranging camera, the infrared ranging camera is connected with the central processing unit, and the central processing unit judges the object distance and the dynamic and static states in a cross processing mode of the infrared ranging camera and a laser radar; and the infrared distance measurement camera processes the primary judgment of the temperature of the living body and assists in measuring the distance, namely the distance from an object to the camera is assisted and measured by the temperature of the infrared heat source, and then the laser radar is used for rapidly judging the secondary judgment and the distinguishing of the movement, thereby judging the physical type and the distance.

5. The system of claim 1, wherein the glasses further comprise a built-in Beidou/GPS positioning device, the Beidou/GPS positioning device is connected to the central processing unit and used for voice navigation for the visually impaired, and the center of the family or the disabled can know the position of the visually impaired through the APP at any time.

6. The system of claim 1, wherein the glasses further comprise a network module, the network module is one or more of 5G, bluetooth and WiFi, and the network module is connected to the central processing unit for indoor positioning and communication with the smart mobile terminal and the internet cloud platform.

7. The intelligent glasses system for the visually impaired according to claim 1, wherein the glasses further comprise a brain-computer interface, the brain-computer interface comprises a bone conduction motor and/or a nerve transmission electrode, the brain-computer interface is connected to the central processing unit for brain-computer communication, and the bone conduction motor further serves as a vibrator for the visually impaired in a noisy environment.

8. A smart eyewear system as claimed in claim 1, wherein the eyewear further comprises a headset connected to the cpu for controlling other devices by voice.

9. A smart eyewear system in accordance with claim 1, wherein the eyewear further comprises a button, the button is connected to the cpu for a one-touch SOS call.

10. The intelligent glasses system for the visually impaired according to claim 1, wherein the glasses further comprise a battery, and the battery is connected to the central processing unit for supplying power; the battery is also provided with a charging interface.

Technical Field

The invention belongs to the field of intelligent auxiliary equipment, and particularly relates to an intelligent glasses system suitable for people with visual impairment.

Background

The smart device that can use for the visually impaired personage in the existing market only has a small amount of intelligent bracelet and wrist-watch, tactile stick etc.. Most of the products are intelligent wearable products applied to common people, and deep concerns and requirements of people with visual impairment are rarely taken care of. The intelligent equipment most attached to the people with visual impairment belongs to intelligent glasses, but the intelligent identification problem of characters, images and object information needs to be solved, and the intelligent identification problem is converted into a prompt problem for the people with visual impairment.

Disclosure of Invention

In view of this, the invention provides an intelligent glasses system suitable for people with vision disorder, which realizes the function of converting a plurality of vision into voice and solves the problem of discomfort of a living environment system of people with vision disorder.

In order to achieve the purpose, the technical scheme of the invention is realized as follows:

an intelligent glasses system suitable for people with visual impairment comprises a central processing unit arranged in glasses, and a three-shot camera, a laser radar and a voice broadcasting device which are connected with the central processing unit; the three-shot camera comprises a main camera, an infrared imaging camera and a depth-of-field gray camera, and the central processing unit performs three-dimensional modeling by combining data collected by the main camera through the depth-of-field gray camera and the infrared imaging camera; the central processing unit judges three-dimensional geometric information by using the main camera and the infrared imaging camera, and obtains a depth distance by using a laser radar for ranging; the central processing unit converts the information into voice and prompts the visually impaired people through the voice broadcasting device.

Further, the three-dimensional modeling comprises the steps of carrying out three-dimensional imaging by using a depth-of-field gray camera, and establishing three-dimensional model data; filling the three-dimensional object in layers by using an infrared imaging camera, and reducing the three-dimensional appearance of the object; identifying a target for the object shape details by using a main camera; and calculating the actual state of the object by background operation on the data, and modeling.

Further, the using the main camera and the infrared imaging camera to judge the three-dimensional geometric information comprises: taking the main camera and the infrared imaging camera as binocular cameras; calibrating the binocular cameras to obtain internal and external parameters and homography matrixes of the two cameras; correcting the original image according to the calibration result, wherein the two corrected images are positioned on the same plane and are parallel to each other; matching pixel points of the two corrected images; and calculating the depth of each pixel according to the matching result, thereby obtaining a depth map.

Furthermore, the glasses are also provided with an infrared ranging camera which is connected with the central processing unit, and the central processing unit judges the distance of an object and the dynamic and static states in a mode of cross processing of the infrared ranging camera and a laser radar; and the infrared distance measurement camera processes the primary judgment of the temperature of the living body and assists in measuring the distance, namely the distance from an object to the camera is assisted and measured by the temperature of the infrared heat source, and then the laser radar is used for rapidly judging the secondary judgment and the distinguishing of the movement, thereby judging the physical type and the distance.

Further, glasses still are equipped with built-in big dipper GPS positioner, big dipper GPS positioner connects central processing unit for the visually impaired personage carries out voice navigation, and makes its family or disabled person center learn visually impaired personage's position through APP at any time.

Further, glasses still are equipped with the network module, the network module is one or more of 5G, bluetooth, wiFi, the network module is connected central processing unit for indoor location to and communicate with intelligent Mobile terminal, thing networking cloud platform.

Further, glasses still are equipped with brain machine interface, brain machine interface includes bone conduction motor and/or nerve transmission electrode, brain machine interface connection central processing unit carries out brain machine communication, the bone conduction motor still is as the electromagnetic shaker for go on for the visually impaired personage in noisy environment.

Furthermore, the glasses are also provided with a headset which is connected with the central processing unit and used for controlling other equipment through voice.

Furthermore, the glasses are also provided with keys, and the keys are connected with the central processing unit and used for one-key SOS calling.

Furthermore, the glasses are also provided with a battery, and the battery is connected with the central processing unit and used for supplying power; the battery is also provided with a charging interface.

Compared with the prior art, the invention has the following advantages:

(1) the invention is suitable for the blind people with disabled vision, the elderly with visual disturbance and the people with visual disturbance caused by other reasons. The visual impairment prompting system can help people with visual impairment to solve problems in life, such as text reading and broadcasting, front object position broadcasting, outgoing navigation voice guidance, shop merchant introduction, obstacle avoidance prompting, AI face recognition, and reminding people with visual impairment in a vibration and voice prompting mode;

(2) the three-shot camera device is adopted, the combination of the double cameras and the infrared camera is adopted, the three-dimensional environment 3D modeling processing is carried out by the depth-of-field imaging capability, the three-dimensional geometric information judgment is carried out by the binocular camera, the physical type and the distance are judged by the infrared distance measuring camera and the laser radar, and an accurate data base is provided for the AI intelligent voice broadcast in the later period;

(3) the glasses are worn, the internal battery can provide sufficient electric energy for the product, and the wearable charger can be connected to the outside for power supply continuation, so that the glasses are suitable for being carried at home and outdoors;

(4) the Bluetooth mobile phone and the intelligent equipment are networked through Bluetooth, other equipment can be controlled through voice, the mobile phone is connected with the voice to control dialing, an indoor and outdoor alternative working mode is adopted, the dual requirements of home and outdoor positioning are met, indoor positioning is used in a family, and the Bluetooth mobile phone has the characteristics of low power consumption, stability, independence and privacy; when the user goes out, the user switches to the outdoor (Beidou and GPS positioning service), the APP and the platform can monitor the position information of the wearer at any time and respond to the SOS emergency call requirement of the wearer. The system is convenient for the contact between the family and the outers and the remote processing of family events.

Drawings

FIG. 1 is a schematic view of a spectacle construction according to an embodiment of the present invention;

fig. 2 is a schematic diagram of an embodiment of the present invention.

Wherein:

1. a spectacle frame; 2. a lens; 3. a front frame; 4. a main camera; 5. an infrared imaging camera; 6. a depth-of-field grayscale camera; 7. a laser radar; 8. an infrared ranging camera; 9. a left temple inner shell; 10. a left temple housing; 11. a right temple inner shell; 12. a right temple housing; 13. a main PCB; 14. an auxiliary PCB; 15. 5G/WIFI/Bluetooth antenna; 16. a Beidou/GPS antenna; 17. a nerve transmission electrode; 18. a bone conduction motor; 19. a data interface; 20. a battery; 21. pressing a key; 22. a headset; 23. expanding the power supply; 24. and expanding the host.

Detailed Description

It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.

The invention is further described with reference to the following figures and specific embodiments.

As shown in figure 1, the invention adopts the form of glasses, a front frame 3 is arranged in front of an original glasses frame 1 and lenses 2 of the glasses, a left glasses leg and a right glasses leg are made into a shell form, the left glasses leg consists of a left glasses leg inner shell 9 and a left glasses leg outer shell 10, the right glasses leg consists of a right glasses leg inner shell 11 and a right glasses leg outer shell 12, and the front frame 3, the left glasses leg and the right glasses leg are connected with a circuit through a channel. The central position has set up main camera 4 on frame 3 in the front, and main camera 4 one side has set up infrared imaging camera 5 and depth of field grey level camera 6, and the opposite side has set up infrared range camera 8 and laser radar 7.

A main PCB13 is arranged in a left earpiece shell formed by the left earpiece inner shell 9 and the left earpiece outer shell 10, a central processing unit is arranged on the main PCB13, and the main camera 4, the infrared imaging camera 5, the depth-of-field grayscale camera 6, the infrared distance measuring camera 8 and the laser radar 7 are connected with the central processing unit; the left earpiece shell is also internally provided with a Beidou/GPS antenna 16, and the shell is provided with a data interface 19 and a nerve transmission electrode 17; the Beidou/GPS antenna 16, the data interface 19 and the nerve transmission electrode 17 are connected with the central processing unit.

An auxiliary PCB14, a 5G/WIFI/Bluetooth antenna 15 and a battery 20 are arranged in a right earpiece shell formed by the right earpiece inner shell 11 and the right earpiece outer shell 12; the shell is provided with a nerve transmission electrode 17, a bone conduction motor 18, a key 21 and an earphone 22; the 5G/WIFI/Bluetooth antenna 15, the battery 20, the nerve transmission electrode 17, the bone conduction motor 18, the keys 21 and the headset 22 are all connected with the central processing unit through an auxiliary PCB 14.

The battery 20 is provided with a charging interface for connecting an expansion power supply 23 (such as a charger).

The invention can also be connected with an expansion host through a 5G/WIFI/Bluetooth antenna 15, and the expansion host can be a wearable device hung on the waist.

As shown in fig. 2, which is a schematic block diagram of the present invention, the central processing unit of the present invention performs three-dimensional modeling by combining data collected by the depth-of-field grayscale camera 6, the infrared imaging camera 5, and the main camera 4; the central processing unit judges three-dimensional geometric information by using the main camera 4 and the infrared imaging camera 5, measures distance by using the laser radar 7 and the infrared distance measuring camera 8, obtains depth distance, and judges physical type and distance; the central processing unit converts the information into voice and prompts the visually impaired through the headset 22.

The central processing unit is supplied power by battery 20, obtain and send locating information for central processing unit by big dipper GPS antenna, central processing unit (A06) is the image of gathering, the distance data is handled and is given the instruction, do voice prompt through headset 22 and report, can also drive the electromagnetic shaker (bone conduction motor 18 can be as the electromagnetic shaker) and indicate in noisy environment, 5G WIFI bluetooth antenna can communicate with intelligent mobile terminal (such as cell-phone etc.), then carry out data exchange and instruction with central processing unit and receive. Meanwhile, the pushing prompt information of the cloud platform of the Internet of things can be transmitted to the central processing unit through the mobile phone, and can be inquired and positioned and an instruction can be played.

The invention can realize brain-computer communication, the central processing unit can control the nerve transmission electrode 17 and the bone conduction motor 18 to send signals to the brain, the electrode can also be implanted in the brain, the 5G/WIFI/Bluetooth antenna is used for communicating with the implanted electrode, and the real object scene is displayed in the brain.

The key 21 can be used as an SOS emergency help key, and after the key 21 is pressed, the central processing unit can communicate with the intelligent mobile terminal through the 5G/WIFI/Bluetooth antenna to push prompt information and position information to a cloud platform or is controlled by the central processing unit to dial and communicate with related contact telephones.

The software algorithm involved in the present invention is explained as follows.

Firstly, modeling:

the principle is that a general outline of a scene in front is described through Convolutional Neural Network (CNN) model processing (through a natural feature description mechanism of the CNN model, image processing is preprocessed, and image processing and visual preprocessing are combined into a whole) (extraction of edges and outlines needs to avoid that edges and outlines of objects are covered by too strong image lines, and textures are weak edge division modes, so that processing is complex) a hierarchical (hierarchical) method is adopted, each edge pixel is associated with more than one edge class, the method is an end-to-end edge learning framework based on a residual error network (ResNet), and is a cross-layer (skip-layer) framework, wherein class edge activation sharing of the top convolutional layer is employed and fused with the same set of bottom features, where a multi-label loss function is used to supervise the fusion of activations.

The basic network adopts a full convolution network framework, a classification module is added at the top of the basic network to serve as a1 x 1 convolution layer, then bilinear upsampling (realized by a K grouping deconvolution layer) is carried out to generate a set of K activation graphs { A1, ·, AK }, wherein each activation graph has the same size as an image; then, the sigmoid cell defined by yk (p) ═ σ (ak (p)) models the probability that the pixel belongs to the k-th class edge.

The algorithm adopts a nested architecture (nested architecture):

1) and replacing the bottom classification module with a feature extraction module.

2) The classification module is placed to apply supervision only on top of the network.

3) Common connections are used instead of slice connections.

4) And the addition of the attribute coefficient enables the calculation result to be more beneficial to algorithm identification.

The method specifically comprises the following steps: the three-dimensional modeling is carried out in a mode of combining the depth-of-field gray camera 6, the infrared imaging camera 5 and the main camera 4; the method comprises the steps of firstly carrying out three-dimensional imaging by using depth of field gray, establishing three-dimensional model data, then carrying out hierarchical filling on a three-dimensional object by using infrared imaging, further reducing the three-dimensional appearance of the object, and finally identifying the appearance details of the object by using a main camera in combination with the state of achieving an identification target. And calculating the actual state of the object by background operation on the data, and restoring the current physical environment by three-dimensional modeling.

II, judging the depth distance:

and judging the three-dimensional geometric information by using the main camera and the infrared imaging camera.

The depth distance judgment adopts a binocular stereo vision principle and is based on a parallax principle, and a method for acquiring three-dimensional geometric information of an object by a plurality of images is adopted. In a machine vision system, in binocular vision, two digital images of surrounding scenery are acquired simultaneously from different angles by two cameras, or two digital images of the surrounding scenery are acquired by a single camera from different angles at different moments, and three-dimensional geometric information of an object can be recovered based on a parallax principle to reconstruct the three-dimensional shape and position of the surrounding scenery. The binocular stereo vision is based on parallax, and three-dimensional information is acquired by a trigonometry principle, namely a triangle is formed between the image planes of two cameras and a north object. The three-dimensional size of the object in the common field of view of the two cameras and the three-dimensional coordinates of the characteristic points of the space object can be obtained by keeping the position relationship between the two cameras.

The invention uses a main camera and an infrared imaging camera as binocular cameras, and the binocular stereoscopic vision flow is as follows:

1. firstly, calibrating a binocular camera to obtain internal and external parameters and a homography matrix of the two cameras;

2. correcting the original image according to the calibration result, wherein the two corrected images are positioned on the same plane and are parallel to each other;

3. matching pixel points of the two corrected images;

4. and calculating the depth of each pixel according to the matching result, thereby obtaining a depth map.

The method adopts a mode of cross processing of the infrared distance measurement camera and the laser radar camera to judge the distance and the dynamic and static states of the object; and the infrared distance measurement camera processes the primary judgment of the temperature of the living body and assists in measuring the distance (the distance from an object to the camera is assisted by the temperature of the infrared heat source), and then the laser radar is used for quickly judging the secondary judgment and the distinguishing of the moving movement. The purpose of judging the physical type and the distance is achieved.

The information data obtained by the software algorithm and the process provide an accurate data base for the AI intelligent voice broadcast.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.