阅读说明：本技术 一种基于集成多传感器手写式的投射数据采集系统及方法 (Projection data acquisition system and method based on integrated multi-sensor handwriting mode ) 是由 侯艳红 张�林 孙岳 于 2021-08-11 设计创作，主要内容包括：本发明涉及一种基于集成多传感器手写式的投射数据采集系统及方法,系统包括电子板以及电子笔；用户手持集成多传感器的电子笔在电子板上书写或者绘画,通过电子笔上的传感器采集用户在书写或绘画过程中的生理特征信息,同时通过电子板实时采集用户书写和绘画的动态过程,对数据进行存储并分析,识别画面内容,并进一步提取画面笔迹特征,结合对应的用户生理特征信息,进行综合处理分析,从而能够对用户整个测试过程的数据进行完整分析,有助于得到客观和准确的测试结果。(The invention relates to a projection data acquisition system and a method based on an integrated multi-sensor handwriting mode, wherein the system comprises an electronic board and an electronic pen; the electronic pen integrating the multiple sensors is held by a user to write or draw on the electronic board, the physiological characteristic information of the user in the writing or drawing process is collected through the sensors on the electronic pen, meanwhile, the dynamic process of the user in writing and drawing is collected through the electronic board in real time, data is stored and analyzed, picture content is identified, picture handwriting characteristics are further extracted, comprehensive processing analysis is carried out by combining the corresponding physiological characteristic information of the user, therefore, the data of the whole test process of the user can be completely analyzed, and objective and accurate test results are obtained.)

1. A system for projection data acquisition based on integrated multi-sensor handwriting, comprising:

the electronic board comprises a touch screen, an outer frame, a support and a signal transmission interface; a pressure sensor is arranged below the touch screen; the frame of the electronic board is also provided with a plurality of keys and contact indicator lamps, wherein the keys comprise a start key, an end key, a playback key, a data storage key and a data export key; the touch indicator lamp is used for lighting when the electronic pen is effectively contacted with the touch screen of the electronic board, indicating that the contact is effective and the data recording is normal, and indicating that the contact is invalid when the touch indicator lamp is not on;

a color selection area, a stroke width selection area and a back and erase button are arranged on a touch screen display area of the electronic board;

the electronic board also comprises a content recognition module, a handwriting characteristic analysis module and a fusion analysis module; analyzing and processing the handwriting, and extracting multi-dimensional characteristics of the handwriting; wherein the content of the first and second substances,

the content recognition module is used for carrying out overall analysis on the writing and drawing contents through a machine vision algorithm to realize a main body target recognition function;

the handwriting characteristic analysis module is used for extracting and analyzing the drawing handwriting characteristics;

and the fusion analysis module is used for fusing the multi-dimensional characteristics of the handwriting on the basis of completing the content identification to obtain an analysis result.

2. The integrated multi-sensor handwriting-based projection data acquisition system of claim 1, wherein said start key when pressed triggers the touch screen to start writing; triggering the touch screen to end recording the handwriting when the end key is pressed; the recorded handwriting is played back when the playback key is pressed; and when the data export key is pressed, exporting handwriting to an external device.

3. The integrated multi-sensor handwriting-based projection data acquisition system of claim 1, wherein said color selection area is used for user selection of a desired color to be used during testing; the stroke width selection area is used for selecting the stroke width which is expected to be used by a user in the test; the back button is used for executing withdrawing operation, the erasing button is used for erasing the track of a part of the area, and when the user is not satisfied with the track of the picture, the erasing function is used for erasing the content of the area.

4. The system for acquiring projection data based on the handwriting of the integrated multi-sensor as claimed in claim 1, wherein the electronic board is configured to accept writing and drawing by an electronic pen, capture and record the handwriting of the electronic pen through a touch screen on the electronic board, and record the occurrence time of each trace point in the handwriting, and the corresponding force and color information, so as to obtain a trace point P (x, y, t, color, width, and pressure), where x and y are horizontal and vertical coordinates, t is the current recording time, color is color, width is stroke width, and pressure is the pressure of the electronic pen; a handwriting formed by a series of points is L ═ { P1, P2, Pi … … Pn }, and n is the number of points on the handwriting;

a plurality of handwriting L form a whole picture A, wherein A is { L1, L2 and Li … … LN }, and N is the number of the handwriting;

the electronic board internally comprises a timer for recording the time t of each point in the handwriting.

5. The integrated multi-sensor handwriting-based projection data acquisition system of claim 1, wherein said electronic board further comprises a data interface for connecting to a data line of the electronic pen to receive data from the electronic pen; or the electronic board is provided with a wireless receiving and transmitting module which performs data transmission with the electronic pen through wireless signals;

the electronic board can also store the handwriting data, and the stored data can be read by the processor at any time and is played back, analyzed and processed;

the electronic pen integrated with the multiple sensors comprises an electronic pen body and a holding part arranged on the electronic pen body, wherein the holding part is provided with multiple sensors including a pressure sensor, a heart rate sensor, a blood oxygen saturation sensor, a skin temperature sensor and a skin temperature sensor; the tail end of the electronic pen is provided with a data transmission line, and the data transmission line is connected to the electronic board through an interface; or, the electronic pen is provided with a wireless transceiver module, and the sensor data is transmitted through the wireless transceiver module.

6. A method for handwritten projection data acquisition using the system of any of claims 1-5, comprising:

step 1, connecting an electronic pen with an electronic board in a wired or wireless mode, starting a power switch of a system, and operating the system;

step 2, the user holds the electronic pen, the thumb is contacted with the blood oxygen sensor, the skin electricity sensor and the skin temperature sensor, and the index finger is connected with the heart rate sensing period pressure sensor; clicking a start key, starting drawing on an electronic board by a user by using an electronic pen with a multi-sensor fusion, recording the drawing handwriting in real time by the electronic board, and recording the editing operation of the user on a picture pen, including withdrawing and erasing operations, and measuring physiological characteristic data of the user by the electronic pen; clicking an end button after drawing is finished;

step 3, when the electronic pen is detected to be in contact with the electronic board, the contact indicator lamp is turned on, the pressure sensor on the electronic pen acquires blood pressure data of the user in real time, the blood oxygen sensor acquires blood oxygen concentration parameters of the user, the heart rate sensor acquires the heart rate of the user, and the physiological data of the user are transmitted to the electronic board;

step 4, the electronic board detects the handwriting of the electronic pen on the touch screen in real time, and the color parameters selected by the electronic pen, the pressure parameters of the electronic pen pressing the touch pad, the position parameters of the handwriting points and the current time parameters are obtained;

step 5, identifying the content of the picture, and performing overall analysis on the written and drawn content through a machine vision algorithm to realize a main body target identification function;

step 6, fusing multi-dimensional characteristics of handwriting on the basis of completing content identification, and outputting collected and analyzed data; the multi-dimensional characteristics of the handwriting mainly comprise handwriting characteristics and physiological characteristics.

7. The projection data acquisition method according to claim 6, wherein in the step 5, the content of the picture is identified, and the written picture content is totally analyzed through a machine vision algorithm to realize a main body target identification function; the method comprises the following specific steps:

step 5.1, performing area division on the image on the electronic board, dividing the image into K × K square areas, sequentially numbering all the square areas from 0, and totally having K × K areas;

step 5.2, constructing an area adjacency matrix, wherein the area adjacency matrix is used for representing the adjacency relation between square areas, continuous handwriting can be used when a user draws the same main body content, and the handwriting jumps when drawing different main bodies; the element of the ith row and the jth column of the area adjacency matrix represents the total times of simultaneously traversing the ith and jth image areas by continuous handwriting; i. j is the sequence number of the region;

step 5.3, merging the image and the adjacent matrix, and firstly, according to the affiliated relationship of the region and the pixel, enabling the dimension of the region adjacent matrix to be K²×K²Expansion to WXHXK². Then, the expanded region adjacency matrix is merged with the image data to obtain W × H × (C + K)²) C represents the number of channels, W is the image width, and H is the image height, and the data is used as the input of a Darknet deep learning prediction model;

step 5.4, extracting image features by using the trained Darknet deep learning prediction model, wherein the Darknet comprises a convolutional layer, a residual error layer and a pooling layer, and extracting the depth features from the image data;

and 5.5, outputting a prediction result, and identifying the target main body in each area according to the image characteristics output by the Darknet by each area, and finally outputting the target type and the target size.

8. The projection data acquisition method according to claim 6, wherein in the step 6, on the basis of completing content recognition, multi-dimensional features of handwriting are fused, and acquired and analyzed data are output; the method specifically comprises the following steps:

step 6.1, extracting all handwriting characteristics in the content main body area;

step 6.2, extracting corresponding user physiological characteristic information according to the handwriting generation and end time;

6.3, calculating the average retention time, the modification times and the jumping times of the handwriting of the user in each main body area and the corresponding physiological characteristic change condition;

and 6, outputting the acquisition and analysis results.

9. The projection data acquisition method as claimed in claim 6, wherein a handwriting feature analysis module is used for extracting and analyzing the painting handwriting features; the handwriting characteristics comprise handwriting appearance characteristics, handwriting time stage characteristics, handwriting space stage characteristics and handwriting modification characteristics; specifically, the method comprises the following steps:

the handwriting appearance characteristics comprise: thickness, regularity, size, lightness and color of handwriting; the thickness of the handwriting is obtained by selecting on a panel; the regularity refers to the degree of smooth handwriting or flow field, and can be obtained by calculating the mean square error of a fitting curve, for example; the handwriting size refers to the area size defined by the maximum width and the maximum height of the handwriting; the weight of the handwriting is obtained through a pressure sensor on the electronic pen;

the handwriting time stage characteristics comprise: stroke point sequence, sequence among the strokes, pause time, speed characteristic, stroke jumping and stroke fluency; the handwriting point sequence comprises the sequence of each point in the handwriting, and the time information is obtained on the handwriting points, so that the sequence of the points on the handwriting can be obtained through the time information; the sequence and the pause time among different handwriting can be obtained by comparing the time of the upper point of the different handwriting; calculating the speed characteristic by dividing the length of the handwriting by the time difference between the first point and the last point of the handwriting; the handwriting jumping refers to the distance between the tail point of the previous handwriting and the starting point of the next handwriting among different handwriting;

the handwriting space characteristics include: handwriting area distribution, breakpoint proportion, breakpoint number, area density and handwriting continuity;

according to the embodiment of the invention, the handwriting distribution area characteristics can be expressed by a matrix, and the distribution quantity of handwriting points in each area is used as the corresponding position element of the matrix; the number of the breakpoints is the stroke points with the length smaller than the preset length, and compared with the total handwriting number, the breakpoint proportion is obtained; the handwriting continuity refers to the ratio of the length of the handwriting to the average length of the handwriting; areal density refers to the density of points per area;

the handwriting modification characteristics comprise correction adjustment, repetition and deletion, and represent the modification processing process of a user; a back and erase button is displayed in a screen area of the electronic board, when a user clicks the back or erase button in the drawing process, the system records the operation and modifies the corresponding handwriting to obtain a modified record edge (mode, L, t), wherein the mode represents back or erase; l represents the corresponding handwriting, and t represents the current time.

10. The projection data acquisition method of claim 6, wherein the physiological characteristic is heart rate, pen-holding pressure, blood oxygen saturation, electrodermal, and electrodermal of the user at the time of drawing.

Technical Field

The invention relates to the field of biological information acquisition terminals, in particular to a system and a method for acquiring projection data based on an integrated multi-sensor handwriting mode.

Background

Projection testing is an important testing technique in the field of psychology, in which a drawing analysis technique in projection testing requires a subject to draw according to a certain task, and an evaluator evaluates the contents or form characteristics of the work, such as house-tree-person testing. The evaluator analyzes the painting of the tested person to conjecture the psychological characteristics of the tested person or diagnose the psychological disorder. The traditional house tree person test is generally completed by using a paper pen test method, and some defects and inconveniences exist in the practical use. Therefore, there is also a technology that proposes an electronic tree man technique, which displays the traditional tree man test on a computer in the form of software by using a processing mode of a multimedia technology, and then analyzes the characteristics of the picture by the computer to achieve the purpose of measurement. However, in the above techniques, only the picture completed by the user is analyzed, and the collection and analysis of the current physiological characteristics of the user drawing and the analysis of the dynamic parameters in the user drawing process are lacked, so that the data is very single, the subjectivity is strong, the influence of the experience of the evaluating personnel is large, and the conclusion is sometimes not objective.

Disclosure of Invention

In order to solve the technical problems, the invention provides a projection data acquisition system and method based on an integrated multi-sensor handwriting mode, an electronic board and an electronic pen; the user holds the electronic pen to write or draw on the electronic board, the physiological characteristic information of the user in the writing or drawing process is collected through the sensor on the electronic pen, meanwhile, the dynamic process of the user in writing and drawing is collected in real time through the electronic board, data is stored and analyzed, picture content is identified, picture handwriting characteristics are further extracted, comprehensive processing and analysis are carried out by combining the corresponding physiological characteristic information of the user, therefore, the data of the whole test process of the user can be completely analyzed, and objective and accurate test results are obtained.

The technical scheme of the invention is as follows: an integrated multi-sensor handwriting based projection data acquisition system comprising:

the electronic board comprises a touch screen, an outer frame, a support and a signal transmission interface; a pressure sensor is arranged below the touch screen; the frame of the electronic board is also provided with a plurality of keys and contact indicator lamps, wherein the keys comprise a start key, an end key, a playback key, a data storage key and a data export key; the contact indicator light is used for lighting when the electronic pen is effectively contacted with the touch screen of the electronic board, indicating that the contact is effective and the data recording is normal, and indicating that the contact is invalid when the contact indicator light is not on, possibly indicating that the contact is poor or the operation of a user is improper;

a color selection area, a stroke width selection area and a back and erase button are arranged on a touch screen display area of the electronic board;

the electronic board further comprises a processor and a memory, wherein the processor is used for analyzing and processing the handwriting and extracting the characteristics of the handwriting, and the electronic board specifically comprises the following components:

the content recognition module is used for carrying out overall analysis on the writing and drawing contents through a machine vision algorithm to realize a main body target recognition function;

the handwriting characteristic analysis module is used for extracting and analyzing the drawing handwriting characteristics;

and the fusion analysis module is used for fusing the multi-dimensional characteristics of the handwriting on the basis of completing the content identification to obtain an analysis result.

Further, when the start key is pressed down, the touch screen is triggered to start recording handwriting; triggering the touch screen to end recording the handwriting when the end key is pressed; the recorded handwriting is played back when the playback key is pressed; and when the data export key is pressed, exporting handwriting to an external device.

Further, the color selection area is used for selecting a color which is expected to be used by a user in the test; the stroke width selection area is used for selecting the stroke width which is expected to be used by a user in the test; the back button is used for executing withdrawing operation, the erasing button is used for erasing the track of a part of the area, and when the user is not satisfied with the track of the picture, the erasing function is used for erasing the content of the area.

Further, the electronic board is used for receiving writing and drawing of the electronic pen, capturing and recording handwriting of the electronic pen through a touch screen on the electronic board, recording occurrence time of each track point in the handwriting, corresponding strength and color information, and obtaining a recording pen track point P (x, y, t, color, width and pressure), wherein x and y are horizontal coordinates and vertical coordinates, t is current recording time, color is color, width is stroke width, and pressure is pressure of the electronic pen; a handwriting formed by a series of points is L ═ { P1, P2, Pi … … Pn }, and n is the number of points on the handwriting;

a plurality of handwriting L form a whole picture A, wherein A is { L1, L2 and Li … … LN }, and N is the number of the handwriting;

the electronic board internally comprises a timer for recording the time t of each point in the handwriting.

Furthermore, the electronic board further comprises a data interface for connecting with a data line of the electronic pen so as to receive data of the electronic pen; or the electronic board is provided with a wireless receiving and transmitting module which performs data transmission with the electronic pen through wireless signals;

the electronic board can also store the handwriting data, and the stored data can be read by the processor at any time and can be played back, analyzed and processed;

the electronic pen comprises an electronic pen body and a holding part arranged on the electronic pen body, wherein a plurality of sensors including a pressure sensor, a heart rate sensor, a blood oxygen saturation sensor, a skin temperature sensor and a skin temperature sensor are arranged at the holding part; the tail end of the electronic pen is provided with a data transmission line, and the data transmission line is connected to the electronic board through an interface; or, the electronic pen is provided with a wireless transceiver module, and the sensor data is transmitted through the wireless transceiver module.

According to another aspect of the present invention, a method for performing handwritten projection data acquisition by using the aforementioned system is provided, which includes the following steps:

step 1, connecting an electronic pen with an electronic board in a wired or wireless mode, starting a power switch of a system, and operating the system;

step 2, the user holds the electronic pen, the thumb is contacted with the blood oxygen sensor, the skin electricity sensor and the skin temperature sensor, and the index finger is connected with the heart rate sensing period pressure sensor; clicking a start key, starting drawing on an electronic board by a user by using an electronic pen with a multi-sensor fusion, recording the drawing handwriting in real time by the electronic board, and recording the editing operation of the user on a picture pen, including withdrawing and erasing operations, and measuring physiological characteristic data of the user by the electronic pen; clicking an end button after drawing is finished;

step 3, when the electronic pen is detected to be in contact with the electronic board, the contact indicator lamp is turned on, the pressure sensor on the electronic pen acquires blood pressure data of the user in real time, the blood oxygen sensor acquires blood oxygen concentration parameters of the user, the heart rate sensor acquires the heart rate of the user, and the physiological data of the user are transmitted to the electronic board;

step 4, the electronic board detects the handwriting of the electronic pen on the touch screen in real time, and the color parameters selected by the electronic pen, the pressure parameters of the electronic pen pressing the touch pad, the position parameters of the handwriting points and the current time parameters are obtained;

step 5, identifying the content of the picture, and performing overall analysis on the written and drawn content through a machine vision algorithm to realize a main body target identification function;

step 6, fusing multi-dimensional characteristics of handwriting on the basis of completing content identification, and outputting collected and analyzed data; the multi-dimensional characteristics of the handwriting mainly comprise handwriting characteristics and physiological characteristics.

Further, in the step 5, content recognition is carried out on the picture, and the writing and drawing content is subjected to overall analysis through a machine vision algorithm to realize a main body target recognition function; the method comprises the following specific steps:

step 5.1, performing area division on the image on the electronic board, dividing the image into K × K square areas, sequentially numbering all the square areas from 0, and totally having K × K areas;

step 5.2, constructing an area adjacency matrix, wherein the area adjacency matrix is used for representing the adjacency relation between square areas, continuous handwriting can be used when a user draws the same main body content, and the handwriting jumps when drawing different main bodies; the element of the ith row and the jth column of the area adjacency matrix represents the total times of simultaneously traversing the ith and jth image areas by continuous handwriting; i. j is the sequence number of the region;

step 5.3, merging the image and the adjacent matrix, and firstly, according to the affiliated relationship of the region and the pixel, enabling the dimension of the region adjacent matrix to be K²×K²Expansion to WXHXK². Then, the expanded region adjacency matrix is merged with the image data to obtain W × H × (C + K)²) C represents the number of channels, W is the image width, and H is the image height, and the data is used as the input of a Darknet deep learning prediction model;

step 5.4, extracting image features by using the trained Darknet deep learning prediction model, wherein the Darknet comprises a convolutional layer, a residual error layer and a pooling layer, and extracting the depth features from the image data;

and 5.5, outputting a prediction result, and identifying the target main body in each area according to the image characteristics output by the Darknet by each area, and finally outputting the target type and the target size.

Further, step 6, on the basis of completing content identification, fusing multi-dimensional characteristics of handwriting and outputting collected and analyzed data; the method specifically comprises the following steps:

step 6.1, extracting all handwriting characteristics in the content main body area;

step 6.2, extracting corresponding user physiological characteristic information according to the handwriting generation and end time;

6.3, calculating the average retention time, the modification times and the jumping times of the handwriting of the user in each main body area and the corresponding physiological characteristic change condition;

and 6, outputting the acquisition and analysis results.

Further, extracting and analyzing the characteristics of the painting handwriting by using a handwriting characteristic analysis module; the handwriting characteristics comprise handwriting appearance characteristics, handwriting time stage characteristics, handwriting space stage characteristics and handwriting modification characteristics; specifically, the method comprises the following steps:

the handwriting appearance characteristics comprise: thickness, regularity, size, lightness and color of handwriting; the thickness of the handwriting is obtained by selecting on a panel; the regularity refers to the degree of smooth handwriting or flow field, and can be obtained by calculating the mean square error of a fitting curve, for example; the handwriting size refers to the area size defined by the maximum width and the maximum height of the handwriting; the weight of the handwriting is obtained through a pressure sensor on the electronic pen;

the handwriting time stage characteristics comprise: stroke point sequence, sequence among the strokes, pause time, speed characteristic, stroke jumping and stroke fluency; the handwriting point sequence comprises the sequence of each point in the handwriting, and the time information is obtained on the handwriting points, so that the sequence of the points on the handwriting can be obtained through the time information; the sequence and the pause time among different handwriting can be obtained by comparing the time of the upper point of the different handwriting; calculating the speed characteristic by dividing the length of the handwriting by the time difference between the first point and the last point of the handwriting; the handwriting jumping refers to the distance between the tail point of the previous handwriting and the starting point of the next handwriting among different handwriting;

the handwriting space characteristics include: handwriting area distribution, breakpoint proportion, breakpoint number, area density and handwriting continuity;

according to the embodiment of the invention, the handwriting distribution area characteristics can be expressed by a matrix, and the distribution quantity of handwriting points in each area is used as the corresponding position element of the matrix; the number of the breakpoints is the stroke points with the length smaller than the preset length, and compared with the total handwriting number, the breakpoint proportion is obtained; the handwriting continuity refers to the ratio of the length of the handwriting to the average length of the handwriting; areal density refers to the density of points per area;

the handwriting modification characteristics comprise correction adjustment, repetition and deletion, and represent the modification processing process of a user; a back and erase button is displayed in a screen area of the electronic board, when a user clicks the back or erase button in the drawing process, the system records the operation and modifies the corresponding handwriting to obtain a modified record edge (mode, L, t), wherein the mode represents back or erase; l represents the corresponding handwriting, and t represents the current time;

further, the physiological characteristics are heart rate, pen holding pressure, blood oxygen saturation, skin electricity and skin temperature of the user during drawing.

Has the advantages that:

according to the system and the method for acquiring the projection data based on the integrated multi-sensor handwriting mode, disclosed by the invention, the physiological characteristic information of a user in the writing or drawing process is acquired through various sensors on the electronic pen, meanwhile, the dynamic process of the writing and drawing of the user is acquired in real time through the electronic board, the data is stored and analyzed, the picture content is identified, the picture handwriting characteristics are further extracted, and the comprehensive processing and analysis are carried out in combination with the corresponding physiological characteristic information of the user, so that the data of the whole testing process of the user can be completely analyzed, and objective and accurate testing results are favorably obtained.

Drawings

FIG. 1 is a schematic diagram of an electronic board and electronic pen of an integrated multi-sensor handwriting-based projection data acquisition system of the present invention;

FIG. 2 is a schematic structural diagram of an electronic pen according to the present invention;

fig. 3 is a flow chart of a method for acquiring projection data based on an integrated multi-sensor handwriting form according to the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments, and all other embodiments obtained by a person skilled in the art based on the embodiments of the present invention belong to the protection scope of the present invention without creative efforts.

According to an embodiment of the present invention, a system for acquiring projection data based on an integrated multi-sensor handwriting mode, as shown in fig. 1, includes an electronic board 1 and an electronic pen 2 integrated with a multi-sensor, where the electronic board includes a touch screen, an outer frame, a support and a signal transmission interface; a pressure sensor is arranged below the touch screen;

the frame of the electronic board is also provided with a plurality of keys and contact indicator lamps, wherein the keys comprise a start key, an end key, a playback key, a data storage key and a data export key;

triggering the touch screen to start recording handwriting when the start key is pressed;

triggering the touch screen to end recording the handwriting when the end key is pressed;

the recorded handwriting is played back when the playback key is pressed;

when the data export key is pressed down, exporting handwriting to external equipment, such as a computer or a U disk;

the contact indicator light is used for lighting when the electronic pen is effectively contacted with the touch screen of the electronic board, indicating that the contact is effective and the data recording is normal, and indicating that the contact is invalid when the contact indicator light is not on, possibly indicating that the contact is poor or the operation of a user is improper;

a color selection area, a stroke width selection area, buttons for backing, erasing and the like are arranged on a touch screen display area of the electronic board;

the color selection area is used for selecting a color which is expected to be used by a user in the test;

the stroke width selection area is used for selecting the stroke width which is expected to be used by a user in the test;

the back button is used for executing withdrawing operation, the erasing button is used for erasing the track of a part of the area, and when a user is not satisfied with the track of the picture, the erasing function can be used for erasing the content of the area;

the electronic board is used for receiving writing and drawing of an electronic pen on the electronic board, capturing and recording handwriting of the electronic pen through a touch screen on the electronic board, and recording occurrence time of each track point in the handwriting, corresponding strength and color information; for example, a point P (x, y, t, color, width, pressure), where x, y are abscissa and ordinate, t is the current recording time, color is color, width is stroke width, and pressure is the pressure of the electronic pen; a handwriting formed by a series of points is L ═ { P1, P2, Pi … … Pn }, and n is the number of points on the handwriting;

a plurality of handwriting L form a whole picture A, wherein A is { L1, L2 and Li … … LN }, and N is the number of the handwriting;

the electronic board internally comprises a timer for recording the time t of each point in the handwriting;

according to the embodiment of the invention, the electronic board further comprises a processor and a memory, wherein the processor is used for identifying the picture content, analyzing and processing the handwriting, and extracting the characteristics of the handwriting, such as the time of each part of the movement, the force of the stroke, the continuity/fluency of the track, the regularity of short points, the jumping performance of the stroke, repeated description, the position (coordinate axis), the area, the symmetry and the like of the outline.

The electronic board also comprises a data interface which is used for connecting with a data line of the electronic pen so as to receive the data of the electronic pen;

or the electronic board is provided with a wireless receiving and transmitting module which performs data transmission with the electronic pen through wireless signals;

the electronic board can also store the handwriting data, and the stored data can be read by the processor at any time and can be played back, analyzed and processed; so that data processing can be performed at any time;

as shown in fig. 2, the electronic pen includes an electronic pen body and a holding portion disposed on the electronic pen body, and a plurality of sensors including a pressure sensor, a heart rate sensor, a blood oxygen saturation sensor, a pico-cell sensor, and a skin temperature sensor are disposed at the holding portion; the black areas in the figure are sensors, which are only schematic and can be arranged according to the actual requirement;

furthermore, a data transmission line is arranged at the tail end of the electronic pen and connected to the electronic board through an interface; or the electronic pen is provided with a wireless transceiver module, and the sensor data is transmitted through the wireless transceiver module;

according to an embodiment of the present invention, a method for performing handwritten projection data acquisition by using the aforementioned system is provided, which includes the following steps:

step 1, connecting an electronic pen with an electronic board in a wired or wireless mode, starting a power switch of a system, and operating the system;

step 2, the user holds the electronic pen, the thumb is contacted with the blood oxygen sensor, the skin electricity sensor and the skin temperature sensor, and the index finger is connected with the heart rate sensing period and the pressure sensor; clicking a start key, starting drawing on an electronic board by a user by using an electronic pen with a multi-sensor fusion, recording the drawing handwriting in real time by the electronic board, and recording the editing operation of the user on a picture pen, including withdrawing and erasing operations, and measuring physiological characteristic data of the user by the electronic pen; clicking an end button after drawing is finished;

step 3, when the electronic pen is detected to be in contact with the electronic board, the contact indicator lamp is turned on, the pressure sensor on the electronic pen acquires blood pressure data of the user in real time, the blood oxygen sensor acquires blood oxygen concentration parameters of the user, the heart rate sensor acquires the heart rate of the user, and the physiological data of the user are transmitted to the electronic board;

step 4, the electronic board detects the handwriting of the electronic pen on the touch screen in real time, and the color parameters selected by the electronic pen, the pressure parameters of the electronic pen pressing the touch pad, the position parameters of the handwriting points and the current time parameters are obtained;

step 5, utilizing a content identification module to identify the content of the picture, and performing overall analysis on the written and drawn content through a machine vision algorithm to realize the main body target identification function;

step 6, on the basis of completing content identification, extracting handwriting characteristics by using a handwriting characteristic extraction module; fusing the multi-dimensional characteristics of the handwriting by using a fusion analysis module, and outputting the acquired and analyzed data; the multi-dimensional characteristics of the handwriting mainly comprise handwriting characteristics and physiological characteristics.

The content recognition module performs overall analysis on the writing and drawing contents through a machine vision algorithm to realize a main body target recognition function. The content identification process steps are shown in fig. 3. Comprises the following steps:

step 1, carrying out area division on the image on the electronic board. In an expert experience manner, the image is divided into K × K dry square regions, for example, K5, K6, or the like. When the center point (black dot) of the content body falls within a certain square area, the area is responsible for recognizing the content body and outputting the type and size of the final body target. All square areas are numbered sequentially in order starting from 0. For example, the first region from the top left corner is 0, and the first region is numbered to K × K in sequence according to a zigzag manner, and there are K × K regions in total;

and 2, constructing a region adjacency matrix. The region adjacency matrix is used to represent the adjacency relationship between the square regions. Continuous handwriting is used when the user draws the same body of content, whereas jumps in handwriting occur when drawing different bodies. Therefore, the pixel connection graph constructed by using the continuous handwriting contributes to improving the accuracy of the subject recognition. The element of the ith row and the jth column of the area adjacency matrix represents the total times of simultaneously traversing the ith and jth image areas by continuous handwriting; i. j is the serial number of the region, and the total number of the regions is K²。

And 3, combining the image with the adjacent matrix. Firstly, according to the affiliated relationship between the regions and the pixels, the dimension of the region adjacency matrix is changed from K²×K²Expansion to WXHXK². And then performing channel expansion on the image data by using the expanded area adjacency matrix, wherein the channel expansion is referred to as channel expansion. The original image is W × H × C, and after the channel expansion, it becomes W × H (C + k)²) To obtain W.times.Hx (C + K)²) C represents the number of channels, which generally refers to the values of three colors of RGB (red, green, and blue). However, there may be a plurality of cases such as RGBA (red green blue alpha)4 dimensions, or 1 dimension of the gray scale system, or CMYK. Depending on the image format. W is the image width and H is the image height, which are inputs to the prediction model.

And 4, extracting image features by using the Darknet deep learning model. The Darknet is a deep learning model constructed by a convolutional layer, a residual layer, a pooling layer and the like, and can well extract depth features from image data. The Darknet deep learning model completes training by adopting known image data in advance.

And 5, outputting a prediction result. And each region identifies the target subject in the region according to the image characteristics output by the Darknet, and finally outputs the target type and the size of the target subject. The target types include buildings, vehicles, people, animals, plants, and the like; the target size is the area occupied by the region (for example, if the coordinate values of the upper left corner and the lower right corner of the box are actually given, the area is calculated according to the coordinate values).

The handwriting characteristic analysis module is used for extracting and analyzing the drawing handwriting characteristics; the handwriting characteristic analysis module has the main functions of summarizing all historical handwriting in the target area on the basis of completing content identification, fusing the multi-dimensional characteristics of the handwriting and obtaining an analysis result for realizing the dynamic portrayal of the user's mind.

The multi-dimensional characteristics of the handwriting mainly comprise handwriting characteristics and physiological characteristics.

The handwriting characteristics mainly comprise handwriting appearance characteristics, handwriting time stage characteristics, handwriting space stage characteristics and handwriting modification characteristics;

the handwriting appearance characteristics include, for example: thickness, regularity, size, lightness and color of handwriting;

for example, the handwriting thickness may be selected on the panel, and the selected handwriting thickness is, for example, 5 pixels, or 10 pixels as the handwriting thickness; the regularity refers to the degree of smooth handwriting or flow field, and can be obtained by calculating the mean square error of a fitting curve, for example; the handwriting size refers to the area size defined by the maximum width and the maximum height of the handwriting; the weight of the handwriting is obtained through a pressure sensor on the electronic pen;

the handwriting time stage characteristics comprise: stroke point sequence, sequence among the strokes, pause time, speed characteristic, stroke jumping and stroke fluency; the handwriting point sequence comprises the sequence of each point in the handwriting, and the time information is obtained on the handwriting points, so that the sequence of the points on the handwriting can be obtained through the time information; the sequence and the pause time among different handwriting can be obtained by comparing the time of the upper point of the different handwriting; calculating the speed characteristic by dividing the length of the handwriting by the time difference between the first point and the last point of the handwriting; the handwriting jumping refers to the distance between the tail point of the previous handwriting and the starting point of the next handwriting among different handwriting;

the handwriting modification characteristics comprise correction adjustment, repetition, deletion and the like, and represent the modification processing process of a user;

according to the embodiment of the invention, a back button and an erasing button are displayed in a screen area of an electronic board, when a user clicks the back button or the erasing button in the drawing process, a system records the operation and modifies the corresponding handwriting to obtain a modified record, namely, an edge (mode, L, t), wherein the mode represents back or erasing; l represents the corresponding handwriting, and t represents the current time;

the handwriting space characteristics include: handwriting area distribution, breakpoint proportion, breakpoint number, area density and handwriting continuity;

according to the embodiment of the invention, the handwriting distribution area characteristics can be expressed by a matrix, and the distribution quantity of handwriting points in each area is used as the corresponding position element of the matrix; the number of the breakpoints is the stroke points with the length smaller than the preset length, and compared with the total handwriting number, the breakpoint proportion is obtained; the handwriting continuity refers to the ratio of the length of the handwriting to the average length of the handwriting; areal density refers to the density of points per area;

the physiological characteristics are the physiological characteristics of the user such as heart rate, pen holding pressure, blood oxygen saturation, skin electricity, skin temperature and the like during drawing.

According to the embodiment of the invention, the processor can further judge different writing and drawing stages of an operator according to the handwriting, and divide the whole handwriting into a plurality of track segments by stage division from time or space or combination of the time and the space to obtain stage characteristics of the handwriting:

for example, in the process of writing and drawing, a user stops writing for a period of time in the middle and then drops the writing, so that the recording time of the handwriting points is discontinuous jump; representing the user's thought, or representing the completion of one goal, to the next goal, i.e., one stage to another;

or, the user can finish drawing from one area and jump to another area to start drawing, and the spatial crossing in the drawing process of the user can be analyzed by analyzing the recording time and the recording area of the stroke point; furthermore, the whole drawing process can be subjected to space crossing analysis to obtain the drawing sequence and stage of each region.

According to an alternative embodiment of the invention, the handwriting is also divided in time in stages as follows:

the initial stage is as follows: positioning a handwriting area of a user, analyzing a layout area and content of the user, and calculating the fluency of each handwriting line;

a modification processing stage: extracting the characteristics of detail adjustment, repetition, deletion and color modification of the picture content by a user;

in the completion phase: extracting the picture content layout of a user, and analyzing the main content, the main color, the background area and the color;

in the finalizing stage: the user performs written description and confirmation on the screen content, and extracts a keyword of the description therein as auxiliary data.

According to another aspect of the present invention, the content recognition module, the handwriting feature extraction module, and the fusion analysis module may be configured as a single chip module, such as an FPGA or other digital chip, or may be stored on a memory as a program module, and the content recognition module and the like are implemented by executing the program module through a processor

Although illustrative embodiments of the present invention have been described above to facilitate the understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the embodiments, but various changes may be apparent to those skilled in the art, and it is intended that all inventive concepts utilizing the inventive concepts set forth herein be protected without departing from the spirit and scope of the present invention as defined and limited by the appended claims.