阅读说明：本技术 一种信息交互方法、交互系统和应用 (Information interaction method, interaction system and application ) 是由 褚莹琨 廖昀 于 2019-09-30 设计创作，主要内容包括：本发明公开了一种信息交互方法、交互系统和应用,属于电子白板领域。针对现有技术中存在的会议系统中电子白板传输数据大,互动带宽高,对网络需求高的问题,本发明提供了一种信息交互方法,主机将画面数据发送给本地和远端进行显示,本地对画面有批注时,主机对本地的批注进行序列化处理,将处理后数据发送至远端,远端结合序列化处理后数据以及对应的坐标和压感值,通过图形处理器计算绘制成图像显示,所述交互方法应用在会议系统的电子白板中,可以实现书写时只发送笔迹数据,所有参会方收到该笔迹数据后将其显示在屏幕上,进而实现多方实时互动式白板协作,且在本地会议和远程会议中均支持。(The invention discloses an information interaction method, an interaction system and application, and belongs to the field of electronic whiteboards. The invention provides an information interaction method, aiming at the problems of large transmission data, high interaction bandwidth and high network demand of an electronic whiteboard in a conference system in the prior art, wherein a host sends picture data to a local terminal and a remote terminal for displaying, when the local terminal annotates a picture, the host carries out serialization processing on the local annotation and sends the processed data to the remote terminal, the remote terminal combines the serialized processed data and corresponding coordinates and pressure values and calculates and draws the data into an image for displaying through a graphic processor, the interaction method is applied to the electronic whiteboard of the conference system, and can realize that only handwriting data is sent during writing, and all participants display the handwriting data on a screen after receiving the handwriting data, thereby realizing multi-party real-time interactive whiteboard cooperation and being supported in a local conference and a remote conference.)

1. The information interaction method is characterized in that a host carries out serialization processing on local annotation drawing line segment information, and a display terminal carries out calculation drawing on the annotation drawing line segment information and annotation related information which are subjected to serialization processing through a graphic processor to form image display.

2. The information interaction method according to claim 1, wherein the specific interaction steps are as follows:

step 1: the host displays the picture locally and the display terminal through the internal display module;

step 2: the user annotates the picture locally, draws line segment information during annotation and sends the line segment information to the host for serialization processing and sending the line segment information to the display terminal;

and step 3: locally sending the annotation related information to a display terminal through a host;

and 4, step 4: and the display terminal carries out deserialization on the received drawing line segment information subjected to serialization processing, adds the annotation related information to the recorded line segment, and uses a graphic processor to calculate and draw the information into an image for display.

3. The information interaction method according to claim 2, wherein the annotation drawing line segment information comprises a line segment ID, a user ID and a timestamp attribute.

4. The information interaction method of claim 2, wherein the annotation-related information comprises coordinates and pressure-sensitive values.

5. The information interaction method according to claim 3 or 4, wherein the annotation drawing line segment information and the annotation related information are transmitted separately.

6. The information interaction method as claimed in claim 2, wherein the local and display terminals use a serialization rule to serialize and deserialize the drawn line segment information.

7. The information interaction method as claimed in claim 6, wherein the host processes the annotation information and sends the processed data to other display terminals through the network module.

8. An information interaction system using the information interaction method as claimed in any one of claims 1 to 7, wherein the system comprises a host computer, a touch display device and an electronic whiteboard device.

9. The information interaction system of claim 8, wherein the host comprises a data processing module, a touch module, a display module and a network module.

10. A conferencing system comprising an information interaction system as claimed in any of claims 8-9.

Technical Field

The present invention relates to the field of electronic whiteboards, and in particular, to an information interaction method, an interaction system, and an application.

Background

The remote conference refers to a video conference, and voice and video images are interactively shared in a remote way. For data interaction in a computer, the traditional method is to transmit the data after converting the data into video, and the image quality and the real-time interactivity are greatly reduced in the traditional method. With the development requirement of meeting content datamation, the interactive writing system appears timely, and the deficiency is made up. The remote conference system mainly comprises a telephone conference, a network conference and a video conference.

In order to provide the participation interactivity of the participants in the conference and improve the efficiency of collaborative communication, some conference electronic tools are often used in the conference. However, there are still many problems to date with the use of electronic tools in meetings. In order to bring the display content of the computer to the projector, the most common method is to use some wired or wireless projection devices to display it on the large screen of the conference room for all participants to share. However, merely sharing a presentation is not enough. For example, if the meeting participants only record the main content of the lecturer, the conference will have low interaction and efficiency. Meanwhile, in the conference process, a presenter may need to perform certain annotation explanation on the presentation content, and the content annotation is supported by installing certain software in the traditional mode. Therefore, the use of an interactive electronic whiteboard becomes particularly important in a conference.

Interactive electronic whiteboards are currently used more and more frequently in meetings, especially in teleconferencing. The electronic whiteboard is combined with a computer and a projector, so that paperless office work and teaching can be realized. It may be a stand-alone touch screen computer that is used independently to perform tasks and operations, or it may be a connectable device that is used as a touch pad to control the computer from a projector.

The ClickShare wireless presentation system, marketed by Barco, and the MAXHUB conference tablet, are more widely used. Its advantages are plug and play, no need of additional software driver, and simple installation and deployment. But it only supports local conferencing and does not support teleconferencing multiparty interaction. And writing interaction is unidirectional, namely after entering the annotating mode, the picture shot by the PC is only intercepted, and only the annotation and the display can be carried out on the interactive electronic panel (touch panel), and the annotation content cannot be simultaneously displayed on the PC. And it cannot support video collaboration in teleconferencing. Writing and commenting can be carried out only on the touch television; likewise, written content is not returned to display on the PC.

Compared with unidirectional electronic whiteboard cooperative interaction, the whiteboard interaction is realized by adopting an auxiliary stream (screen projection by a computer), namely, a sharing party opens a blank page and writes on the page, and intercepts the page and sends the page as a video stream to a participant. The method has better compatibility, and the content can be seen as long as the auxiliary stream can be seen no matter whether the opposite party (including a third party) supports the electronic whiteboard/writing. But the disadvantage is also obvious, because it needs to intercept the complete image picture as video stream to send, even if the user does not write anything any more, the video stream will be sent continuously, so the occupied bandwidth and the requirement for the network are higher.

The Chinese patent applies for a content interaction method and a system of a video conference, the patent number CN201310069962.8, the publication date 2018, 4 and 6, and discloses a content interaction method and a system of a video conference, which are more important practical services expanded on the basis of using a double-stream function, namely, on the basis of a whiteboard interaction service, terminals of participants perform label interaction in content shared by auxiliary videos, so that the problem of data interaction in a video conference is solved. Therefore, the invention can enable all parties in the video conference to communicate more fully, enables the video conference terminal to have stronger function and is closer to the goal of realizing meeting in the same conference room at any time and any place. The invention adopts the intercepted complete image picture as the video stream for transmission, the transmission data is huge, and the bandwidth occupation is large; and the whiteboard cooperation needs to be transferred through the conference convener terminal, and the data transmission is complex.

Disclosure of Invention

1. Technical problem to be solved

Aiming at the problems of large transmission data, high interactive bandwidth and high network requirement of an electronic whiteboard in a conference system in the prior art, the invention provides an information interaction method, an interaction system and application, which can realize that all equipment start an independent electronic whiteboard program, only handwriting data is sent during writing, all participants display the handwriting data on a screen after receiving the handwriting data, and further realize multiparty real-time interactive whiteboard cooperation, and are supported in a local conference and a remote conference.

2. Technical scheme

The purpose of the invention is realized by the following technical scheme.

A host carries out serialization processing on local annotation drawing line segment information, and a display terminal carries out calculation drawing on the local annotation drawing line segment information and annotation related information through a graphic processor according to the serialization processing to form image display. According to the invention, during data interaction, the annotation data is serialized, only the annotated handwriting information is sent during interaction, the image information is shared through auxiliary streams, the real-time interaction occupies small bandwidth, and the requirement on a network is low.

Further, the specific interaction steps are as follows:

step 1: the host displays the picture locally and the display terminal through the internal display module;

step 2: the user annotates the picture locally, draws line segment information during annotation and sends the line segment information to the host for serialization processing and sending the line segment information to the display terminal;

and step 3: locally sending the annotation related information to a display terminal through a host;

and 4, step 4: and the display terminal carries out deserialization on the received drawing line segment information subjected to serialization processing, adds the annotation related information to the recorded line segment, and uses a graphic processor to calculate and draw the information into an image for display.

The comment information and the video picture are completely displayed separately in two layers. Firstly, displaying a video picture, and then covering annotation information on the video picture, wherein the background of the annotation information is transparent, and the annotation information is displayed as if handwriting information is fused in the video picture. The data receiving of the video picture and the annotation information is completely separated, the video picture and the annotation information do not need to be used simultaneously, a user can only use the video picture or only use the whiteboard annotation, and the difference between the data receiving and the data displaying is that the background of the annotation information is default white and is not transparent, and the color is adjustable. The drawing line segment information and the coordinate pressure sensing value are sent separately so as to achieve the real-time handwriting display of the local display terminal equipment and the display terminal equipment. The method comprises the steps of sending segment information to a display terminal from the beginning of a stroke falling of a local device user to inform that a new segment is generated, obtaining point coordinate information along with the sliding of the stroke, drawing points one by one to form a line, synchronously sending point information to the display terminal, judging which segment the point belongs to when the display terminal receives the point information, and adding the point information to the segment for drawing. Therefore, the drawing points of the local display terminal and the display terminal can be synchronized.

Further, the annotation drawing line segment information includes a line segment ID, a user ID, and a time stamp attribute. The annotation drawing line segment information is divided into an attribute part and an ID part after being subjected to serialization processing, wherein the attribute part comprises a line segment ID, a user ID and a timestamp attribute.

Further, the annotation-related information includes coordinates and pressure-sensitive values. The coordinate and pressure sensing value and the annotation drawing line segment information are sent separately to ensure the real-time transmission of the annotation information, from the beginning of the stroke of a local device user, the line segment information is sent to a display terminal to inform that a new line segment is generated, point coordinate information is obtained along with the sliding of the pen, the points are drawn one by one to form a line, meanwhile, the point information is synchronously sent to the display terminal, and the display terminal judges which line segment the point belongs to when receiving the point information and adds the point information to the line segment for drawing. Therefore, the drawing points of the local display terminal and the display terminal can be synchronized.

Further, the annotation drawing line segment information and the annotation related information are transmitted separately. The line segment information and the point coordinate pressure-sensitive value data are transmitted separately, so that the real-time performance of data transmission is guaranteed, if the data are transmitted together, in order to take complete point coordinate information, the data can be transmitted to the display terminal only after a line is completely drawn locally, and the real-time performance of display of both sides cannot be achieved.

Furthermore, the local terminal and the display terminal use a serialization rule to carry out serialization and deserialization on the drawn line segment information. The local terminal and the display terminal use the same serialization rule, and the same rule serialization and deserialization are adopted when the line segment information is drawn.

Furthermore, the host processes the annotation information and sends the processed data to other display terminals through the network module. The host computer receives the annotation information, carries out serialization processing, then sends the annotation information to other display terminals through the network module, and simultaneously calculates, draws and generates handwriting information according to the annotation related information through the GPU to be displayed locally.

The information interaction system comprises a host, touch display equipment and electronic whiteboard equipment. The local display device is a touch display device, and the display terminal is an electronic whiteboard device. In the interaction method of the electronic whiteboard, when a user marks on the touch display equipment, draws line segment data, transmits the line segment data to the host for serialization processing and then transmits the line segment data to the electronic whiteboard equipment, the electronic whiteboard equipment receives and memorizes the line segment information, the touch display equipment randomly transmits coordinates and pressure sensitivity values to the electronic whiteboard equipment, the electronic whiteboard equipment receives the line segment information and then adds the line segment information to the recorded line segment, and an image is calculated and drawn through a GPU (graphic processing Unit) and displayed on the electronic whiteboard equipment. The electronic whiteboard device and the touch display device can be multiple, the electronic whiteboard devices allow different people to annotate, and the touch display devices can be used for expanded display. The electronic whiteboard device can be a tablet or a PC, and is an accessory using mode connected with a host, and the accessory can synchronously display video and annotation information on the host and also input annotations. The electronic whiteboard device and the touch display device are not communicated, data are firstly and synchronously transmitted to the host, and then the host carries out transmission synchronization.

Furthermore, the host comprises a data processing module, a touch module, a display module and a network module. The data processing module comprises a media module and a cooperation module, wherein the media module provides audio and video services and transmits audio and video to each conferencing terminal of the conference through an SIP protocol; and the cooperation module is responsible for processing the annotation data and transmitting the data to each cooperation device through a TCP-based private protocol. The touch control module and the display module are respectively connected with the touch control display device and the host, the data of the touch control display device are transmitted to the host through the touch control module, and the processed data are transmitted to the touch control display device through the display module by the host. In the invention, a user can use the touch display equipment as a local end to carry out annotation marking and interact with other conference-participating ends in real time; the electronic whiteboard equipment can also be used as a local end for annotation marking, and information is interactively displayed on the touch display equipment.

The invention discloses a conference system, which is an application of an information interaction system and comprises the information interaction system. The conference system is a local conference system or a remote conference system. The conference system supports local conferences and remote conferences, each conference room is connected with the cloud server during the remote conferences, and different conference rooms are registered in the designated cloud servers and connected with each other to form interaction. The conference system comprises an information interaction system and corresponding conference input and output equipment, wherein the conference input and output equipment comprises a microphone, a camera and audio playing equipment. In addition to the physical video conference room, personal mobile devices such as mobile phones, personal PCs or tablet computers may also be used to attend conferences and to perform annotation tagging.

The invention only sends handwriting data in information interaction, and any party can freely write and share auxiliary stream in the touch display device during image sharing.

3. Advantageous effects

Compared with the prior art, the invention has the advantages that:

compared with other technical schemes, the electronic whiteboard in the technical scheme can be applied to a teleconference, an independent electronic whiteboard program is started through all equipment, only handwriting data is sent during writing, and all participants display the handwriting data on a screen after receiving the handwriting data. The data transmission only needs to transmit each handwriting, the data volume is small, the data transmission only needs to be carried out once, new data does not need to be sent when the user stops writing, the dependence on bandwidth is low, meanwhile, the situation that pictures seen by participants become fuzzy due to network deterioration is avoided, and the requirement on the network is low.

Drawings

FIG. 1 is a flow chart of the present invention for tagging interactions with shared content during a conference;

FIG. 2 is an electronic whiteboard collaboration application in a local conference of the present invention;

FIG. 3 illustrates an electronic whiteboard collaboration application in a teleconference, in accordance with the present invention;

FIG. 4 is a flow chart of the marker interaction of the present invention.

Detailed Description

The invention is described in detail below with reference to the drawings and specific examples.