阅读说明：本技术 多设备协同和一对多控制的一个新方法和系统 (A new method and system for multi-device coordination and one-to-many control ) 是由 林征 孔淋 于 2021-08-06 设计创作，主要内容包括：本发明公开一个多设备协同和一对多控制的新方法和系统,包括：建立一个系统,含有多个智能设备同处一个网络,通过协作,由其中的一个设备遥控其他设备,协同完成某个任务,系统其中一个设备担任数据源角色,提供数据(流)服务,另一个担任控制器角色,其余的为受控设备,受控设备负责处理从数据服务器输出的内容。操作步骤概要为：1)各个设备启动以后,以一定的时间间隔在网络上发布自身在等待的信息,直到被控制器发现；2)控制器发现一个设备以后,通过询问设备,获得设备相关属性和服务参数,调用专门的设备识别分类规则,对设备分类,每一类别单独成为一个群；3)控制器对每个群发指令,指令保证到达每个群成员；指令包括控制指令和同步指令；4)收到指令的设备按照规则执行指令；5)控制器继续对所有设备监控。(The invention discloses a new method and a system for multi-device cooperation and one-to-many control, which comprises the following steps: a system is established, a plurality of intelligent devices are in the same network, one device is used for remotely controlling other devices through cooperation to cooperatively complete a certain task, one device in the system plays a role of a data source to provide data (stream) service, the other device plays a role of a controller, the rest devices are controlled devices, and the controlled devices are responsible for processing contents output from a data server. The outline of the operation steps is as follows: 1) after each device is started, the information waiting for the device is issued on the network at certain time intervals until the device is discovered by the controller; 2) after finding a device, the controller acquires the relevant attributes and service parameters of the device by inquiring the device, calls a special device identification classification rule, classifies the device, and forms each class into a group independently; 3) the controller sends an instruction to each group, and the instruction ensures that each group member is reached; the instructions comprise control instructions and synchronization instructions; 4) the equipment receiving the instruction executes the instruction according to the rule; 5) the controller continues to monitor all devices.)

1. A method for multiple devices to co-operate in a network, one of the devices remotely controlling the other devices to jointly perform a task, the method comprising:

a. a plurality of devices are located in a network, and the devices are intelligent devices with storage, operation, display and network communication capabilities, and are usually mobile tablets or computers;

b. one of the devices takes the role of a data source and provides data or data stream services; the other assumes the controller role; the rest is controlled equipment; all devices have the function of executing the most basic network communication, namely, the function of receiving and sending the binary data;

c. the controlled device is a player, can obtain data or data stream from the specified address to play, have and play, pause, restart the function at least;

d. the data service is provided by a server which has a fixed address and is accessible to all the devices, and the data server at least supports HTTP;

e. the controller controls the equipment to act and also controls the equipment to be synchronous; the controller has not only the common functional units of user operation interface, communication, instruction generation, data storage and operation, but also an equipment identification and classification unit, which has the function of storing equipment classification logic definitions, namely rules, and calculating how to divide groups; the rule may be a simple fixed correspondence, for example, it may be a complex function that specifies that the device a belongs to the group G, and maps the device to a certain group F (a, b, …, k) by operation according to the device characteristic parameters (a, b, …, k);

f. at any one time, only one smart device in the system can become a controller, but the roles of the devices can be switched, e.g., the player can become a controller at the next time and vice versa;

g. after each device is started, the information waiting for the device is issued on the network at certain time intervals until the device is discovered by the controller;

h. after finding a device, the controller sends a series of queries to the device through the network, determines the service and the capability range which can be provided by the device, and classifies the device according to preset rules; each category is individually grouped into a group; each group contains one or more functionally similar devices;

i. the controller sends a control instruction to the classified equipment, and the control instruction is sent aiming at the group and can reach all members; the control command can be an action or a synchronous command; these instructions include, but are not limited to:

a. playing a data source or data stream address;

b. the player and the data server are connected with parameters, such as necessary connection parameters of user authentication, initialization, authority and the like;

c. initiating an action, such as start of play;

d. reading the current situation and condition of the equipment;

e. changing the attributes of the device, such as output interpretation degree;

the synchronous command is a motion command with a time stamp and indicates that the motion occurs at a certain future time;

j. the player receives the instruction and executes the instruction according to the appointed rule, and then enters the idle waiting state again after finishing the instruction;

k. and d, the controller continues to monitor the new equipment and repeats the operation started in the step h.

2. The method of claim (1), wherein step (h) further comprises:

after finding a device, the controller determines the device attribute and service capability by sending command query, then calls a device classification rule to carry out identification and classification operation, and classifies the player; the classification is performed by the device identification and classification unit, taking into account factors including, but not limited to:

a. equipment type, manufacturer and model;

b. a system, a processor, a memory;

c. network speed, screen type, direction, size;

d. other capabilities of the device related to the application;

e. device user and application related attributes, etc.;

each category is individually grouped into a cluster; a cluster may contain one or more devices.

3. The method of claim (1) (2), wherein step (i) further comprises:

m, the controller simultaneously sends out control instructions to a plurality of groups, and different groups can have different instructions; commands issued to the group will reach each group member;

n. the concurrent instruction is issued using: forming a thread pool by using a limited number of threads, and simultaneously sending instructions to a plurality of groups in a thread concurrent mode; if the thread pool has N threads, sending instructions to N groups at the same time; within a cluster, sequentially commanding each device; after completing one group, the thread immediately processes the next group; the size N of the thread pool depends on the hardware configuration of the controller, 2 to 4 threads are suggested to correspond to one CPU core, too many threads can generate extra thread management and scheduling overhead, and too few threads can cause the CPU to be under-worked and slow down the processing speed;

step (n) is further, the instructions in the group are sent in a linear and asynchronous mode, namely the sending thread does not need to wait for the instruction to return a result, the sending thread continuously sends the result to the end, and then the sending thread is attentively waited for the instruction result; in addition, any device error is considered as the failure of the group transmission;

p. if there is an error, the user will be prompted for the error, decided by the user whether to retry or abort.

4. A system for a plurality of devices to co-operate in a network to enable one of the devices to remotely control another device to collectively perform a task, comprising:

q. multiple devices are located in a network, and the devices are intelligent devices with storage, operation, display and network communication capabilities, and are usually mobile tablets or computers;

one of the devices takes the role of a data source and provides data or data stream services; the other assumes the controller role; the rest is controlled equipment; all devices have the function of executing the most basic network communication, namely, the function of receiving and sending the binary data;

the controlled device is a player, can obtain data or data stream from the specified address and play, have and play, pause, restart the function at least;

t, the data service is provided by a server which has a fixed address and can be accessed by all the equipment, and the data server at least supports HTTP;

the controller controls the equipment to act and also controls the equipment to be synchronous; the controller has not only the common functional units of user operation interface, communication, instruction generation, data storage and operation, but also an equipment identification and classification unit, which has the function of storing equipment classification logic definitions, namely rules, and calculating how to divide groups; the rule may be a simple correspondence, for example, it may be a complex function that specifies that the device a belongs to the group G, and maps the device to a certain group F (a, b, …, k) by operation according to the device characteristic parameter (a, b, …, k);

at any one time, only one smart device in the system can become a controller, however, the roles of the devices can be switched, e.g., the player can become a controller at the next time and vice versa;

w. after each device is started, issuing self-waiting information on the network at certain time intervals until the information is discovered by the controller;

x, after finding a device, the controller sends a series of queries to the device through the network, determines the service and the capability range which can be provided by the device, and classifies the device according to preset rules; each category is individually grouped into a group; each group contains one or more functionally similar devices;

y. the controller sends out control command, the control command is sent out aiming at the group and reaches all the group members; the control command can be an action or a synchronous command; these instructions include, but are not limited to:

a. playing a data source or data stream address;

b. the player and the data server are connected with parameters, such as necessary connection parameters of user authentication, initialization, authority and the like;

c. initiating an action, such as start of play;

d. reading the current situation and condition of the equipment;

e. changing the attributes of the device, such as output interpretation degree;

the synchronous command is a motion command with a time stamp and indicates that the motion occurs at a certain future time;

z. the player receives the command and executes the command according to the appointed rule;

the controller continues to listen for new equipment, repeating the above operations.

5. The system of claim (4), wherein step (x) further comprises:

b, after finding a device, the controller determines the device attribute and service capability by sending command query, calls a device classification rule to perform identification and classification operation, and classifies the player; the classification is mainly performed by the device identification and classification unit, taking into account factors including but not limited to:

a. equipment type, manufacturer and model;

b. a system, a processor, a memory;

c. network speed, screen type, direction, size;

d. other capabilities of the device related to the application;

e. device user and application related attributes, etc.;

each category is individually grouped into a group; a group contains one or more functionally similar devices.

6. The system of claim (4) (5), wherein step (y) further comprises:

cc. the controller sends control commands to multiple groups simultaneously, and different groups can have different commands; commands issued to the group will reach each group member;

dd. the concurrent instruction is issued using the following: forming a thread pool by using a limited number of threads, and simultaneously sending instructions to a plurality of groups in a thread concurrent mode; if the thread pool has N threads, sending instructions to N groups at the same time; within a cluster, sequentially commanding each device; after completing one group, the thread immediately processes the next group; the size N of the thread pool depends on the hardware configuration of the controller, 2 to 4 threads are suggested to correspond to one CPU core, too many threads can generate extra thread management and scheduling overhead, and too few threads can cause the CPU to be under-worked and slow down the processing speed;

ee. step (dd) further, the instruction in the group is sent in a linear and asynchronous manner, that is, the sending thread does not need to wait for the instruction to return a result, and continuously sends the result to the sending thread, and then the sending thread is attentive to wait for the instruction result; in addition, any device error is considered as the failure of the group transmission;

ff. if there is an error, the user will be prompted for the error, and the user will decide whether to retry or abort.

The technical field is as follows:

the invention discloses a method and a system for cooperation among a plurality of intelligent devices in an internet environment, and relates to intelligent classification of the devices and one-to-many control in a group unit.

Background art:

mutual identification, pairing, communication and control of intelligent mobile devices are the basis for the intelligent devices to cooperatively complete a specific task. Establishing communication between networked devices typically requires establishing communication via a common third party, i.e., a central server, or via some common rules in a peer-to-peer session between the devices. A typical example of the point-to-point protocol is the UPnP (Universal Plug-and-Play) protocol. The purpose is to enable equipment produced by different brands and different manufacturers to be directly communicated. One important application in the field of home entertainment is to use an intelligent device such as a mobile phone to control another device to play video, namely screen projection in popular terms. The control information and the playing data of the screen are generally transmitted through a wireless network. In general, a media source, a playback device, and a controller are required to implement remote playback. The media source is a media server providing data or data streams, the playback device is a media player, and the controller is a means for remotely controlling the player and the media source. The controllers of the existing market products generally only realize one-to-one control, namely the controller and the player are paired one-to-one, and the controller instruction is only effective for the paired player. The remote control of the smart television by the typical mobile phone is performed in this mode:

turning on a remote control function of the mobile phone;

selecting media content to be played on the mobile phone;

pressing a remote play key or a push key to transmit media to a television for playing;

the handset continues to monitor and remotely control the playback.

The one-to-one mode is sufficient in home television remote control applications. It is noted that there are many limitations to simply applying a one-to-one mode to control multiple devices in other situations, such as an electronic classroom with multiple controlled devices. This involves that the controlled devices may have different device capabilities, take different roles, have different permissions, etc., which is much more complex than a one-to-one scenario. In addition, the control instruction is sent to a plurality of devices to form a potentially serious performance bottleneck, and the synchronous execution of one cooperative task by the plurality of devices also has many challenges; we also note that the published literature is rarely detailed in the grouping of devices in a multi-device scenario, and it is difficult to find a generally accepted theoretical consensus and best practice, and there is a wide area for continued research in this area.

To address these problems, we propose a one-to-many control and coordination method. The core problem solved by the method is how to carry out rule-based classification according to the characteristics of the equipment, and then control instructions are sent to the equipment which is already divided into the groups in a unique mode to cooperatively complete the task.

The invention content is as follows:

the multiple devices are in the same network, and one device remotely controls other devices through cooperation to jointly complete a certain task. For convenience of description, we make the simplest assumption: one of the devices takes the role of a data source and provides data or data stream services; the other one takes the role of a controller, and the rest are controlled devices which are responsible for processing data acquired from a data server, and the presented content can be regarded as a multimedia player.

Further, all devices have the capability to perform the most basic network communication, i.e., binary data messaging.

Further, the data streaming service is provided by a server having a fixed address and accessible to all devices. The controlled device is a player, can obtain data or data stream from the designated address to play, and at least has the functions of playing pause and restarting. The controller controls both device activity and device synchronization. The controller has, in addition to the usual functional units for user interface, communication, command generation, data storage and calculation, in particular, a device identification and classification unit, whose function is to store device classification logic definitions, i.e. rules, calculating how to divide the groups. The rules may be simple fixed correspondences, such as specifying that device a belongs to group G, or complex functions that operate to map devices to a group F (a, b, …, k) based on device characteristic parameters (a, b, …, k).

The schematic diagram of the logic unit of the controller is shown in the figure I.

The mutual identification and control method of the equipment comprises the following steps:

1. after each device is started, the information waiting for the device is issued on the network at certain time intervals until the device is discovered by the controller;

2. after the controller discovers a device, it sends a series of inquiry questions to the device via the network, determines the range of services and capabilities that the device can provide, and classifies the device. The classification is mainly performed by the recognition and classification unit, taking into account factors including, but not limited to:

a. equipment type, manufacturer and model;

b. a system, a processor, a memory;

c. network speed, screen type, direction, size;

d. other capabilities of the device related to the application;

e. device user application-related attributes, and the like.

Each category is individually grouped into a group. A group contains one or more functionally similar devices.

3. The controller sends out a control instruction, and the control instruction is sent out aiming at the group and can reach all group members; the control command can be an action or a synchronous command; the action instructions include, but are not limited to:

a. a player data source address;

b. parameters are connected with a data server in an abutting mode to complete necessary steps of user authentication, connection initialization, content authorization and the like;

c. initiating a player action, such as start playing;

d. reading the current situation and condition of the equipment;

e. changing the attributes of the device, such as output interpretation degree;

the synchronous instruction is to add the reserved time into the action instruction and force the instruction to start to execute at the reserved time;

4. the equipment receiving the instruction executes the instruction according to an agreed rule;

5. the controller continues to monitor the new device and repeats the above operations.

The group sending instruction (3) can be realized by the following modes:

6. the method for sending the single-thread instruction in the group comprises the following steps:

a. acquiring a contact way or a connection port of the playing equipment;

b. sending an instruction to the playing device;

c. repeating the instruction of all group members of the a and b straight paths to finish sending;

d. waiting for a return status for each instruction;

e. the processing may be erroneous;

f. repeating d, e until all instructions return to state.

Note that this is a linear and asynchronous instruction sending mode, and the instruction sending thread continuously sends out instruction without waiting for instruction return, so as to avoid idle of the instruction sending thread and maximally utilize the waiting time.

7. For a group containing a plurality of devices, a limited number of threads are used to form a thread pool, and a plurality of groups are simultaneously instructed in a thread concurrent mode. If the thread pool has N threads, sending instructions to N groups at the same time; in a group, sequentially sending instructions to each device, and the method is the same as the step (1); after one group is completed, the thread immediately processes the next group. The size N of the thread pool depends on the hardware configuration of the controller, and it is recommended that 2 to 4 threads correspond to one CPU core, too many threads will generate extra thread management scheduling overhead, and too few threads will cause the CPU to run insufficiently and slow down the processing speed. The optimal number of threads can be obtained through experiments;

8. the error of any device in the step (7) is considered as the failure of the group transmission;

9. if there is an error, the user is prompted for the error, and the user decides whether to retry or abort.

The logic flow diagram of the thread executing step (7, 8, 9) is shown in figure (two).

Note that multi-device synchronization imposes a much higher demand on instruction transmission speed than one-to-one control. The faster the transmission speed, the shorter the time interval between the first and last device receiving the command, the shorter the time the synchronization action needs to wait, and the finer the synchronization can be performed. Let us see an example of a web-lesson: the teacher needs to play a section of the movie and explain the movie while playing. In order to avoid the synchronization of the sound taught by the teacher and the video seen by the students, the video on all devices is required to start playing at the same time. If the instructions are sent slowly, the time for waiting for all the instructions to be sent out is long, and the class giving efficiency is greatly reduced. Therefore, optimization of the transmission speed of the synchronous command is not indispensable,

(7) the presence of (b) is not only beneficial for solving the problem, but is absolutely necessary from the practical point of view. It is supposed that if dozens of devices in a classroom need tens of minutes to finish the mass sending of playing instructions, the class is not used. Therefore, the superposition effect of the instruction transmission delay brought by multiple devices is needed to be seriously faced by all control schemes.

Combining the above descriptions, the overall process flow of the embodiment specific to each type of equipment is as follows:

1, a controller:

a. after starting, the information waiting for the device is issued on the network at certain time intervals, and the device waits for echoes from other devices;

b. for each replied device, inquiring parameters and attributes of the device in a question-answering mode, and dividing the device into groups according to classification rules;

c. sending instructions to the player and the data server;

d. monitoring the instruction execution state of the equipment, and making a response to the error;

e. the above steps are repeated.

2, the player:

a. after starting, the information waiting for the user is issued on the network at certain time intervals;

b. after receiving the calling information of the controller, providing various information such as required equipment, service, users and the like for the controller in a conventional mode;

c. continuing to wait for the operation instruction;

d. after receiving the instruction sent by the controller, executing the instruction, and waiting for the trigger time if the instruction is a synchronous instruction;

e. handling errors that can be handled locally;

f. if an error which cannot be solved locally is encountered, reporting the error to the controller;

g. the above steps are repeated.

Data server (server has fixed address):

a. after starting, entering a waiting state, waiting for a controller initialization instruction, or responding to a data service request;

b. receiving a controller initialization instruction, and executing a server initialization program;

c. receiving a service request, and providing data or data stream service according to the parameters and protocol specification of the request;

d. the above steps are repeated.

Description of the drawings:

FIG. 1: schematic diagram of controller logic unit.

FIG. 2: flow diagram of mass issue instruction.

FIG. 3: a remote classroom system with one-to-many control capability is illustrated.

FIG. 4: the abstract of the specification is a flow diagram.

The implementation case is as follows:

figure (three) is a schematic diagram of a remote classroom system application with one-to-many control capability.

The application is a remote classroom using a mobile tablet and a PC. In the application, a teacher and a student respectively have a mobile tablet or a computer, a classroom uses a smart television as a white board, and all devices are networked; there is a teacher in the classroom who uses a PC, a plurality of students who use a tablet, and a smart tv that is used as an electronic whiteboard. In this application, the textbook, practice problem, reference book and other classroom data are all from a server, the content is multimedia, the server has a fixed address, and the multimedia content is output by HTTP protocol. All equipment is provided with software adopting the one-to-many control and system method, equipment discovery, identification and classification between the flat panel and the television and between the flat panel and the computer are automatic, a teacher PC plays a role in controlling the equipment discovery, identification and classification work needs the identification and classification unit 104, and the communication unit 103 is combined for completing the equipment discovery, identification and classification work. Once discovered, the student device establishes a point-to-point connection with the controller, and the communication unit is mainly responsible for the part of the work. When in class, the teacher opens the lecture, shares the screen content of the teacher to the flat panel and the television screen of the student, and simultaneously shares the head portrait video of the teacher to the student; these teaching actions of the teacher on their own computer include, but are not limited to: opening a file, turning pages, switching windows, playing videos and marking circles on the screen generate controller instructions, and the controller instructions are sent to all students and televisions to update the contents of the screens of the students and the televisions. This part of the work group needs to be completed by the instruction generating unit 102 and the communication unit 103 in cooperation. Teachers may also distribute exercises, share pictures, web pages, and the like. The students can share the findings, answers and the like with classmates under the condition of being allowed by the teachers.

In this case, the student tablet has two major systems, namely iOS and Android, and since the video stream formats accessed by the student tablet are different, it is necessary to distinguish the two different tablet systems; in addition, some students are in the same class with the teacher, and the rest are in remote class at other places, so that the students in class do not see the head portrait video of the teacher, and therefore, it is necessary to distinguish two classes of students with different geographic addresses. Finally we need four categories: respectively iOS-local, iOS-remote, Android-local, and Android-remote. The word expression of the classification rule is:

"if the device system is X, the location is Y, the device belongs to the group X-Y"

Wherein X, Y is a string variable, X allows the value "iOS" or "Android", and Y allows the value "local", or "remote". The video access instruction sent by the teacher computer can have corresponding parameters according to different characteristics of the four groups.

To summarize:

under the multi-device application environment, the cooperation between devices has many limitations if the traditional one-to-one mode is adopted for sequential control. This involves that the controlled devices may have different device capabilities, take different roles, have different permissions, etc., which is much more complex than a one-to-one scenario. In addition, there is a serious potential performance bottleneck for the control commands to be sent to multiple devices, and a more ingenious solution is needed.

The discussion of the publications, also lacking details of classifying packets in a multi-device scenario, makes it difficult to find a consensus of efficient collaboration of multiple devices.

To address these problems, we propose a one-to-many control and coordination method. The core problem solved by the method is how to carry out rule-based classification according to the characteristics of the equipment, and then control instructions are sent to the equipment which is already divided into the groups in a unique mode to cooperatively complete the task.

It is desirable to disclose the method and system, which is helpful for the colleagues to derive the inspiration from the method and system, and promote the rapid development of the intelligent device cooperation technology.