阅读说明：本技术 用于所存储内容的网络控制媒体上传的方法和系统 (Method and system for network controlled media upload of stored content ) 是由 R·荣 D·朗布莱德 P·伊斯贝格 于 2019-07-26 设计创作，主要内容包括：当多个流传输电子设备被注册为在相同流传输事件内时,媒体制作元件(12)组合来自多个流传输电子设备的媒体内容。媒体制作元件(12)可以使用多个电子设备提供的流传输内容的选择和组合来制作组合的实时视频流或组合的媒体作品。参与事件的上行链路数据流传输的电子设备可以将数据段实时地发送到媒体制作元件(12),并且它还可以在本地存储数据段。网络辅助设备(42)或媒体制作元件(12)可以从电子设备请求非实时存储的数据段。(A media production element (12) combines media content from multiple streaming electronic devices when the multiple streaming electronic devices are registered within the same streaming event. The media production element (12) may produce a combined real-time video stream or combined media production using a selection and combination of streaming content provided by a plurality of electronic devices. An electronic device participating in the uplink data streaming of the event may send the data segments to the media production element (12) in real time, and it may also store the data segments locally. The network auxiliary device (42) or the media production element (12) may request the non-real-time stored data segments from the electronic device.)

1. A method for providing data from an electronic device (14) to a media production service in a network, the method comprising:

registering, by the electronic device (14), an uplink data streaming session to an event, wherein the data streaming session comprises a plurality of data segments and the event is live, in which live a plurality of electronic devices registered to the event simultaneously compose respective uplink media streams and transmit the respective uplink media streams to the media composition service;

storing, by the electronic device (14), a data segment of the plurality of data segments in a local memory, wherein the data segment corresponds to a time period during the event;

receiving, by the electronic device (14), a request to transmit the stored data segment; and

in response to receiving the request, transmitting, by the electronic device (14), the stored data segment to the media production service.

2. The method of claim 1, further comprising:

receiving, by the electronic device (14), event coordinated uplink network assistance data from a network assistance service, wherein the event coordinated uplink network assistance data relates to the time period during the event; and

modifying, by the electronic device (14), at least one transmission characteristic of the data streaming session during the time period based on the event coordinating uplink network assistance data.

3. The method of claim 1 or 2, wherein the request to transmit the stored data segment is received from a network assistance service.

4. The method of claim 1 or 2, wherein the request to transmit the stored data segment is received from the media production service.

5. The method of any of claims 2 to 4, wherein modifying at least one transmission characteristic of the data streaming session involves suspending the data streaming session during the time period.

6. The method of any of claims 2 to 4, wherein modifying at least one transmission characteristic of the data streaming session involves transmitting a low quality version of the data segment during the time period, and

storing the data segment involves storing a high quality version of the data segment in the local memory.

7. The method of any of claims 1-6, further comprising: a query is received from a network assistance service, wherein the query requests confirmation of whether the stored data segment is available.

8. The method of claim 7, further comprising: in response to receiving the query confirming whether the stored data segment is available, sending a confirmation notification to the network assistance service.

9. The method according to any of claims 2 to 8, wherein the network assisted service is implemented in a base station (30) or connected to the base station (30).

10. An electronic device comprising a wireless interface (18) and control circuitry (32), the control circuitry (32) configured to perform the method of any of claims 1-9.

11. A method for providing event coordinated uplink network assistance to a plurality of electronic devices in a network during an event, the event being live, by a network assistance service, the plurality of electronic devices being registered to the live and the plurality of electronic devices simultaneously producing respective uplink media streams from the live and sending the respective uplink media streams to a media production service, the method comprising:

receiving, by the network assistance service, a registration request from an electronic device of the plurality of electronic devices for a data streaming session corresponding to the event, wherein the data streaming session comprises a plurality of data segments; and

transmitting, by the network assistance service, a request to the electronic device for a stored data segment of the plurality of data segments, the stored data segment corresponding to a time period during the event.

12. The method of claim 11, further comprising:

determining, by the network assistance service, event coordinated uplink network assistance data related to the time period during the event, wherein the event coordinated uplink network assistance data is based at least in part on network resources available for the plurality of data streaming sessions during the time period; and

transmitting, by the network assistance service, the event coordinated upstream network assistance data to the electronic device.

13. The method of claim 12, wherein the event coordinated uplink network assistance data comprises a command to suspend the data streaming session during the time period.

14. The method of claim 12, wherein the event-coordinated uplink network assistance data comprises a command to transmit a low quality version of the data segment during the time period, and

the request made to the electronic device for the stored data segment corresponding to the time period is a request for a high quality version of the data segment.

15. The method of any of claims 11-14, wherein the network assistance service is hosted by a DASH-aware network element (DANE) (42).

16. The method of any of claims 11 to 14, wherein the network assistance service is hosted by a framework of real-time uplink streaming (FLUS).

17. The method of any of claims 11 to 16, further comprising:

determining, by the network assistance service, that network resources are insufficient for the electronic device to transmit the data segment during the time period; and

determining, by the network assistance service, that network resources are sufficient for the electronic device to transmit the data segment at a time subsequent to the time period, wherein a request for the stored data segment is transmitted to the electronic device in response to determining that network resources are sufficient.

18. The method of any of claims 11 to 17, further comprising: sending, by the network assistance service, a query to the electronic device, wherein the query requests confirmation of whether the stored data segment is available from the electronic device.

19. The method of claim 18, further comprising: receiving, by the network assistance service, an acknowledgement notification from the electronic device in response to the query, wherein the acknowledgement notification acknowledges that the stored data segment is available, and sending a request for the stored data segment in response to receiving the acknowledgement notification.

20. The method according to any of claims 11-19, wherein the network assisted service is implemented in a base station (30) or connected to the base station (30).

21. A network element (42), the network element (42) hosting a network assistance service, the network element (42) comprising a network interface (52) and a control circuit (44), the control circuit (44) configured to perform the method of any of claims 11-20.

22. A system comprising an electronic device according to claim 10 and a network element (42) according to claim 21.

23. A method of making a composite media work, the method comprising:

receiving, by a media production service, a data segment from each of a plurality of electronic devices participating in an event-coordinated uplink streaming session related to an event, the one or more data segments corresponding to one or more time periods during the event, and the event being a live, the plurality of electronic devices being registered to the live and the plurality of electronic devices simultaneously producing respective uplink media streams from the live and transmitting the respective uplink media streams to a media production service;

requesting, by a media production service, a stored data segment from one of the plurality of electronic devices, the stored data segment corresponding to a time period of the one or more time periods; and

receiving, by a media production service, the stored data segment from the electronic device.

24. The method of claim 23, further comprising:

creating, by a media production service, the composite media work, the creating including inserting the stored data segment into the composite media work for the time period.

25. A media production element (12), the media production element (12) hosting a media production service, the media production element (12) comprising a network interface (28) and control circuitry (20), the control circuitry (20) being configured to perform the method of any of claims 23 to 24.

Technical Field

The technology of the present disclosure relates generally to wireless communication between electronic devices in a network environment, and more particularly to a method of controlling media upload of stored content related to an event having multiple uplink streams and related devices.

Background

In a network, such as a cellular or mobile network, a client device may stream media content to a media production entity. For example, a professional camera or portable electronic device used to report real-time sports, music, or news events may provide an audiovisual stream in an uplink transmission. Other devices may also stream audiovisual content to the network in real-time, such as a wireless handheld device in the form of a smart phone, tablet computer, or similar device.

In these examples, the client device that is the source of the media stream is typically configured to transmit data at the highest bit rate that the client can generate. Such uplink media streaming can create various problems where multiple client devices are streaming media content intended for collaborative media production. One problem is that the network node adjusting the streaming will assume that the streaming media content is only coming from individual clients, and therefore each individual streaming link should be optimized individually.

In the case where a plurality of clients stream data in a coordinated manner, there is a need to further improve streaming.

Disclosure of Invention

Features are disclosed that can enhance the quality of experience (QoE) of an uplink media streaming service if multiple client devices are streaming content intended for collaborative production. The disclosed features may improve the final combined media work and QoE, particularly when operating on a network that shares spectrum resources among multiple users (e.g., in a wireless mobile or cellular network in accordance with any appropriate set of standards, such as those promulgated by the 3 GPP).

According to aspects of the present disclosure, a method for providing data from an electronic device to a media production service in a network, the method comprising: registering a data streaming session corresponding to an event with a network assistance service, wherein the data streaming session comprises a plurality of data segments; storing a data segment of the plurality of data segments in a local memory, wherein the data segment corresponds to a time period during the event; receiving a request to transmit the stored data segment; and sending the stored data segment to the media production service in response to receiving the request.

According to one embodiment of the method, the method further comprises: receiving event coordinated uplink network assistance data from the network assistance service, wherein the event coordinated uplink network assistance data relates to a time period during the event; and coordinating uplink network assistance data based on the event, modifying at least one transmission characteristic of the data streaming session during the time period.

According to one embodiment of the method, the request to transmit the stored data segment is received from the network assistance service.

According to one embodiment of the method, the request to send the stored data segment is received from the media production service.

According to one embodiment of the method, modifying at least one transmission characteristic of the data streaming session involves suspending the data streaming session during the time period.

According to one embodiment of the method, modifying at least one transmission characteristic of the data streaming session involves sending a low quality version of the data segment during the time period, and storing the data segment involves storing a high quality version of the data segment in the local memory.

According to one embodiment of the method, the method further comprises: receiving a query from the network assistance service, wherein the query requests confirmation of whether the stored data segment is available.

According to one embodiment of the method, the method further comprises: in response to receiving the query confirming whether the stored data segment is available, sending a confirmation notification to the network assistance service.

According to one embodiment of the method, the network assisted service is implemented in or connected to a base station.

According to one embodiment, the method is performed by an electronic device.

According to aspects of the present disclosure, a method for providing event coordinated uplink network assistance to a plurality of electronic devices in a network through a network assistance service during streaming of data from the plurality of electronic devices during an event, the method comprising: receiving a registration request from one of the plurality of electronic devices for a data streaming session corresponding to the event, wherein the data streaming session includes a plurality of data segments; and transmitting a request to the electronic device for a stored data segment of the plurality of data segments, the stored data segment corresponding to a time period during the event.

According to one embodiment, the method further comprises: determining event coordinated uplink network assistance data related to the time period during the event, wherein the event coordinated uplink network assistance data is based at least in part on network resources available for the plurality of data streaming sessions during the time period; and transmitting the event coordinated uplink network assistance data to the electronic device.

According to one embodiment, the event coordinated uplink network assistance data comprises a command for suspending the data streaming session during the time period.

According to one embodiment, the event coordinated uplink network assistance data comprises a command to transmit a low quality version of the data segment during the time period, and the request made to the electronic device for the stored data segment corresponding to the time period is a request for a high quality version of the data segment.

According to one embodiment, the network assistance service is hosted by a DASH-aware network element (DANE).

According to one embodiment, the network assistance service is hosted by a framework for real-time uplink streaming (FLUS).

According to one embodiment, the method further comprises: determining that network resources are insufficient for the electronic device to transmit the data segment during the time period; and determining that network resources are sufficient for the electronic device to transmit the data segment at a time subsequent to the time period, wherein, in response to determining that network resources are sufficient, a request for the stored data segment is transmitted to the electronic device.

According to one embodiment, the method further comprises sending a query to the electronic device, wherein the query requests confirmation whether the stored data segment is available from the electronic device.

According to one embodiment, the method further comprises receiving a confirmation notification from the electronic device in response to the query. The acknowledgement confirms that the stored data segment is available and, in response to receiving the acknowledgement, sends a request for the stored data segment.

According to one embodiment, the network assistance service is implemented in or connected to a base station.

According to one embodiment, the network assistance service is hosted by a network element.

According to aspects of the present disclosure, a method of making a composite media work, the method comprising: receiving one or more data segments from one or more electronic devices participating in an event-coordinated uplink streaming session related to an event, the one or more data segments corresponding to one or more time periods during the event; requesting a stored data segment from an electronic device of the one or more electronic devices, the stored data segment corresponding to a time period of the one or more time periods; and receiving the stored data segment from the electronic device.

According to one embodiment, the method further comprises inserting the stored data segment into the combined media piece at the time segment.

According to one embodiment, the method is performed by a media production component.

Drawings

Fig. 1 is a schematic block diagram of a representative network system that performs network-assisted uplink media transport for collaborative media production.

Fig. 2 is a representative high-level architecture diagram of network assistance for uplink media transport in conjunction with media production.

Fig. 3 is a schematic diagram of a representative embodiment of a network-assisted network system that performs uplink media transport for collaborative media production.

Fig. 4 is a schematic diagram of another representative embodiment of a network-assisted network system that performs uplink media transmission for collaborative media production.

Fig. 5 is a schematic diagram of another representative embodiment of a network-assisted network system that performs uplink media transmission for collaborative media production.

Fig. 6 is a diagram illustrating two representative uplink data flows made up of multiple data segments.

Fig. 7 is a signaling diagram of a representative method of network assisted media upload of stored content.

FIG. 8 is a flow diagram of a representative method of providing data from an electronic device to a media production service.

Fig. 9 is a flow diagram of a representative methodology for providing event coordinated uplink network assistance to a plurality of electronic devices via a network assistance service.

FIG. 10 is a flow chart of a representative method of making a composite media work.

Detailed Description

A. Introduction to

Embodiments will now be described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. It will be understood that the figures are not necessarily to scale. Features that are described and/or illustrated with respect to one embodiment may be used in the same way or in a similar way in one or more other embodiments and/or in combination with or instead of the features of the other embodiments.

Various embodiments of systems and methods for performing event coordinated network assisted wireless radio communication including for uplink streaming are described below in conjunction with the appended figures. Event coordinated network assistance for uplink streaming may be performed in an automated manner by various services on the network. Event coordinated network assistance for uplink streaming may improve uplink streaming performance.

B. System architecture

FIG. 1 is a schematic diagram of an exemplary network system 10 for implementing the disclosed technology. It should be understood that the illustrated system is representative and that other systems may be used to implement the disclosed techniques. The exemplary network system 10 includes a media production element 12 for data streamed from an electronic device 14 over an uplink. Media production element 12 will typically be a server, a component of a server, or other device located in and/or managed by the core network of a wireless mobile or cellular network operator. The client for the received media stream may also be reached via the operator's core network or over a more extensive network, such as the internet.

The electronic device 14 transmits data and control signaling via the base station 30 or other access point operating in accordance with cellular or mobile device network protocols, such as, but not limited to, protocols promulgated by the third generation project partnership (3 GPP). Exemplary base stations 30 may be Long Term Evolution (LTE) base stations, commonly referred to as enhanced node bs (enodebs or enbs) or next generation node bs (gnbs). The base station 30 serves one or more electronic devices, including the electronic device 14. The base station 12 may support communication between electronic devices and a network medium 16, through which network medium 16 electronic devices may communicate with other electronic devices, servers, within the operator's core network or connected via the internet or the like. One device with which electronic device 14 may communicate via network medium 16 is media production element 12. It will be understood that other locations of the media production element 12 in the network architecture are possible. As an example of another location of the media production element 12, the media production element 12 may be a server on the Internet for streaming services that are not managed by a wireless network operator.

The source of the streamed data may be any client or terminal of the base station 12. An exemplary electronic device 14 that may be the source of the streamed data may be a User Equipment (UE), as described in accordance with general terminology adopted in the 3GPP specifications. The electronic device 14 may be any kind of electronic device that provides a media stream via a wireless interface 18 (e.g., a 3GPP wireless modem). Exemplary electronic devices 14 include, but are not limited to, smart phones, tablet computers, desktop or laptop computers, cameras, and the like. In the case of a camera, the camera may be included in a drone, may be operated by a person, may be an autonomous surveillance camera, and the like. In one embodiment, the camera may be a professional type camera used by news or real-time event reporting organizations. The electronic device 14 may also be a source of a contribution feed transmitted via an uplink, such as an external radio or mobile studio, that transmits the contribution feed as a set of multiple camera or other device feeds or as pre-made edits and audio mixes of the event being reported. In these contribution feed scenarios, multiple electronic devices may also be providing contribution feeds related to the same event. The camera may include a wireless interface 18 or may be operatively connected to the wireless interface 18 through a local interface, for example using a cable or electrical connector. In the case where the cameras are operated by a news or real-time event report organization, the organization may arrange with the operator of the mobile network to provide enhanced bandwidth capacity to support streaming of audiovisual content. For example, a camera may access multiple carriers in a network to deliver content using carrier aggregation. Wireless networks may also provide sufficient bandwidth capacity for media streams in a single carrier or channel.

Media production element 12 may include operational components for performing wireless communication and other functions of media production element 12. For example, media production element 12 may include control circuitry 20, with control circuitry 20 being responsible for overall operation of media production element 12, including controlling media production element 12 to perform operations applicable to media production element 12, and described in detail below. The control circuit 20 includes a processor 22 that executes code 24, such as an operating system and/or other applications. The functionality described in this disclosure and applicable to media production element 12 may be implemented as part of code 24 or as part of other dedicated logic operations of media production element 12. Depending on the nature and configuration of the media production element 12, the logical functions and/or hardware of the media production element 12 may be implemented in other ways. Thus, the illustrated and described methods are merely examples, and other methods may be used, including, but not limited to, implementing the control circuit 20 as or including hardware (e.g., a microprocessor, a microcontroller, a Central Processing Unit (CPU), etc.) or a combination of hardware and software (e.g., a system on a chip (SoC), an Application Specific Integrated Circuit (ASIC), etc.).

Code 24 and any stored data (e.g., data associated with the operation of media production element 12) may be stored on memory 26. The code 24 may be embodied in the form of an executable logic routine (e.g., a software program) that is stored as a computer program product on a non-transitory computer readable medium (e.g., memory 26) of the media production element 12 and executed by the processor 22. The functions described as being performed by media production element 12 may be considered methods performed by media production element 12.

The memory 26 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, a Random Access Memory (RAM), or other suitable device. In a typical arrangement, the memory 26 includes non-volatile memory for long-term data storage and volatile memory that serves as system memory for the control circuit 20. The memory 26 is considered to be a non-transitory computer readable medium.

The media production element 12 includes communication circuitry that enables the media production element 12 to establish various communication connections. For example, media production element 12 may have a network communication interface 28 to communicate with network medium 16.

The electronic device 14 may include operational components for performing various functions of the electronic device 14, including wireless communication with the base station 30 and any other devices with which the electronic device 14 may communicate. One function of the electronic device 14 is network assisted uplink streaming and/or event coordination network assisted uplink streaming, as will be described in more detail. The electronic device 14 may include, among other components, control circuitry 32, the control circuitry 32 being responsible for overall operation of the electronic device 14, including controlling the electronic device 14 to perform the operations described in detail below. The control circuitry 32 includes a processor 34 that executes code 36, such as an operating system and/or other applications. The functions described in this disclosure may be implemented as part of the code 36 or as part of other dedicated logical operations of the electronic device 14. Depending on the nature and configuration of the electronic device 14, the logical functions and/or hardware of the electronic device 14 may be implemented in other ways. Thus, the illustrated and described methods are merely examples, and other methods may be used, including, but not limited to, implementing the control circuit 32 as or including hardware (e.g., a microprocessor, a microcontroller, a Central Processing Unit (CPU), etc.) or a combination of hardware and software (e.g., a system on a chip (SoC), an Application Specific Integrated Circuit (ASIC), etc.).

The code 36 and any stored data (e.g., data associated with operation of the electronic device 14) may be stored on the memory 38. The code 36 may be embodied in the form of an executable logic routine (e.g., a software program) that is stored as a computer program product on a non-transitory computer readable medium (e.g., memory 38) of the electronic device 14 and executed by the processor 34. The functions described as being performed by the electronic device 14 may be considered a method performed by the electronic device 14.

The memory 38 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, a Random Access Memory (RAM), or other suitable device. In a typical arrangement, the memory 38 includes non-volatile memory for long-term data storage and volatile memory that serves as system memory for the control circuit 32. The memory 38 is considered to be a non-transitory computer readable medium.

The electronic device 14 includes communication circuitry that enables the electronic device 14 to establish various communication connections. For example, the electronic device 14 includes a wireless interface 18, and communicates wirelessly with the base station 30 via the wireless interface 18. Other communications may be established with the electronic device 14, such as Wi-Fi communications, wired connections, and so forth. Wireless interface 18 may include radio circuitry including one or more radio frequency transceivers (also referred to as modems), at least one antenna assembly, and any appropriate tuners, impedance matching circuits, and any other components as needed to support various frequency bands and radio access technologies.

Other components of the electronic device 14 may include, but are not limited to, a user input device (e.g., buttons, keypad, touch surface, etc.), a display, a microphone, a speaker, sensors, jacks or electrical connectors, a rechargeable battery and power supply unit, a SIM card, a motion sensor (e.g., an accelerometer or gyroscope), a GPS receiver, and any other suitable components. In particular, the electronic device 14 may include a camera assembly 40 having all suitable optical and electronic imaging components, as well as components such as a video processor, video and audio encoders, and the like.

In some implementations, the logic (e.g., as implemented by the control circuitry 20) that controls the wireless communication interface 18 is a so-called lower layer protocol architecture and may include a physical layer, a medium access layer, a radio resource control layer. This may be referred to as a modem entity of the electronic device 14. Further, in some implementations, the logic that controls the video media client and corresponding video content encoding, as well as any related functions, is a so-called higher layer architecture and may include an IP layer, an application layer, and the like. The logic may be referred to as an application entity and may include an uplink video streaming client. With this separation in layers, in some implementations, wireless modem entity communications with cellular base station 30 are logically separated from application entity data communications performed with media production element 12.

Network system 10 may include a network auxiliary device 42, and network auxiliary device 42 may provide network auxiliary services to any network elements including electronic device 14 and/or media production element 12 during uplink streaming. In an exemplary embodiment, the network auxiliary device 42 may be a DASH-aware network element (DANE). Although DASH refers to dynamic adaptive streaming over hypertext transfer protocol (HTTP), DANEs may provide network assistance for uplink streaming performed according to additional and/or other protocols, such as HTTP real-time streaming (HLS), real-time transport protocol (RTP), real-time messaging protocol (RTMP), etc. In another exemplary embodiment, the network auxiliary device 42 may be implemented as part of the 3GPP framework for real-time uplink streaming (FLUS).

Network auxiliary device 42 may include operational components for performing various functions of network auxiliary device 42, including providing uplink network auxiliary services to electronic device 14 and/or media production element 12. The network accessory 42 may include, among other components, control circuitry 44, which control circuitry 44 is responsible for overall operation of the network accessory 42, including controlling the network accessory 42 to perform the operations described in detail below. The control circuitry 44 includes a processor 46 that executes code 48, such as an operating system and/or other applications. The functionality described in this disclosure may be implemented as part of code 48 or as part of other dedicated logical operations of network auxiliary device 42. Depending on the nature and configuration of the network auxiliary device 42, the logical functions and/or hardware of the network auxiliary device 42 may be implemented in other ways. Thus, the illustrated and described methods are merely examples, and other methods may be used, including, but not limited to, implementing the control circuit 44 as or including hardware (e.g., a microprocessor, a microcontroller, a Central Processing Unit (CPU), etc.) or a combination of hardware and software (e.g., a system on a chip (SoC), an Application Specific Integrated Circuit (ASIC), etc.).

The code 48 and any stored data (e.g., data associated with the operation of the network accessory 42) may be stored on the memory 50. The code 48 may be embodied in the form of an executable logic routine (e.g., a software program) that is stored as a computer program product on a non-transitory computer readable medium (e.g., memory 50) of the network accessory 42 and executed by the processor 46. The functions described as being performed by the network assistance device 42 may be considered a method performed by the network assistance device 42.

The memory 50 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, a Random Access Memory (RAM), or other suitable device. In a typical arrangement, the memory 50 includes non-volatile memory for long-term data storage and volatile memory that is used as system memory for the control circuit 44. The memory 50 is considered to be a non-transitory computer readable medium.

The network accessory 42 includes communication circuitry that enables the network accessory 42 to establish various communication connections. For example, network auxiliary device 42 includes a network interface 52 through which network interface 52 communicates with media production element 12, electronic device 14, and any other devices via network medium 16.

In the illustrated embodiment, the network auxiliary device 42 is implemented as a server that is out of band of a media delivery path of content transmitted from the electronic device 14 to the media production element 12 via an uplink stream. With the out-of-band architecture, communications between electronic device 14 and network auxiliary device 42 may be independent of communications between electronic device 14 and media production element 12. Thus, communication between electronic device 14 and network accessory 42 may occur in a different communication path and/or data link than the communication path and/or data link used for the uplink flow. In this manner, the network auxiliary device 42 may be provided at multiple locations in the network system 10. For example, the network accessory 42 may communicate with the electronic device 14 through the illustrated base station 30, through a different base station, or directly with the electronic device 14. In another embodiment, the functionality of the network auxiliary device 42 may be incorporated into the media production element 12.

The network system 10 may also include a third party media element 54. Data streamed from the electronic device 14 to the media production component 12 (via the base station 30 and its uplink data path) may be passed to the third party media component 54. In one embodiment, media production element 12 temporarily buffers streaming data received from electronic device 14 and communicates the data over network medium 16 using an appropriate protocol, the details of which need not be discussed in detail for purposes of this disclosure. Where the electronic device 14 is a camera, the third-party media element 54 may be part of a television studio that processes, stores, and/or redistributes audiovisual content received from the electronic device 14 to media players (e.g., televisions, smart phones, handheld devices, etc.). The end user may consume the content on the media player in a "tape-delayed" or "real-time" manner. In other cases, the third party media element 54 may be an end-user device, or may be one of many end-user devices that receive a media stream from the media production element 12 or other device (e.g., a retransmission server connected to the media production element 12) via broadcast or multicast retransmission.

C. Event coordinated network assistance for uplink streaming

Signaling

Fig. 2 is a high-level architecture diagram of an uplink network assistance system that may be implemented for event coordinated uplink network assistance, which shows some of the signal flows of the system and may be referred to in connection with the following description as a representative method for event coordinated uplink network assistance.

In one embodiment, to assist in providing an improved quality of experience for uplink video streaming, a signaling path for Network Assistance (NA) may be established. Thus, to implement the NA functionality, a signaling method between the electronic device 14 and the network accessory 42 may be established. In one embodiment, such a NA signaling method is established between the NA client 56 of the electronic device 14 and the network accessory 42. In the first direction, the network assistance device 42 may transmit event coordinated uplink network assistance data to the NA client 56 of the electronic device 14, as described below. In the second direction, NA client 56 of electronic device 14 may send a message to network accessory 42. Such messages may include event registration requests, requests for event coordinated uplink network assistance data, or device-related information. For example, the device-related information may include information about media quality levels that can be obtained, such as frame rate, video and audio resolution, and so forth.

In one embodiment, the media production element 12 and the uplink client 58 of the electronic device share a data path. In the first direction, the media production element 12 may send a message to the uplink client 58 of the electronic device 14. For example, the media production element 12 message may include a request for a segment of the media stream, a pause or resume stream command, and the like. In the second direction, the uplink client 58 of the electronic device 14 may send an uplink media stream to the media production element 12. In some implementations, media production element 12 may process the media stream from electronic device 14 as well as media streams from other electronic devices to create a combined media production. The combined media work may be sent to the third party media element 54. The third party media element 54 may further edit, view, display, or distribute the combined media production. It should be understood that the communication paths and functions described for network auxiliary device 42 and media production element 12 may exist interchangeably between the two elements, or the communication paths and functions may exist in a single element.

General operation

Some events may involve multiple electronic devices transmitting data simultaneously in an area where a large number of electronic devices are present. In these cases, even though the wireless network may be comprised of multiple base stations, the uplink data capacity is likely to be the limiting factor in the uplink data rate achievable on each electronic device-to-base station link. Also, due to channel fading due to the nature of the radio link, the achievable data rate in the uplink direction may vary over time even without taking into account the total uplink cell load. Using prior art systems, each connected electronic device will try to optimize its own transmission to create as high an individual quality as possible, regardless of any other electronic devices, which requires as much uplink data capacity as is available for each client.

The media production element 12 operates a media production service and may be configured to combine multiple real-time media streams of an event when multiple electronic devices are streaming the multiple real-time media streams. When multiple streaming electronic devices are registered within the same streaming event, media production element 12 combines media content from the multiple streaming electronic devices. The media production element 12 may produce a combined real-time video stream or combined media production using a selection and combination of streaming content provided by a plurality of electronic devices. In these embodiments, the network assistance device 42 may determine event coordination uplink network assistance data with the purpose of preferentially allocating resources to client/electronic devices 14 that are currently used to combine media pieces and allocating less resources to client/electronic devices 14 that are not currently used for those combined media pieces. Rather than optimizing individual links, event coordinated uplink network assistance data is determined by considering aggregate capacity and production requirements of the media production element 12.

Example network architectures for various embodiments are provided in fig. 3-5. In general, several different architectures may be used to implement the different functions of network assistance and media production, and it should be understood that these functions may be implemented as separate nodes or within any other node.

Turning now to fig. 3, the network assistance device 42 and the media production element 12 are implemented on a node that is located on the operator network but separate from the base station 30 (e.g., eNB or gNB). Both network auxiliary device 42 and media production element 12 may communicate with base station 30 and/or directly with electronic device 14. Network auxiliary device 42 and media production element 12 may communicate and provide functionality to base station 30 as well as second base station 60. The base station 30 and the second base station 60 may or may not provide services to electronic devices corresponding to the same event.

Turning now to fig. 4, the network assistance device 42 may be implemented in or connected to the base station 30 (e.g., eNB or gNB) and thus have information about the scheduling load or other indication of the available uplink transmission capacity. In some embodiments, the network assistance device 42 may be implemented in or connected to the second base station 60, and the second base station 60 may or may not provide service to electronic devices corresponding to the same event.

Turning now to fig. 5, the network accessory 42 may be implemented in the base station 30 or connected to the base station 30 as described above with respect to fig. 4. In addition, the media production component 12 may be implemented in a third party media component 54 or connected to the third party media component 54. In these embodiments, the media production element 12 is not part of the carrier network and may be integrated with the third party media element 54.

Regardless of the network architecture arrangement, the event coordinated uplink network assistance system and method may function as described below. A media streaming client 58 residing in a sending device (e.g., electronic device 14) may establish an uplink media streaming session pertaining to a certain event. In addition, one or more additional streaming clients at the event may also establish an uplink media streaming session belonging to the event. Examples of such events are concerts, sporting events and news events. This event registration may be accomplished by a network function, such as a network accessory 42 for video streaming. In one example, the network assistance device 42 may be a DASH-aware network element (DANE), wherein a video streaming session may be initiated. In another example, the network assistance device 42 may be part of a 3GPP framework for real-time uplink streaming (plus). In some embodiments, event registration to create a link between a media streaming sub-device to network auxiliary device 42 may be performed directly between the media streaming sub-device and network auxiliary device 42. For example, the event registration process may be performed using a DNS query to identify the IP address of the associated network auxiliary device 42, or indirectly via another node (such as media production element 12).

In some embodiments, the network auxiliary device 42 and the media production element 12 may be defined as different entities, but they may also be the same network entity, meaning, for example, that registration is performed directly with respect to the network auxiliary device 42, and that the network auxiliary device 42 may also directly include the media production element 12.

When the electronic device 14 is registered/launched into an event, the electronic device 14 will be able to receive event coordination uplink network assistance data specific to the event from the network assistance device 42. The event specific information may include information on a streaming session of the event itself, and may also include information on an overall event to which a plurality of electronic devices may be connected. Such information may include, but is not limited to, information about the number of electronic devices registered to the event, the total allocated network resources for the event, or information about radio link performance over an established link of the radio link (as compared to other client-to-network links). Such information may be provided periodically by the network assistance server in a request-response manner, wherein the electronic device 14 receives event information from the network assistance device 42 based on the request for the information. Information may also be "pushed" from the network accessory 42 to the electronic device 14, i.e., the information may be delivered without an explicit request from the electronic device 42. When the electronic device 14 receives such event-specific information, it may adjust the transmission of its content in order to optimize the overall combined media work and/or optimize the wireless network load and/or reduce energy consumption in the electronic device 14.

In some embodiments, the event coordination uplink network assistance data is shared once during registration. In other embodiments, the event coordinated uplink network assistance data may be shared multiple times. For example, the network assistance device 42 may transmit assistance information at a periodic time, or the network assistance device 42 may respond with assistance information upon receiving an information request from the electronic device 14. The event coordination uplink network assistance data may provide the electronic device 14 with additional information for the streaming session that enables improved streaming performance for the individual electronic device 14 and the entire group of electronic devices that are also registered to the event. For example, the electronic device 14 may use the event coordinated uplink network assistance data to adjust (adapt) its transmission during the streaming event. Such an adjustment may be an increase or decrease in streaming video resolution and/or frame rate, a selection of a media codec, a selection of a media operation point, a selection of a media quality, a selection of a data rate, a selection of a compression level, a pause of a data streaming session, or a resume of a data streaming session. The event coordination uplink network assistance data may depend on the number of connected electronic devices, e.g. the number of electronic devices within a given geographical area or connected to a certain base station, or registered to a certain event. As a benefit to the electronic device 14, this may reduce device energy consumption during streaming. In some implementations, another incentive to register with the production of the event may be to gain access to the final combined media production, and/or to gain revenue share if the combined media production is released.

Further, when the electronic device 14 is registered in a streaming event and is active, the event coordinated uplink network assistance data received by the electronic device 14 may also include an uplink transmission command or recommendation from the network assistance device 42 coordinating the event. Such an uplink transmission command or recommendation may be received as part of a request-response procedure. In particular, the transmission command or recommendation is executed to coordinate a plurality of uplink streaming electronic devices to produce a combined media work. Such coordination may include providing requests/recommendations regarding streaming thereof to the electronic device 14 to enable the network auxiliary device 42, the media production element 12, and/or the third party media element 54 to optimize the combined media production. In some cases, not all uplink media streams registered to an event are each important, so some of the uplink media streams may, for example, have their media rate reduced or suspended at certain times. However, for media production in the third party media component 54 or the media production component 12, it is still important to see as much of the media stream as possible in order to quickly respond to changes in the media stream. The media production element 12 or the third party media element 54 may send media optimization information to the network auxiliary device 42 to indicate the priority of each media stream to the network auxiliary device 42. The network auxiliary device 42 may continuously coordinate the quality level, pause or resume of each stream to provide the best possible input for production to the media production element 12 or third party media element 54, but still alleviate network load.

Some embodiments also include electronic device 14 to network accessory 42 information sharing. In these embodiments, the electronic device 14 may provide the device-related information to the network accessory 42. The device-related information may include information regarding the level of media quality available to electronic device 14. This may include the available frame rate, video resolution, audio resolution, etc. This information may be provided to the network assistance device 42 so that the network assistance device 42 may select or recommend between available media quality levels. This communication may occur when electronic device 14 registers its uplink data flow with network auxiliary device 42 as part of an event.

In addition to providing requests/recommendations to the electronic device 14 regarding its streaming, the network assistance device 42 may also request appropriate priorities in scheduling and/or other network resource allocations from other network functions, including radio access functions. For example, if the network auxiliary device 42 detects that the electronic device registered to the event requires additional resources, such as radio access capacity, in order to provide a media stream of acceptable quality as determined by the media production element 12, the resources may be transferred from other traffic. In other implementations, the network accessory 42 may provide information to other network functions indicating a priority for a given electronic device 14. In this manner, the network auxiliary device 42 may influence the priority of data streams of certain events relative to data streams of other events or relative to other unrelated traffic in the network. Moreover, the network auxiliary device 42 may affect the priority of data associated with certain electronic devices as compared to other electronic devices. In some embodiments, the network accessory 42 or any other network function may provide information to the electronic device 14 participating in events related to anticipated or given network priorities or changes in network priorities. Such information provided to the electronic device 14 may indicate to the electronic device 14 whether event participation affects data priorities of devices within the network. Such information may indicate a time period for such a priority change, e.g., a length of time for which the priority change is valid.

A composite media work may be constructed by selecting segments provided over time by media streams from different electronic devices. In some embodiments, media production element 12 constructs a composite media work. In other embodiments, the third party media element 54 constructs a composite media work. The media production element 12 and/or the third party media element 54 may assign priorities to the various streams based on whether individual streams are currently being used as part of the combined media production, whether the streams may be used as part of the combined media production in the near future, or whether individual streams are not currently needed. Such priorities may be communicated to the network assistance device 42 so that the network assistance device 42 may use the priorities to determine event coordinated uplink network assistance data to send to the corresponding electronic device 14. In an example, if a segment currently selected for combining a media piece is being transmitted from one electronic device 14 at a given point in time, the network assistance device 42 may instruct one or more other electronic devices to transmit uplink segments at a medium to low quality and/or frame rate during that time, as it may want to select any of these electronic devices for upcoming media segments. Further, since the network assistance device 42 may not currently consider any of their uplink media segments important or suitable for use in combining media pieces, the network function may instruct one or more other electronic devices to suspend streaming or not send uplink media segments for a given period of time. The choice of quality/rate level and/or transmission suspension may be decided based on the media stream priority of the media production element 12 and an estimate or measurement of the overall uplink capacity.

In some embodiments, another network/device interaction exists not only in real-time uplink transmissions, but also for stored content. For example, if the uplink data flow of the electronic device 14 has been suspended based on a recommendation from the network assistance service and, therefore, has not been sent in a real-time session during a period of time, the network assistance service may request that the suspended data segment be uploaded at a later time. Further, if the frame rate/resolution of the data during the real-time uplink transmission is limited based on the recommendation from the network assistance service, the electronic device 14 may store higher quality data segments for the corresponding time segments. Higher quality data segments may be requested later (e.g., when network capacity is not limited) by a network-assisted service or by a media production service. As used herein, the terms data stream and data segment may refer to any type of data, including but not limited to audio data, video data, or a combination of audio and video data.

Turning now to FIG. 6, a diagram of a representative embodiment is presented to display a first data stream produced by a first electronic device 62 and a second data stream produced by a second electronic device 64. Two data streams are registered to an event using a network assistance service. During the first time period 66, the first electronic device 62 makes the data segment A1, and the second electronic device 64 makes the data segment B1. For example, a media production service running on media production element 12 may indicate a preference for data segment A1 over data segment B1. In another example, the network assistance service may decide to utilize its network resources to receive content from a particular electronic device (e.g., first electronic device 62) within an event. Events derived from the network assistance service coordinate the uplink network assistance data recommending that the first electronic device 62 upload data segment a1, and the second electronic device 64 suspend uploading data segment B1. However, although the contents of data segment B1 are not transmitted, the second electronic device 64 may store data segment B1 locally within the device. The stored data segments may be stored in the electronic device 64, for example, in the local memory 38 or on a memory card. In some implementations, the network assistance service may send a local storage request to request that the electronic device store content that is not needed by the network assistance service or the media production service locally during the time period of the real-time streaming event. Alternatively, the electronic device may decide to store the content locally on its own, or by default.

During the second time period 68, the media production service or the network assistance service may indicate a preference for data segment B2 over data segment A2. Events derived from the network assistance service coordinate the uplink network assistance data recommending that the second electronic device 64 upload the data segment B2, and the first electronic device 62 suspend uploading the data segment a 2. However, the contents of data segment A2, although not transmitted, may be stored locally within the device. The stored data segments may be stored in the electronic device 62, for example, in the local memory 38 or on a memory card. Similarly, during the third time period 70, the media production service or the network assistance service indicates a preference for data segment a3 or B3, and the electronic device may react as described above.

In another embodiment, the electronic device may transmit the low-quality version of the data segment in real-time, but store the high-quality version of the data segment in the local memory. For example, during the third time period 70, the first electronic device 62 may transmit data segment a3 in real-time at a reduced bit rate due to limited network resources during the real-time uplink streaming session. Even if the first electronic device 62 transmits the data segment A3 at a reduced bit rate, the first electronic device 62 may locally store a high quality version of the data segment A3. When network conditions are more favorable, the network assistance service or the media production service may request that the first electronic device 62 send a high quality version of data segment A3 to the media production service.

After the time period 68 has elapsed, the media production service may decide that it desires a data segment that has not yet been uploaded as part of the real-time uplink data stream. In some cases, the media production service or network assistance service may request that the stored data segment be sent when network conditions are more favorable. For example, the media production service may decide that it desires data segment A2. The media production service and the network assistance service may interact with the electronic device to request and receive the stored data segments in a manner described below.

Turning now to fig. 7, the network assistance protocol may include functions and commands for requesting stored content from devices participating in an uplink streaming event. The network assistance service on the network assistance device 42 may send a query 72 to the first electronic device 62 asking if the stored data segment is available. In some implementations, the query 72 may include a query for the availability of multiple data segments (e.g., B1, A2, and B3). In response to receiving the query 72, the first electronic device 62 may determine whether the stored data segment is locally available. If the requested stored data segment is locally available, the first electronic device 62 may send a confirmation notification 74 to the network assistance service. After receiving the confirmation notification 74, the network assistance service may send a request 76 to the first electronic device 62. The request 76 requests the first electronic device 62 to send the stored data segment to the media production service on the media production component 12. In some embodiments, the request 76 may include a request for multiple available data segments. In response to the request 76, the first electronic device 62 initiates transmission 78 of the stored data segment that has been requested to the media production service.

In some embodiments, the method may avoid (bypass) queries 72 and confirmation notifications 74. In these embodiments, the network assistance service sends the request 76 to the first electronic device 62. If the requested data segment is not available, the first electronic device 62 may send a message to the network assistance service informing the network assistance service that the data segment is not available. If the requested data segment is available, the first electronic device 62 may proceed with the sending 78 of the data segment to the media production service.

Another feature relates to the network assistance service and/or the media production service sending event specific information to the event registration electronic device 14 for display to a user of the electronic device as part of a user interface visible on the display 14 of the electronic device. The event specific information may include an indication of the number of devices connected in the event, a request to start or stop a shot, a custom message from an administrator, and an indication of whether segments are currently being uploaded from a device. Further, the network assistance service may indicate to the electronic device 14 that certain requests for the electronic device 14 should be provided to such a user interface (if possible). In this manner, the network assistance service may specifically request that the electronic device 14 provide specific information or commands to the user of the electronic device 14, or indicate that the network assistance service does not allow the electronic device 14 to share information within the user interface. In further embodiments, the network assistance service or media production service may send control signals to the electronic device 14 to control aspects of the electronic device 14, such as camera commands including pan, tilt, zoom, or focus commands, record commands, pause commands, and stop commands, among others.

Fig. 8 and 9 depict methods performed by the network elements disclosed herein. It should be understood that any of the following methods may be performed alone, simultaneously, or in combination with each other.

Turning now to fig. 8, an embodiment of a method for providing data from an electronic device to a media production service in a network is disclosed. At reference numeral 80, the electronic device registers a data streaming session corresponding to the event with the network assistance service. The data streaming session includes a plurality of data segments. At reference numeral 82, the electronic device stores a data segment of the plurality of data segments in a local memory. The data segment corresponds to a time period during the event. At reference numeral 84, the electronic device receives a request to transmit the stored data segment. The request may be received from a network assistance service or a media production service. At reference numeral 86, the electronic device sends the stored data segment to the media production service in response to receiving the request.

Turning now to fig. 9, an embodiment of a method for providing event coordinated uplink network assistance to a plurality of electronic devices in a network during streaming of data from the plurality of electronic devices during an event by a network assistance service is disclosed. At reference numeral 90, a network assistance service receives a registration request for a data streaming session corresponding to an event from an electronic device of a plurality of electronic devices. The data streaming session includes a plurality of data segments. At reference numeral 92, the network assistance service sends a request to the electronic device for a stored data segment of the plurality of data segments. The stored data segment corresponds to a time period during the event.

Turning now to FIG. 10, an embodiment of a method for making a combined media work is disclosed. At reference numeral 100, a media production service receives one or more data segments from one or more electronic devices participating in an event-coordinated upstream streaming session related to an event. The one or more data segments correspond to one or more time segments during the event. At reference numeral 102, a media production service requests a stored data segment from one of one or more electronic devices. The stored data segment corresponds to one of the one or more time segments. At reference numeral 104, the media production service receives the stored data segment from the electronic device. In some implementations, the media production service may then insert the stored data segments into the combined media production at corresponding time periods.

D. Conclusion

Although certain embodiments have been shown and described, it is understood that equivalents and modifications that fall within the scope of the appended claims will occur to others skilled in the art upon the reading and understanding of this specification.