阅读说明：本技术 一种用于认知无线网络的切片资源分配方法及系统 (Slice resource allocation method and system for cognitive wireless network ) 是由 郑伟军 张勇 陈嵘 王文华 郄文博 马滕腾 钱啸 陈鼎 方景辉 吴国庆 唐锦江 于 2020-05-26 设计创作，主要内容包括：本发明实施例提供一种用于认知无线网络的切片资源分配方法及系统。该方法包括：基于增强型移动宽带切片和超高可靠超低时延通信切片,建立认知无线网络切片资源分配模型；基于Actor-Critic深度强化学习算法对所述认知无线网络切片资源分配模型进行深度强化学习,得到切片资源分配最优解；其中,所述Actor-Critic深度强化学习算法包括定义用户状态和当前时刻到下一时刻的动作,并由所述用户状态和所述动作构建系统奖励函数。本发明实施例通过在认知网络资源分配中,结合切片技术和Actor-Critic深度强化学习算法,在有限的频谱资源和受限的发射功率情况下,对资源进行最优分配,使得系统吞吐量最大。(The embodiment of the invention provides a slice resource allocation method and system for a cognitive radio network. The method comprises the following steps: establishing a cognitive wireless network slice resource allocation model based on the enhanced mobile broadband slice and the ultra-high reliable ultra-low time delay communication slice; performing deep reinforcement learning on the cognitive wireless network slice resource allocation model based on an Actor-Critic deep reinforcement learning algorithm to obtain an optimal slice resource allocation solution; the Actor-Critic deep reinforcement learning algorithm comprises a user state and an action from the current moment to the next moment, and a system reward function is constructed by the user state and the action. In the embodiment of the invention, in the cognitive network resource allocation, the slicing technology and the Actor-Critic deep reinforcement learning algorithm are combined, and the resources are optimally allocated under the conditions of limited spectrum resources and limited transmitting power, so that the system throughput is maximum.)

1. A slice resource allocation method for a cognitive wireless network, comprising:

establishing a cognitive wireless network slice resource allocation model based on the enhanced mobile broadband slice and the ultra-high reliable ultra-low time delay communication slice;

performing deep reinforcement learning on the cognitive wireless network slice resource allocation model based on an Actor-Critic deep reinforcement learning algorithm to obtain an optimal slice resource allocation solution; the Actor-Critic deep reinforcement learning algorithm comprises a user state and an action from the current moment to the next moment, and a system reward function is constructed by the user state and the action.

2. The slice resource allocation method for the cognitive wireless network according to claim 1, wherein the Actor-critical based depth reinforcement learning algorithm is used for performing depth reinforcement learning on the cognitive wireless network slice resource allocation model to obtain an optimal slice resource allocation solution; the Actor-Critic deep reinforcement learning algorithm includes defining a user state and an action from a current time to a next time, and constructing a system reward function from the user state and the action, and the Actor-Critic deep reinforcement learning algorithm further includes:

and obtaining a full-connection neural network model to construct an Actor-critical deep reinforcement learning algorithm network.

3. The slice resource allocation method for the cognitive radio network according to claim 1 or 2, wherein the establishing of the cognitive radio network slice resource allocation model based on the enhanced mobile broadband slice and the ultra-high-reliability ultra-low-latency communication slice specifically comprises:

defining the throughput of a master user of an enhanced mobile broadband and the interruption probability of the master user of ultra-high reliable ultra-low time delay communication;

and defining a system optimization target and a system constraint condition based on the master user throughput and the master user interruption probability and taking the maximum system throughput as a target, and constructing the cognitive wireless network slice resource allocation model.

4. The slice resource allocation method for cognitive wireless networks according to claim 3, wherein the defining of the throughput of the primary user of the enhanced mobile broadband and the interruption probability of the primary user of the ultra-high reliable ultra-low latency communication further comprises:

the master user throughput is obtained by the signal-to-interference-and-noise ratio of any enhanced mobile broadband master user bandwidth and any enhanced mobile broadband master user on any channel, wherein the signal-to-interference-and-noise ratio is obtained by the channel gain from any enhanced mobile broadband master user transmitter to a master user receiver, the channel gain from any enhanced mobile broadband master user transmitter to a secondary user receiver, the transmitting power of any enhanced mobile broadband master user transmitter on any channel and the transmitting power of any enhanced mobile broadband secondary user transmitter on any channel;

the master user interruption probability is obtained by the delay time of any ultra-high reliable ultra-low delay communication user, the maximum delay time of any ultra-high reliable ultra-low delay communication user and the maximum data arrival rate.

5. The slice resource allocation method for the cognitive radio network as claimed in claim 3, wherein the step of defining a system optimization objective and a system constraint condition based on the throughput of the primary user and the interruption probability of the primary user with the maximum system throughput as an objective comprises the steps of:

maximizing the sum of the throughputs of all secondary users in the system as the system optimization goal;

defining the rate of any enhanced mobile broadband user not to be lower than a first preset value;

defining the probability that any ultra-high reliable ultra-low delay communication user does not meet low delay to be smaller than a second preset value;

defining that one secondary user can only occupy one channel;

defining that the secondary user transmitter power does not exceed a third preset value.

6. The slice resource allocation method for the cognitive wireless network according to claim 1, wherein the Actor-critical based depth reinforcement learning algorithm is used for performing depth reinforcement learning on the cognitive wireless network slice resource allocation model to obtain an optimal slice resource allocation solution; the Actor-critical deep reinforcement learning algorithm includes defining a user state and an action from a current time to a next time, and a system reward function is constructed by the user state and the action, and specifically includes:

defining all secondary users as intelligent agents and signal-to-interference-and-noise ratio state functions of all primary users at any moment;

based on the SINR state function, obtaining an action function of the intelligent agent from the current moment to the next moment, wherein the action function comprises the state representation of sub-carrier occupied by a sub-user at any moment and the power state representation of the sub-user at any moment;

and setting the reward function of the intelligent agent as the sum of the throughputs of all secondary users, and obtaining the result of the reward function according to whether the enhanced mobile broadband user meets a rate constraint condition and whether the ultra-high-reliability ultra-low time delay communication user meets a power constraint condition.

7. The slice resource allocation method for the cognitive wireless network according to claim 2, wherein the obtaining of the fully-connected neural network model to construct an Actor-critical deep reinforcement learning algorithm network specifically comprises:

acquiring a three-layer linear neural network, wherein the number of neurons of an input layer is a first preset parameter, the number of neurons of a middle hidden layer is a second preset parameter, the input layer and the middle hidden layer adopt ReLU as an activation function, the number of neurons of an output layer is a third preset parameter, and the output layer adopts sigmoid and softmax as activation functions;

and respectively constructing an Actor network and a criticic network based on the three layers of linear neural networks.

8. A system for slice resource allocation for cognitive wireless networks, comprising:

the construction module is used for establishing a cognitive wireless network slice resource allocation model based on the enhanced mobile broadband slice and the ultra-high-reliability ultra-low time delay communication slice;

the solving module is used for carrying out deep reinforcement learning on the cognitive wireless network slice resource allocation model based on an Actor-Critic deep reinforcement learning algorithm to obtain an optimal slice resource allocation solution; the Actor-Critic deep reinforcement learning algorithm comprises a user state and an action from the current moment to the next moment, and a system reward function is constructed by the user state and the action.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps of the slice resource allocation method for cognitive wireless networks according to any of claims 1 to 7.

10. A non-transitory computer readable storage medium, having a computer program stored thereon, wherein the computer program, when being executed by a processor, implements the steps of the slice resource allocation method for cognitive wireless networks according to any one of claims 1 to 7.

Technical Field

The invention relates to the technical field of wireless communication, in particular to a slice resource allocation method and system for a cognitive wireless network.

Background

With the rapid development of wireless communication technology, the use of wireless devices (e.g., vehicles, mobile phones, tablets, and various wireless sensors) has increased rapidly, promoting the development of fifth generation (5G) wireless communication, in which 5G wireless networks, the data rate is expected to be 10 times higher than the current rate, and strong connectivity and 100% coverage are expected to provide better quality of service and user experience. While in practice spectrum resources are limited, spectrum usage is regulated for security and stability considerations. Spectrum access rights are typically granted to licensed users and unlicensed users are not allowed to send and receive data over unlicensed regions of the spectrum. Therefore, a contradiction occurs between the limitation of spectrum resources and the increase of the number of users, and how to intelligently allocate resources in the cognitive wireless network becomes a hot point of research.

In cognitive wireless networks, unauthorized users (secondary users) are allowed to communicate within a licensed region of the spectrum as long as that portion of the spectrum is not used by authorized users (primary users). The network slicing technology is one of important characteristics of a 5G network, essentially, a physical network of an operator is divided into a plurality of virtual networks by the network slicing technology, each virtual network is divided according to different service requirements, such as time delay, safety, bandwidth, reliability and the like, three application fields are provided by flexibly coping with different application scenes, such as the 5G network, and the enhanced mobile broadband, the ultra-high reliable ultra-low time delay communication and the large-scale Internet of things meet different communication characteristics and communication requirements.

In addition, reinforcement learning is a branch of artificial intelligence, also called reinforcement learning, and refers to a kind of problem of continuously learning from (with) interaction and a method for solving the problem. The reinforcement learning problem may be described as an agent continuously learning from interactions with the environment to accomplish a particular goal (e.g., to achieve a maximum reward value). At present, the reinforcement learning algorithm is widely applied in the fields of games, communication, medicine and the like. The existing common reinforcement learning methods are divided into two categories, namely, modeled reinforcement learning and modeless reinforcement learning, and according to the characteristic of complex communication scene, the modeless reinforcement learning method is generally adopted, and the modeless reinforcement learning is specifically subdivided into the following categories: the strategy learning method based on the value function comprises a dynamic programming method, a Monte Carlo method, a time sequence difference learning method, a Q learning method and a deep Q learning method; the learning method based on the strategy function comprises a reinforcee algorithm and a reinforcee algorithm with a reference line. Generally, a method based on a value function, such as a Q learning method, may cause over-estimation during policy updating, has a certain influence on convergence, and is good at handling the dispersion problem, whereas a method based on a policy function is more stable during policy updating, but the latter method is difficult to perform sufficient sampling because the solution space of the policy function is larger, causes a larger variance, and is easy to converge to a locally optimal solution.

Disclosure of Invention

The embodiment of the invention provides a slice resource allocation method and system for a cognitive radio network, which are used for solving the problems in the prior art or at least partially solving the problems.

In a first aspect, an embodiment of the present invention provides a slice resource allocation method for a cognitive wireless network, including:

establishing a cognitive wireless network slice resource allocation model based on the enhanced mobile broadband slice and the ultra-high reliable ultra-low time delay communication slice;

performing deep reinforcement learning on the cognitive wireless network slice resource allocation model based on an Actor-Critic deep reinforcement learning algorithm to obtain an optimal slice resource allocation solution; the Actor-Critic deep reinforcement learning algorithm comprises a user state and an action from the current moment to the next moment, and a system reward function is constructed by the user state and the action.

Further, performing deep reinforcement learning on the cognitive wireless network slice resource allocation model based on an Actor-Critic deep reinforcement learning algorithm to obtain an optimal slice resource allocation solution; the Actor-Critic deep reinforcement learning algorithm includes defining a user state and an action from a current time to a next time, and constructing a system reward function from the user state and the action, and the Actor-Critic deep reinforcement learning algorithm further includes:

and obtaining a full-connection neural network model to construct an Actor-critical deep reinforcement learning algorithm network.

Further, the establishing of the cognitive radio network slice resource allocation model based on the enhanced mobile broadband slice and the ultra-high reliable ultra-low time delay communication slice specifically includes:

defining the throughput of a master user of an enhanced mobile broadband and the interruption probability of the master user of ultra-high reliable ultra-low time delay communication;

and defining a system optimization target and a system constraint condition based on the master user throughput and the master user interruption probability and taking the maximum system throughput as a target, and constructing the cognitive wireless network slice resource allocation model.

Further, the defining of the throughput of the primary user of the enhanced mobile broadband and the primary user interruption probability of the ultra-high-reliability ultra-low-delay communication further includes:

the master user throughput is obtained by the signal-to-interference-and-noise ratio of any enhanced mobile broadband master user bandwidth and any enhanced mobile broadband master user on any channel, wherein the signal-to-interference-and-noise ratio is obtained by the channel gain from any enhanced mobile broadband master user transmitter to a master user receiver, the channel gain from any enhanced mobile broadband master user transmitter to a secondary user receiver, the transmitting power of any enhanced mobile broadband master user transmitter on any channel and the transmitting power of any enhanced mobile broadband secondary user transmitter on any channel;

the master user interruption probability is obtained by the delay time of any ultra-high reliable ultra-low delay communication user, the maximum delay time of any ultra-high reliable ultra-low delay communication user and the maximum data arrival rate.

Further, defining a system optimization target and a system constraint condition based on the throughput of the main user and the interruption probability of the main user and with the maximum system throughput as a target, and constructing the cognitive radio network slice resource allocation model, specifically comprising:

maximizing the sum of the throughputs of all secondary users in the system as the system optimization goal;

defining the rate of any enhanced mobile broadband user not to be lower than a first preset value;

defining the probability that any ultra-high reliable ultra-low delay communication user does not meet low delay to be smaller than a second preset value;

defining that one secondary user can only occupy one channel;

defining that the secondary user transmitter power does not exceed a third preset value.

Further, performing deep reinforcement learning on the cognitive wireless network slice resource allocation model based on an Actor-Critic deep reinforcement learning algorithm to obtain an optimal slice resource allocation solution; the Actor-critical deep reinforcement learning algorithm includes defining a user state and an action from a current time to a next time, and a system reward function is constructed by the user state and the action, and specifically includes:

defining all secondary users as intelligent agents and signal-to-interference-and-noise ratio state functions of all primary users at any moment;

based on the SINR state function, obtaining an action function of the intelligent agent from the current moment to the next moment, wherein the action function comprises the state representation of sub-carrier occupied by a sub-user at any moment and the power state representation of the sub-user at any moment;

and setting the reward function of the intelligent agent as the sum of the throughputs of all secondary users, and obtaining the result of the reward function according to whether the enhanced mobile broadband user meets a rate constraint condition and whether the ultra-high-reliability ultra-low time delay communication user meets a power constraint condition.

Further, the obtaining of the fully-connected neural network model to construct an Actor-critical deep reinforcement learning algorithm network specifically includes:

acquiring a three-layer linear neural network, wherein the number of neurons of an input layer is a first preset parameter, the number of neurons of a middle hidden layer is a second preset parameter, the input layer and the middle hidden layer adopt ReLU as an activation function, the number of neurons of an output layer is a third preset parameter, and the output layer adopts sigmoid and softmax as activation functions;

and respectively constructing an Actor network and a criticic network based on the three layers of linear neural networks.

In a second aspect, an embodiment of the present invention provides a slice resource allocation system for a cognitive wireless network, including:

the construction module is used for establishing a cognitive wireless network slice resource allocation model based on the enhanced mobile broadband slice and the ultra-high-reliability ultra-low time delay communication slice;

the solving module is used for carrying out deep reinforcement learning on the cognitive wireless network slice resource allocation model based on an Actor-Critic deep reinforcement learning algorithm to obtain an optimal slice resource allocation solution; the Actor-Critic deep reinforcement learning algorithm comprises a user state and an action from the current moment to the next moment, and a system reward function is constructed by the user state and the action.

In a third aspect, an embodiment of the present invention provides an electronic device, including:

the chip resource allocation method for the cognitive wireless network comprises the following steps of a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the steps of any one of the chip resource allocation methods for the cognitive wireless network.

In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements any of the steps of the slice resource allocation method for a cognitive wireless network.

According to the slice resource allocation method and system for the cognitive radio network, provided by the embodiment of the invention, in the cognitive network resource allocation, the slice technology and the Actor-Critic deep reinforcement learning algorithm are combined, and the resources are optimally allocated under the conditions of limited spectrum resources and limited transmitting power, so that the system throughput is maximum.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

Fig. 1 is a flowchart of a slice resource allocation method for a cognitive wireless network according to an embodiment of the present invention;

fig. 2 is a schematic diagram of an Actor and critical network provided in an embodiment of the present invention;

fig. 3 is a structural diagram of a slice resource allocation system for a cognitive wireless network according to an embodiment of the present invention;

fig. 4 is a block diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Aiming at the defects in the prior art, the embodiment of the invention provides a slice resource allocation method for a cognitive radio network, which realizes the joint power and channel allocation of secondary users while ensuring the service of a main user, and enables the throughput of all secondary users in the system to be maximum.

Fig. 1 is a flowchart of a slice resource allocation method for a cognitive radio network according to an embodiment of the present invention, as shown in fig. 1, including:

s1, establishing a cognitive radio network slice resource allocation model based on the enhanced mobile broadband slice and the ultra-high reliable ultra-low time delay communication slice;

s2, performing deep reinforcement learning on the cognitive wireless network slice resource allocation model based on an Actor-Critic deep reinforcement learning algorithm to obtain an optimal slice resource allocation solution; the Actor-Critic deep reinforcement learning algorithm comprises a user state and an action from the current moment to the next moment, and a system reward function is constructed by the user state and the action.

Specifically, resource allocation of two application scenes in a 5G network, namely enhanced mobile broadband and ultra-high-reliability ultra-low-delay communication, is considered, network resource slicing is carried out, corresponding resource allocation problems are mapped into a general reinforcement learning algorithm model, a cognitive radio network slice resource allocation model is established, optimization objectives and constraint conditions of the system are included, further, a deep reinforcement learning resource allocation method-CNAC algorithm based on Actor-Critic is provided, a reward function setting mechanism is provided, the constraint conditions and the optimization objectives in the cognitive radio network slice resource allocation model are simultaneously placed into a reward function, and an optimal resource allocation solution of the cognitive radio system is obtained through solving.

In the embodiment of the invention, in the cognitive network resource allocation, the slicing technology and the Actor-Critic deep reinforcement learning algorithm are combined, and the resources are optimally allocated under the conditions of limited spectrum resources and limited transmitting power, so that the system throughput is maximum.

Based on the above embodiment, the method further includes, before step S2:

and obtaining a full-connection neural network model to construct an Actor-critical deep reinforcement learning algorithm network.

Specifically, the Actor-Critic based deep reinforcement learning algorithm adopts a neural network structure, and comprises two networks, namely an Actor network and a Critic network, which adopt the same network structure.

Based on any of the above embodiments, step S1 in the method specifically includes:

defining the throughput of a master user of an enhanced mobile broadband and the interruption probability of the master user of ultra-high reliable ultra-low time delay communication;

and defining a system optimization target and a system constraint condition based on the master user throughput and the master user interruption probability and taking the maximum system throughput as a target, and constructing the cognitive wireless network slice resource allocation model.

Wherein, the definition of the throughput of the master user of the enhanced mobile broadband and the interrupt probability of the master user of the ultra-high reliable ultra-low time delay communication further comprises:

the master user throughput is obtained by the signal-to-interference-and-noise ratio of any enhanced mobile broadband master user bandwidth and any enhanced mobile broadband master user on any channel, wherein the signal-to-interference-and-noise ratio is obtained by the channel gain from any enhanced mobile broadband master user transmitter to a master user receiver, the channel gain from any enhanced mobile broadband master user transmitter to a secondary user receiver, the transmitting power of any enhanced mobile broadband master user transmitter on any channel and the transmitting power of any enhanced mobile broadband secondary user transmitter on any channel;

the master user interruption probability is obtained by the delay time of any ultra-high reliable ultra-low delay communication user, the maximum delay time of any ultra-high reliable ultra-low delay communication user and the maximum data arrival rate.

The method comprises the following steps of establishing a cognitive radio network slice resource allocation model by defining a system optimization target and a system constraint condition based on the throughput of a main user and the interruption probability of the main user and taking the maximum system throughput as a target, and specifically comprises the following steps:

maximizing the sum of the throughputs of all secondary users in the system as the system optimization goal;

defining the rate of any enhanced mobile broadband user not to be lower than a first preset value;

defining the probability that any ultra-high reliable ultra-low delay communication user does not meet low delay to be smaller than a second preset value;

defining that one secondary user can only occupy one channel;

defining that the secondary user transmitter power does not exceed a third preset value.

Specifically, a cognitive radio network slice resource allocation model is first established, where enhanced mobile broadband (eMBB) slice users and ultra-high reliable ultra-low latency communication (URLLC) slice users are considered.

Defining the throughput of a master user m of the eMBB slice to meet the following conditions:

c_m,k(t)≥μ₀，m∈M₁

wherein the content of the first and second substances,

c_m,k(t) represents the data transmission rate of the mth user on channel k, g_m,kAnd g_nm,kRespectively representing the channel gain from the transmitter of primary user m to the receiver of primary user m and the channel gain from the transmitter of primary user m to the receiver of secondary user n, p_m(k) Indicating the transmission power, p, of the primary user transmitter on the k-th channel_n,k(t) represents the transmit power of the secondary user transmitter on the k-th channel, B_mDenotes the bandwidth of user m, B denotes the bandwidth of the whole cognitive system, μ₀Representing the lowest throughput requirement for user m.

For URLLC slice users, assuming that the arrival process of slice 2 user packets can be represented by an M/M/1/∞ queuing system and that the packet length follows an exponential distribution, the probability of interruption for primary user M ism∈M₂. Wherein d is_mRepresenting the delay time of user m, d_m,βRepresents the maximum delay time, r, of user m_mRepresenting the maximum data arrival rate.

Further, the allocation method aims at the maximum system throughput, and the proposed optimization aims and constraint conditions are as follows:

here, constraint C₁Indicating that eMB slice user rate can not be lower than mu at the lowest₀，C₂The probability of representing that the URLLC slice user does not meet the low delay is less than the minimum value tau, C₃Meaning that one channel can be occupied by at most one secondary user, C₄Transmit power constraints for the secondary user transmitters.

Based on any of the above embodiments, step S2 in the method specifically includes:

defining all secondary users as intelligent agents and signal-to-interference-and-noise ratio state functions of all primary users at any moment;

based on the SINR state function, obtaining an action function of the intelligent agent from the current moment to the next moment, wherein the action function comprises the state representation of sub-carrier occupied by a sub-user at any moment and the power state representation of the sub-user at any moment;

and setting the reward function of the intelligent agent as the sum of the throughputs of all secondary users, and obtaining the result of the reward function according to whether the enhanced mobile broadband user meets a rate constraint condition and whether the ultra-high-reliability ultra-low time delay communication user meets a power constraint condition.

Specifically, on the basis of the foregoing embodiment, a deep reinforcement learning resource allocation method based on Actor-Critic, namely a CNAC algorithm, is proposed, in which all secondary users are regarded as an agent, SINRs of all primary users at time t are used as states, and s is used as a state_tExpressed as:

s_t＝{SINR₁(t),SINR₂(t),...,SINR_M(t)}

intelligent slave s_tTo s_t+1The actions taken are represented as:

indicating the situation where the secondary user occupies a sub-carrier at time t,

representing the secondary user power situation.

Since the problem is aimed at maximizing the cognitive system throughput and considering the difference in user traffic requirements, i.e., different constraints, for the two slices, the reward function r(s) of the agent is used here according to the lagrange duality method_t,a_t) Setting as sum of throughputs of all secondary users

If the eMBB user meets the rate constraint and the URLLC user meets the power constraint, the reward is set as the sum of the throughput of the secondary user; when the eMBB user does not meet the rate constraint requirement, the reward is set to be 0; when URLLC user does not reach the rate requirement, reward is set to 0.

Based on any one of the above embodiments, the obtaining of the fully-connected neural network model to construct the Actor-Critic reinforcement learning algorithm network specifically includes:

acquiring a three-layer linear neural network, wherein the number of neurons of an input layer is a first preset parameter, the number of neurons of a middle hidden layer is a second preset parameter, the input layer and the middle hidden layer adopt ReLU as an activation function, the number of neurons of an output layer is a third preset parameter, and the output layer adopts sigmoid and softmax as activation functions;

and respectively constructing an Actor network and a criticic network based on the three layers of linear neural networks.

Specifically, the Actor-Critic deep reinforcement learning algorithm network includes two networks, which are an Actor network and a Critic network respectively, as shown in fig. 2, the Actor is based on Policy algorithm and has a function of making a decision, the Critic evaluates the decision of the Actor, generates a TD error according to a state, an action and a reward, and then guides the decision after the Actor.

It can be understood that the Actor network and Critic network adopt the same neural network structure, the CNAC algorithm neural network part, the main body adopts a three-layer linear neural network, the neuron number of the input layer is 16, the activation function is relu, the neuron number of the middle hidden layer is 30, the activation function is relu, the neuron number of the output layer is 12, and two activation functions of sigmoid and softmax are used.

The neural network adopts the dropout technology, the generalization capability of the network is improved, meanwhile, the variance of the network is reduced, and the occurrence of overfitting is prevented. In order to speed up the training of the network, an AdamaOptimizers optimizer is adopted in the back propagation process of the network.

Based on any one of the embodiments, a simulation experiment is performed on the basis of the embodiment of the invention, and a comparison experiment is performed by using a DQN (deep Qnetwork) algorithm, and the experiment result shows that the proposed CNAC algorithm result can be converged more quickly and is better in stability and interruption rate.

Fig. 3 is a structural diagram of a slice resource allocation system for a cognitive radio network according to an embodiment of the present invention, as shown in fig. 3, including: a construction module 31 and a solution module 32; wherein:

the construction module 31 is used for establishing a cognitive radio network slice resource allocation model based on the enhanced mobile broadband slice and the ultra-high reliable ultra-low time delay communication slice; the solving module 32 is used for carrying out deep reinforcement learning on the cognitive wireless network slice resource allocation model based on an Actor-Critic deep reinforcement learning algorithm to obtain an optimal slice resource allocation solution; the Actor-Critic deep reinforcement learning algorithm comprises a user state and an action from the current moment to the next moment, and a system reward function is constructed by the user state and the action.

The system provided by the embodiment of the present invention is used for executing the corresponding method, the specific implementation manner of the system is consistent with the implementation manner of the method, and the related algorithm flow is the same as the algorithm flow of the corresponding method, which is not described herein again.

In the embodiment of the invention, in the cognitive network resource allocation, the slicing technology and the Actor-Critic reinforcement learning algorithm are combined, and the resources are optimally allocated under the conditions of limited spectrum resources and limited transmitting power, so that the system throughput is maximum.

Fig. 4 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 4: a processor (processor)410, a communication Interface 420, a memory (memory)430 and a communication bus 440, wherein the processor 410, the communication Interface 420 and the memory 430 are communicated with each other via the communication bus 440. The processor 410 may call logic instructions in the memory 430 to perform the following method: establishing a cognitive wireless network slice resource allocation model based on the enhanced mobile broadband slice and the ultra-high reliable ultra-low time delay communication slice; performing deep reinforcement learning on the cognitive wireless network slice resource allocation model based on an Actor-Critic deep reinforcement learning algorithm to obtain an optimal slice resource allocation solution; the Actor-Critic deep reinforcement learning algorithm comprises a user state and an action from the current moment to the next moment, and a system reward function is constructed by the user state and the action.

In addition, the logic instructions in the memory 430 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

In another aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented to perform the transmission method provided in the foregoing embodiments when executed by a processor, and for example, the method includes: establishing a cognitive wireless network slice resource allocation model based on the enhanced mobile broadband slice and the ultra-high reliable ultra-low time delay communication slice; performing deep reinforcement learning on the cognitive wireless network slice resource allocation model based on an Actor-Critic deep reinforcement learning algorithm to obtain an optimal slice resource allocation solution; the Actor-Critic deep reinforcement learning algorithm comprises a user state and an action from the current moment to the next moment, and a system reward function is constructed by the user state and the action.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.