阅读说明：本技术 通用透明代理方法 (Universal transparent proxy method ) 是由 贺桂友 柳遵梁 于 2019-08-27 设计创作，主要内容包括：本发明公开了一种通用透明代理方法，包括如下步骤：通过Linux机器连接客户端和服务端；配置用户脚本文件，启动透明代理服务；数据接收发送模块接收数据包，修改数据包，然后上发Freebsd协议栈进行处理，最后上发异步网络框架模块后进入用户层，进行用户层数据处理。本发明具有如下有益效果：本发明使用Freebsd协议栈处理数据，提高了效率，部署简单，开发便捷；在Freebsd协议栈基础上提供异步网络开发接口库Libnet和库LibProxy，Libnet处理网络数据收发，LibProxy用于对Libnet封装，将网络复杂情况交给Libnet处理，便于开发人员进行二次开发。(The invention discloses a universal transparent proxy method, which comprises the following steps: connecting a client and a server through a Linux machine; configuring a user script file and starting a transparent proxy service; and the data receiving and sending module receives the data packet, modifies the data packet, sends a Freebsd protocol stack for processing, and finally sends an asynchronous network frame module and then enters a user layer for processing the data of the user layer. The invention has the following beneficial effects: the invention uses the Freebsd protocol stack to process data, improves the efficiency, has simple deployment and convenient development; the method is characterized in that an asynchronous network development interface library Libnet and a library LibProxy are provided on the basis of a Freebsd protocol stack, the Libnet processes network data receiving and transmitting, the LibProxy is used for packaging the Libnet and handing network complex conditions to the Libnet for processing, and developers can conveniently carry out secondary development.)

1. A universal transparent proxy method is characterized by comprising an IGB network card driving module (1), a data receiving and sending module, a Freebsd protocol stack (2) and an asynchronous network frame module (3); the method comprises the following steps:

(1-1) connecting a client (4) and a server (5) through a Linux machine, starting a Linux system, and automatically running an automatic script;

(1-2) configuring a user script file, operating a user program, starting a DPDK service and starting a transparent proxy service;

(1-3) the data receiving and sending module receives the data packet, modifies the data packet and then sends the Freebsd protocol stack;

(1-4) the Freebsd protocol stack processes the modified data packet and then sends the data packet to the asynchronous network framework module;

and (1-5) the data packet is sent to an asynchronous network frame module and then enters a user layer (8) for user layer data processing.

2. The generic transparent proxy method of claim 1, wherein the specific steps of step (1-2) are as follows:

(2-1) configuring network card information by using script config.ini;

(2-2) configuring the maximum connection number supported by the transparent proxy, the number of the local proxy port and the number of the network card in and out by using script config.yaml;

(2-3) using server.yaml to configure the IP and Port of the proxy server;

and (2-4) running a user program, starting DPDK service and starting transparent proxy service.

3. The generic transparent proxy method of claim 2, wherein the specific steps of step (1-3) are as follows:

(3-1) the data receiving and sending module receives the data packet from the IGB network card driving module, analyzes the data packet, judges according to the network card in-out number configured by the script config.yaml,

if the received data packet is the client direction data, taking out the destination IP and the destination Port field, and turning to the step (3-2);

if the received data packet is the data in the direction of the server, taking out the source IP and the source Port field, and turning to the step (3-2);

(3-2) forming the extracted IP and Port fields into a key, performing Hash search, filtering a data packet according to the IP and Port fields of the proxy server configured by the script server.yaml, if the key does not conform to the configured IP and Port fields of the proxy server, performing two-layer forwarding on the data packet, and turning to the step (3-1); if the key is in accordance with the IP and Port fields of the configured proxy server, the step (3-3) is carried out;

(3-3) the packet judges whether it is an initial request or not based on the SYN and ACK tags,

if the data packet is a primary request of the client, storing a network card number, a network card MAC, a source IP, a source Port field, a destination IP, a destination Port field and a Vlan tag field into a hash table, and turning to the step (3-4);

if the data packet is not the primary request, the step (3-4) is carried out;

and (3-4) replacing the target IP and target Port fields in the client request data packet according to the local IP and the local proxy Port configured by the configuration files config.yaml and config.yaml, recalculating the check code, and uploading the Freebsd protocol stack.

4. The generic transparent proxy method of claim 1, wherein the specific steps of step (1-4) are as follows:

(4-1) the Freebsd protocol stack analyzes the protocol of the data packet and discards illegal data packets;

(4-2) after receiving the data packet of the TCP three-way handshake of the client, the Freebsd protocol stack sends a response data packet;

(4-3) extracting the destination IP and destination Port fields of the response data packet to form a key, searching a Hash table of the key formed by the IP and Port fields requested by the client,

if the search is successful, indicating that the response data packet is sent to the client side, replacing the source IP and source Port fields of the response data packet to be the IP of the service side and Port fields of the service side, and turning to the step (4-5);

if the search fails, the source IP and the source Port field of the response data packet are taken out to form a Key, and the step (4-4) is carried out;

(4-4) looking up a Hash table of a key consisting of a local IP and a Port which establish TCP connection between a local machine and a server side,

if the search is successful, indicating that the response data packet is sent to the server side, replacing the source IP and source Port fields of the response data packet as the IP of the client side and Port fields of the client side; turning to the step (4-5);

if the search fails, no data is replaced, and the number 0 network port is directly sent;

and (4-5) delivering the data packet with the replaced IP and Port fields to a network card, carrying out check code hardware operation by the network card, and uploading the data packet to the asynchronous network frame module.

5. The generic transparent proxy method of claim 1, wherein the specific steps of step (1-5) are as follows:

(5-1) the data packet enters a user layer after being uploaded to the asynchronous network framework module, the LibProxy library establishes bidirectional connection between the client and the server by using a Libnet function, and the Libnet function is an asynchronous function in the asynchronous network framework module;

(5-2) receiving and transmitting data from and to the client using the LibProxy library function;

and (5-3) receiving data from the server and transmitting data to the server by using a LibProxy library function.

6. The generic transparent proxy method of claim 1, 2, 3, 4 or 5, further comprising the steps of:

(6-1) carrying out validity check and security verification on the data sent by the client;

and (6-2) carrying out sensitive information desensitization processing on data sent by the server.

Technical Field

The invention relates to the technical field of network communication, in particular to a universal transparent proxy method which is simple in configuration, convenient and fast to deploy and convenient for secondary development.

Background

The traditional transparent agent method needs to perform rule configuration on a firewall such as iptables, build a network bridge and control a user program, is very dependent on a network environment, needs to be redeployed when the network environment is changed in the later period, is very inconvenient to implement, and needs to know the field network topology environment in detail; acquiring related information such as vlan id and the like according to a field network topology environment; a Vlan number needs to be configured for bond binding; the multi-Vlan network and the load balancing multi-line node are not supported; the later maintenance cost is large.

Disclosure of Invention

The invention provides a universal transparent proxy method which is simple in configuration, convenient and fast in deployment and convenient for secondary development, and aims to overcome the defects that implementation and deployment are inconvenient and the network environment is complex in the prior art.

In order to achieve the purpose, the invention adopts the following technical scheme:

a universal transparent proxy method comprises an IGB network card driving module, a data receiving and sending module, a Freebsd protocol stack and an asynchronous network frame module; the method comprises the following steps:

(1-1) connecting a client and a server through a Linux machine, starting a Linux system, and automatically running an automatic script;

(1-2) configuring a user script file, operating a user program, starting a DPDK service and starting a transparent proxy service;

(1-3) the data receiving and sending module receives the data packet, modifies the data packet and then sends the Freebsd protocol stack;

(1-4) the Freebsd protocol stack processes the modified data packet and then sends the data packet to the asynchronous network framework module;

and (1-5) the data packet is sent to an asynchronous network framework module and then enters a user layer for user layer data processing.

The invention establishes bidirectional connection between the client and the server, and sends the data of the client to the server, and the data of the server to the client, the proxy server only plays a transfer role in the middle, but is transparent, and both the server and the client can not sense the existence of the transparent proxy server.

Preferably, the specific steps of step (1-2) are as follows:

(2-1) configuring network card information by using script config.ini;

(2-2) configuring the maximum connection number supported by the transparent proxy, the number of the local proxy port and the number of the network card in and out by using script config.yaml;

(2-3) using server.yaml to configure the IP and Port of the proxy server;

and (2-4) running a user program, starting DPDK service and starting transparent proxy service.

Preferably, the specific steps of step (1-3) are as follows:

(3-1) the data receiving and sending module receives the data packet from the IGB network card driving module, analyzes the data packet, judges according to the network card in-out number configured by the script config.yaml,

if the received data packet is the client direction data, taking out the destination IP and the destination Port field, and turning to the step (3-2);

if the received data packet is the data in the direction of the server, taking out the source IP and the source Port field, and turning to the step (3-2);

(3-2) forming the extracted IP and Port fields into a key, performing Hash search, filtering a data packet according to the IP and Port fields of the proxy server configured by the script server.yaml, if the key does not conform to the configured IP and Port fields of the proxy server, performing two-layer forwarding on the data packet, and turning to the step (3-1); if the key is in accordance with the IP and Port fields of the configured proxy server, the step (3-3) is carried out;

(3-3) the packet judges whether it is an initial request or not based on the SYN and ACK tags,

if the data packet is a primary request of the client, storing a network card number, a network card MAC, a source IP, a source Port field, a destination IP, a destination Port field and a Vlan tag field into a hash table, and turning to the step (3-4);

if the data packet is not the primary request, the step (3-4) is carried out;

and (3-4) replacing the target IP and target Port fields in the client request data packet according to the local IP and the local proxy Port configured by the configuration files config.yaml and config.yaml, recalculating the check code, and uploading the Freebsd protocol stack.

Preferably, the specific steps of steps (1-4) are as follows:

(4-1) the Freebsd protocol stack analyzes the protocol of the data packet and discards illegal data packets;

(4-2) after receiving the data packet of the TCP three-way handshake of the client, the Freebsd protocol stack sends a response data packet;

(4-3) extracting the destination IP and destination Port fields of the response data packet to form a key, searching a Hash _ Port _ recHash table, wherein the Hash _ Port _ recv is the Hash table of the key formed by the IP and Port fields requested by the client,

if the search is successful, indicating that the response data packet is sent to the client side, replacing the source IP and source Port fields of the response data packet to be the IP of the service side and Port fields of the service side, and turning to the step (4-5);

if the search fails, the source IP and the source Port field of the response data packet are taken out to form a Key, and the step (4-4) is carried out;

(4-4) looking up a Hash table of a key consisting of a local IP and a Port which establish TCP connection between a local machine and a server side,

if the search is successful, indicating that the response data packet is sent to the server side, replacing the source IP and source Port fields of the response data packet as the IP of the client side and Port fields of the client side; turning to the step (4-5);

if the search fails, no data is replaced, and the number 0 network port is directly sent;

and (4-5) delivering the data packet with the replaced IP and Port fields to a network card, carrying out check code hardware operation by the network card, and uploading the data packet to the asynchronous network frame module.

Preferably, the specific steps of step (1-5) are as follows:

(5-1) the data packet enters a user layer after being uploaded to the asynchronous network framework module, the LibProxy library establishes bidirectional connection between the client and the server by using a Libnet function, and the Libnet function is an asynchronous function in the asynchronous network framework module;

(5-2) receiving and transmitting data from and to the client using the LibProxy library function;

and (5-3) receiving data from the server and transmitting data to the server by using a LibProxy library function.

Preferably, the method further comprises the following steps:

(6-1) carrying out validity check and security verification on the data sent by the client;

and (6-2) carrying out sensitive information desensitization processing on data sent by the server.

Therefore, the invention has the following beneficial effects: the invention uses the Freebsd protocol stack to process data, improves the efficiency, has simple deployment and convenient development; the method is characterized in that a conventional asynchronous network development interface library Libnet and a library LibProxy are provided on the basis of a Freebsd protocol stack, the Libnet processes network data receiving and transmitting, the LibProxy is used for packaging the Libnet, only a data processing interface is provided, and network complex conditions are handed to the Libnet for processing, so that the complexity of a network environment is overcome, and developers can conveniently perform secondary development.

Drawings

FIG. 1 is a flow chart of the present invention;

FIG. 2 is a network location diagram of the present invention;

fig. 3 is a timing diagram of the operation of the present invention.

In the figure: the system comprises an IGB network card driving module 1, a Freebsd protocol stack 2, an asynchronous network frame module 3, a client 4, a server 5, a transparent proxy server 6, a network 7 and a user layer 8.

Detailed Description

The invention is further described in the following detailed description with reference to the drawings in which:

the embodiment shown in fig. 1 is a general transparent proxy method, and includes an IGB network card driver module (1), a data receiving and sending module, a Freebsd protocol stack (2), and an asynchronous network framework module (3); the method comprises the following steps:

step 100, connecting a client and a server through a Linux machine, starting a Linux system, and automatically running an automatic script;

as shown in fig. 2, firstly, two network cables are plugged into the Linux machine, one is for the client (4) and the other is for the server (5), the Linux operating system is started, the server and the client are connected through the transparent proxy server (6) and the network (7), an automatic script is automatically run, and the script comprises the insertion of an IGB network card drive module, the configuration of a large-page memory and the binding of the network card.

(1) IGB network card drive module: IGB _ uio.ko appears when the source code of the Intel development tool is compiled in a centralized manner, a command insmod IGB _ uio.ko is loaded into a Linux kernel, and an IGB network card driving module is used for realizing the functions of controlling and operating the network card to send and receive data.

(2) Configuring a large-page memory: the default of the traditional Linux memory page is 4KB in size, so that the TLB Cache Miss (page table buffer loss) can be caused, the size of the modified page is 2MB, the TLB Cache Miss can be reduced, and the method is a performance optimization technology, and the specific operation command is as follows:

mkdir-p/mnt/huge

mount-t hugetlbfs nodev/mnt/huge

echo2048>/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_h ugepages

(3) binding a network card: after the Linux operating system is started, the kernel can take over the network card, the kernel does not control the network card any more, but gives management to the IGB network card driver module, so that the network card needs to be bound to the IGB driver module, and the control right is given to the IGB driver module with the following commands:

./dpdk-devbind.py--bind＝igb_uio mc1

./dpdk-devbind.py--bind＝igb_uio mc2

mc1 and mc2 are network card names, and dpdk-devidbind. py is a binding script file;

step 200, configuring a user script file, running a user program, starting DPDK service and starting transparent proxy service;

step 201, configuring network card information by using script config.ini, wherein the network card information comprises IP, subnet mask, gateway and CPU mask; IP and gateway address are private addresses, CPU mask is the CPU core on which the appointed program runs, preventing the task from switching among a plurality of CPUs and causing performance loss;

step 202, configuring the maximum connection number supported by the transparent proxy, the number of the local proxy port and the network card in and out by using script config.yaml; the maximum number of connections depends on the system hardware resources; the local proxy port is a TCP port to which the client requests data to be sent; the configuration of the network card access number is to distinguish whether the data is in the client direction or the server direction, so as to optimize the code to analyze the data packet and avoid repeated inquiry;

step 203, using server.yaml to configure the IP and Port of the proxy server; IP and Port form Key for Hash search;

step 204, running the user program, starting the DPDK service, and starting the transparent proxy service, where the program command is as follows:

./user_program 0>log0 2>&1&

./arp-p 0-s 10.10.10.254 10:0c:29:86:48:53

initializing physical hardware configuration by using int ret rte _ eal _ init (argc, argv) when a program is started; init _ lcore _ conf () sets the running CPU core; the init _ mem _ pool () configuration program uses the memory; initializing the network card by ret ═ init _ port _ start (), setting specific parameters, such as automatically calculating check codes; using rte _ Hash _ create (& Hash _ params) to create a fast Hash lookup table for packet transceiving IP and Port fast replacement; configuring the relationship between the gateway ip and the mac by using a special arp command so as to send out a data packet and start transparent proxy service;

step 300, a data receiving and sending module receives a data packet, modifies the data packet and then sends a Freebsd protocol stack;

step 301, the data receiving and sending module receives the data packet from the IGB network card driving module, analyzes the data packet, and determines according to the network card access number configured by the script config.yaml,

if the received data packet is client direction data, the destination IP and the destination Port field are taken out, and the step 302 is carried out;

if the received data packet is the data of the server side direction, taking out the source IP and the source Port field, and turning to step 302;

step 302, forming the extracted IP and Port fields into a key, performing Hash search, filtering a data packet according to the IP and Port fields of the proxy server configured by the script server.yaml, if the key does not conform to the IP and Port fields of the configured proxy server, performing two-layer forwarding on the data packet, and turning to step 301; if the key conforms to the configured IP and Port fields of the proxy server, go to step 303;

step 303, the data packet determines whether it is the initial request according to the SYN and ACK tags,

if the data packet is a primary request of the client, storing a network card number, a network card MAC, a source IP, a source Port field, a destination IP, a destination Port field and a Vlan tag field into a hash table, and turning to step 304;

if the data packet is not the primary request, go to step 304;

and step 304, replacing the target IP and the target Port field in the client request data packet according to the local IP and the local agent Port configured by the configuration file config.yaml and config.yaml, recalculating the check code, and uploading the Freebsd protocol stack.

Step 400, the Freebsd protocol stack processes the modified data packet and then sends the asynchronous network frame module;

step 401, the Freebsd protocol stack performs protocol analysis on the data packet, and discards an illegal data packet;

step 402, after receiving a data packet of a three-way handshake of a client TCP, a Freebsd protocol stack sends a response data packet;

step 403, extracting the destination IP and the destination Port field of the response packet to form a key, looking up a Hash table of the key formed by the IP and the Port field requested by the client,

if the search is successful, indicating that the response data packet is sent to the client side, replacing the source IP and source Port fields of the response data packet as the IP of the service side and Port fields of the service side, and turning to step 405; if the search fails, the source IP and the source Port field of the response data packet are taken out to form a Key, and the step 404 is carried out;

step 404, look up Hash _ Port _ send Hash table, Hash _ Port _ send is Hash table of key composed by local IP and Port which establishes TCP connection between local and server,

if the search is successful, indicating that the response data packet is sent to the server side, replacing the source IP and source Port fields of the response data packet as the IP of the client side and Port fields of the client side; go to step 405;

if the search fails, no data is replaced, and the number 0 network port is directly sent;

step 405, the data packet with the replaced IP and Port fields is delivered to the network card, the network card performs check code hardware operation, and the asynchronous network frame module is uploaded.

Step 500, the data packet is sent to an asynchronous network frame module and then enters a user layer (8) for user layer data processing;

step 501, a data packet is uploaded to an asynchronous network frame module and then enters a user layer, a LibProxy library establishes bidirectional connection between a client and a server by using a Libnet function, and the Libnet function is an asynchronous function in the asynchronous network frame module;

the LibProxy library accepts new client connection by using a Libnet asynchronous function void async _ accept (mc _ tcp _ ptcp, callback _ cb); establishing new connection from a local machine to a server by using void async _ connect (end _ point & dpkp, end _ point & ep, callback _ cb cb cb), and establishing bidirectional connection between a client and a server;

step 502, using a LibProxy library function to receive data from a client and send data to the client;

collecting data from the client by using a void onRecvFromClient (std: week _ ptr < BaseSessessionPtr, const char pData and uint32_ t cbSize) function in the Libproxy library; sending data to a client by using an int32_ tsendToClient (const BasePuptTr & pktPtr) function;

step 503, using LibProxy library function to receive data from the server and send data to the server;

collecting data from the server by using a void onRecvFromServer (std: week _ ptr < BaseSessessino > sessionPtr, const char pData and uint32_ t cbSize) function in the Libproxy library; the int32_ tsendToServer (const BasePKPtr & pktPtr) function sends data to the server.

And the LibProxy library receiving and sending functions respectively call the Libnet library receiving and sending functions. The Libnet asynchronous network library API is the same as the traditional network development API. When a user closes the TCP connection, the Libnet can automatically clear the bottom-layer hash invalid information record, and the user layer does not sense the hash invalid information record.

The Libnet part interface is as follows

Void async_accept(mc_tcp*ptcp,callback_cb)；

Void async_connect(end_point&dpdkep,callback_cb cb)；

Void async_connect(end_point&dpdkep,end_point&ep,callback_cb cb)；

Void async_read(std::shared_ptr<buffer>pbuffer,callback_cb cb)；

Void async_write(std::shared_ptr<buffer>pbuffer,callback_cb cb)；

Void async_close(callback_cb cb)；

Void async_close_dpdk(callback_cb cb)；

Void async_add_protect_server(end_point ep,callback_cb cb)；

Bool is_close()；

The data of the client side is sent to the server side, the data of the server side is sent to the client side, the proxy server plays a transfer role in the middle, and the proxy server is transparent and does not exist for the two sides; in the actual development process, the transparent proxy server adds a filtering mechanism according to the self service requirement; for example, the data sent by the client is subjected to validity check and security verification, so that hackers are prevented from invading the system. And carrying out sensitive information desensitization processing on data sent by the server, and preventing information leakage and the like.

As shown in fig. 3, after the client sends a TCP connection and filters the proxy data packet, the IP and PORT of the data packet are replaced, the check code is recalculated, the data packet enters the Freebsd protocol stack, the three-way handshake is completed, the asynchronous network framework module is notified, the user layer is uploaded, a new connection is established, and at this time, the user sends a new connection to the server, and in the process, the user does not need to do too many operations, and only needs to receive and send data to the SOCKET, and all IP and PORT replacement is already completed by operations at the receiving and sending entries at the bottom layer; the HASH table is automatically updated, invalid data are automatically cleared after connection is disconnected, and an upper layer user cannot sense the invalid data.

It should be understood that this example is for illustrative purposes only and is not intended to limit the scope of the present invention. Further, it should be understood that various changes or modifications of the present invention may be made by those skilled in the art after reading the teaching of the present invention, and such equivalents may fall within the scope of the present invention as defined in the appended claims.