阅读说明：本技术 综合检测大数据漏洞和不安全配置的脆弱性扫描系统 (Vulnerability scanning system for comprehensively detecting big data bugs and unsafe configurations ) 是由 董丽萍 余睿渊 于 2021-08-10 设计创作，主要内容包括：本申请实施例提供一种综合检测大数据漏洞和不安全配置的脆弱性扫描系统,包括：资产管理模块、漏洞检测模块、配置核查模块、报表管理模块、快捷升级模块、分布式管理模块；本申请能够对主流大数据组件进行漏洞扫描和安全配置合规性检查,包括Hadoop、Spark、Hbase、Solr、ES等,从而能够及时发现大数据组件中存在的安全漏洞和不安全配置,及时通过安全加固,提升大数据平台的安全保障水平,满足如等级保护、行业规范等政策法规的安全建设要求。(The embodiment of the application provides a vulnerability scanning system for comprehensively detecting big data bugs and unsafe configurations, which comprises: the system comprises an asset management module, a vulnerability detection module, a configuration checking module, a report management module, a quick upgrading module and a distributed management module; according to the method and the device, vulnerability scanning and safety configuration compliance checking can be performed on the mainstream big data assembly, and the device comprises Hadoop, Spark, Hbase, Solr, ES and the like, so that safety vulnerabilities and unsafe configurations existing in the big data assembly can be found in time, the safety guarantee level of a big data platform is improved through safety reinforcement in time, and the safety construction requirements of policy and regulations such as level protection and industrial specifications are met.)

1. A vulnerability scanning system to comprehensive detection of big data vulnerabilities and unsafe configurations, the system comprising:

the asset management module is used for discovering a survival host, network equipment and a database in a target network, automatically generating network topology and checking detailed information of each asset;

the vulnerability detection module is used for carrying out security vulnerability detection on the big data assembly and generating vulnerability description and vulnerability repair suggestions;

the configuration checking module is used for carrying out safety configuration compliance check on each component of the big data environment and determining unsafe configuration in the big data platform component;

the report management module is used for analyzing the scanning result in the form of a report and a graph to obtain a vulnerability risk level, a vulnerability category, a vulnerability description, a vulnerability type and a vulnerability solution;

the quick upgrading module is used for carrying out online upgrading, local upgrading and timed upgrading on the leak library and the software through a network or a local data packet;

and the distributed management module is used for issuing a scanning task to the lower engine, receiving a scanning result uploaded by the lower engine, performing unified analysis and generating an integral big data vulnerability scanning report.

2. The comprehensive vulnerability scanning system of detecting big data vulnerabilities and unsafe configurations of claim 1, wherein the vulnerability detection module is further configured to perform security vulnerability detection of remotely executed code vulnerabilities, command injection vulnerabilities, access control vulnerabilities, authority promotion vulnerabilities, denial of service vulnerabilities, information leakage vulnerabilities on big data components.

3. The vulnerability scanning system of comprehensive detection of big data vulnerabilities and unsafe configurations of claim 1, wherein the configuration verification module is further configured to perform a safety configuration compliance check on big data acquisition components, big data storage components, big data processing components of the big data environment, determine unsafe configurations of user access right control, log record integrity, file right minimization, account right minimization, service connection number limitation, transmission encryption, interface authentication in the big data platform component.

4. The comprehensive vulnerability scanning system of detecting big data vulnerabilities and unsafe configurations of claim 1, wherein the distributed management module further comprises:

and the self-evaluation unit is used for automatically creating a scanning task by a subordinate engine, and scanning and risk evaluation on real-time and timed big data vulnerability of the large-scale network.

5. The comprehensive vulnerability scanning system of detecting big data vulnerabilities and unsafe configurations of claim 1, further comprising:

the vulnerability early warning unit is used for notifying a user in a mail or telephone mode when the latest high-risk vulnerability information is published and providing corresponding preventive measures;

the vulnerability scanning unit is used for carrying out vulnerability scanning on the target big data platform, detecting vulnerability and unsafe configuration of the big data by adopting a risk assessment model, finding vulnerability, carrying out priority sequencing and generating a vulnerability scanning report;

the vulnerability fixing unit is used for providing a safe configuration suggestion of the system and an effective downloading link of the patch;

and the vulnerability auditing unit is used for tracking, recording and verifying the effect of vulnerability management, and simultaneously starting a timing scanning task to perform comparative analysis and effect verification.

Technical Field

The application relates to the field of big data, in particular to a vulnerability scanning system for comprehensively detecting big data bugs and unsafe configurations.

Background

With the global acceleration of digital economy and the rapid development of related technologies such as 5G, artificial intelligence, internet of things and the like, the global data volume has exploded. According to statistics and predictions of the international authority Statista, the global data production is predicted to reach 47ZB in 2020, and by 2035, this number will reach 2142 ZB. Big data is remodeling a new world situation, is known as 'diamond mine in 21 century', and is more a national basic strategic resource. Big data in various industries are applied to wind and cloud surge, and the big data plays an increasingly greater role in national economic development. Along with the wide application of big data, the big data security problem is increasingly highlighted.

Since large data packets contain complex sensitive data, more potential attackers are attracted. And a large amount of data is gathered, so that more important data can be obtained by one successful attack, and the attack yield is increased. With the explosion of the internet and big data applications, the system is attacked, the data is lost and the personal information is leaked, and the underground data transaction is black and grey, which also causes a great deal of data abuse and phishing events.

How to help users find the vulnerability of a big data platform component before hacking, and ensuring safe and reliable operation of big data service becomes a problem which needs to be solved urgently.

Disclosure of Invention

Aiming at the problems in the prior art, the vulnerability scanning system for comprehensively detecting the big data vulnerability and unsafe configuration can carry out vulnerability scanning and safety configuration compliance inspection on a main stream big data assembly, wherein the vulnerability scanning and safety configuration compliance inspection comprises Hadoop, Spark, Hbase, Solr, ES and the like, so that the safety vulnerability and unsafe configuration existing in the big data assembly can be found in time, the safety guarantee level of a big data platform is improved through safety reinforcement in time, and the safety construction requirements of policy and regulations such as level protection, industrial specifications and the like are met.

In order to solve at least one of the above problems, the present application provides the following technical solutions:

in a first aspect, the present application provides a vulnerability scanning system for comprehensive detection of big data vulnerabilities and unsafe configurations, comprising:

the asset management module is used for discovering a survival host, network equipment and a database in a target network, automatically generating network topology and checking detailed information of each asset;

the vulnerability detection module is used for carrying out security vulnerability detection on the big data assembly and generating vulnerability description and vulnerability repair suggestions;

the configuration checking module is used for carrying out safety configuration compliance check on each component of the big data environment and determining unsafe configuration in the big data platform component;

the report management module is used for analyzing the scanning result in the form of a report and a graph to obtain a vulnerability risk level, a vulnerability category, a vulnerability description, a vulnerability type and a vulnerability solution;

the quick upgrading module is used for carrying out online upgrading, local upgrading and timed upgrading on the leak library and the software through a network or a local data packet;

and the distributed management module is used for issuing a scanning task to the lower engine, receiving a scanning result uploaded by the lower engine, performing unified analysis and generating an integral big data vulnerability scanning report.

Further, the vulnerability detection module is also used for performing security vulnerability detection of remote execution code vulnerabilities, command injection vulnerabilities, access control vulnerabilities, authority promotion vulnerabilities, denial of service vulnerabilities, and information leakage vulnerabilities on the big data component.

Further, the configuration checking module is further configured to perform security configuration compliance check on a big data acquisition component, a big data storage component and a big data processing component of the big data environment, and determine insecure configuration of user access permission control, log record integrity, file permission minimization, account permission minimization, service connection number limitation, transmission encryption and interface authentication in the big data platform component.

Further, the distributed management module further comprises:

and the self-evaluation unit is used for automatically creating a scanning task by a subordinate engine, and scanning and risk evaluation on real-time and timed big data vulnerability of the large-scale network.

Further, still include:

the vulnerability early warning unit is used for notifying a user in a mail or telephone mode when the latest high-risk vulnerability information is published and providing corresponding preventive measures;

the vulnerability scanning unit is used for carrying out vulnerability scanning on the target big data platform, detecting vulnerability and unsafe configuration of the big data by adopting a risk assessment model, finding vulnerability, carrying out priority sequencing and generating a vulnerability scanning report;

the vulnerability fixing unit is used for providing a safe configuration suggestion of the system and an effective downloading link of the patch;

and the vulnerability auditing unit is used for tracking, recording and verifying the effect of vulnerability management, and simultaneously starting a timing scanning task to perform comparative analysis and effect verification.

According to the technical scheme, vulnerability scanning and safety configuration compliance inspection including Hadoop, Spark, Hbase, Solr, ES and the like are carried out on the main-flow big data assembly, so that the safety vulnerability and the safety configuration existing in the big data assembly can be found in time, the safety guarantee level of the big data platform is improved through safety reinforcement in time, and the safety construction requirements of policy and regulations such as level protection, industrial specifications and the like are met.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a block diagram of one embodiment of a vulnerability scanning system for comprehensive detection of large data vulnerabilities and unsafe configurations;

FIG. 2 is a second block diagram of a vulnerability scanning system for comprehensive detection of big data vulnerabilities and unsafe configurations in an embodiment of the present application;

FIG. 3 is a third block diagram of a vulnerability scanning system for comprehensive detection of large data vulnerabilities and unsafe configurations in an embodiment of the present application;

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

Consider that large data packets contain complex sensitive data, which attracts more potential attackers. And a large amount of data is gathered, so that more important data can be obtained by one successful attack, and the attack yield is increased. With the outbreak of Internet and big data application, the system is attacked, data loss and personal information leakage are happened occasionally, and the problem of massive data abuse and phishing events is caused by the black and grey product of underground data transaction.

In order to perform vulnerability scanning and security configuration compliance inspection on a mainstream big data assembly, including Hadoop, Spark, Hbase, Solr, ES and the like, so as to timely discover security vulnerabilities and insecure configurations existing in the big data assembly, and timely promote the security guarantee level of a big data platform through security reinforcement, and meet the security construction requirements of policy and regulations such as level protection, industry specifications and the like, the application provides an embodiment of a vulnerability scanning system for comprehensively detecting the big data vulnerabilities and insecure configurations, and referring to fig. 1, the vulnerability scanning system for comprehensively detecting the big data vulnerabilities and insecure configurations specifically comprises the following contents:

the asset management module 10 is used for discovering a survival host, network equipment and a database in a target network, automatically generating network topology and checking detailed information of each asset;

optionally, the asset management includes major data platform components such as Hadoop, Spark, Hbase, Solr, ES, and the like, which are mainstream, and accurately identifies attributes including an IP address, a port, an operating system, a software version, a responsible person, a region, and the like, so as to prepare for further vulnerability scanning. The system can automatically generate network topology, and can also carry out later-stage manual modification to check the detailed information of each asset. The method supports the export and import of the assets, facilitates the rapid discovery and statistics of the information assets of the whole network by users, and knows the safety risk level of each asset at a glance.

The vulnerability detection module 20 is used for carrying out security vulnerability detection on the big data assembly and generating vulnerability description and vulnerability repair suggestions;

optionally, vulnerability characteristics are extracted based on vulnerability research on big data, and a big data vulnerability scanning function is formed. The vulnerability scanning system can detect security vulnerabilities of big data components, including Hadoop, Spark, Hbase, Solr, ES and the like of mainstream, and provides detailed vulnerability description and vulnerability repair suggestions. The large data vulnerability includes a remote execution code vulnerability, a command injection vulnerability, an access control vulnerability, an authority promotion vulnerability, a denial of service vulnerability, an information leakage vulnerability and the like. The security vulnerability existing in the big data platform can be found by the user in time, and the security vulnerability can be prevented in the bud through security reinforcement.

The configuration checking module 30 is used for performing safety configuration compliance check on each component of the big data environment and determining unsafe configuration in the big data platform component;

optionally, the vulnerability scanning system may further perform a safety configuration compliance check for each component of the big data environment, and cover the big data acquisition component (Kafka, flux), the big data storage component (Hbase, Hive, HDFS, Impala), and the big data processing component (Yarn & MR, Spark, Storm, Zookeeper). Therefore, unsafe configurations in the big data platform assembly are discovered, and safety baseline requirements such as user access authority control, log record integrity, file authority minimization, account authority minimization, service connection number limitation, transmission encryption, interface authentication and the like are included. And the user is helped to improve the safety protection level of the big data platform through configuration optimization.

The report management module 40 is used for analyzing the scanning result in the form of a report and a graph to obtain a vulnerability risk level, a vulnerability category, a vulnerability description, a vulnerability type and a vulnerability solution;

optionally, the vulnerability scanning system analyzes the scanning result in a form of report and graph, and can predefine, self-define, multi-angle and multi-level analyze the scanning result. And providing a perfect vulnerability risk level, vulnerability category, vulnerability description, vulnerability type and vulnerability solution. The system provides international authority records about the vulnerability (including CVE number support), and a link associated with the vendor patch. The report forms provide styles of administrative personnel, technicians, safety experts, user-defined report forms and the like, and the output report forms comprise: HTML, DOC, PDF, etc.

The quick upgrading module 50 is used for performing online upgrading, local upgrading and timed upgrading on the leak library and software through a network or a local data packet;

optionally, the vulnerability scanning system may perform online upgrade, local upgrade, and timed upgrade on the vulnerability database and the software through a network or a local data packet by using a product upgrade module built in the program. Therefore, the system can timely and accurately detect the newly published bugs, and the safety protection level of the big data platform is improved.

And the distributed management module 60 is used for issuing a scanning task to the lower engine, receiving a scanning result uploaded by the lower engine, performing unified analysis, and generating an integral big data vulnerability scanning report.

Optionally, with the gradual enlargement and the gradual complexity of the network scale, the construction of a core level network, a department level network, a terminal/personal user level network, and the filtering mechanisms such as a firewall, a switch, etc. exist among the networks, most of the detection data packets sent by the network vulnerability management system will be filtered by the network devices, and the scanning timeliness and the scanning accuracy are reduced. Aiming at the distributed complex network, the vulnerability scanning system provides a distributed management function, and can issue scanning tasks to the lower engine, receive scanning results uploaded by the lower engine, perform unified analysis and generate an integral big data vulnerability scanning report. The lower engine can also automatically create a scanning task to meet the requirement of self evaluation. Therefore, real-time and timed big data vulnerability scanning and risk assessment of a large-scale network are realized.

Further, the vulnerability detection module is also used for performing security vulnerability detection of remote execution code vulnerabilities, command injection vulnerabilities, access control vulnerabilities, authority promotion vulnerabilities, denial of service vulnerabilities, and information leakage vulnerabilities on the big data component.

Further, the configuration checking module is further configured to perform security configuration compliance check on a big data acquisition component, a big data storage component and a big data processing component of the big data environment, and determine insecure configuration of user access permission control, log record integrity, file permission minimization, account permission minimization, service connection number limitation, transmission encryption and interface authentication in the big data platform component.

Further, the distributed management module 60 further includes:

and the self-evaluation unit 61 is used for automatically creating a scanning task by a subordinate engine, and scanning and risk evaluation on real-time and timed big data vulnerability of a large-scale network.

Further, still include:

the vulnerability early warning unit 71 is used for notifying a user in a mail or telephone mode when the latest high-risk vulnerability information is published, and providing corresponding preventive measures;

the vulnerability scanning unit 72 is used for carrying out vulnerability scanning on the target big data platform, detecting vulnerability and unsafe configuration of the big data by adopting a risk assessment model, finding vulnerability, carrying out priority sequencing and generating a vulnerability scanning report;

a bug fixing unit 73, configured to provide a security configuration suggestion of the system and an effective download link of the patch;

and the vulnerability auditing unit 74 is used for tracking, recording and verifying the success of vulnerability management, and simultaneously starting a timing scanning task to perform comparative analysis and success verification.

Alternatively, security management needs to be continuously improved for a long time. Safety management is not only a technology, but more importantly, vulnerability risks are controlled through a flow system. The loop process of vulnerability management can be divided into vulnerability early warning, vulnerability scanning, vulnerability repairing and vulnerability auditing.

(1) Vulnerability early warning: when the latest high-risk vulnerability information is published, the application notifies the user by means of mail or telephone at the first time and provides corresponding preventive measures. Meanwhile, a product upgrade package is provided, and the completeness of a vulnerability knowledge base is ensured;

(2) vulnerability scanning: vulnerability scanning is carried out on a target big data platform by means of the vulnerability scanning system, a risk assessment model at the front edge is adopted to detect vulnerability and unsafe configuration of big data, vulnerability is found in time, priority ranking is carried out, and a vulnerability scanning report is generated;

(3) and (3) vulnerability repair: the vulnerability scanning system provides a vulnerability repairing scheme with strong operability, and the vulnerability repairing scheme comprises a security configuration proposal of the system, an effective downloading link of a patch and the like, so that a user can repair the vulnerability in time and efficiently;

(4) and (4) vulnerability auditing: the vulnerability management also needs to provide a complete auditing mechanism, so that a user can conveniently track, record and verify the success of the vulnerability management, supervise and urge the user to repair the vulnerability, and simultaneously start a timing scanning task to perform comparative analysis and success verification. All these processes are fully automated, thereby ensuring the overall work efficiency of vulnerability management.

Examples are as follows:

meanwhile, the vulnerability scanning system can be deployed at any place of a network through B/S mode management, the vulnerability scanning system can normally work as long as a target big data platform to be subjected to security assessment can be accessed, and the detection range covers the mainstream big data platform assembly. In consideration of security, it is generally suggested to deploy a scanning system for vulnerability of big cloud data at a core switch by-pass, so as to detect various security vulnerabilities and insecure configurations existing in a big data platform component in time and prevent the vulnerability from happening in the bud. Therefore, the safety guarantee level of the large data platform is improved, and the increasing safety requirements of various service systems are met.

As can be seen from the above description, the vulnerability scanning system for comprehensively detecting big data vulnerabilities and unsafe configurations provided in the embodiment of the present application can perform vulnerability scanning and safety configuration compliance inspection on the main-flow big data component, including Hadoop, Spark, Hbase, Solr, ES, and the like, so as to timely discover the safety vulnerabilities and unsafe configurations existing in the big data component, and timely enhance the safety guarantee level of the big data platform through safety reinforcement, thereby meeting the safety construction requirements of policy and regulations such as level protection and industrial specifications.

In order to perform vulnerability scanning and security configuration compliance inspection on a mainstream big data assembly, including Hadoop, Spark, Hbase, Solr, ES and the like, on a hardware level, so that security vulnerabilities and insecure configurations existing in the big data assembly can be found in time, the security guarantee level of a big data platform can be improved in time through security reinforcement, and the security construction requirements of policy and regulations such as level protection and industry specifications are met, the application provides an embodiment of electronic equipment for realizing all or part of contents in a vulnerability scanning system for comprehensively detecting the big data vulnerabilities and insecure configurations, and the electronic equipment specifically comprises the following contents:

a processor (processor), a memory (memory), a communication Interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete mutual communication through the bus; the communication interface is used for realizing information transmission between a vulnerability scanning system for comprehensively detecting big data bugs and unsafe configuration and relevant equipment such as a core service system, a user terminal, a relevant database and the like; the logic controller may be a desktop computer, a tablet computer, a mobile terminal, and the like, but the embodiment is not limited thereto. In this embodiment, the logic controller may be implemented with reference to the embodiment of the vulnerability scanning system for comprehensive detection of the big data vulnerability and the unsafe configuration and the embodiment of the vulnerability scanning system for comprehensive detection of the big data vulnerability and the unsafe configuration in the embodiment, and the contents thereof are incorporated herein, and repeated details are not repeated here.

It is understood that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a Personal Digital Assistant (PDA), an in-vehicle device, a smart wearable device, and the like. Wherein, intelligence wearing equipment can include intelligent glasses, intelligent wrist-watch, intelligent bracelet etc..

In practical applications, part of the vulnerability scanning system for comprehensively detecting the big data vulnerability and the unsafe configuration may be executed on the electronic device side as described above, or all operations may be completed in the client device. The selection may be specifically performed according to the processing capability of the client device, the limitation of the user usage scenario, and the like. This is not a limitation of the present application. The client device may further include a processor if all operations are performed in the client device.

The client device may have a communication module (i.e., a communication unit), and may be communicatively connected to a remote server to implement data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, the server may also include a server on an intermediate platform, for example, a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may include a single computer device, or may include a server cluster formed by a plurality of servers, or a server structure of a distributed apparatus.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.