阅读说明：本技术 一种面向威胁场景的智能化安全事件关联分析系统 (Intelligent security event correlation analysis system for threat scene ) 是由 刘家豪 吕华辉 杨航 刘欣 陈华军 明哲 张佳发 梁段 陈锋 于 2020-11-16 设计创作，主要内容包括：本发明提供一种面向威胁场景的智能化安全事件关联分析系统,包括海量格式事件存储模块、分析模块、可视化展示模块；所述分析模块包括事件关联分析模块、综合威胁分析模块、攻击链分析模块、攻击路径分析模块、Web攻击深度分析模块、网络流量元数据行为分析模块、网络异常行为分析模块。本发明所提供的分析系统可以汇总和合理化威胁数据自动筛选出攻陷指标(IOC)作为可机读威胁情报(MRTI),并且使用现存的日志对比匹配以便轻松发现不常见的趋势或线索,并对其有效执行操作。通过将团队、流程和工具结合在一起,系统平台为安全团队提供了对于威胁来自哪里前所未有的视野,并可以从头到尾跟踪整个事件,通过报告,可指导安全响应并进行阻断。(The invention provides an intelligent security event correlation analysis system for a threat scene, which comprises a mass format event storage module, an analysis module and a visual display module; the analysis module comprises an event correlation analysis module, a comprehensive threat analysis module, an attack chain analysis module, an attack path analysis module, a Web attack depth analysis module, a network flow metadata behavior analysis module and a network abnormal behavior analysis module. The analysis system provided by the invention can summarize and rationalize threat data and automatically screen out attack and subsidence Indexes (IOCs) as machine-readable threat intelligence (MRTI), and uses the existing log comparison and matching so as to easily find unusual trends or clues and effectively execute operation on the unusual trends or clues. By combining teams, processes, and tools, the system platform provides the security team with an unprecedented view of where the threat came from, and can track the entire event from beginning to end, with reports that can guide security responses and block it.)

1. An intelligent security event correlation analysis system for threat scenes is characterized by comprising a mass format event storage module, an analysis module and a visual display module;

the mass format event storage module supports a single event collector and a plurality of event collectors, collects mass log data by adopting a parallel event pipeline collection mode by means of hardware multi-core characteristics, and quickly performs event parallel pipeline processing by means of a cache by adopting an asynchronous non-blocking event collection mode;

the analysis module comprises an event correlation analysis module, a comprehensive threat analysis module, an attack chain analysis module, an attack path analysis module, a Web attack depth analysis module, a network flow metadata behavior analysis module and a network abnormal behavior analysis module;

the event correlation analysis module comprises a rule-based correlation analysis module, a context-based correlation analysis module and a behavior-based correlation analysis module;

the association analysis module based on the rules carries out rule matching through an event association engine, identifies the attack and violation processes of known modes, supports the establishment of single event rules and multi-event rules, and realizes single event association and multi-event association; the single event correlation is to carry out rule matching on the event stream which accords with a single rule through the single event correlation; the multi-event association is to perform complex event rule matching on event streams conforming to a plurality of rules through the multi-event association, wherein the rules are called as combination rules, and are specifically analyzed and matched in the following way:

(1.1) unsure of pattern matching of finite automata based on Nondeteritiative Fine Automation (NFA);

(1.2) syntax compilation of CQL (Continuous Query Language) based on Extended Back-Naur Form (EBNF) Extended BNF paradigm;

the context-based association analysis module associates the security event with the actual operation environment of the current network and service, and identifies the security threat through information correlation analysis, and specifically includes the following analysis contents:

(2.1) asset-based contextual association: associating the IP address in the event with an asset name, an asset value, an asset type and a custom asset tag, wherein the asset type comprises a user custom asset type;

(2.2) vulnerability-based context association: associating the security event with the current vulnerability information of the target asset to which the event aims, wherein the vulnerability information comprises port association and vulnerability number association;

(2.3) context association based on network alerts: associating the security event with the current alarm information of the target asset or the initiated source asset and the current network alarm information;

(2.4) topology-based context association: according to the characteristic that the network fault propagates along the network topology level, the network fundamental fault source diagnosis is automatically carried out through the distribution of a large number of network alarm events in the topology space and the sequence of propagation time;

the association analysis module based on the behaviors specifically includes the following analysis contents:

(3.1) dynamic baseline analysis: according to historical data, firstly establishing a single-period database profile baseline which is a curve and consists of a plurality of data profile points, wherein each profile point represents a sampling time point, and if a new actual measured value does not exceed the range of the baseline, updating an old profile value through a weighted average algorithm; discarding if the new actual measurement value exceeds the baseline range, and not participating in the calculation of the new contour value; the above steps are repeated, and the baseline is always in dynamic change;

(3.2) predictive analysis: by adopting a detection model based on a time window confidence interval, the method continuously adjusts and approaches by self in actual operation, automatically eliminates abnormal historical data in a historical time window, and realizes high coincidence between the historical time window data and the actual normal flow behavior characteristics of the network, thereby improving the accuracy of alarming abnormal behavior;

the comprehensive threat analysis module comprises the following contents: the system adopts an analysis mode based on a scene, wherein the scene defines a main body of behavior analysis, and an analysis index and a threshold value for triggering behavior early warning are adopted on the main body; the main body of the behavior analysis is the asset IP; behavior analysis indexes represent key indexes of a certain behavior, suspicious behaviors are found through monitoring and analyzing the indexes, and the number or the proportion of events of a certain type or characteristic is used as a characteristic index;

the attack chain analysis module is used for mining multi-step attack behavior occurrence modes from a historical data warehouse and then realizing online attack intention identification through real-time mode matching and attack association, and specifically comprises the following steps: data mining is carried out in a historical data warehouse, a trend item and a period item alarm are filtered, then the alarm is classified according to main attribute information, attack type identification is carried out on the classified alarm category so as to generate a high-grade security event, an attack scene time window is utilized to convert a security event alarm database into a candidate attack sequence set, and a multi-step attack behavior occurrence sequence pattern is excavated from the candidate attack sequence set by utilizing an improved Apriori-all sequence pattern mining algorithm; different attack behavior sequence modes reflect different attack step behavior generation modes of multi-step attack, and attack scene reconstruction, online attack intention and attack strategy identification are realized through real-time mode matching and attack correlation;

the attack path analysis module adopts a heuristic scene reconstruction technology, and specifically comprises the following contents:

(4.1) for the summarized alarm events, firstly, performing association among the alarm events based on the existing attack scene knowledge base;

(4.2) for the attack sequence in the matching, if the newly received alarm can be matched with the existing attack sequence, directly associating; if the alarm can not be matched with the existing attack sequence but can be matched with the subsequent events of a certain attack sequence, matching the newly received alarm with the sequence and generating an event type with a 'virtual alarm' mark missing;

(4.3) determining the event type of the virtual alarm according to the description of the attack path diagram, determining the description of the time range of the virtual alarm according to the time of related alarm before and after the virtual alarm, performing backtracking analysis by using original data, searching whether the alarm event which is not reported exists, and adding the alarm event into the attack sequence again if the alarm event which is not reported is found until the complete attack path diagram is matched;

the Web attack depth analysis module comprises the following contents: carrying out deep analysis by using a fuzzy comprehensive evaluation method from the session state, the request data, the response data analysis, the identity authentication, the file uploading, the access frequency, the WAF attack log and the dark net connection based on an improved weighted naive Bayes classification algorithm related to characteristics, and mining the attack which is missed to report by the traditional means;

the network traffic metadata behavior analysis module specifically comprises the following contents: establishing a flow periodic baseline and an aperiodic baseline automatically through modeling and analyzing the flow behavior of the internal network, automatically discovering the interconnection relationship of equipment, predicting network congestion, adopting a support network optimization decision, and discovering network abnormality based on the baseline;

the network abnormal behavior analysis module specifically comprises botnet monitoring discovery, lost host discovery, slow scanning monitoring, malicious mail discovery and diffusion analysis;

the visual display module comprises the following contents: and generating a topological graph showing an audit data source, reflecting the network topological relation of the audit data source, marking the log quantity and the alarm event quantity of each audit data source on a topological node, and inquiring the details of the log and the alarm information by clicking the topological node by an administrator.

2. The intelligent security event correlation analysis system for threat scenarios according to claim 1, wherein: the logical expression of the rule includes the following operators or keywords but is not limited to: equal to, not equal to, greater than, less than, not greater than, not less than, located between, belonging to, containing, FollowBy.

3. The intelligent security event correlation analysis system for threat scenarios according to claim 1, wherein: the rules support statistical counting functions, specify fixed and varying event attributes during statistics, and correlate events that meet certain statistical rules.

4. The intelligent security event correlation analysis system for threat scenarios according to claim 1, wherein: the botnet monitoring discovery is judged as the botnet according to the following judgment standards:

(5.1) the data volume transmitted by one-time communication connection between the Bot and the BotMaster is small, and the data packet is small;

(5.2) the data packet size is relatively fixed, and the types are few;

(5.3) Beacon behavior exists between Bot and BotMaster, which is shown in that the interval between two visits is relatively consistent;

(5.4) Bot will repeatedly request a connection because of loss of contact with BotMaster, resulting in a large difference in the number of flows in both directions;

(5.5) the communication between Bot and BotMaster is more uniform in time distribution and less affected by Day/Night;

(5.6) Bot will be controlled by BotMaster to engage in malicious activities;

(5.7) there is similarity in the sequence of behavior between different bots accessing the BotMaster.

5. The intelligent security event correlation analysis system for threat scenarios according to claim 1, wherein: the discovery of the lost host is specifically to discover the lost host through the extraction capability of APT on the characteristics of the malicious file CallBack and the capability of real-time synchronization of the characteristics and IDS.

6. The intelligent security event correlation analysis system for threat scenarios according to claim 1, wherein: the visual display module can also visually display a safety management evaluation curve of each safety domain or service system changing along with time, can perform ring ratio analysis and cross-safety domain or service system same ratio analysis, supports drilling of index items for each key management index, and realizes focusing from macro to micro.

Technical Field

The invention relates to network security analysis, in particular to an intelligent security event correlation analysis system for a threat scene.

Background

With the continuous expansion of network scale, the current network plays an increasingly important role in social life; meanwhile, the network security problem is increasingly highlighted and gradually becomes a key problem to be solved urgently for further development of network services and applications. In addition, as network intrusion and attack behaviors develop towards the trends of distribution, scale, complication, indirection and the like, threats and losses caused by network viruses, Dos/DDos attacks and the like are increasingly large, and many scientific researchers and organizations have come to realize that the whole network security condition cannot be monitored in real time only by relying on the existing network security products.

As is well known, with the continuous scale expansion of business and IT infrastructure and the development of new technology, the scale of domestic power grid power communication networks is getting larger and larger, and the traditional security event and information management system based on the relational database technology and similar log auditing systems or security management systems cannot meet the processing requirements of high-speed mass events. The method mainly includes that after a certain number of events is exceeded, the traditional event management technology cannot complete real-time acquisition and storage of all information events, is limited by the computing capacity of a single system, massive real-time data cannot be effectively correlated and analyzed, and false alarm and missing alarm can be generated, so that safety attack cannot be effectively found. Historical data cannot be effectively analyzed, time is consumed for historical query and retrieval of mass data by adopting a relational database technology, and generation of a report usually consumes several hours, which cannot meet the daily safety work requirement of safety analysts.

At present, domestic analysis systems are basically established in different production links, and are supplemented with common and simple tools, such as databases, report tools, even Excel (data table tool) and the like, so that production data are directly analyzed to know the operation condition of enterprises. The inevitable problem is that the data sources in the enterprise are scattered, and the analysis system built on this basis is necessarily isolated. However, effective association and comprehensive analysis are lacked among the information islands, and a unified view of enterprise data cannot be formed. The method is weak in analysis angle and depth, and correlation analysis and prediction analysis.

Therefore, there is a need for an analysis system that can implement an agile and fast response manner to cope with the threat of growing, large-scale, high priority, and can implement comprehensive analysis and research from the perspective of "globalization".

Disclosure of Invention

In order to solve the problems in the prior art, the invention aims to provide an intelligent security event correlation analysis system facing a threat scene, which can analyze and detect various real-time and historical network threat events in a power communication network and provide a visual analysis report.

In order to achieve the above object, the technical solution of the present invention is implemented as follows:

an intelligent security event correlation analysis system facing threat scenes comprises a mass format event storage module, an analysis module and a visual display module;

the mass format event storage module supports a single event collector and a plurality of event collectors, collects mass log data by adopting a parallel event pipeline collection mode by means of hardware multi-core characteristics, and quickly performs event parallel pipeline processing by means of a cache by adopting an asynchronous non-blocking event collection mode. The performance of event acquisition and pretreatment is greatly improved;

the analysis module comprises an event correlation analysis module, a comprehensive threat analysis module, an attack chain analysis module, an attack path analysis module, a Web attack depth analysis module, a network flow metadata behavior analysis module and a network abnormal behavior analysis module;

the analysis system also provides a visual display module, and a user customizes association rules based on logic expressions and statistical conditions through a built-in visual editor, so that all log fields can participate in association.

The event correlation analysis module comprises a rule-based correlation analysis module, a context-based correlation analysis module and a behavior-based correlation analysis module;

the association analysis module based on the rules carries out rule matching through an event association engine, identifies the attack and violation processes of known modes, supports the establishment of single event rules and multi-event rules, and realizes single event association and multi-event association; the single event correlation is to carry out rule matching on the event stream which accords with a single rule through the single event correlation; the multi-event association is to perform complex event rule matching on event streams conforming to a plurality of rules through the multi-event association, wherein the rules are called as combination rules, and are specifically analyzed and matched in the following way:

(1.1) unsure of pattern matching of finite automata based on Nondeteritiative Fine Automation (NFA);

(1.2) syntax compilation of CQL (Continuous Query Language) based on Extended Back-Naur Form (EBNF) Extended BNF paradigm; the CQL syntax format is similar to SQL, with independent syntax keys, e.g., representing timing, time windows;

the logical expression of the rule includes the following operators or keywords but is not limited to: equal to, not equal to, greater than, less than, not greater than, not less than, located between, belonging to, containing, FollowBy.

The rules support statistical counting functions, specify fixed and varying event attributes during statistics, and correlate events that meet certain statistical rules.

The context-based association analysis module associates the security event with the actual operation environment of the current network and service, and identifies the security threat through information correlation analysis, and specifically includes the following analysis contents:

(2.1) asset-based contextual association: associating the IP address in the event with an asset name, an asset value, an asset type and a custom asset tag, wherein the asset type comprises a user custom asset type;

(2.2) vulnerability-based context association: associating the security event with the current vulnerability information of the target asset to which the event aims, wherein the vulnerability information comprises port association and vulnerability number association;

(2.3) context association based on network alerts: associating the security event with the current alarm information of the target asset or the initiated source asset and the current network alarm information;

(2.4) topology-based context association: according to the characteristic that the network fault propagates along the network topology level, the Root Cause (Root Cause) of the network is automatically diagnosed by the distribution of a large number of network alarm events in the topology space and the sequence of propagation time;

the association analysis module based on the behaviors specifically includes the following analysis contents:

(3.1) dynamic baseline analysis: according to historical data, firstly establishing a single-period database profile baseline which is a curve and consists of a plurality of data profile points, wherein each profile point represents a sampling time point, and if a new actual measured value does not exceed the range of the baseline, updating an old profile value through a weighted average algorithm; discarding if the new actual measurement value exceeds the baseline range, and not participating in the calculation of the new contour value; the above steps are repeated, and the baseline is always in dynamic change;

(3.2) predictive analysis: the detection model based on the time window confidence interval is adopted, the abnormal historical data in the historical time window are automatically removed by continuous self-adjustment and approximation in actual operation, the high coincidence between the historical time window data and the actual normal flow behavior characteristics of the network is realized, and the accuracy of alarming the abnormal behavior is improved.

The comprehensive threat analysis module comprises the following contents: the system adopts an analysis mode based on a scene, wherein the scene defines a main body of behavior analysis, and an analysis index and a threshold value for triggering behavior early warning are adopted on the main body; the main body of the behavior analysis is the asset IP; the behavior analysis indexes represent key indexes of a certain behavior, and suspicious behaviors are found through monitoring and analyzing the indexes, wherein the number or the proportion of events of a certain type or characteristic is used as a characteristic index. For example: the user can pay attention to the quantity mutation of Apache server logs, the quantity mutation of router logs, the quantity mutation of configuration change logs and the quantity mutation of login logs.

The attack chain analysis module is used for mining multi-step attack behavior occurrence modes from a historical data warehouse and then realizing online attack intention identification through real-time mode matching and attack association, and specifically comprises the following steps: data mining is carried out in a historical data warehouse, a trend item and a period item alarm are filtered, then the alarm is classified according to main attribute information, attack type identification is carried out on the classified alarm category so as to generate a high-grade security event, an attack scene time window is utilized to convert a security event alarm database into a candidate attack sequence set, and a multi-step attack behavior occurrence sequence pattern is excavated from the candidate attack sequence set by utilizing an improved Apriori-all sequence pattern mining algorithm; different attack behavior sequence modes reflect different attack step behavior generation modes of multi-step attack, and attack scene reconstruction, online attack intention, attack strategy identification and the like are achieved through real-time mode matching and attack correlation.

The attack path analysis module adopts a heuristic scene reconstruction technology, and specifically comprises the following contents:

(4.1) for the summarized alarm events, firstly, performing association among the alarm events based on the existing attack scene knowledge base;

(4.2) for the attack sequence in the matching, if the newly received alarm can be matched with the existing attack sequence, directly associating; if the alarm can not be matched with the existing attack sequence but can be matched with the subsequent events of a certain attack sequence, matching the newly received alarm with the sequence and generating an event type with a 'virtual alarm' mark missing;

(4.3) determining the event type of the virtual alarm according to the description of the attack path diagram, determining the description of the time range of the virtual alarm according to the time of related alarm before and after the virtual alarm, performing backtracking analysis by using original data, searching whether the alarm event which is not reported exists, and adding the alarm event into the attack sequence again if the alarm event which is not reported is found until the complete attack path diagram is matched;

web access logs of an external portal website are analyzed to find Web attack behaviors, the Web attack behaviors can be found and attack tracing can be carried out, and Web application under Web attack is positioned; the capability of detecting and analyzing the password detection behavior; further comprising at least: botnet behavior detection, scanning behavior detection, password detection behavior detection, ARP deception behavior detection, DDOS attack behavior detection, malicious information release behavior detection, malicious website access behavior detection, sensitive information stealing behavior detection and the like.

The Web attack depth analysis module comprises the following contents: from session state (illegal cookie/cookie parameter tampering/forced session expiration/location change within session validity/browser change within session validity etc.), request data (malformed request body/multiple encodings/anomalies/anomalous request method/illegal URI data/anomalous request header/parameter anomalies/character set anomalies), response data analysis (anomalous response header/error code/title alteration/dynamic content alteration/response time delay etc.), identity authentication (default username/password/multiple usernames/high frequency login attempt/login failure etc.), file upload (file size anomaly/number anomaly/suffix anomaly), access frequency (request response delay, anomalous request interval, anomalous request flow/resource utilization increase), WAF attack logs and dark web connections (malicious domain names/hosts/URLs and the like) are deeply analyzed by a fuzzy comprehensive evaluation method based on an improved weighted naive Bayes classification algorithm related to features, and attacks which are missed in reporting by a traditional means are mined.

The network traffic metadata behavior analysis module specifically comprises the following contents: the method comprises the steps of automatically establishing a flow periodic baseline and a non-periodic baseline through modeling and analyzing the flow behavior of an internal network, automatically discovering the interconnection relation of equipment, predicting network congestion, adopting a support network optimization decision, and discovering network abnormality based on the baseline.

The network abnormal behavior analysis module specifically comprises botnet monitoring discovery, lost host discovery, slow scanning monitoring, malicious mail discovery and diffusion analysis.

Botnet (Botnet) refers to a one-to-many controllable network formed between a controller and an infected host by infecting a large number of hosts with bot program (Botnet) viruses using one or more propagation means. That is, a hacker organizes tens of thousands of trapped machines into one control node by using a self-written distributed denial of service attack program, and the control node is used for sending fake packets or junk data packets to make a predetermined attack target paralyze and refuse to service. Typically, worm viruses may also be utilized to form botnets. An attacker propagates bots through various ways to infect a large number of hosts on the internet, and the infected hosts receive the attacker's instructions through a control channel to form a botnet. The name botnet is used to more visually identify the nature of such hazards: the computers are unconsciously driven and directed by people like zombie groups in ancient Chinese legends, and become a tool utilized by people.

The botnet monitoring discovery is judged as the botnet according to the following judgment standards:

(5.1) the data volume transmitted by one-time communication connection between the Bot and the BotMaster is small, and the data packet is small;

(5.2) the data packet size is relatively fixed, and the types are few;

(5.3) Beacon behavior exists between Bot and BotMaster, which is shown in that the interval between two visits is relatively consistent, but the heartbeat cycles of different hosts may not be consistent;

(5.4) Bot will repeatedly request a connection because of loss of contact with BotMaster, resulting in a large difference in the number of flows in both directions;

(5.5) the communication between Bot and BotMaster is more uniform in time distribution and less affected by Day/Night;

(5.6) Bot is controlled by BotMaster to engage in malicious activities, such as DDoS, Spam, PE Download, attack behaviors, etc.;

(5.7) there is similarity in the sequence of behavior between different bots accessing the BotMaster.

The discovery of the lost host is specifically to discover the lost host through the extraction capability of APT on the characteristics of the malicious file CallBack and the capability of real-time synchronization of the characteristics and IDS.

The slow scan monitoring utilizes big data technology to identify slow scan attacks. Unlike the fast scan attack, it uses a slow-down half-connection attack to bypass the monitoring of the security device to obtain the resources that can be acquired. Since for conventional security devices, data information can only be monitored for a short period of time, the relevant attack behavior of slow scanning cannot be identified. The big data technology allows a large amount of long-term bottom data content to be stored, provides high-efficiency query and analysis capability, and can accurately identify the slow scanning attack.

The malicious mail discovering and spreading analysis utilizes the full-network monitoring capability and the APT analysis capability on the malicious files to analyze the spreading and influence of the malicious files based on the relation association of the receivers and the senders.

The visual display module comprises the following contents:

and generating a topological graph showing an audit data source, reflecting the network topological relation of the audit data source, marking the log quantity and the alarm event quantity of each audit data source on a topological node, and inquiring the details of the log and the alarm information by clicking the topological node by an administrator.

For the security data, the administrator can track the source and destination IP addresses and mark the IP addresses on the displayed world map.

The administrator can also conduct behavior analysis on the logs within a period of time, and the administrator can assist in positioning safety problems from a macroscopic perspective by generating a behavior analysis diagram to visually display the association relationship among the massive logs.

The method can evaluate the overall network security of the client from a macroscopic perspective, evaluate the level of the overall security management construction, provide decision support for improving the security protection capability of the client, and simultaneously provide decision support for improving the maturity of the information security management system construction of the client.

The method comprises the steps of obtaining a security domain or service system security management construction level rating through calculation of a group of hierarchical indexes representing the security domain or service system security management construction level, and accordingly showing the construction maturity of an information security management system of the security domain or service system.

The set of hierarchical indexes representing the safety management construction level is called as key management indexes, and each index item establishes a measurement standard aiming at a certain type of safety event.

The method can visually display a security management evaluation curve of each security domain or business system changing along with time, and can perform ring ratio analysis and same ratio analysis across the security domains or business systems. And the drill-down of the index item is supported for each key management index, and the focusing from macro to micro is realized.

The analysis system provided by the invention can summarize and rationalize threat data and automatically screen out attack and subsidence Indexes (IOCs) as machine-readable threat intelligence (MRTI), and uses the existing log comparison and matching so as to easily find unusual trends or clues and effectively execute operation on the unusual trends or clues. By combining teams, processes, and tools, the system platform provides the security team with an unprecedented view of where the threat came from, and can track the entire event from beginning to end, with reports that can guide security responses and block it. A large amount of time spent tracking false alarms generated by traditional situational awareness platforms is saved.

Drawings

FIG. 1 is a schematic diagram of an event correlation analysis module of an intelligent security event correlation analysis system for threat scenario according to the present invention;

FIG. 2 is a schematic diagram of a Web attack depth analysis module according to the present invention.

Detailed Description

The application provides an intelligent security event correlation analysis system for threat scenes, which comprises a mass format event storage module, an analysis module and a visual display module;

by adopting a distributed mass log acquisition technology, a single event collector and a plurality of event collectors can be simultaneously supported. The performance of event collection is further improved by means of the distributed collector. The mass format event storage module supports a single event collector and a plurality of event collectors, collects mass log data by adopting a parallel event pipeline collection mode by means of hardware multi-core characteristics, and quickly performs event parallel pipeline processing by means of a cache by adopting an asynchronous non-blocking event collection mode. The performance of event acquisition and pretreatment is greatly improved;

by means of the built-in distributed event access agent, the system can perform distributed parallel storage on a large number of events, on one hand, the storage performance is improved, and on the other hand, the storage capacity is also improved.

The big data analysis aiming at the mass events can comprise data acquisition, data sorting, data analysis, data storage, data visualization and the like. The data analysis can be further divided into real-time data analysis, data retrieval and data history analysis. Data analysis is central to BDA (big data analysis).

In the aspect of data real-time analysis, the system adopts a CEP engine based on stream processing technology. Stream processing has natural advantages when performing real-time analysis, especially when compared to data-based analysis. The stream has a high real-time.

In the aspect of data history analysis, the situation awareness platform for enlightening stars supports both batch analysis and interactive analysis. The system adopts a data automatic aggregation technology (also called a data extraction technology) to improve the historical analysis efficiency of mass data.

In the aspect of data retrieval (data query), the system adopts unique task-driven segmented event query technology.

The system also adopts a distributed query technology, and the query speed is improved through parallel query. And the upper layer adopts a task-driven segmented event query technology. This technique has three features: and the task driving, sectional type query and progressive loading greatly improve the query speed and the user experience.

The system also constructs an intelligent security event correlation analysis algorithm

The intelligent security event analysis is mainly embodied through intelligent security event correlation analysis. Event correlation refers to finding the relationships existing in a large number of events and extracting a small number of events that are really important from these large number of events. By means of an advanced intelligent event correlation analysis engine, the system can perform security event correlation analysis on all the normalized log streams uninterruptedly in real time. The system provides three event correlation analysis modules, which are respectively: rule-based association analysis, context-based association analysis, and behavior-based association analysis. The specific relevant contents of the three event correlation analysis modules are shown in fig. 1.

The analysis module comprises an event correlation analysis module, a comprehensive threat analysis module, an attack chain analysis module, an attack path analysis module, a Web attack depth analysis module, a network flow metadata behavior analysis module and a network abnormal behavior analysis module.

The analysis system also provides a visual display module, and a user customizes association rules based on logic expressions and statistical conditions through a built-in visual editor, so that all log fields can participate in association.

The intelligent security event analysis is mainly embodied through intelligent security event correlation analysis. Event correlation refers to finding the relationships existing in a large number of events and extracting a small number of events that are really important from these large number of events. By means of an advanced intelligent event correlation analysis engine, the system can perform security event correlation analysis on all the normalized log streams uninterruptedly in real time. The system provides three event correlation analysis modules, which are respectively: rule-based association analysis, context-based association analysis, and behavior-based association analysis. Specific relevant contents of the three event correlation analysis modules are shown in fig. 1, and the event correlation analysis modules include a rule-based correlation analysis module, a context-based correlation analysis module and a behavior-based correlation analysis module.

The association analysis module based on the rules carries out rule matching through an event association engine, identifies the attack and violation processes of known modes, supports the establishment of single event rules and multi-event rules, and realizes single event association and multi-event association; the single event correlation is to carry out rule matching on the event stream which accords with a single rule through the single event correlation; the multi-event association is to perform complex event rule matching on event streams conforming to a plurality of rules through the multi-event association, wherein the rules are called as combination rules, and are specifically analyzed and matched in the following way:

(1.1) unsure of pattern matching of finite automata based on Nondeteritiative Fine Automation (NFA);

(1.2) syntax compilation of CQL (Continuous Query Language) based on Extended Back-Naur Form (EBNF) Extended BNF paradigm; the CQL syntax format is similar to SQL, with independent syntax keys, e.g., representing timing, time windows;

the logical expression of the rule includes the following operators or keywords but is not limited to: equal to, not equal to, greater than, less than, not greater than, not less than, located between, belonging to, containing, FollowBy.

The rules support a statistical counting function, can specify fixed and variable event attributes during statistics, and can correlate events that meet certain statistical rules.

The context-based association analysis module associates the security event with the actual operation environment of the current network and service, and identifies the security threat through information correlation analysis, and specifically includes the following analysis contents:

(2.1) asset-based contextual association: associating the IP address in the event with an asset name, an asset value, an asset type and a custom asset tag, wherein the asset type comprises a user custom asset type;

(2.2) vulnerability-based context association: associating the security event with the current vulnerability information of the target asset to which the event aims, wherein the vulnerability information comprises port association and vulnerability number association;

(2.3) context association based on network alerts: associating the security event with the current alarm information of the target asset or the initiated source asset and the current network alarm information;

(2.4) topology-based context association: according to the characteristic that the network fault propagates along the network topology level, the Root Cause (Root Cause) of the network is automatically diagnosed by the distribution of a large number of network alarm events in the topology space and the sequence of propagation time;

establishing respective specific baselines for various data, providing better information and faster detection of abnormal activities for IT computing environments of the data, so that an administrator can customize event baselines, observe event behavior trends more specifically and perform early warning on suspicious behaviors, and based on the above, the behavior-based association analysis module specifically comprises the following analysis contents:

(3.1) dynamic baseline analysis: calculating to obtain a periodic baseline according to historical data, firstly establishing a single-period database contour line, wherein the curve consists of a plurality of data contour points, each contour point represents a sampling time point, and if a new actual measurement value does not exceed the range of the baseline, updating an old contour value through a weighted average algorithm; discarding if the new actual measurement value exceeds the baseline range, and not participating in the calculation of the new contour value; the above steps are repeated, and the baseline is always in dynamic change;

(3.2) predictive analysis: the detection model based on the time window confidence interval is adopted, the abnormal historical data in the historical time window are automatically removed by continuous self-adjustment and approximation in actual operation, the high coincidence between the historical time window data and the actual normal flow behavior characteristics of the network is realized, and the accuracy of alarming the abnormal behavior is improved.

The comprehensive threat analysis module comprises the following contents: the system adopts an analysis mode based on a scene, wherein the scene defines a main body of behavior analysis, and an analysis index and a threshold value for triggering behavior early warning are adopted on the main body; the main body of the behavior analysis is the asset IP; the behavior analysis indexes represent key indexes of a certain behavior, and suspicious behaviors are found through monitoring and analyzing the indexes, wherein the number or the proportion of events of a certain type or characteristic is used as a characteristic index. For example: the user can pay attention to the quantity mutation of Apache server logs, the quantity mutation of router logs, the quantity mutation of configuration change logs and the quantity mutation of login logs.

The system is internally provided with 14 templates of behavior analysis scenes, and a user can instantiate the templates according to the requirement of the user to establish any number of analysis scene instances.

The system also comprises a monitoring and evaluating module, which is used for accurately and efficiently detecting the malignant attack behavior of the known threat event and reporting the detection result to the monitoring and evaluating module of the system for monitoring and evaluating, so that the clue mining of the threat event is realized, and data support is provided for the system operation, thereby effectively processing the threat; the known threats include: viruses, trojans, worms, botnets, buffer overflow attacks, DDoS, scanning probes, spoofing hijacking, SQL injection, XSS, website horse hanging, abnormal traffic.

The method comprises the following steps of carrying out fine-grained detection on malignant attack behaviors of unknown threat events of malicious codes, specifically: the method comprises the steps of carrying out detection on unknown malicious codes, nested attack detection, Trojan worm virus identification, secret channel detection and detection of various unknown vulnerabilities (0-day) by utilizing behaviors.

The method adopts double detection of static detection and dynamic detection or multiple detection engines to detect the core step of the APT attack, and reports the detection result to the system for comprehensive threat analysis, so as to realize the clue mining of the threat event, provide data support for the system operation and effectively discover the APT attack. The various detection engines include: binary detection, heap-spray detection, ROP utilization detection, sensitive API detection, stack detection, Shell code detection, sandbox detection.

The dynamic sandbox detection engine has various virtual machine environments, the execution of an application program and the execution of attack codes in a malicious file are simulated, and all contents of the malicious file are monitored and recorded. So that the content and intent of the attack event can be known. The recorded behaviors comprise registry operation, file operation, vulnerability utilization mode, API calling sequence, network behavior, process thread operation, other behaviors which harm the system and detailed behavior report of the PE file contained in the malicious file.

Threat situation analysis, also known as threat KPI analysis. The system obtains a threat index through calculation of a group of key threat indexes, and describes a threat index curve with time, so as to represent the network security threat state and the development trend of a certain network area within a period of time.

The system establishes a set of dynamic multi-dimensional threat index system, and distinguishes the current threat cause through a pareto analysis method, so that key threat factors are drilled down layer by layer from macro to middle and then to micro until a key safety event causing threat situation abnormity is positioned.

The information security products comprise a firewall, a security router, an intrusion detection system, a host security system, an anti-virus system and the like, and generate huge redundant, scattered and independent security event data such as alarms and logs, and the event data is mistaken, missed and mixed, has low hit rate, and brings huge challenges to a security operation and maintenance team to accurately and quickly respond to events.

Therefore, the attack chain analysis module is used for mining multi-step attack behavior occurrence patterns from a historical data warehouse and realizing online attack intention identification through real-time pattern matching and attack association, and specifically comprises the following steps: data mining is carried out in a historical data warehouse, trend item and period item alarms are filtered, the alarms are classified according to main attribute information, attack type identification is carried out on the classified alarm types to generate high-grade security events, an attack scene time window is utilized to convert a security event alarm database into a candidate attack sequence set, and a multi-step attack behavior occurrence sequence pattern is mined from the candidate attack sequence set by utilizing an improved Apriori-all sequence pattern mining algorithm. Different attack behavior sequence modes reflect different attack step behavior generation modes of multi-step attack, and attack scene reconstruction, online attack intention, attack strategy identification and the like are achieved through real-time mode matching and attack correlation.

A complete APT attack process comprises stages of probing, attack implementation and authority acquisition, sensitive data collection and return, log clearing and the like before attack, and a real-time monitoring subsystem is possibly triggered to generate one or more independent alarm messages in each stage. The threat degree of the alarm information is not high in isolation, and the alarm information is easily submerged in mass alarm information; but if correlated, may mark a well-defined APT behavior.

Existing APT attack scenes are summarized, a comprehensive scene knowledge base is established, and then potential APT attack behaviors are detected based on the summarized real-time alarm information. An APT attack scene is established by adopting an attack scene knowledge base mode, and due to the complexity of an attack means and a network environment in an actual environment, matching of a complete attack path diagram is often not performed due to the loss of an alarm event when scene matching is performed, so that the APT scene establishment fails. Therefore, the attack path analysis module adopts a heuristic scene reconstruction technology to solve the problem, and specifically includes the following contents:

(4.1) for the summarized alarm events, firstly, performing association among the alarm events based on the existing attack scene knowledge base;

(4.2) for the attack sequence in the matching, if the newly received alarm can be matched with the existing attack sequence, directly associating; if the alarm can not be matched with the existing attack sequence but can be matched with the subsequent events of a certain attack sequence, matching the newly received alarm with the sequence and generating an event type with a 'virtual alarm' mark missing;

(4.3) determining the event type of the virtual alarm according to the description of the attack path diagram, determining the description of the time range of the virtual alarm according to the time of related alarm before and after the virtual alarm, performing backtracking analysis by using original data, searching whether the alarm event which is not reported exists, and adding the alarm event into the attack sequence again if the alarm event which is not reported is found until the complete attack path diagram is matched;

web access logs of an external portal website are analyzed to find Web attack behaviors, the Web attack behaviors can be found and attack tracing can be carried out, and Web application under Web attack is positioned; the capability of detecting and analyzing the password detection behavior; further comprising at least: botnet behavior detection, scanning behavior detection, password detection behavior detection, ARP deception behavior detection, DDOS attack behavior detection, malicious information release behavior detection, malicious website access behavior detection, sensitive information stealing behavior detection and the like.

The Web attack depth analysis module comprises the following contents: from session state (illegal cookie/cookie parameter tampering/forced session expiration/location change within session validity/browser change within session validity etc.), request data (malformed request body/multiple encodings/anomalies/anomalous request method/illegal URI data/anomalous request header/parameter anomalies/character set anomalies), response data analysis (anomalous response header/error code/title alteration/dynamic content alteration/response time delay etc.), identity authentication (default username/password/multiple usernames/high frequency login attempt/login failure etc.), file upload (file size anomaly/number anomaly/suffix anomaly), access frequency (request response delay, anomalous request interval, anomalous request flow/resource utilization increase), WAF attack logs and dark web connections (malicious domain names/hosts/URLs and the like) are deeply analyzed by a fuzzy comprehensive evaluation method based on an improved weighted naive Bayes classification algorithm related to features, and attacks which are missed in reporting by a traditional means are mined.

The system adopts the Flow (Flow) analysis idea and related technical means to monitor Flow information, display Flow topology, formulate a Flow compliance detection strategy, discover abnormal Flow and application behaviors, and fully store Flow information, thereby ensuring the normality and order of the network and assisting in preventing APT attack from the Flow safety perspective. Compared with the traditional analysis technology, the method has the advantages of long history period of the analyzed data and high analysis response speed. The access/visited behavior of the asset may be automatically and continuously analyzed and asset information may be determined as to what services the asset runs, what applications and ports are open, and who communicates the most frequently with the asset.

The network traffic metadata behavior analysis module specifically comprises the following contents: the method comprises the steps of automatically establishing a flow periodic baseline and a non-periodic baseline through modeling and analyzing the flow behavior of an internal network, automatically discovering the interconnection relation of equipment, predicting network congestion, adopting a support network optimization decision, and discovering network abnormality based on the baseline.

The network abnormal behavior analysis module specifically comprises botnet monitoring discovery, lost host discovery, slow scanning monitoring, malicious mail discovery and diffusion analysis.

Botnet (Botnet) refers to a one-to-many controllable network formed between a controller and an infected host by infecting a large number of hosts with bot program (Botnet) viruses using one or more propagation means. That is, a hacker organizes tens of thousands of trapped machines into one control node by using a self-written distributed denial of service attack program, and the control node is used for sending fake packets or junk data packets to make a predetermined attack target paralyze and refuse to service. Typically, worm viruses may also be utilized to form botnets. An attacker propagates bots through various ways to infect a large number of hosts on the internet, and the infected hosts receive the attacker's instructions through a control channel to form a botnet. The name botnet is used to more visually identify the nature of such hazards: the computers are unconsciously driven and directed by people like zombie groups in ancient Chinese legends, and become a tool utilized by people.

The botnet monitoring discovery is judged as the botnet according to the following judgment standards:

(5.1) the data volume transmitted by one-time communication connection between the Bot and the BotMaster is small, and the data packet is small;

(5.2) the data packet size is relatively fixed, and the types are few;

(5.3) Beacon behavior exists between Bot and BotMaster, which is shown in that the interval between two visits is relatively consistent, but the heartbeat cycles of different hosts may not be consistent;

(5.4) Bot will repeatedly request a connection because of loss of contact with BotMaster, resulting in a large difference in the number of flows in both directions;

(5.5) the communication between Bot and BotMaster is more uniform in time distribution and less affected by Day/Night;

(5.6) Bot is controlled by BotMaster to engage in malicious activities, such as DDoS, Spam, PE Download, attack behaviors, etc.;

(5.7) there is similarity in the sequence of behavior between different bots accessing the BotMaster.

The discovery of the lost host is specifically to discover the lost host through the extraction capability of APT on the characteristics of the malicious file CallBack and the capability of real-time synchronization of the characteristics and IDS.

The discovery of the lost host is specifically to discover the lost host through the extraction capability of APT on the characteristics of the malicious file CallBack and the capability of real-time synchronization of the characteristics and IDS.

The slow scan monitoring utilizes big data technology to identify slow scan attacks. Unlike the fast scan attack, it uses a slow-down half-connection attack to bypass the monitoring of the security device to obtain the resources that can be acquired. Since for conventional security devices, data information can only be monitored for a short period of time, the relevant attack behavior of slow scanning cannot be identified. The big data technology allows a large amount of long-term bottom data content to be stored, provides high-efficiency query and analysis capability, and can accurately identify the slow scanning attack.

The malicious mail discovering and spreading analysis utilizes the full-network monitoring capability and the APT analysis capability on the malicious files to analyze the spreading and influence of the malicious files based on the relation association of the receivers and the senders.

The visual display module comprises the following contents:

and generating a topological graph showing an audit data source, reflecting the network topological relation of the audit data source, marking the log quantity and the alarm event quantity of each audit data source on a topological node, and inquiring the details of the log and the alarm information by clicking the topological node by an administrator.

For the security data, the administrator can track the source and destination IP addresses and mark the IP addresses on the displayed world map.

The administrator can also conduct behavior analysis on the logs within a period of time, and the administrator can assist in positioning safety problems from a macroscopic perspective by generating a behavior analysis diagram to visually display the association relationship among the massive logs.

The method can evaluate the overall network security of the client from a macroscopic perspective, evaluate the level of the overall security management construction, provide decision support for improving the security protection capability of the client, and simultaneously provide decision support for improving the maturity of the information security management system construction of the client.

The method comprises the steps of obtaining a security domain or service system security management construction level rating through calculation of a group of hierarchical indexes representing the security domain or service system security management construction level, and accordingly showing the construction maturity of an information security management system of the security domain or service system.

The set of hierarchical indexes representing the safety management construction level is called as key management indexes, and each index item establishes a measurement standard aiming at a certain type of safety event.

The method can visually display a security management evaluation curve of each security domain or business system changing along with time, and can perform ring ratio analysis and same ratio analysis across the security domains or business systems. And the drill-down of the index item is supported for each key management index, and the focusing from macro to micro is realized.

The analysis system provided by the invention can provide high-efficiency threat information, accurately discover an internal lost host, help a security team to quickly and accurately locate the threat, provide rich context information related to the threat for analysis and response, and help a client to realize the following detection targets:

1. a data collection system driven by the results.

The threat information center takes the output of actual detection and analysis capability as a drive, the biggest difference with the situation perception platform is that the dust collector is not data, and excessive data collection only generates larger noise and affects the performance. Only necessary multi-source data are collected, investment, operation and maintenance cost is greatly reduced, the construction period is shortened, the quick effect can be achieved, and the management layer is helped to gradually establish information for large data safety construction. It is also the first intention of domestic famous threat intelligence companies to target on focused threats and intelligence drive.

2. The threat can be detected rapidly and accurately, and the controlled host can be found.

The connection between the lost host and the controlled end is accurately found in network flow by extracting attack infrastructures such as malicious domain names and IP (Internet protocol) and the like which are distributed all over the world based on mass data and a powerful analyst team. In addition, the DGA algorithm of the deep learning method is applied to find out the access to the malicious dynamically generated domain name. TIP also helps to locate a lost host by performing malware and trojan discovery in a specified directory and process on the host side.

3. And the analysis capability of the client on the threat event is improved by combining the threat intelligence data and the mass basic data.

The basic capabilities of massive network basic data, hacker organization figures and the like required by enterprise security analysis are effectively implanted into a local TIP platform, and a whole set of capability system for threat analysis is formed for a client.

Through combining together situation awareness platform and threat information platform, the enterprise can be unified behind intelligence driven's defence all people, process and technique, obtains powerful effect:

and (4) identifying important threats, namely collecting internal logs of the enterprise, and combining the logs with threat intelligence to quickly identify which feedback is most suitable for the enterprise environment.

The nature of the threat can be better understood, namely the capability of exceeding a situation awareness platform is realized, the situation and the associated rich context are added for the alarm and the event, the enterprise can be better assisted in knowing the risk, and the more targeted reflection is made.

And enriching the internal capacity of the enterprise by sharing threat data and using reliable information sources.

Data sharing and storage the platform is used as a knowledge base for historical threat data to help cope with emerging or persistent threats.

Optimizing workflow and orchestration using platform workflow to drive actions through integration with other security infrastructures, translating its own event data into internal threat intelligence (which is the most valuable intelligence). And to eliminate fragmentation from a central platform and manage the enterprise security infrastructure.

The technical solutions described above only represent the preferred technical solutions of the present invention, and some possible modifications to some parts of the technical solutions by those skilled in the art all represent the principles of the present invention, and fall within the protection scope of the present invention.