阅读说明：本技术 一种基于云计算分布式网络视频数据挖掘采集系统 (Distributed network video data mining and collecting system based on cloud computing ) 是由 黄铁成 于 2021-09-03 设计创作，主要内容包括：本发明提供一种基于云计算分布式网络视频数据挖掘采集系统,包括互联网大平台,互联网大平台连接有数据采集模块,数据采集模块连接有数据预处理模块,数据预处理模块连接有管理调度模块,管理调度模块连接有终端设备,本发明：通过设有的数据采集模块,爬虫或开放API采集数据将爬取的信息传至接收数据并通过发送请求发送到资源解析处,资源解析将信息中的时间、音频、视屏和位置信息进行分类别收集,解析后的信息发送到统计站进行统计,敏感信息剔除后保留在存储箱内部,设有的数据预处理模块,将获取的数据进行解码处理,管理调度模块将处理后的数据进行分类统计,最终分布存储。(The invention provides a distributed network video data mining and collecting system based on cloud computing, which comprises an internet large platform, wherein the internet large platform is connected with a data collecting module, the data collecting module is connected with a data preprocessing module, the data preprocessing module is connected with a management scheduling module, and the management scheduling module is connected with terminal equipment, and the cloud computing distributed network video data mining and collecting system comprises the following components: through the data acquisition module that is equipped with, crawler or open API data collection will crawl the information and pass to the received data and send resource analysis department through sending the request, resource analysis carries out the classification with time, audio frequency, look screen and the positional information in the information and collects, information after the analysis is sent to the statistics station and is counted, sensitive information rejects the back and remains in the storage box inside, the data preprocessing module that is equipped with decodes the data that will acquire, the management scheduling module carries out classification statistics with the data after handling, final distributed storage.)

1. The utility model provides a video data mining collection system based on cloud computing distributed network, includes big platform of internet (1), its characterized in that: the internet large platform (1) is connected with a data acquisition module (2), the data acquisition module (2) is connected with a data preprocessing module (3), the data preprocessing module (3) is connected with a management scheduling module (4), and the management scheduling module (4) is connected with a terminal device (5).

2. The video data mining and collecting system based on the cloud computing distributed network according to claim 1, wherein: the data acquisition module (2) comprises a crawler or an open API acquisition data (21), the crawler or the open API acquisition data (21) are connected with a receiving data (22), the receiving data (22) are connected with a sending request (23), the receiving data (22) are connected with a resource analysis (24), the sending request (23) are connected with the resource analysis (24), and the resource analysis (24) is connected with a storage box (25).

3. The video data mining and collecting system based on the cloud computing distributed network according to claim 2, wherein: the resource analysis (24) is respectively connected with a time acquisition unit (241), an audio acquisition unit (242), a video acquisition unit (243) and a position acquisition unit (244), the time acquisition unit (241), the audio acquisition unit (242), the video acquisition unit (243) and the position acquisition unit (244) are all connected with a statistic station (26), and the statistic station (26) is connected with an evaluation unit (27).

4. The video data mining and collecting system based on the cloud computing distributed network according to claim 3, wherein: the evaluation part (27) is connected with a garbage can (28), the evaluation part (27) is connected with a transmission station (29), and the transmission station (29) is connected with a storage box (25).

5. The video data mining and collecting system based on the cloud computing distributed network according to claim 1, wherein: the data preprocessing module (3) comprises a decoding unit (31), a noise cleaning module (32) and a resolution (33), wherein the noise cleaning module (32) is connected with a QNC260 module (34).

6. The video data mining and collecting system based on the cloud computing distributed network according to claim 5, wherein: the decoding unit (31) comprises a MUC processor (35), a data receiving port (36) is connected to the MUC processor (35), a decoding processor (37) is connected to the MUC processor (35), the data receiving port (36) is connected to the inside of the decoding processor (37), an analog-to-digital converter (38) is bidirectionally connected to the decoding processor (37), and a digital-to-analog converter (39) is connected to the decoding processor (37).

7. The video data mining and collecting system based on the cloud computing distributed network according to claim 1, wherein: the management scheduling module (4) comprises a data backup (41), a category statistic (42), a site mark (43) and a distributed storage (44), wherein the data backup (41), the category statistic (42) and the site mark (43) are all connected with the distributed storage (44), and the distributed storage (44) is connected with a copy port (45).

8. The video data mining and collecting system based on the cloud computing distributed network according to claim 1, wherein: the terminal equipment (5) is connected with an administrator scheduling (7).

9. The video data mining and collecting system based on the cloud computing distributed network according to claim 1, wherein: the data acquisition module (2), the data preprocessing module (3), the management scheduling module (4), the terminal equipment (5) and the internet large platform (1) are in network transmission connection through a local area network (6).

10. The video data mining and collecting system based on the cloud computing distributed network according to claim 1, wherein: the data acquisition module (2), the data preprocessing module (3), the management scheduling module (4), the terminal equipment (5) and the administrator scheduling module (7) are electrically connected with an external power supply.

Technical Field

The invention relates to the technical field of data mining, in particular to a distributed network video data mining and collecting system based on cloud computing.

Background

Data mining is translated into data exploration and data mining. It is a method to analyze a large amount of data stored in an enterprise through a mathematical model to find out different customers or market segments and to analyze consumer's preferences and behaviors. It is a step in database knowledge discovery. Data mining generally refers to a process of automatically searching for information hidden therein with special relationships from a large amount of data. Mainly comprises 3 steps of data preparation, rule searching and rule representation. The existing data mining task can not screen and eliminate abnormal and sensitive information in the massive data mining process, and the data types can not be distinguished and counted, so that the data can not be effectively sorted.

Disclosure of Invention

The invention aims to provide a cloud computing-based distributed network video data mining and collecting system, aiming at solving the problems that abnormal and sensitive information existing in the process of mining mass data cannot be screened and eliminated, data types cannot be distinguished and counted again, and data cannot be effectively sorted in the existing data mining task.

In order to achieve the purpose, the invention provides the following technical scheme: the utility model provides a video data excavates collection system based on cloud calculates distributing type network, includes the big platform of internet, the big platform of internet is connected with data acquisition module, data acquisition module is connected with data preprocessing module, data preprocessing module is connected with management scheduling module, management scheduling module is connected with terminal equipment.

In order to make the network data crawled, as a preferred scheme of the invention: the data acquisition module comprises a crawler or an open API (application program interface) acquisition data, the crawler or the open API acquisition data are connected with receiving data, the receiving data are connected with a sending request, the receiving data are connected with a resource analysis, the sending request is connected with the resource analysis, and the resource analysis is connected with a storage box.

In order to classify the information data, as a preferred embodiment of the present invention: the resource analysis is respectively connected with time acquisition, audio acquisition, video screen acquisition and position acquisition, the time acquisition, the audio acquisition, the video screen acquisition and the position acquisition are all connected with a statistics station, and the statistics station is connected with an evaluation part.

In order to make the retention of the screened data, as a preferred embodiment of the present invention: the evaluation part is connected with a garbage can, the evaluation part is connected with a transmission station, and the transmission station is connected with the storage box.

In order to convert the collected data, as a preferred aspect of the present invention: the data preprocessing module comprises a decoding unit, noise cleaning and resolution, and the noise cleaning is connected with the QNC module.

In order to further process the data, as a preferred scheme of the invention: the decoding unit comprises an MUC processor, the MUC processor is connected with a data receiving port, the MUC processor is connected with a decoding processor, the data receiving port is connected to the inside of the decoding processor, the decoding processor is bidirectionally connected with an analog-to-digital converter, and the decoding processor is connected with a digital-to-analog converter.

In order to classify the mined information, as a preferred scheme of the present invention: the management scheduling module comprises data backup, category statistics, site marks and distributed storage, wherein the data backup, the category statistics and the site marks are all connected with the distributed storage, and the distributed storage is connected with a copy port.

In order to apply the mined data, as a preferred scheme of the invention: the terminal equipment is connected with an administrator for scheduling.

In order to transmit the modules to each other, as a preferred scheme of the invention: the data acquisition module, the data preprocessing module, the management scheduling module, the terminal equipment and the internet large platform are in transmission connection through a local area network.

In order to operate the electronic device, as a preferred embodiment of the present invention: the data acquisition module, the data preprocessing module, the management scheduling module, the terminal equipment and the administrator scheduling module are all electrically connected with the external power supply.

Compared with the prior art, the distributed network video data mining and collecting system based on cloud computing provided by the invention has the following beneficial effects:

1) crawling information is collected by a crawler or an open API through an arranged data collection module and is transmitted to received data and is transmitted to a resource analysis position through a transmission request, time, audio, video and position information in the information is collected by the resource analysis in a classified mode, the analyzed information is transmitted to a statistics station for statistics, useless or sensitive information is removed, and the rest information data is reserved in a storage box through a transmission station and provides a data source for the next processing;

2) through the data preprocessing module that is equipped with, the data that will acquire are decoded, carry out transform processing to the data that need convert, and the flaw in audio frequency and the visual screen is handled, and the processing of making an uproar is fallen and is handled the improvement image quality definition for the data of digging has better usability, and the management scheduling module that is equipped with carries out categorised statistics with the data after handling, and to the page link after crawling and website etc. mark, and final distribution storage conveniently copies the analysis.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:

FIG. 1 is a schematic view of the overall structure of the present invention;

FIG. 2 is a schematic diagram of a data acquisition module according to the present invention;

FIG. 3 is a block diagram of a data preprocessing module according to the present invention;

FIG. 4 is a diagram illustrating a decoding unit according to the present invention;

FIG. 5 is a diagram illustrating a structure of a management scheduling module according to the present invention.

In the figure: 1. an internet large platform; 2. a data acquisition module; 21. collecting data by a crawler or an open API; 22. receiving data; 23. sending a request; 24. analyzing resources; 241. collecting time; 242. audio acquisition; 243. collecting a view screen; 244. position collection; 25. a storage box; 26. a statistics station; 27. an evaluation unit; 28. a trash can; 29. a transmission station; 3. a data preprocessing module; 31. a decoding unit; 32. noise cleaning; 33. resolution ratio; 34. a QNC260 module; 35. a MUC processor; 36. a data receiving port; 37. a decoding processor; 38. an analog/digital converter; 39. a digital-to-analog converter; 4. a management scheduling module; 41. data backup; 42. carrying out category statistics; 43. marking the station; 44. distributed storage; 45. copying a port; 5. a terminal device; 6. a local area network; 7. and (5) scheduling by an administrator.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example 1

Referring to fig. 1-5, the present invention provides the following technical solutions: the utility model provides a video data excavates collection system based on cloud calculates distributing type network, includes the big platform of internet 1, and big platform of internet 1 is connected with data acquisition module 2, and data acquisition module 2 is connected with data preprocessing module 3, and data preprocessing module 3 is connected with management scheduling module 4, and management scheduling module 4 is connected with terminal equipment 5.

In this embodiment: the data acquisition module 2 comprises a crawler or open API acquisition data 21, the crawler or open API acquisition data 21 is connected with a receiving data 22, the receiving data 22 is connected with a sending request 23, the receiving data 22 is connected with a resource analysis 24, the sending request 23 is connected with the resource analysis 24, and the resource analysis 24 is connected with a storage box 25.

Specifically, data on the internet large platform 1 is extensively crawled by using a crawler or an open API to collect data 21, the obtained information is directly sent to a resource analysis 24 for analysis through receiving data 22, partial data needing authorization is sent to a sending request 23, the sending request 23 is linked to the resource analysis 24 through copying to analyze resources, and the analyzed resources are stored in a storage box 25 in advance.

In this embodiment: the resource analysis 24 is connected with a time acquisition 241, an audio acquisition 242, a view acquisition 243, and a position acquisition 244, respectively, the time acquisition 241, the audio acquisition 242, the view acquisition 243, and the position acquisition 244 are connected with a statistical station 26, and the statistical station 26 is connected with an evaluation unit 27.

Specifically, the resource analysis 24 analyzes the information, decomposes data in the information, records analysis time by the time acquisition 241, acquires images and sounds in the information by the audio acquisition 242 and the video acquisition 243, records acquired link positions by the position acquisition 244, and finally sends the information to the data transmission statistical station 26 for statistical data amount, and the statistical station 26 sends the information to the evaluation part 27 for data evaluation.

In this embodiment: the evaluation section 27 is connected to a trash can 28, the evaluation section 27 is connected to a transfer station 29, and the transfer station 29 is connected to the storage box 25.

Specifically, the evaluated data is processed, part of sensitive information and garbage information are removed, the information is delivered to the interior of the garbage can 28, and the rest required information is transmitted to the interior of the storage box 25 through the transmission station 29 and provides resource calling for the data preprocessing module 3.

In this embodiment: the data preprocessing module 3 comprises a decoding unit 31, a noise cleaning 32 and a resolution 33, wherein the noise cleaning 32 is connected with a QNC260 module 34.

Specifically, the evaluated data enters the data preprocessing module 3, and is cleaned by the noise cleaning module 32, the noise in the audio can be reduced to the maximum extent by the QNC260 module 34 connected to the noise cleaning module 32, and the data to be converted is processed by the decoding unit 31.

In this embodiment: the decoding unit 31 includes a MUC processor 35, the MUC processor 35 is connected to a data receiving port 36, the MUC processor 35 is connected to a decoding processor 37, the data receiving port 36 is connected to the inside of the decoding processor 37, the decoding processor 37 is bidirectionally connected to an analog/digital converter 38, and the decoding processor 37 is connected to a digital/analog converter 39.

Specifically, the data to be decoded is processed by the MUC processor 35, the audio data stream in the analog-to-digital converter 38 and the digital-to-analog converter 39 is decoded and restored to an analog audio-video signal by the decoding processor 37, and the processed information is transmitted to the outside through the data receiving port 36 for further processing.

In this embodiment: the management scheduling module 4 includes a data backup 41, a category statistic 42, a site mark 43, and a distributed storage 44, where the data backup 41, the category statistic 42, and the site mark 43 are all connected to the distributed storage 44, and the distributed storage 44 is connected to a copy port 45.

Specifically, the information data after being subjected to screening processing and cleaning and decoding enters the management scheduling module 4, the data is backed up through the data backup 41, category statistics 42 is performed, the sites of the information are marked through the site markers 43, the number of subsequent revisits is conveniently counted, and the processed data is finally stored through the distributed storage 44.

In this embodiment: the terminal device 5 is connected with an administrator schedule 7.

Specifically, the terminal device 5 receives the processed information, and finally overlaps the administrator schedule 7, and allocates and uses the mined information resource by the administrator schedule 7.

In this embodiment: the data acquisition module 2, the data preprocessing module 3, the management scheduling module 4, the terminal device 5 and the internet large platform 1 are in network transmission connection through the local area network 6.

Specifically, in the data acquisition process, the preprocessing process and the management scheduling process, the modules and the terminal device 5 all realize data transmission and operation processing through the local area network 6, and the internet large platform 1 provides an information acquisition place for data acquisition through the local area network 6.

In this embodiment: the data acquisition module 2, the data preprocessing module 3, the management scheduling module 4, the terminal device 5 and the administrator scheduling module 7 are electrically connected with an external power supply.

Specifically, after the data acquisition module 2, the data preprocessing module 3, the management scheduling module 4, the terminal device 5 and the administrator scheduling module 7 are connected with the external power supply, the device starts to enter an operating state to perform mining and carding on the data, and when the device is disconnected with the external power supply, the device stops operating.

Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that changes may be made in the embodiments and/or equivalents thereof without departing from the spirit and scope of the invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.