阅读说明：本技术 数据一般化装置、数据一般化方法、程序 (Data generalization device, data generalization method, and program ) 是由 长谷川聪 于 2020-02-20 设计创作，主要内容包括：提供能够仅使用小容量的主存储装置,将大规模数据高速地进行一般化处理的数据一般化装置。将属性值进行一般化的数据一般化装置包括：排序部,按照辅助存储装置上的一般化层级中的属性值的排列顺序,对辅助存储装置上的属性值进行排序；属性值取得部,将排序后的属性值的一部分从辅助存储装置取得到主存储装置上；一般化层级取得部,将作为一般化层级的一部分、即与取得到主存储装置上的属性值中的处理对象的属性值对应的一般化层级从辅助存储装置取得到主存储装置上；一般化处理部,基于取得到主存储装置上的属性值和取得到主存储装置上的一般化层级,执行一般化处理；以及再排序部,将一般化处理后的属性值移动到辅助存储装置,排序为原来的顺序。(A data generalization device capable of performing generalization processing of large-scale data at high speed using only a small-capacity main storage device is provided. The data generalization device for generalizing the attribute value includes: a sorting unit that sorts the attribute values on the auxiliary storage device in accordance with an order of arrangement of the attribute values in a generalized hierarchy on the auxiliary storage device; an attribute value acquisition unit that acquires a part of the sorted attribute values from the auxiliary storage device to the main storage device; a generalization hierarchy acquisition unit that acquires, from the auxiliary storage device to the main storage device, a generalization hierarchy that is a part of the generalization hierarchy and corresponds to the attribute value of the processing target among the attribute values acquired to the main storage device; a generalization processing unit that executes a generalization process based on the attribute values acquired in the main storage device and the generalization hierarchy acquired in the main storage device; and a re-ordering unit that moves the generalized attribute values to the auxiliary storage device and orders the attribute values in the original order.)

1. A data generalization apparatus that generalizes attribute values, comprising:

a sorting unit that sorts the attribute values on the auxiliary storage device in an order of arrangement of the attribute values in a generalized hierarchy on the auxiliary storage device;

an attribute value acquisition unit that acquires a part of the sorted attribute values from the secondary storage device to a primary storage device;

a generalization level acquisition unit that acquires, from the secondary storage device to the primary storage device, a generalization level corresponding to the attribute value to be processed out of the attribute values acquired to the primary storage device as a part of the generalization level;

a generalization processing unit that executes a generalization process based on the attribute value acquired to the main storage and the generalization level acquired to the main storage; and

and a re-ordering unit that moves the generalized attribute values to the auxiliary storage device and orders the attribute values in the original order.

2. The data generalization apparatus according to claim 1,

the generalized hierarchy acquiring unit updates the generalized hierarchy in the main storage device based on an LRU algorithm.

3. A data generalization method that generalizes attribute values, comprising:

a sorting step of sorting the attribute values on the auxiliary storage device in accordance with an order of arrangement of the attribute values in a generalized hierarchy on the auxiliary storage device;

an attribute value acquisition step of acquiring a part of the sorted attribute values from the secondary storage device to a primary storage device;

a generalization level acquisition step of acquiring, from the secondary storage device to the primary storage device, a generalization level corresponding to the attribute value to be processed out of the attribute values acquired to the primary storage device as a part of the generalization level;

a generalization processing step of performing a generalization process based on the retrieval of the attribute value on the main storage and the retrieval of the generalization level on the main storage; and

and a re-ordering step of moving the attribute values after the generalization processing to the auxiliary storage device and ordering the attribute values to the original order.

4. A program for causing a computer to function as the data generalization apparatus according to claim 1 or claim 2.

Technical Field

The present invention relates to a technique for hiding individual data on a database, and relates to a data generalization device, a data generalization method, and a program.

Background

Techniques are known for hiding individual data of a database by a certain method. For example, the hiding process of the database can be performed by performing a generalization process on each value of the database. The generalization processing is to generalize values. For example, as shown in the example of fig. 1, the gender [ male ] is generalized to [ human ], and the age [13] is generalized to [11-20 ]. In most of the generalization processes, a tree structure representing a generalized/specified relationship of values called a generalization hierarchy is prepared, and values are rewritten by tracing back the tree structure (raising the hierarchy). For example, in the case of the age hierarchy of the figure, the number of 13 is [11-20] when increasing by 1 hierarchy, and [11-40] when increasing by 2 hierarchies. The number of levels that are increased in the generalization is referred to as a level increase number.

Documents of the prior art

Non-patent document

Non-patent document 1: xianchuan ji, history of xianchuan, history of Iso field, yingchen, maozui and yangmu, jiyao, waozui, early leideng, hetian and hesheng, xilianchuan you, "study and initial evaluation of conversational nonspecific manipulations of large-scale data using nonsequential database engine (な nonspecific manipulations of pique of いた da データ for nonsequential データ ベ ー ス エ ン ジ ン を, initial value of な nonspecific manipulations of pique と by nonsequential database engine)", study in data engineering society, p377, society for electronic information and communications, 2018

Disclosure of Invention

Problems to be solved by the invention

There are roughly two ways to achieve generalization. One is a method implemented by configuring a database to be an object on a primary storage and a generalized hierarchy. Another is a method of configuring a database to be an object and a generalized hierarchy on a secondary storage device, and configuring a small amount of data on a primary storage device as appropriate.

The former can be executed at high speed, but in the case of a large-scale database, the capacity of the main storage device is insufficient, and thus a generalized process cannot be executed. The latter method is a method of arranging a database on a secondary storage device mainly using an RDBMS (relational database management system) or the like and realizing a generalized process using SQL operations (non-patent document 1), for example. However, when the generalized processing is realized in SQL without preparing SQL operations dedicated to the generalized processing, it takes time and effort, and the efficiency is low (low speed) because the operation is not dedicated.

Therefore, in the present invention, a data generalization device capable of performing generalization processing of large-scale data at high speed by using only a small-capacity main storage device is provided.

Means for solving the problems

The data generalization device of the present invention is a data generalization device for generalizing an attribute value, and includes an ordering unit, an attribute value acquisition unit, a generalization hierarchy acquisition unit, a generalization processing unit, and a re-ordering unit.

The sorting unit sorts the attribute values on the auxiliary storage device in order of arrangement of the attribute values in the generalized hierarchy on the auxiliary storage device. The attribute value acquisition unit acquires a part of the sorted attribute values from the auxiliary storage device to the main storage device. The generalization level acquisition unit acquires, from the auxiliary storage device to the main storage device, a generalization level corresponding to an attribute value to be processed out of the attribute values acquired to the main storage device as a part of the generalization level. The generalization processing unit executes generalization processing based on the attribute value acquired to the main storage device and the generalization level acquired to the main storage device. The re-sorting unit moves the attribute values after the generalization processing to the auxiliary storage device and sorts the attribute values into the original order.

Effects of the invention

According to the data generalization device of the present invention, it is possible to generalize large-scale data at high speed using only a small-capacity main storage device.

Drawings

Fig. 1 is a diagram illustrating a generalized process.

Fig. 2 is a diagram illustrating a prediction process (lookup processing).

Fig. 3 is a block diagram showing the configuration of the data generalization apparatus according to embodiment 1.

Fig. 4 is a flowchart showing the operation of the data generalization apparatus according to embodiment 1.

Fig. 5 is a diagram showing an example of update based on the LRU algorithm of a generalized hierarchy on a main storage device.

Detailed Description

Hereinafter, embodiments of the present invention will be described in detail. The same reference numerals are assigned to components having the same functions, and redundant description thereof is omitted.

< summary of processing >

In the following embodiments, a data generalization apparatus, a data generalization method, and a program are disclosed which can perform generalization processing of large-scale data at high speed in a small-capacity main storage device. The data generalization apparatus, the data generalization method, and the program according to embodiment 1 realize high-speed processing by devising an algorithm and a data structure in consideration of a characteristic that a sequential access of an auxiliary storage device (e.g., HDD) is faster than a random access, a characteristic that the auxiliary storage device (e.g., HDD) is slower than a main storage device (e.g., memory), and a characteristic that the auxiliary storage device (e.g., HDD) has a larger storage capacity than the main storage device (e.g., memory).

In the following embodiments, the generalized process is processed for each attribute on the database. In the following, for convenience of explanation, a database of attributes is assumed. In addition, when a plurality of attributes are to be generalized, the generalization processing for one attribute may be performed for each attribute.

[ step 1]

The attributes of the database to be generalized are arranged on the auxiliary storage device. In addition, the generalized hierarchy is similarly arranged on the secondary storage device. The values of the attributes are arranged on the auxiliary storage device so that the values become continuous. Here, "configured continuously …" means that data is configured continuously so as not to be interspersed with other data. When each value of the attribute is referred to, it is assumed that the attribute is continuously accessed, and when data is read from the auxiliary storage device to the main storage device, prediction processing is performed.

Specifically, as shown in the example of the prediction processing in fig. 2, when attribute values such as ages 18, 23, 34, and 13 are stored in the auxiliary storage device, it is assumed that these are sequentially read out to the main storage device. In the example of this figure, it is assumed that the main storage is capable of holding a maximum of 2 elements. First, as shown in fig. 2A, when the value of the 0 th index is accessed, the values of the 0 th and 1 st indices are transferred to the main storage device, and the value of the 0 th index of the main storage device is acquired. Next, in the case of accessing the value of the 1 st index, data is read from the main storage. After that, when the value of the 2 nd index is accessed, since there is no matching data in the main storage device, as shown in fig. 2B, the values of the 2 nd and 3 rd indexes are transferred from the auxiliary storage device to the main storage device, and the value of the 0 th index in the main storage device (matching the 2 nd index in the auxiliary storage device) is read. In this case, when data is read from the auxiliary storage device, the prediction process is executed assuming continuous access.

When reading a part of the generalization hierarchy for each value (for example, in the case of the generalization hierarchy of fig. 1 having a value of 13, a tree structure Used in the process of acquiring 13 → [11-20] → [11-40 ]) from the auxiliary storage device, it is preferable to update the generalization hierarchy based on an LRU (Least Recently Used) algorithm.

[ step 2]

The object attributes are sorted in an external sort (sort).

[ step 3]

Each attribute value of the object attribute is continuously (sequential) referred to, and a part of the corresponding generalization hierarchy is obtained for each value, and the generalization processing is performed.

[ step 4]

And returning the attribute values after the generalization processing to the original sequence.

Pre-ordered in step 2 so that the same values appear in successive states. When the values are the same, the same portion may be used for the generalization layer, and the content that has been used most recently by the LRU algorithm is arranged on the main storage device, so that the number of times of accessing the auxiliary storage device can be reduced, and the generalization processing can be efficiently performed.

In addition, the attributes are represented below as vectors, with a ^ appended to the text like a ^. In addition, this is a representation that facilitates the document creation software, meaning that in the case of a ^ for example, [ number 1] with raise suppression added to bold

The ith element of the additional vector is denoted as a_i. The peripheral represents the number of elements of the vector a as a. The generalized hierarchy is represented in uppercase and appended "^" after the character (e.g., X ^). In addition, this is a representation that facilitates the document creation software, in the case of X ^ representing the [ number 2] meaning that raise suppression symbols are appended to bold

Set to the value a of a certain attribute_iThe value of the generalized hierarchy of the hierarchy ascending number j is represented as

[ number 3]

For example, in fig. 1, when the AGE (AGE) is generalized by X ^ and the attribute value 13 is generalized by the number of hierarchy increases of 2, the hierarchy is X ^

[ number 4]

Example 1

The configuration of the data generalization apparatus according to embodiment 1 is described below with reference to fig. 3. As shown in the figure, the data generalization device 1 of the present embodiment includes a main storage device 10A, an auxiliary storage device 10B, a sorting unit 11, an attribute value acquisition unit 12, a generalization level acquisition unit 13, a generalization processing unit 14, and a reordering unit 15. The operation of each component will be described below with reference to fig. 4.

< sequence section 11 >

The sorting unit 11 sorts the attribute values on the auxiliary storage device 10B in the order of arrangement of the attribute values in the generalized hierarchy on the auxiliary storage device 10B (S11). The "arrangement order of attribute values in a generalized hierarchy" means, for example, an arrangement order of leaves indicating the end of a tree structure of a generalized hierarchy. In the example of fig. 1, the leaves at the end of the tree are arranged in the order 13, 18, 23, 34 … ….

< Attribute value acquisition unit 12 >

The attribute value acquisition unit 12 acquires a part of the sorted attribute values from the auxiliary storage device 10B to the main storage device 10A (S12). In the present embodiment, the attribute value acquisition unit 12 stores a part of the sorted attribute values in a vector c ^ (described later) prepared in advance in the main storage device 10A.

< generalized hierarchical level acquisition part 13 >

The generalization hierarchy acquiring unit 13 acquires a generalization hierarchy corresponding to the attribute value to be processed, which is a part of the generalization hierarchy, among the attribute values acquired in the main storage 10A, from the auxiliary storage 10B to the main storage 10A (S13). Note that the generalized hierarchy acquiring unit 13 acquires a part of the generalized hierarchy corresponding to the attribute value of the processing target from the auxiliary storage device 10B to the main storage device 10A, not for all the generalized hierarchies.

In the present embodiment, the generalization level acquisition unit 13 stores a part of the generalization level in a vector d ^ (described later) prepared in advance in the main storage 10A. Preferably, the generalized hierarchy acquisition unit 13 updates the generalized hierarchy in the main storage device 10A based on the LRU algorithm.

< generalized processing section 14 >

The generalization processing unit 14 executes the generalization processing based on the attribute value acquired in the main storage 10A and the generalization level acquired in the main storage 10A (S14).

< reordering portion 15 >

The re-sorting unit 15 moves the attribute values after the generalization process to the auxiliary storage device 10B and sorts them into the original order (S15).

< Algorithm 1: Large-Scale data generalization Algorithm >

The record number N, the attribute a ^ which is the generalization object, the generalization hierarchy X ^ corresponding to the attribute, the hierarchy ascending number j, and the element number M used in the main storage 10A

Output generalized attribute b ^

[1] The object attribute a is configured on the auxiliary storage device in such a manner that the values become continuous. Here, "the values are arranged so as to be continuous" means that data is arranged continuously so that other data is not included. A generalized hierarchy X is configured on the secondary storage 10B. A vector c, d of element number M is prepared on a main storage device 10A. In addition, c ^ is for the attribute and d ^ is for the vector on primary storage 10A of the generalized hierarchy.

[2] The object attribute a is sorted in an external sort. As the external sort, for example, there is a merge sort or the like (corresponding to S11 in fig. 4).

[3]:for i＝1to N do

[4]Using algorithm 2 to obtain the value a of the ith attribute_i(corresponding to S12 in fig. 4).

[5]The algorithm 3 is used to obtain the sum of_iValue of corresponding generalized hierarchy

[ number 5 ]

(corresponding to S13 in fig. 4).

[6] Set to the value of the ith attribute

[ number 6]

(corresponding to S14 in fig. 4).

[7]:end for

[8] The property b to be generalized becomes the original permutation of a (equivalent to S15 of FIG. 4).

[9]:return b^

< Algorithm 2: value acquisition based on prediction algorithm >)

Input is attribute a ^ configured on secondary storage 10B, location i of element to reference, vector c ^ on primary storage 10A

Output value a_i

[1] If the ith element does not exist in c ^ then n

[2] The i th element to the i + | c | -1 element are obtained from the auxiliary storage device and stored in c ^ 1.

[3]:end if

[4]A within return c ^ a_i

< Algorithm 3: value retrieval based on LRU Algorithm >

Input generalized level X ^ configured on the auxiliary storage 10B, value a of attribute representing position of generalized level to be referred to_iAnd the number of hierarchy rises j, the vector on main storage d ^ d

Output:

[ number 7 ]

[1]:if a_iDoes not exist in d ^ then

[2] If d ^ is full the then

[3] Replacing the least utilized elements in d ^ with

[ number 8]

[4]:else

[5]A is prepared by_iAnd the value of the generalized hierarchy corresponding thereto

[ number 9 ]

Is arranged into

[ number 10 ]

And added to d ^ d.

[6]:end if

[7]:end if

[8] Within return d ^ [ number 11 ]

[ number 12 ] of

As shown in the example of fig. 5, for example, when the attribute values (T, U, V, W) are stored for the vector c on the main storage device 10A and the attribute values (Q ← T, Q ← U, R ← V) which are part of the generalization hierarchy are stored for the vector d on the main storage device 10A, the R ← V with the smallest number of reference times is replaced with the one with a new generalization hierarchy by the LRU algorithm (S ← W). In addition, since the generalized hierarchy stored in the vector d in the main storage device 10A is already used at the time of replacement and is not reused, there is no problem in replacing the vector with any algorithm other than the LRU algorithm.

< Effect >

In the generalization of data, it is common to move data from the auxiliary storage device to the main storage device, and after performing calculation on the main storage device, to move data again from the main storage device to the auxiliary storage device. In addition, when accessing the auxiliary storage device, it is important to perform continuous access instead of random access for high speed, so that the processing efficiency does not fall behind.

According to the data generalization device 1 of the present embodiment, by sorting data in advance or the like and selecting a manner of calling data from the auxiliary storage device 10B such as an attribute value and a generalization level in a form suitable for generalization processing, the number of accesses to the auxiliary storage device 10B can be reduced, efficient continuous access can be achieved when accessing the auxiliary storage device 10B, and large-scale data can be generalized at high speed using only the small-capacity main storage device 10A.

< appendix >)

The apparatus of the present invention, for example, as a single hardware entity, has: an input Unit to which a keyboard and the like can be connected, an output Unit to which a liquid crystal display and the like can be connected, a communication Unit to which a communication device (for example, a communication cable) capable of communicating with the outside of a hardware entity can be connected, a CPU (Central Processing Unit (CPU) which may have a cache memory, a register, or the like), a RAM (random access memory) or a ROM (read only memory) as a memory, an external storage device as a hard disk, and a bus connected to enable interaction of data among the input Unit, the output Unit, the communication Unit, the CPU, the RAM, the ROM, and the external storage device. Further, a device (drive) capable of reading and writing a storage medium such as a CD-ROM may be provided in the hardware entity as necessary. As a physical entity having such hardware resources, there is a general-purpose computer or the like.

The external storage device of the hardware entity stores a program necessary for realizing the above-described functions, data necessary for processing the program, and the like (the external storage device is not limited to the external storage device, and may be stored in a ROM as a dedicated storage device for a read program, for example). Data and the like obtained by processing these programs are appropriately stored in the RAM, the external storage device, and the like.

In the hardware entity, each program stored in an external storage device (or ROM or the like) and data necessary for processing the program are read into a memory as necessary, and are appropriately interpreted and executed/processed by a CPU. As a result, the CPU realizes specific functions (each of the components shown in the … section, … section, and the like).

The present invention is not limited to the above-described embodiments, and can be modified as appropriate within a scope not departing from the gist of the present invention. The processes described in the above embodiments may be executed in time series in accordance with the order described, or may be executed in parallel or individually in accordance with the processing capability of the apparatus that executes the processes or the need.

As described above, when the processing function in the hardware entity (the apparatus of the present invention) described in the above embodiment is realized by a computer, the processing content of the function to be possessed by the hardware entity is described by a program. Then, by executing the program on the computer, the processing functions in the above-described hardware entities are realized on the computer.

The program describing the processing content may be stored in a computer-readable storage medium in advance. Examples of the computer-readable storage medium include a magnetic storage device, an optical disk, an magneto-optical storage medium, and a semiconductor memory. Specifically, for example, a hard disk device, a flexible disk, a magnetic tape, or the like can be used as a magnetic storage device, and a DVD (Digital Versatile Disc), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disc Read Only Memory), a CD-R (Recordable/ReWritable), an MO (Magneto-Optical Disc), or the like can be used as an Optical Disc, an MO (magnetic-Optical Disc), or the like can be used as an Optical-magnetic storage medium, and an EEP-ROM (Electronically Erasable programmable Read Only Memory), or the like can be used as a semiconductor Memory.

The distribution of the program is performed by, for example, selling, transferring, and lending a portable storage medium such as a DVD or a CD-ROM on which the program is stored. Further, the program may be stored in a storage device of a server computer, and the program may be distributed by transferring the program from the server computer to another computer via a network.

The computer executing such a program first temporarily stores a program stored in a portable storage medium or a program transferred from a server computer in its own storage device, for example. Then, when executing the processing, the computer reads the program stored in its own storage medium and executes the processing in accordance with the read program. In addition, as another execution form of the program, the computer may directly read the program from the portable storage medium, execute the processing according to the program, and may sequentially execute the processing according to the received program each time the program is transferred from the server computer to the computer. Further, the processing function may be realized by only the execution instruction and the result acquisition without transferring the program from the server computer to the computer, and the processing may be executed by a so-called ASP (Application Service Provider) type Service. In addition, the program of the present embodiment includes information conforming to the program (data or the like that specifies the nature of the processing of the computer, not a direct instruction to the computer) as information used for the processing of the electronic computer.

In this embodiment, although a hardware entity is configured by executing a specific program on a computer, at least a part of the processing contents may be implemented only in hardware.