Converting an ARFF file into a transaction database in SPMF format (SPMF documentation)

The ARFF format is used by some data mining software. To provide interoperability, the GUI interface and command line interface of SPMF can read the ARFF file format since version 0.93 of SPMF and this is totally transparent to the user. When the user run an algorithm on an ARFF file, it is automatically converted. But if a file is used multiple times, it is desirable to convert the file once to avoid the overhead of repeated conversions. Hence, this example explains how to convert an ARFF file to a transaction database in SPMF format so that it can be reused multiple times. This works only for algorithms that take a transaction database as input (see details below).

How to run this example?

To run this example:

If you are using the graphical interface, (1) choose the "Convert_ARFF_file_to_transaction_database" algorithm, (2) select the input file "test.arff", (3) set the output file name (e.g. "output.txt") (4) set the maximum number of transactions to 10 and (5) click "Run algorithm".
If you want to execute this example from the command line, then execute this command:
java -jar spmf.jar run Convert_ARFF_file_to_transaction_database test.arff output.txt 10 in a folder containing spmf.jar and the example input file test.arff.

What is the input?

The input is a file in ARFF format. The file considered in this example is called test.arff and has the following content:

@RELATION emp_proj

@ATTRIBUTE ssn STRING
@ATTRIBUTE pNumber STRING
@ATTRIBUTE hours STRING
@ATTRIBUTE eName STRING
@ATTRIBUTE pName STRING
@ATTRIBUTE pLocation STRING

@DATA
'ssn:333445555','pNumber:2' ,'hours:10.0','eName:Franklin Wong' ,'pName:ProductY','pLocation:Sugarland'
'ssn:333445555','pNumber:3' ,'hours:10.0','eName:Franklin Wong' ,'pName:ProductZ','pLocation:Houston'
'ssn:333445555','pNumber:10','hours:10.0','eName:Franklin Wong' ,'pName:ProductX','pLocation:Stafford'
'ssn:333445555','pNumber:20','hours:10.0','eName:Franklin Wong' ,'pName:Reorganization','pLocation:Houston'
'ssn:123456789','pNumber:1' ,'hours:32.5','eName:Ramesh Narayan','pName:ProductX','pLocation:Bellaire'
'ssn:123456789','pNumber:2' ,'hours:7.5' ,'eName:Ramesh Narayan','pName:ProductY','pLocation:Sugarland'
'ssn:666884444','pNumber:3' ,'hours:40.0','eName:Ramesh Narayan','pName:ProductZ','pLocation:Houston'
'ssn:453453453','pNumber:1' ,'hours:20.0','eName:Joyce English' ,'pName:ProductX','pLocation:Bellaire'
'ssn:453453453','pNumber:2' ,'hours:20.0','eName:Joyce English' ,'pName:ProductY','pLocation:Sugarland'

The test ARFF file, named "emp_proj," represents a relational database linking employees to the projects they work on. It contains six attributes: a social security number (ssn) identifying each employee, a project number (pNumber), the number of hours worked on the project (hours), the employee's name (eName), the project name (pName), and the project location (pLocation). The dataset includes nine records involving three employees — Franklin Wong, Ramesh Narayan, and Joyce English — who are assigned to various projects such as ProductX, ProductY, ProductZ, and Reorganization, located in different cities including Sugarland, Houston, Stafford, and Bellaire. Each record describes one employee-project assignment along with the corresponding hours worked. When converted to SPMF's transaction database format, each record becomes a transaction where the attribute-value pairs (e.g., "ssn:333445555", "pName:ProductY") are treated as items.

It is important to note that the full ARFF format is supported except that (1) the character "=" is forbidden and (2) escape characters are not considered.

What is the output?

The output is a transaction database in SPMF format:

@CONVERTED_FROM_ARFF
@RELATION_NAME=emp_proj=
@ATTRIBUTE=ssn=STRING=
@ATTRIBUTE=pNumber=STRING=
@ATTRIBUTE=hours=STRING=
@ATTRIBUTE=eName=STRING=
@ATTRIBUTE=pName=STRING=
@ATTRIBUTE=pLocation=STRING=
@ITEM=1=ssn='ssn:333445555'
@ITEM=2=pNumber='pNumber:2'
@ITEM=3=hours='hours:10.0'
@ITEM=4=eName='eName:Franklin Wong'
@ITEM=5=pName='pName:ProductY'
@ITEM=6=pLocation='pLocation:Sugarland'
1 2 3 4 5 6
@ITEM=7=pNumber='pNumber:3'
@ITEM=8=pName='pName:ProductZ'
@ITEM=9=pLocation='pLocation:Houston'
1 3 4 7 8 9
@ITEM=10=pNumber='pNumber:10'
@ITEM=11=pName='pName:ProductX'
@ITEM=12=pLocation='pLocation:Stafford'
1 3 4 10 11 12
@ITEM=13=pNumber='pNumber:20'
@ITEM=14=pName='pName:Reorganization'
1 3 4 9 13 14
@ITEM=15=ssn='ssn:123456789'
@ITEM=16=pNumber='pNumber:1'
@ITEM=17=hours='hours:32.5'
@ITEM=18=eName='eName:Ramesh Narayan'
@ITEM=19=pLocation='pLocation:Bellaire'
11 15 16 17 18 19
@ITEM=20=hours='hours:7.5'
2 5 6 15 18 20
@ITEM=21=ssn='ssn:666884444'
@ITEM=22=hours='hours:40.0'
7 8 9 18 21 22
@ITEM=23=ssn='ssn:453453453'
@ITEM=24=hours='hours:20.0'
@ITEM=25=eName='eName:Joyce English'
11 16 19 23 24 25
2 5 6 23 24 25

The file begins with a header section containing metadata about the conversion. It starts with the tag @CONVERTED_FROM_ARFF, followed by the relation name (@RELATION_NAME=emp_proj=) and the list of attributes (@ATTRIBUTE) corresponding to the six fields from the original ARFF file.

Each unique attribute-value pair encountered in the data is then assigned a unique integer identifier, declared using @ITEM tags (e.g., @ITEM=1=ssn='ssn:333445555', @ITEM=2=pNumber='pNumber:2', etc.). These @ITEM declarations appear incrementally throughout the file as new attribute-value pairs are encountered for the first time.

Each of the nine original records is converted into a transaction represented as a line of space-separated integers, where each integer corresponds to one of the attribute-value pairs present in that record. For example, the first transaction "1 2 3 4 5 6" represents Franklin Wong working 10 hours on ProductY in Sugarland. Importantly, the conversion reuses item identifiers across transactions when the same attribute-value pair appears in multiple records, which allows pattern mining algorithms to discover associations between shared attribute values across different employee-project assignments.

The converted file can then be taken as input by frequent itemset mining and association rule mining algorithms such as Apriori and FPGrowth. You may see the other examples of the documentation to learn how to run such algorithms.