Calculate Statistics for a Sequence Database with Utility Information(SPMF documentation)
This example explains how to calculate statistics for a Sequence Database with Utility Information using the SPMF open-source data mining library.
How to run this example?
- If you are using the graphical interface, (1) choose the "Calculate_stats_for_a_sequence_database_with_utility" algorithm, (2) choose the input file DataBase_HUSRM.txt (3) click "Run algorithm".
- If you want to execute this example from the command line,
then execute this command:
java -jar spmf.jar run Calculate_stats_for_a_sequence_database_with_utility DataBase_HUSRM.txt no_output_file in a folder containing spmf.jar and the input file DataBase_HUSRM.txt. - If you are using the source code version of SPMF, launch the file "MainTestStatsSequenceDatabaseUtility.java" in the package ca.pfv.SPMF.tests.
What is this tool?
This tool is a tool for generating statistics about a sequence database with utility information, as used by algorithms such as HUSRM and USpan.
What is the input?
The input is a sequence database with utility information.
Let's consider the following sequence database consisting of 4 sequences of transactions (s1,s2, s3, s4) and 7 items (1, 2, 3, 4, 5, 6, 7). This database is provided in the text file "DataBase_HUSRM.txt" in the package ca.pfv.spmf.tests of the SPMF distribution.
| Sequence | Sequence utility | |
| s1 | {1[1],2[4]},{3[10]},{6[9]},{7[2]},{5[1]} | 27 |
| s2 | {1[1],4[12]},{3[20]},{2[4]},{5[1],7[2]} | 40 |
| s3 | {1[1]},{2[4]},{6[9]},{5[1]} | 15 |
| s4 | {1[3],2[4],3[5]},{6[3],7[1]} | 16 |
Each line of the database is a sequence:
- each sequence is an ordered list of transactions, such that transactions are enclosed by {} in this Example
- each transaction contains a set of items represented by integers
- each item is annotated with a utility value (e.g. sale profit), indicated between squared brackets [ ].
- the sum of the utilities (e.g. profit) of all items in the sequence is also indicated (the "sequence utility" column)
What are real-life examples of such a database? A typical example is a database containing sequences of customer transactions. Imagine that each sequence represents the transactions made by a customer. The first customer named "s1" bought items 1 and 2, and those items respectively generated a profit of 1$ and 4$. Then, the customer bought item 3 for 10$. Then, the customer bought item 6 for 9 $. Then, the customer bought items 7 for 2$. Then the customer bought item 5 for 1 $.
What is the output?
The output is statistics about the database. For example, if we use the tool on the previous database given as example, we get the following statistics:
============ SEQUENCE UTILITY DATABASE STATS ==========
File size (MB): 0.00
Number of sequences : 4
Max item: 7
Average number of itemsets per sequence : 3.75 standard deviation: 1.0897247358851685 variance: 1.1875000000000002
Average number of items per itemset : 1.4 standard deviation: 0.6110100926607788 variance: 0.37333333333333346
Average utility per item: 4.666666666666667 standard deviation: 4.734205151099645 variance: 22.41269841269841
Average utility per sequence: 24.5 standard deviation: 10.111874208078342 variance: 102.25
What is the input file format?
The input file format is defined as follows. It is a text file.
- Each line represents a sequence of transactions.
- Each transaction is separated by the keyword -1.
- A transaction is a list of items (positive integers) separated by single spaces and where each item is annotated with a generated sale profit indicated between square brackets [ ]. The sale profit is a positive integer.
- In a transaction, it is assumed that items are sorted according to some order (eg. alphabetical order).
- Each sequence ends by the keyword "-2". Then, it is followed by the keyword "SUtility:" followed by the sum of the utility (profit) of all items in that sequence.
For example, for the previous example, the input file is defined as follows:
@FILETYPE="Sequence database with utility values"
@SOURCE="SPMF SOFTWARE https://philippe-fournier-viger.com/spmf/"
1[1] 2[4] -1 3[10] -1 6[9] -1 7[2] -1 5[1] -1 -2 SUtility:27
1[1] 4[12] -1 3[20] -1 2[4] -1 5[1] 7[2] -1 -2 SUtility:40
1[1] -1 2[4] -1 6[9] -1 5[1] -1 -2 SUtility:15
1[3] 2[4] 3[5] -1 6[3] 7[1] -1 -2 SUtility:16
The first two lines are optional metadata. The first line indicate the type of file and the second line describes the source of the data. Then it is the data section. consider the first line of data. It means that the first customer bought items 1 and 2, and those items respectively generated a profit of 1$ and 4$. Then, the customer bought item 3 for 10$. Then, the customer bought item 6 for 9 $. Then, the customer bought items 7 for 2$. Then the customer bought item 5 for 1 $. Thus, this customer has made 5 transaction. The total utility (profit) generated by that sequence of transactions is 1$ + 4$ + 10$ + 9$ + 2$ + 1$ = 27 $.