Converting a Text file into a transaction database in SPMF format (SPMF documentation)

To provide interoperability, the GUI interface and command line interface of SPMF can read text files since version 0.93 of SPMF and this is totally transparent to the user as long as the input text file has the extension ".text". When the user run an algorithm, the input text file is automatically converted. But if a file is used multiple times, it is desirable to convert the file once to avoid the overhead of repeated conversions. Hence, this example explains how to convert a TEXT file to a sequence database in SPMF format so that it can be reused multiple times. This works only for algorithms that take a sequence database as input such as sequential pattern mining and sequential rule mining algorithms.

How to run this example?

To run this example:

If you are using the graphical interface, (1) choose the "Convert_TEXT_file_to_sequence_database" algorithm, (2) select the input file "moby_fragment.txt", (3) set the output file name (e.g. "output.txt") (4) set the maximum number of sequences to 100 and (5) click "Run algorithm".
If you want to execute this example from the command line, then execute this command:
java -jar spmf.jar run Convert_TEXT_file_to_sequence_database moby_fragment.txt output.txt 100 in a folder containing spmf.jar and the example input file moby_fragment.txt.

What is the input?

The input is a file in text file. The file considered in this example is called moby_fragment.txt and has the following content:

“All ye mast-headers have before now heard me give orders about a white whale. Look ye! d’ye see this Spanish ounce of gold?”—holding up a broad bright coin to the sun—“it is a sixteen dollar piece, men. D’ye see it? Mr. Starbuck, hand me yon top-maul.”
While the mate was getting the hammer, Ahab, without speaking, was slowly rubbing the gold piece against the skirts of his jacket, as if to heighten its lustre, and without using any words was meanwhile lowly humming to himself, producing a sound so strangely muffled and inarticulate that it seemed the mechanical humming of the wheels of his vitality in him.
Receiving the top-maul from Starbuck, he advanced towards the main-mast with the hammer uplifted in one hand, exhibiting the gold with the other, and with a high raised voice exclaiming: “Whosoever of ye raises me a white-headed whale with a wrinkled brow and a crooked jaw; whosoever of ye raises me that white-headed whale, with three holes punctured in his starboard fluke—look ye, whosoever of ye raises me that same white whale, he shall have this gold ounce, my boys!”
“Huzza! huzza!” cried the seamen, as with swinging tarpaulins they hailed the act of nailing the gold to the mast.
“It’s a white whale, I say,” resumed Ahab, as he threw down the topmaul: “a white whale. Skin your eyes for him, men; look sharp for white water; if ye see but a bubble, sing out.”
All this while Tashtego, Daggoo, and Queequeg had looked on with even more intense interest and surprise than the rest, and at the mention of the wrinkled brow and crooked jaw they had started as if each was separately touched by some specific recollection.
“Captain Ahab,” said Tashtego, “that white whale must be the same that some call Moby Dick.”
“Moby Dick?” shouted Ahab. “Do ye know the white whale then, Tash?”
“Does he fan-tail a little curious, sir, before he goes down?” said the Gay-Header deliberately.
“And has he a curious spout, too,” said Daggoo, “very bushy, even for a parmacetty, and mighty quick, Captain Ahab?”
“And he have one, two, three—oh! good many iron in him hide, too, Captain,” cried Queequeg disjointedly, “all twiske-tee be-twisk, like him—him—” faltering hard for a word, and screwing his hand round and round as though uncorking a bottle—“like him—him—”
“Corkscrew!” cried Ahab, “aye, Queequeg, the harpoons lie all twisted and wrenched in him; aye, Daggoo, his spout is a big one, like a whole shock of wheat, and white as a pile of our Nantucket wool after the great annual sheep-shearing; aye, Tashtego, and he fan-tails like a split jib in a squall. Death and devils! men, it is Moby Dick ye have seen—Moby Dick—Moby Dick!”
“Captain Ahab,” said Starbuck, who, with Stubb and Flask, had thus far been eyeing his superior with increasing surprise, but at last seemed struck with a thought which somewhat explained all the wonder. “Captain Ahab, I have heard of Moby Dick—but it was not Moby Dick that took off thy leg?”

This is an excerpt from the Moby Dick book.

What is the output?

The output is a sequence database in SPMF format:

@CONVERTED_FROM_TEXT
@ITEM=1=all
@ITEM=2=ye
@ITEM=3=mastheaders
@ITEM=4=have
@ITEM=5=before
@ITEM=6=now
@ITEM=7=heard
@ITEM=8=me
@ITEM=9=give
@ITEM=10=orders
@ITEM=11=about
@ITEM=12=a
@ITEM=13=white
@ITEM=14=whale
1 -1 2 -1 3 -1 4 -1 5 -1 6 -1 7 -1 8 -1 9 -1 10 -1 11 -1 12 -1 13 -1 14 -1 -2
@ITEM=15=look
15 -1 2 -1 -2
@ITEM=16=d
@ITEM=17=see
@ITEM=18=this
@ITEM=19=spanish
@ITEM=20=ounce
@ITEM=21=of
@ITEM=22=gold
16 -1 2 -1 17 -1 18 -1 19 -1 20 -1 21 -1 22 -1 -2
@ITEM=23=holding
@ITEM=24=up
@ITEM=25=broad
@ITEM=26=bright
@ITEM=27=coin
@ITEM=28=to
@ITEM=29=the
@ITEM=30=sun
@ITEM=31=it
@ITEM=32=is
@ITEM=33=sixteen
@ITEM=34=dollar
@ITEM=35=piece
@ITEM=36=men
23 -1 24 -1 12 -1 25 -1 26 -1 27 -1 28 -1 29 -1 30 -1 31 -1 32 -1 12 -1 33 -1 34 -1 35 -1 36 -1 -2
16 -1 2 -1 17 -1 31 -1 -2
@ITEM=37=mr
37 -1 -2
@ITEM=38=starbuck
@ITEM=39=hand
@ITEM=40=yon
@ITEM=41=topmaul
38 -1 39 -1 8 -1 40 -1 41 -1 -2
@ITEM=42=while
@ITEM=43=mate
@ITEM=44=was
@ITEM=45=getting
@ITEM=46=hammer
@ITEM=47=ahab
@ITEM=48=without
@ITEM=49=speaking
@ITEM=50=slowly
@ITEM=51=rubbing
@ITEM=52=against
@ITEM=53=skirts
@ITEM=54=his
@ITEM=55=jacket
@ITEM=56=as
@ITEM=57=if
@ITEM=58=heighten
@ITEM=59=its
@ITEM=60=lustre
@ITEM=61=and
@ITEM=62=using
@ITEM=63=any
@ITEM=64=words
@ITEM=65=meanwhile
@ITEM=66=lowly
@ITEM=67=humming
@ITEM=68=himself
@ITEM=69=producing
@ITEM=70=sound
@ITEM=71=so
@ITEM=72=strangely
@ITEM=73=muffled
@ITEM=74=inarticulate
@ITEM=75=that
@ITEM=76=seemed
@ITEM=77=mechanical
@ITEM=78=wheels
@ITEM=79=vitality
@ITEM=80=in
@ITEM=81=him
42 -1 29 -1 43 -1 44 -1 45 -1 29 -1 46 -1 47 -1 48 -1 49 -1 44 -1 50 -1 51 -1 29 -1 22 -1 35 -1 52 -1 29 -1 53 -1 21 -1 54 -1 55 -1 56 -1 57 -1 28 -1 58 -1 59 -1 60
-161 -1 48 -1 62 -1 63 -1 64 -1 44 -1 65 -1 66 -1 67 -1 28 -1 68 -1 69 -1 12 -1 70 -1 71 -1 72 -1 73 -1 61 -1 74 -1 75 -1 31 -1 76 -1 29 -1 77 -1 67 -1 21 -1 29 -1 78
-1 21 -1 54 -1 79 -1 80 -1 81 -1 -2
@ITEM=82=receiving
@ITEM=83=from
@ITEM=84=he
@ITEM=85=advanced
@ITEM=86=towards
@ITEM=87=mainmast
@ITEM=88=with
@ITEM=89=uplifted
@ITEM=90=one
@ITEM=91=exhibiting
@ITEM=92=other
@ITEM=93=high
@ITEM=94=raised
@ITEM=95=voice
@ITEM=96=exclaiming
@ITEM=97=whosoever
@ITEM=98=raises
@ITEM=99=whiteheaded
@ITEM=100=wrinkled
@ITEM=101=brow
@ITEM=102=crooked
@ITEM=103=jaw
@ITEM=104=three
@ITEM=105=holes
@ITEM=106=punctured
@ITEM=107=starboard
@ITEM=108=fluke
@ITEM=109=same
@ITEM=110=shall
@ITEM=111=my
@ITEM=112=boys
82 -1 29 -1 41 -1 83 -1 38 -1 84 -1 85 -1 86 -1 29 -1 87 -1 88 -1 29 -1 46 -1 89 -1 80 -1 90 -1 39 -1 91 -1 29 -1 22 -1 88 -1 29 -1 92 -1 61 -1 88 -1 12 -1 93 -1 94 -1
95 -1 96 -1 97 -1 21 -1 2 -1 98 -1 8 -1 12 -1 99 -1 14 -1 88 -1 12 -1 100 -1 101 -1 61 -1 12 -1 102 -1 103 -1 97 -1 21 -1 2 -1 98 -1 8 -1 75 -1 99 -1 14 -1 88 -1 104
-1 105 -1 106 -1 80 -1 54 -1 107 -1 108 -1 15 -1 2 -1 97 -1 21 -1 2 -1 98 -1 8 -1 75 -1 109 -1 13 -1 14 -1 84 -1 110 -1 4 -1 18 -1 22 -1 20 -1 111 -1 112 -1 -2
@ITEM=113=huzza
113 -1 -2
113 -1 -2
@ITEM=114=cried
@ITEM=115=seamen
@ITEM=116=swinging
@ITEM=117=tarpaulins
@ITEM=118=they
@ITEM=119=hailed
@ITEM=120=act
@ITEM=121=nailing
@ITEM=122=mast
114 -1 29 -1 115 -1 56 -1 88 -1 116 -1 117 -1 118 -1 119 -1 29 -1 120 -1 21 -1 121 -1 29 -1 22 -1 28 -1 29 -1 122 -1 -2
@ITEM=123=s
@ITEM=124=i
@ITEM=125=say
@ITEM=126=resumed
@ITEM=127=threw
@ITEM=128=down
31 -1 123 -1 12 -1 13 -1 14 -1 124 -1 125 -1 126 -1 47 -1 56 -1 84 -1 127 -1 128 -1 29 -1 41 -1 12 -1 13 -1 14 -1 -2
@ITEM=129=skin
@ITEM=130=your
@ITEM=131=eyes
@ITEM=132=for
@ITEM=133=sharp
@ITEM=134=water
@ITEM=135=but
@ITEM=136=bubble
@ITEM=137=sing
@ITEM=138=out
129 -1 130 -1 131 -1 132 -1 81 -1 36 -1 15 -1 133 -1 132 -1 13 -1 134 -1 57 -1 2 -1 17 -1 135 -1 12 -1 136 -1 137 -1 138 -1 -2
@ITEM=139=tashtego
@ITEM=140=daggoo
@ITEM=141=queequeg
@ITEM=142=had
@ITEM=143=looked
@ITEM=144=on
@ITEM=145=even
@ITEM=146=more
@ITEM=147=intense
@ITEM=148=interest
@ITEM=149=surprise
@ITEM=150=than
@ITEM=151=rest
@ITEM=152=at
@ITEM=153=mention
@ITEM=154=started
@ITEM=155=each
@ITEM=156=separately
@ITEM=157=touched
@ITEM=158=by
@ITEM=159=some
@ITEM=160=specific
@ITEM=161=recollection
1 -1 18 -1 42 -1 139 -1 140 -1 61 -1 141 -1 142 -1 143 -1 144 -1 88 -1 145 -1 146 -1 147 -1 148 -1 61 -1 149 -1 150 -1 29 -1 151 -1 61 -1 152 -1 29 -1 153 -1 21 -1 29 -1 100 -1 101 -1 61 -1 102 -1 103 -1 118 -1 142 -1 154 -1 56 -1 57 -1 155 -1 44 -1 156 -1 157 -1 158 -1 159 -1 160 -1 161 -1 -2
@ITEM=162=captain
@ITEM=163=said
@ITEM=164=must
@ITEM=165=be
@ITEM=166=call
@ITEM=167=moby
@ITEM=168=dick
162 -1 47 -1 163 -1 139 -1 75 -1 13 -1 14 -1 164 -1 165 -1 29 -1 109 -1 75 -1 159 -1 166 -1 167 -1 168 -1 -2
167 -1 168 -1 -2
@ITEM=169=shouted
169 -1 47 -1 -2
@ITEM=170=do
@ITEM=171=know
@ITEM=172=then
@ITEM=173=tash
170 -1 2 -1 171 -1 29 -1 13 -1 14 -1 172 -1 173 -1 -2
@ITEM=174=does
@ITEM=175=fantail
@ITEM=176=little
@ITEM=177=curious
@ITEM=178=sir
@ITEM=179=goes
174 -1 84 -1 175 -1 12 -1 176 -1 177 -1 178 -1 5 -1 84 -1 179 -1 128 -1 -2
@ITEM=180=gayheader
@ITEM=181=deliberately
163 -1 29 -1 180 -1 181 -1 -2
@ITEM=182=has
@ITEM=183=spout
@ITEM=184=too
@ITEM=185=very
@ITEM=186=bushy
@ITEM=187=parmacetty
@ITEM=188=mighty
@ITEM=189=quick
61 -1 182 -1 84 -1 12 -1 177 -1 183 -1 184 -1 163 -1 140 -1 185 -1 186 -1 145 -1 132 -1 12 -1 187 -1 61 -1 188 -1 189 -1 162 -1 47 -1 -2
@ITEM=190=two
@ITEM=191=oh
61 -1 84 -1 4 -1 90 -1 190 -1 104 -1 191 -1 -2
@ITEM=192=good
@ITEM=193=many
@ITEM=194=iron
@ITEM=195=hide
@ITEM=196=disjointedly
@ITEM=197=twisketee
@ITEM=198=betwisk
@ITEM=199=like
@ITEM=200=faltering
@ITEM=201=hard
@ITEM=202=word
@ITEM=203=screwing
@ITEM=204=round
@ITEM=205=though
@ITEM=206=uncorking
@ITEM=207=bottle
@ITEM=208=corkscrew
192 -1 193 -1 194 -1 80 -1 81 -1 195 -1 184 -1 162 -1 114 -1 141 -1 196 -1 1 -1 197 -1 198 -1 199 -1 81 -1 81 -1 200 -1 201 -1 132 -1 12 -1 202 -1 61 -1 203 -1 54
-1 39 -1 204 -1 61 -1 204 -1 56 -1 205 -1 206 -1 12 -1 207 -1 199 -1 81 -1 81 -1 208 -1 -2
@ITEM=209=aye
@ITEM=210=harpoons
@ITEM=211=lie
@ITEM=212=twisted
@ITEM=213=wrenched
@ITEM=214=big
@ITEM=215=whole
@ITEM=216=shock
@ITEM=217=wheat
@ITEM=218=pile
@ITEM=219=our
@ITEM=220=nantucket
@ITEM=221=wool
@ITEM=222=after
@ITEM=223=great
@ITEM=224=annual
@ITEM=225=sheepshearing
@ITEM=226=fantails
@ITEM=227=split
@ITEM=228=jib
@ITEM=229=squall114 -1 47 -1 209 -1 141 -1 29 -1 210 -1 211 -1 1 -1 212 -1 61 -1 213 -1 80 -1 81 -1 209 -1 140 -1 54 -1 183 -1 32 -1 12 -1 214 -1 90 -1 199 -1
12 -1 215 -1 216 -1 21 -1 217 -1 61 -1 13 -1 56 -1 12 -1 218 -1 21 -1 219 -1 220 -1 221 -1 222 -1 29 -1 223 -1 224 -1 225 -1 209 -1 139 -1 61 -1 84 -1 226 -1 199
-1 12 -1 227 -1 228 -1 80 -1 12 -1 229 -1 -2
@ITEM=230=death
@ITEM=231=devils
230 -1 61 -1 231 -1 -2
@ITEM=232=seen
36 -1 31 -1 32 -1 167 -1 168 -1 2 -1 4 -1 232 -1 167 -1 168 -1 167 -1 168 -1 -2
@ITEM=233=who
@ITEM=234=stubb
@ITEM=235=flask
@ITEM=236=thus
@ITEM=237=far
@ITEM=238=been
@ITEM=239=eyeing
@ITEM=240=superior
@ITEM=241=increasing
@ITEM=242=last
@ITEM=243=struck
@ITEM=244=thought
@ITEM=245=which
@ITEM=246=somewhat
@ITEM=247=explained
@ITEM=248=wonder
162 -1 47 -1 163 -1 38 -1 233 -1 88 -1 234 -1 61 -1 235 -1 142 -1 236 -1 237 -1 238 -1 239 -1 54 -1 240 -1 88 -1 241 -1 149 -1 135 -1 152 -1 242 -1 76 -1 243 -1 88 -1 12 -1 244 -1 245 -1 246 -1 247 -1 1 -1 29 -1 248 -1 -2
@ITEM=249=not
@ITEM=250=took
@ITEM=251=off
@ITEM=252=thy
@ITEM=253=leg
162 -1 47 -1 124 -1 4 -1 7 -1 21 -1 167 -1 168 -1 135 -1 31 -1 44 -1 249 -1 167 -1 168 -1 75 -1 250 -1 251 -1 252 -1 253 -1 -2

The file begins with the tag @CONVERTED_FROM_TEXT, indicating that this sequence database was generated by converting a text file.

Each unique word encountered in the text is then assigned a unique integer identifier, declared using @ITEM tags (e.g., @ITEM=1=all, @ITEM=2=ye, @ITEM=3=mastheaders, etc.). Words are converted to lowercase and stripped of punctuation before being assigned an identifier. These @ITEM declarations appear incrementally throughout the file, as new words are encountered for the first time while reading the text from beginning to end.

Each sentence found in the input text is converted into a sequence, where each word of the sentence becomes an item, and each item is followed by -1, which is the itemset separator. The special symbol -2 indicates the end of a sequence. For example, the first sequence:
1 -1 2 -1 3 -1 4 -1 5 -1 6 -1 7 -1 8 -1 9 -1 10 -1 11 -1 12 -1 13 -1 14 -1 -2

represents the sentence "All ye mast-headers have before now heard me give orders about a white whale.", where each number refers to the corresponding word through the @ITEM mapping shown above the sequence.

Note that new @ITEM declarations are written just before the sequence in which they first appear, which is why they appear interspersed throughout the file rather than all being grouped at the top. Also note that some sentences contain very few words (e.g., "Huzza!" is converted to the single-item sequence 113 -1 -2), since exclamations and short dialogue are treated as their own sequences, in the same way as longer sentences.

Importantly, the conversion reuses item identifiers across sequences when the same word appears in multiple sentences (e.g., the word "ye", assigned item id 2, reappears in many sequences throughout the file). This allows sequential pattern mining and sequential rule mining algorithms to discover patterns and associations between words that are reused across different sentences of the text.

The converted file can then be taken as input by sequential pattern mining and sequential rule mining algorithms such as PrefixSpan, CM-SPADE and RuleGrowth. You may see the other examples of the documentation to learn how to run such algorithms.