Of particular interest arer the labels with intermediate frequency (between 1 and 99 occurrences). I separated those into a file .labels-rare.dic. 8cc8o 8ccoe 8o8orm 8oHccoe 8oHoeo 8oHor 8oeoro 8oero 8orcco8o Hoccorom Hoecc8 Horomo Poro cPccor ccPoeo cccHoe cccocHco cccor8o cccoro8 cccoroe ccoHoro ccoccro ccoer ccoerom o8r oHcc8occcHoe oHcc8oe oHcccc8o oHcccco oHccccor oHccco8o oHccco8or oHcccor oHcco8oer oHcco8or oHccocc8o oHccocco oHccooro oHccoror oHccr oHco8occcHco oHco8oer oHcoeo oHcoeor oHcoeroe oHcooe8o oHcooeo oHcoor oHcoroe oHcororor oHo8oe oHo8or oHo8oro oHoHo oHoHoe oHocHco oHocco oHoccor oHoe8oe oHoe8or oHoecc8 oHoecc8o oHoeccoe8o oHoeo8 oHoeo8o oHoeoe oHoeoeo oHoeok oHoeom oHoeor oHoeoroe oHoeoror oHoer oHoero oHom8om oHomoHom oHorcccHo oHorcco8 oHorcco8o oHorccok oHorccor oHorco oHoroe8o oHorok oHorom oHoror oHororo oHroe oPccco8o oPcccor oPcco8oo oPccor oPccorom oPoeror oPorom occc8oe8om occo8o8o occoro oecPco oecccccco oeorom oeoror ooPcco oorcccor oorom oroe8 ororcco8om ororoeo qHoe qoHcro rccoor ro8or ro8ororo roPoe roe8ok roeP roeccror roeoe roeoer roer roero rorccor roroeo roromr rororo rr I thought of removing those labels that were superstrings of others in the same set, e.g. removing "oHoecc8o" since there is already "oHoecc8". Here is a (rather convoluted) recipe to do that: cp -p .labels-rare.dic .labs @ i = 0 while ( $i < 4 ) @ i = $i + 1 cat .labs \ | enum-proper-substrings \ | sort | uniq \ > .subs.$i bool 1.2 .labs .subs.$i \ > .subs-oc.$i if ( -z .subs-oc.$i ) break cat .labs \ | fgrep -v -f .subs-oc.$i \ > .prop.$i cat .subs-oc.$i >> .prop.$i cat .prop.$i \ | sort | uniq \ > .labs end Here is what it would remove: bool 1-2 .labels-rare.dic .labs Hoccorom ccoerom oHcc8occcHoe oHccccor oHccco8or oHcoeor oHo8oro oHoHoe oHoccor oHoecc8 oHoecc8o oHoeo8o oHoeoeo oHoeoroe oHoeoror oHoero oHorcco8o oHoroe8o oHororo oPccorom oPorom ororoeo ro8ororo roeoer roero But it seems that those labels are intersting on their own, and in some cases much larger than the relevant substrings. So I decided to leave them in. We should also look at the "rarest" labels (with less than 25 occurrences): 8o8orm 8oHccoe 8oHoeo 8oero 8orcco8o Hoccorom cPccor ccPoeo cccor8o cccoro8 ccoccro ccoerom o8r oHcc8occcHoe oHcccc8o oHccccor oHccco8or oHcco8oer oHccocc8o oHccooro oHccoror oHco8occcHco oHco8oer oHcoeor oHcoeroe oHcooe8o oHcooeo oHcoor oHcoroe oHcororor oHo8oe oHo8or oHo8oro oHoHoe oHocHco oHoccor oHoeccoe8o oHoeo8 oHoeo8o oHoeoeo oHoeok oHoeom oHoeoroe oHoeoror oHoero oHom8om oHorcccHo oHorcco8 oHorcco8o oHorccok oHorccor oHorco oHoroe8o oHorok oHroe oPccco8o oPcccor oPcco8oo oPccorom oPoeror oPorom occc8oe8om occo8o8o oecPco oecccccco oorcccor ororcco8om qHoe qoHcro rccoor ro8or ro8ororo roPoe roe8ok roeP roeccror roeoer rorccor roromr Now let's find all occurrences of these rare labels in the parags text: cat .parags-j-ecc.evt \ | enum-word-locations .labels-rare.dic \ | sort -b +2 -3n \ > .label-rare-occurrences.idx cat .parags-j-ecc.evt \ | enum-word-locations .labels-rarest.dic \ | sort -b +2 -3n \ > .label-rarest-occurrences.idx Let's tabulate their frequencies per page: foreach f ( '' -rare -rarest ) cat .label${f}-occurrences.idx \ | gawk '/./ { print $1 }' \ | sed -e 's/\..*>/>/g' \ | sort | uniq -c | expand \ | compute-freqs \ | sort +0 -1nr \ > .label${f}-refs-by-panel.frq end --- .label-rare-refs-by-panel.frq ------------------------ 93 0.030 78 0.025 63 0.020 60 0.019 60 0.019 57 0.018 55 0.017 53 0.017 51 0.016 50 0.016 49 0.016 49 0.016 48 0.015 44 0.014 42 0.013 41 0.013 40 0.013 39 0.012 39 0.012 37 0.012 36 0.011 36 0.011 35 0.011 35 0.011 35 0.011 34 0.011 34 0.011 34 0.011 33 0.010 30 0.010 28 0.009 27 0.009 26 0.008 25 0.008 25 0.008 24 0.008 24 0.008 24 0.008 24 0.008 23 0.007 23 0.007 22 0.007 22 0.007 21 0.007 21 0.007 20 0.006 20 0.006 20 0.006 19 0.006 18 0.006 18 0.006 17 0.005 17 0.005 17 0.005 16 0.005 16 0.005 15 0.005 15 0.005 15 0.005 15 0.005 15 0.005 15 0.005 15 0.005 14 0.004 14 0.004 14 0.004 14 0.004 14 0.004 14 0.004 14 0.004 14 0.004 13 0.004 13 0.004 13 0.004 13 0.004 13 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 11 0.003 11 0.003 11 0.003 11 0.003 11 0.003 11 0.003 11 0.003 11 0.003 10 0.003 10 0.003 10 0.003 10 0.003 10 0.003 10 0.003 10 0.003 10 0.003 9 0.003 9 0.003 9 0.003 9 0.003 9 0.003 9 0.003 9 0.003 9 0.003 9 0.003 9 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 7 0.002 7 0.002 7 0.002 7 0.002 7 0.002 7 0.002 7 0.002 7 0.002 7 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 3 0.001 3 0.001 3 0.001 2 0.001 1 0.000 1 0.000 1 0.000 1 0.000 1 0.000 ---------------------------------------------------------- --- .label-rarest-refs-by-panel.frq ------------------------ 28 0.041 18 0.026 15 0.022 13 0.019 13 0.019 12 0.017 12 0.017 11 0.016 10 0.014 10 0.014 10 0.014 9 0.013 9 0.013 9 0.013 8 0.012 8 0.012 8 0.012 8 0.012 8 0.012 8 0.012 8 0.012 7 0.010 7 0.010 7 0.010 7 0.010 7 0.010 6 0.009 6 0.009 6 0.009 6 0.009 6 0.009 6 0.009 6 0.009 6 0.009 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 ------------------------------------------------------------ Let's compute the density of label references per page (number of references divided by number of text characters in page): foreach f ( '' -rare -rarest ) cat .label${f}-refs-by-panel.frq \ | sort +2 -3 \ > .foo join \ -a 1 -e 00 \ -j1 3 -j2 1 \ -o0,1.1,1.2,2.2 \ .foo .panels.nchars \ > .bar${f} cat .bar${f} \ | gawk '/./ {printf "%-8s %5d %5.3f %5d\n", $1, $2, $3, int(1000*$2/$4 + 0.5)}' \ | sort -b +3 -4nr \ > .label${f}-refs-by-panel.rfrq end --- .label-refs-by-panel.rfrq ------------------------ 146 0.005 348 546 0.020 294 178 0.006 293 69 0.003 273 636 0.023 269 99 0.004 268 436 0.016 267 136 0.005 261 97 0.004 261 152 0.006 256 563 0.020 252 106 0.004 249 95 0.003 242 106 0.004 240 131 0.005 234 76 0.003 229 100 0.004 224 37 0.001 224 105 0.004 224 491 0.018 223 45 0.002 223 511 0.019 219