Hacking at the Voynich manuscript Notebook - volume 11 Warning: these notebooks aren't strictly chronological logs. Sometimes I go back and redo things, clarify comments, delete garbage, etc. 97-10-09 stolfi =============== I have split Landini's file into one chunk per page csplit \ --prefix 'chunk-' \ --suffix '%03d.evt' \ - '^# *$' '{*}' and then futher edited it manually, splitting each page into homogeneous "textual units" (all normal text, all labels, etc.) The files are L16/fNNN and L16/fNNN.L, where fNNN is the panel number (as in f85r1) and L is the location code within that panel. Files without location code contain general comments about the panel. See L16/README for a detailed description of the files and my editings. 97-10-12 stolfi =============== Did a preliminarey version of the label location map, later redone. 97-10-14 stolfi =============== 97-10-14 stolfi =============== Ditto. An intermezzo: prompted by John Grove, let's compute the frquency of single-letter words in the manuscript cat .voyn.fsg \ | /n/gnu/bin/tr ' =./' '\012\012\012\012' \ | egrep '^.$' \ | sort | uniq -c | expand \ | compute-freqs \ | sort +0 -1nr \ > .single.frq 13 0.245 R 9 0.170 2 7 0.132 E 5 0.094 G 5 0.094 O 4 0.075 4 3 0.057 8 2 0.038 * 2 0.038 D 2 0.038 S 1 0.019 6 Now for something else again. 97-10-19 stolfi =============== I translated the unit files to the ECC encoding: mkdir L16-ecc foreach f ( L16/f[0-9]* ) echo "$f -> L16-ecc/${f:t}" cat ${f} \ | fsg2ecc \ > L16-ecc/${f:t} end Let's collect the textual units that comtain text or labels of each hand: cat L16/page-table.dir \ | egrep '^[^:]*:[^:]*:[^:]*:[^:]*:[^:]*(labels|words):' \ > .units-labels.dir cat L16/page-table.dir \ | egrep '^[^:]*:[^:]*:[^:]*:[^:]*:[^:]*parags:' \ > .units-parags.dir cat L16/page-table.dir \ | egrep '^[^:]*:[^:]*:[^:]*:[^:]*:[^:]*(lines|titles):' \ > .units-lines.dir foreach let ( A B X ) set pat = "${let}" if ( "${let}" == "X" ) set pat = '[^:]*\?' cat L16/page-table.dir \ | egrep ':'"${pat}"':[^:]*:[^:]*(labels|words):' \ > .units-labels-${let}.dir cat L16/page-table.dir \ | egrep ':'"${pat}"':[^:]*:[^:]*parags:' \ > .units-parags-${let}.dir cat L16/page-table.dir \ | egrep ':'"${pat}"':[^:]*:[^:]*(lines|titles):' \ > .units-lines-${let}.dir end Let's gather all panel numbers that occur in the text. cat L16/page-table.dir \ | sed -e 's/:.*//g' -e 's/\..*$//g' \ | uniq \ > .panels.dir cat L16/page-table.dir \ | egrep '^[^:]*:[^:]*:[^:]*:[^:]*:[^:]*(labels|words):' \ | sed -e 's/:.*//g' -e 's/\..*$//g' \ > .panels-labels.dir cat L16/page-table.dir \ | egrep '^[^:]*:[^:]*:[^:]*:[^:]*:[^:]*parags:' \ | sed -e 's/:.*//g' -e 's/\..*$//g' \ > .panels-parags.dir cat L16/page-table.dir \ | egrep '^[^:]*:[^:]*:[^:]*:[^:]*:[^:]*(lines|titles):' \ | sed -e 's/:.*//g' -e 's/\..*$//g' \ > .panels-lines.dir From them we create a script to convert panel numbers to sequential page numbers (000 to 263): foreach f ( '' '-parags' '-labels' '-lines' ) echo '#\! /n/gnu/bin/sed -f' \ > panel${f}-to-page cat .panels${f}.dir \ | gawk 'BEGIN {pg=0} /./ {printf"s/<%s>/<%03d>/g\n", $1, pg; pg++}' \ >> panel${f}-to-page chmod a+x panel${f}-to-page end --- panel-to-page ------------------------ #! /n/gnu/bin/sed -f s//<000>/g s//<001>/g s//<002>/g s//<003>/g s//<004>/g s//<005>/g s//<006>/g s//<007>/g ... s//<260>/g s//<261>/g s//<262>/g s//<263>/g s//<264>/g s//<265>/g s//<266>/g ------------------------------------------ 97-10-20 stolfi =============== Let's now create a consensus version for each unit. It turns out that the transcription code ";J>" is already used and means Jim Reeds. I will use ";S" for my consensus. mkdir L16-ecc-x foreach f ( L16-ecc/f[0-9]* ) set g = "L16-ecc-x/${f:t}" echo "$f -> $g" cat ${f} \ | make-consensus-interlin \ > ${g}~~ cat ${g}~~ \ | egrep ';S>|^#' \ > ${g} end Let's concatenate all the paragraph locations into a single file: cat .units-parags.dir \ | sed \ -e 's/:.*$//g' \ -e 's:^:L16-ecc-x/:g' \ > .tmp cat `cat .tmp` \ > .parags-j-ecc.evt cat .parags-j-ecc.evt \ | egrep '^<' \ | sed \ -e 's/^<.*> *//g' \ -e 's/ *//g' \ | dicio-wc lines words bytes ------ ------- --------- 3918 3918 168489 Note that the count above includes newlines, so we actually have 164571 Voynich characters in the parags file. Now let's extract the good label text: cat .units-labels.dir \ | sed \ -e 's/:.*$//g' \ -e 's:^:L16-ecc-x/:g' \ > .tmp cat `cat .tmp` \ > .labels-j-ecc.evt extract-words-from-interlin \ -chars "8coqHPemrwkij" \ .labels-j-ecc.evt \ .labels-j-ecc lines words bytes file ------ ------- --------- ------------ 986 986 4243 .labels-j-ecc.wds 277 277 2234 .labels-j-ecc.dic 282 282 2201 .labels-j-ecc-gut.wds 225 225 1831 .labels-j-ecc-gut.dic 652 652 1630 .labels-j-ecc-fun.wds 2 2 5 .labels-j-ecc-fun.dic 52 52 412 .labels-j-ecc-bad.wds 50 50 398 .labels-j-ecc-bad.dic 2526 2526 12954 total Digraph counts: TT 8 c o q H P e m r k i j ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- 282 . 21 26 200 4 3 1 1 . 26 . . . 8 108 11 . 6 88 . . . . . 2 1 . . c 342 1 16 165 120 . 15 8 . . 8 . . 9 o 787 119 48 24 24 . 170 20 156 36 171 19 . . q 4 . . . 3 . 1 . . . . . . . H 194 . . 60 131 . . . 2 . 1 . . . P 31 2 . 14 15 . . . . . . . . . e 159 37 18 21 63 . 4 1 . 1 13 . 1 . m 38 23 3 1 10 . . . . . 1 . . . r 226 71 2 22 125 . 1 . . 1 4 . . . k 20 16 . 3 1 . . . . . . . . . i 1 . . . . . . 1 . . . . . . j 9 2 . . 7 . . . . . . . . . ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- TOT 2201 282 108 342 787 4 194 31 159 38 226 20 1 9 Next-symbol probability (× 99): TT 8 c o q H P e m r k i j -- -- -- -- -- -- -- -- -- -- -- -- -- -- 99 . 7 9 70 1 1 . . . 9 . . . 8 99 10 . 6 81 . . . . . 2 1 . . c 99 . 5 48 35 . 4 2 . . 2 . . 3 o 99 15 6 3 3 . 21 3 20 5 22 2 . . q 99 . . . 74 . 25 . . . . . . . H 99 . . 31 67 . . . 1 . 1 . . . P 99 6 . 45 48 . . . . . . . . . e 99 23 11 13 39 . 2 1 . 1 8 . 1 . m 99 60 8 3 26 . . . . . 3 . . . r 99 31 1 10 55 . . . . . 2 . . . k 99 79 . 15 5 . . . . . . . . . i 99 . . . . . . 99 . . . . . . j 99 22 . . 77 . . . . . . . . . -- -- -- -- -- -- -- -- -- -- -- -- -- -- TOT 99 13 5 15 35 0 9 1 7 2 10 1 0 0 Previous-symbol probability (× 99): TT 8 c o q H P e m r k i j -- -- -- -- -- -- -- -- -- -- -- -- -- -- 13 . 19 8 25 99 2 3 1 . 11 . . . 8 5 4 . 2 11 . . . . . 1 5 . . c 15 . 15 48 15 . 8 26 . . 4 . . 99 o 35 42 44 7 3 . 87 64 97 94 75 94 . . q 0 . . . . . 1 . . . . . . . H 9 . . 17 16 . . . 1 . . . . . P 1 1 . 4 2 . . . . . . . . . e 7 13 17 6 8 . 2 3 . 3 6 . 99 . m 2 8 3 . 1 . . . . . . . . . r 10 25 2 6 16 . 1 . . 3 2 . . . k 1 6 . 1 . . . . . . . . . . i 0 . . . . . . 3 . . . . . . j 0 1 . . 1 . . . . . . . . . -- -- -- -- -- -- -- -- -- -- -- -- -- -- TOT 99 99 99 99 99 99 99 99 99 99 99 99 99 99 Symbol entropy: 2.764 Next-symbol entropy: 2.020 Now, let's make a list of all labels therein. Multiword labels (where words are separated by "-") will be entered as a single word, as well as separate words. /bin/rm -f .labels.def First, the labels without word breaks: cat .labels-j-ecc.evt \ | remove-comments-from-evt \ | sed \ -e 's/ *//g' \ -e 's/;[A-Z]>/>/g' \ -e 's/[-=]//g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>/> /g' \ > .labels1.def Second, the labels split at word boundaries: cat .labels-j-ecc.evt \ | remove-comments-from-evt \ | /n/gnu/bin/sed \ -e 's/ *//g' \ -e 's/;[A-Z]>/>/g' \ -e 's/[=-]$//g' \ -e 's/^/@/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/@\(<[^>]*>\)\([^ @-][^ @-]*\)[ -][ -]*/@\1\2@\1/g' \ -e 's/@\(<[^>]*>\)\([^ @-][^ @-]*\)[ -][ -]*/@\1\2@\1/g' \ -e 's/@\(<[^>]*>\)\([^ @-][^ @-]*\)[ -][ -]*/@\1\2@\1/g' \ -e 's/@\(<[^>]*>\)\([^ @-][^ @-]*\)[ -][ -]*/@\1\2@\1/g' \ -e 's/@\(<[^>]*>\)\([^ @-][^ @-]*\)[ -][ -]*/@\1\2@\1/g' \ -e 's/>/> /g' \ | tr '@' '\012' \ | egrep '.' \ > .labels2.def Now merge the two files, and insert sequential page numbers: cat .labels1.def .labels2.def \ | sort | uniq \ | sed \ -e 's/<\(.*\)> \(.*\)/<\1> \2 {\1}/g' \ -e 's/\.[^>]*> */> /g' \ | panel-to-page \ | tr '{}' '<>' \ > .labels.def Keep only the first definition of each label as its "official position": cat .labels.def \ | sort +1 -2 +0 -1n \ | gawk 'BEGIN{b=""} /./ {if(b!=$2) {print; b=$2; next}}' \ | sort \ > .labels-first.def Collect the labels proper: cat .labels.def \ | gawk '/./ {print $2}' \ | sort | uniq \ | egrep -v '\?' \ > .labels.dic dicio-wc .labels.dic lines words bytes file ------ ------- --------- ------------ 231 231 1906 .labels.dic cat .labels.dic\ | gawk 'BEGIN{m=0} /./{s=length($0);m=(s>m?s:m);next} END{print ("max len " m)}' max len 18 Now let's find all occurrences of the labels in the parags text: cat .parags-j-ecc.evt \ | enum-word-locations .labels.dic \ | sort -b +2 -3n \ > .label-occurrences.idx Let's tabulate the reference frequencies per label: cat .label-occurrences.idx \ | gawk '/./ { print $4 }' \ | sort | uniq -c | expand \ | compute-freqs \ | sort +0 -1nr \ > .label-refs-by-label.frq --- .label-refs-by-label.frq ------------------------ 4774 0.173 oHo 2151 0.078 ccoe 1540 0.056 oHom 1162 0.042 oHoe 976 0.035 oHor 944 0.034 rom 896 0.033 oHcc8o 748 0.027 oroe 737 0.027 cccco 716 0.026 cccoe 646 0.023 oror 612 0.022 8oro ... ..... ........ 4 0.000 oHoeoror 4 0.000 oHoroe8o 3 0.000 ccPoeo 3 0.000 ccoccro 3 0.000 oPcco8oo 3 0.000 oPccorom 3 0.000 roPoe 3 0.000 roeoer 2 0.000 oHcc8occcHoe 2 0.000 oHco8oer 2 0.000 oHcooe8o 2 0.000 occc8oe8om 2 0.000 qHoe 2 0.000 qoHcro 2 0.000 roromr 1 0.000 8o8orm 1 0.000 Hoccorom 1 0.000 oHcco8oer 1 0.000 oHco8occcHco 1 0.000 oHcoeroe 1 0.000 oHcooeo 1 0.000 oHcororor 1 0.000 oHorccok 1 0.000 oHorco 1 0.000 oHroe 1 0.000 oPoeror 1 0.000 oecccccco 1 0.000 ororcco8om 1 0.000 ro8ororo 1 0.000 roeccror ----------------------------------------------------- Obviously "oHo", "ccoe", "oHom", "oHoe", etc. are not really labels; they are either common words (function words? "Star"? "Plant"? "Day"?), or common letter groups that got split off by accident. Let's list the unreferenced labels: cat .label-occurrences.idx \ | gawk '/./ { print $4 }' \ | sort | uniq \ | bool 2-1 - .labels.dic \ > .labels-unref.dic 8ccccorocccPoeom 8cccoe8o 8oHocHc 8oHoecj 8orokcjoe cc8or8omo ccoHcco8orr ccorcHcoroe ccoroeiP eHcccPoe eHcccPoeooPcco o8orcc8oom oHcc8ccor oHccoroeokcco oHcoHcororo oHcooeoeroroeo oHcoorororoeo oHcororoo8o oHe8ok oHeorcjo oHo8rco8occPooeo oHoHokom oHoecHo oHoecPoromok oHoeccHco oHoeccoHorokcjo oHoecjor oHoeoPccoroe oHoeoorok oHooccooHoeoHcoeor oHorccorrr oHorcjo oHorcororo oHorcr oHoroecHoeHoo oHoroecjo oPcc8ocjo oPcccoroe oPccoe8k oPccorok oPocPcor oPoeo8om oPoeo8omrr oPoeoro oPoeoror oPorHo8oe oPoroeor occcHoroHoem8 occcrororcco occoeccorok ocoeoecroror omoHoeoccor oo8cco ooeccccj oomororo qoHoomocHcco roeomcccoe roeooHoro8 room8oeo8 rrcHcrcccHo Note that these "labels" are distinctly longer than those that do occur, and are almost certainly multiword phrases. Now let's prepare a map showing for each label its occurrences in the running text. First, a block-based map: setenv BLOCKSZ 1646 cat .label-occurrences.idx \ | sort -b +3 -4 \ | gawk '/./ { print int($3 / '"$BLOCKSZ"') + 1, $4 }' \ | make-word-location-map \ -v MAXLEN=18 \ -v CTWD=1 \ -v NBLOCKS=100 \ > .label-by-block.map Add a column with the panel where each label was first defined: cat .labels-first.def \ | sed -e 's/\.[^ >]*>/>/g' \ | tr -d '<>' \ | sort +1 -2 \ > .foo join \ -a 1 -e '000' \ -j1 4 -j2 2 \ -o0,2.3,2.1,1.1,1.2,1.3,1.5 \ .label-by-block.map .foo \ | gawk \ '/./ {printf "%-18s %-6s %-3s %5d %5.1f %5.1f %s\n", $1,$2,$3,$4,$5,$6,$7}' \ > .label-by-block-def.map Some comments: * The occurrences of a rarer label in the text usually form one or more tight clusters. * The labels occuring in some pages are relatively common in the text, those in other pages do not occur at all in the text. I computed the number of text characters in each panel: cat .units-parags.dir \ | sed -e 's/:.*$//g' \ > .ups /bin/rm -f .tmp foreach f ( `cat .panels.dir` ) set pp = ( `egrep $f'[.]' .ups` ) echo "$f $pp" > /dev/stderr echo "<$f> `( cd L16-ecc-x && cat $f $pp ) | count-text-chars`" >> .tmp end cat .tmp \ | sort \ > .panels.nchars Let's make a table that gives the range of byte offsets for each panel. Each line has the sequential panel number, the physical panel number, the first offset, and the last offset plus one. cat .panels.nchars \ | sed -e 's/<\(.*\)>/<\1> {\1}/g' \ | panel-to-page \ | tr '{}' '<>' \ | sort +0 -1n \ | gawk 'BEGIN {a=0} /./ {b = a+$3; print $1, $2, a, b; a=b}' \ > .panels.chrange Let's make tables that map block index to panel number and vice-versa: echo 'block size = '$BLOCKSZ cat .panels.chrange \ | tr -d '<>' \ | gawk '/./ {printf "s/<%s>/<%03d>/g\n", $2, 1+int($3/'"$BLOCKSZ"')}' \ > panel-to-block chmod a+x panel-to-block echo 'block size = '$BLOCKSZ cat .panels.chrange \ | grep -v '' \ | tr -d '<>' \ | gawk '/./ {printf "%03d %s\n", 1+int($3/'"$BLOCKSZ"'), $2}' \ | gawk 'BEGIN {n=0} /./ {while($1>n){n++;printf "s/<%03d>/<%s>/g\n", n,$2}}' \ > block-to-panel chmod a+x block-to-panel Formatting block-to-panel as a header: cat block-to-panel \ | tr '<>/' ' ' \ | gawk '/./ {print $2, $3}' \ | sed -e 's/ f\([0-9][0-9]*\)/ f\1 /g' \ | format-block-map-header \ > .block-map-header Computing frequencies of references per panel: cat .label-occurrences.idx \ | gawk '/./ { print $1 }' \ | sed -e 's/\..*>/>/g' \ | sort | uniq -c | expand \ | compute-freqs \ | sort +0 -1nr \ > .label-refs-by-panel.frq --- .label-refs-by-panel.frq ------------------------ 636 0.023 563 0.020 546 0.020 511 0.019 498 0.018 494 0.018 492 0.018 491 0.018 482 0.017 475 0.017 465 0.017 ... ..... ........ 30 0.001 30 0.001 29 0.001 28 0.001 28 0.001 27 0.001 27 0.001 22 0.001 20 0.001 19 0.001 19 0.001 15 0.001 12 0.000 11 0.000 ----------------------------------------------------- There are a few pages that are particularly rich in label references. The panels from f103 on are "starred paragraphs", and so is f58r. Panel f86v6 is on the back of the big fold-out. 97-10-21 stolfi =============== Of particular interest arer the labels with intermediate frequency (between 1 and 99 occurrences). I separated those into a file .labels-rare.dic. 8cc8o 8ccoe 8o8orm 8oHccoe 8oHoeo 8oHor 8oeoro 8oero 8orcco8o Hoccorom Hoecc8 Horomo Poro cPccor ccPoeo cccHoe cccocHco cccor8o cccoro8 cccoroe ccoHoro ccoccro ccoer ccoerom o8r oHcc8occcHoe oHcc8oe oHcccc8o oHcccco oHccccor oHccco8o oHccco8or oHcccor oHcco8oer oHcco8or oHccocc8o oHccocco oHccooro oHccoror oHccr oHco8occcHco oHco8oer oHcoeo oHcoeor oHcoeroe oHcooe8o oHcooeo oHcoor oHcoroe oHcororor oHo8oe oHo8or oHo8oro oHoHo oHoHoe oHocHco oHocco oHoccor oHoe8oe oHoe8or oHoecc8 oHoecc8o oHoeccoe8o oHoeo8 oHoeo8o oHoeoe oHoeoeo oHoeok oHoeom oHoeor oHoeoroe oHoeoror oHoer oHoero oHom8om oHomoHom oHorcccHo oHorcco8 oHorcco8o oHorccok oHorccor oHorco oHoroe8o oHorok oHorom oHoror oHororo oHroe oPccco8o oPcccor oPcco8oo oPccor oPccorom oPoeror oPorom occc8oe8om occo8o8o occoro oecPco oecccccco oeorom oeoror ooPcco oorcccor oorom oroe8 ororcco8om ororoeo qHoe qoHcro rccoor ro8or ro8ororo roPoe roe8ok roeP roeccror roeoe roeoer roer roero rorccor roroeo roromr rororo rr I thought of removing those labels that were superstrings of others in the same set, e.g. removing "oHoecc8o" since there is already "oHoecc8". Here is a (rather convoluted) recipe to do that: cp -p .labels-rare.dic .labs @ i = 0 while ( $i < 4 ) @ i = $i + 1 cat .labs \ | enum-proper-substrings \ | sort | uniq \ > .subs.$i bool 1.2 .labs .subs.$i \ > .subs-oc.$i if ( -z .subs-oc.$i ) break cat .labs \ | fgrep -v -f .subs-oc.$i \ > .prop.$i cat .subs-oc.$i >> .prop.$i cat .prop.$i \ | sort | uniq \ > .labs end Here is what it would remove: bool 1-2 .labels-rare.dic .labs Hoccorom ccoerom oHcc8occcHoe oHccccor oHccco8or oHcoeor oHo8oro oHoHoe oHoccor oHoecc8 oHoecc8o oHoeo8o oHoeoeo oHoeoroe oHoeoror oHoero oHorcco8o oHoroe8o oHororo oPccorom oPorom ororoeo ro8ororo roeoer roero But it seems that those labels are intersting on their own, and in some cases much larger than the relevant substrings. So I decided to leave them in. We should also look at the "rarest" labels (with less than 25 occurrences): 8o8orm 8oHccoe 8oHoeo 8oero 8orcco8o Hoccorom cPccor ccPoeo cccor8o cccoro8 ccoccro ccoerom o8r oHcc8occcHoe oHcccc8o oHccccor oHccco8or oHcco8oer oHccocc8o oHccooro oHccoror oHco8occcHco oHco8oer oHcoeor oHcoeroe oHcooe8o oHcooeo oHcoor oHcoroe oHcororor oHo8oe oHo8or oHo8oro oHoHoe oHocHco oHoccor oHoeccoe8o oHoeo8 oHoeo8o oHoeoeo oHoeok oHoeom oHoeoroe oHoeoror oHoero oHom8om oHorcccHo oHorcco8 oHorcco8o oHorccok oHorccor oHorco oHoroe8o oHorok oHroe oPccco8o oPcccor oPcco8oo oPccorom oPoeror oPorom occc8oe8om occo8o8o oecPco oecccccco oorcccor ororcco8om qHoe qoHcro rccoor ro8or ro8ororo roPoe roe8ok roeP roeccror roeoer rorccor roromr Now let's find all occurrences of these rare labels in the parags text: cat .parags-j-ecc.evt \ | enum-word-locations .labels-rare.dic \ | sort -b +2 -3n \ > .label-rare-occurrences.idx cat .parags-j-ecc.evt \ | enum-word-locations .labels-rarest.dic \ | sort -b +2 -3n \ > .label-rarest-occurrences.idx Let's tabulate their frequencies per page: foreach f ( '' -rare -rarest ) cat .label${f}-occurrences.idx \ | gawk '/./ { print $1 }' \ | sed -e 's/\..*>/>/g' \ | sort | uniq -c | expand \ | compute-freqs \ | sort +0 -1nr \ > .label${f}-refs-by-panel.frq end --- .label-rare-refs-by-panel.frq ------------------------ 93 0.030 78 0.025 63 0.020 60 0.019 60 0.019 57 0.018 55 0.017 53 0.017 51 0.016 50 0.016 49 0.016 49 0.016 48 0.015 44 0.014 42 0.013 41 0.013 40 0.013 39 0.012 39 0.012 37 0.012 36 0.011 36 0.011 35 0.011 35 0.011 35 0.011 34 0.011 34 0.011 34 0.011 33 0.010 30 0.010 28 0.009 27 0.009 26 0.008 25 0.008 25 0.008 24 0.008 24 0.008 24 0.008 24 0.008 23 0.007 23 0.007 22 0.007 22 0.007 21 0.007 21 0.007 20 0.006 20 0.006 20 0.006 19 0.006 18 0.006 18 0.006 17 0.005 17 0.005 17 0.005 16 0.005 16 0.005 15 0.005 15 0.005 15 0.005 15 0.005 15 0.005 15 0.005 15 0.005 14 0.004 14 0.004 14 0.004 14 0.004 14 0.004 14 0.004 14 0.004 14 0.004 13 0.004 13 0.004 13 0.004 13 0.004 13 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 12 0.004 11 0.003 11 0.003 11 0.003 11 0.003 11 0.003 11 0.003 11 0.003 11 0.003 10 0.003 10 0.003 10 0.003 10 0.003 10 0.003 10 0.003 10 0.003 10 0.003 9 0.003 9 0.003 9 0.003 9 0.003 9 0.003 9 0.003 9 0.003 9 0.003 9 0.003 9 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 8 0.003 7 0.002 7 0.002 7 0.002 7 0.002 7 0.002 7 0.002 7 0.002 7 0.002 7 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 6 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 5 0.002 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 4 0.001 3 0.001 3 0.001 3 0.001 2 0.001 1 0.000 1 0.000 1 0.000 1 0.000 1 0.000 ---------------------------------------------------------- --- .label-rarest-refs-by-panel.frq ------------------------ 28 0.041 18 0.026 15 0.022 13 0.019 13 0.019 12 0.017 12 0.017 11 0.016 10 0.014 10 0.014 10 0.014 9 0.013 9 0.013 9 0.013 8 0.012 8 0.012 8 0.012 8 0.012 8 0.012 8 0.012 8 0.012 7 0.010 7 0.010 7 0.010 7 0.010 7 0.010 6 0.009 6 0.009 6 0.009 6 0.009 6 0.009 6 0.009 6 0.009 6 0.009 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 5 0.007 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 4 0.006 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 3 0.004 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 2 0.003 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 1 0.001 ------------------------------------------------------------ Let's compute the density of label references per page (number of references divided by number of text characters in page): foreach f ( '' -rare -rarest ) cat .label${f}-refs-by-panel.frq \ | sort +2 -3 \ > .foo join \ -a 1 -e 00 \ -j1 3 -j2 1 \ -o0,1.1,1.2,2.2 \ .foo .panels.nchars \ > .bar${f} cat .bar${f} \ | gawk '/./ {printf "%-8s %5d %5.3f %5d\n", $1, $2, $3, int(1000*$2/$4 + 0.5)}' \ | sort -b +3 -4nr \ > .label${f}-refs-by-panel.rfrq end --- .label-refs-by-panel.rfrq ------------------------ 146 0.005 348 546 0.020 294 178 0.006 293 69 0.003 273 636 0.023 269 99 0.004 268 436 0.016 267 136 0.005 261 97 0.004 261 152 0.006 256 563 0.020 252 106 0.004 249 95 0.003 242 106 0.004 240 131 0.005 234 76 0.003 229 100 0.004 224 37 0.001 224 105 0.004 224 491 0.018 223 45 0.002 223 511 0.019 219 104 0.004 218 87 0.003 215 395 0.014 215 58 0.002 215 79 0.003 215 482 0.017 213 75 0.003 213 475 0.017 211 ... ... ..... ... 30 0.001 89 31 0.001 87 35 0.001 87 22 0.001 85 53 0.002 84 19 0.001 82 38 0.001 80 33 0.001 76 19 0.001 73 30 0.001 54 11 0.000 49 12 0.000 39 ------------------------------------------------------ --- .label-rare-refs-by-panel.rfrq ------------------------ 35 0.011 58 24 0.008 57 93 0.030 50 10 0.003 50 20 0.006 47 12 0.004 47 14 0.004 44 16 0.005 43 14 0.004 42 20 0.006 42 11 0.003 41 15 0.005 41 24 0.008 40 15 0.005 38 18 0.006 38 15 0.005 37 14 0.004 36 6 0.002 36 14 0.004 35 57 0.018 35 12 0.004 34 78 0.025 33 17 0.005 33 17 0.005 32 9 0.003 31 12 0.004 31 10 0.003 31 14 0.004 31 8 0.003 30 22 0.007 30 ... ... ..... ... 2 0.001 8 4 0.001 8 4 0.001 8 14 0.004 8 7 0.002 8 12 0.004 7 1 0.000 6 4 0.001 6 1 0.000 4 1 0.000 2 1 0.000 2 1 0.000 2 ----------------------------------------------------------- --- .label-rarest-refs-by-panel.rfrq ------------------------ 5 0.007 20 12 0.017 20 7 0.010 19 3 0.004 18 5 0.007 16 8 0.012 15 5 0.007 15 28 0.041 15 3 0.004 15 3 0.004 15 3 0.004 14 4 0.006 14 5 0.007 13 5 0.007 13 5 0.007 12 4 0.006 12 8 0.012 11 5 0.007 11 4 0.006 11 4 0.006 11 4 0.006 10 4 0.006 10 5 0.007 10 3 0.004 9 3 0.004 9 5 0.007 9 4 0.006 9 4 0.006 9 2 0.003 9 4 0.006 9 ... ... ..... ... 4 0.006 2 2 0.003 2 3 0.004 2 1 0.001 2 1 0.001 2 1 0.001 1 1 0.001 1 2 0.003 1 1 0.001 1 1 0.001 1 1 0.001 1 2 0.003 1 1 0.001 0 1 0.001 0 ------------------------------------------------------------- 97-10-22 stolfi =============== Let's make a joint file with all three counts: foreach f ( '' -rare -rarest ) cat .label${f}-refs-by-panel.rfrq \ | sort +0 -1 \ > .foo${f} end join \ -a 1 -a 2 -e 00 \ -j1 1 -j2 1 \ -o0,1.2,1.4,2.2,2.4 \ .foo .foo-rare \ > .bar join \ -a 1 -a 2 -e 00 \ -j1 1 -j2 1 \ -o0,1.2,1.3,1.4,1.5,2.2,2.4 \ .bar .foo-rarest \ > .baz cat .baz \ | sed -e 's/<\(.*\)>/<\1> {\1}/g' \ | panel-to-page \ | tr -d '{}<>' \ | sort +0 -1n \ | gawk '/./ {printf "%03d %-6s %5d %5d %5d %5d %5d %5d\n", $1,$2,$3,$4,$5,$6,$7,$8}' \ > .label-refs-by-panel.jfrq --- .label-refs-by-panel.jfrq ------------------------ 001 f1r 116 122 12 13 2 2 002 f1v 78 197 14 35 5 13 003 f2r 67 157 12 28 3 7 004 f2v 28 103 5 18 1 4 005 f3r 90 159 12 21 5 9 006 f3v 62 160 6 15 2 5 007 f4r 37 123 4 13 2 7 008 f4v 35 87 1 2 0 0 009 f5r 22 85 5 19 0 0 010 f5v 38 176 5 23 2 9 ... ...... ... ... ... ... ... ... 262 f114v 289 147 24 12 5 3 263 f115r 445 188 60 25 15 6 264 f115v 342 157 37 17 1 0 265 f116r 465 193 51 21 10 4 ------------------------------------------------------ Let's make histograms of those ratios, sorted by page position. cat .label-refs-by-panel.jfrq \ | make-label-ref-graphs \ -v MAX1=348 -v MAX2=58 -v MAX3=20 \ > .label-refs-by-panel.jhis --- .label-refs-by-panel.jhis ------------------------ 001 f1r 116 122 ooo...... 12 13 oo....... 2 2 ......... 002 f1v 78 197 ooooo.... 14 35 ooooo.... 5 13 ooooo.... 003 f2r 67 157 oooo..... 12 28 oooo..... 3 7 ooo...... 004 f2v 28 103 oo....... 5 18 oo....... 1 4 o........ 005 f3r 90 159 oooo..... 12 21 ooo...... 5 9 oooo..... ... ... ... ... ... ... ... ... ... ... ... 261 f114r 382 170 oooo..... 35 16 oo....... 6 3 o........ 262 f114v 289 147 ooo...... 24 12 o........ 5 3 o........ 263 f115r 445 188 oooo..... 60 25 ooo...... 15 6 oo....... 264 f115v 342 157 oooo..... 37 17 oo....... 1 0 ......... 265 f116r 465 193 oooo..... 51 21 ooo...... 10 4 o........ ------------------------------------------------------ The most label-rich and label-poor pages are ALL LABELS 146 0.005 348 546 0.020 294 178 0.006 293 30 0.001 54 11 0.000 49 12 0.000 39 UNDER 100 35 0.011 58 24 0.008 57 93 0.030 50 10 0.003 50 20 0.006 47 12 0.004 47 14 0.004 44 16 0.005 43 14 0.004 42 20 0.006 42 11 0.003 41 1 0.000 2 1 0.000 2 1 0.000 2 UNDER 25 5 0.007 20 12 0.017 20 7 0.010 19 3 0.004 18 5 0.007 16 8 0.012 15 5 0.007 15 28 0.041 15 3 0.004 15 3 0.004 15 1 0.001 1 1 0.001 1 2 0.003 1 1 0.001 1 1 0.001 1 1 0.001 1 2 0.003 1 1 0.001 0 1 0.001 0 The UNDER 100 class is distributed fairly uniformly among the most labelliferous pages. The UNDER 25 class has a steeper ditribution. The "starred paragraph" pages (incuding f58r) are not exceptionally labelliferous in relative terms; their absolute counts are high only because they contain a lot of paragraphical text. Page f40r has many labels of the ALL and UNDER 100 classes, but only 2 of the UNDER 25 class. On the other hand, f99v is label-rich in all three classes. The same can be said of page f58r, except for a modest drop in the UNDER-25 class. Let's find WHICH labels were mentioned on pages f58r and f99v. I had previously written a gawk script "show-occurrences" to show the occurrences of a bunch of words in a text. let's run it, just as a check: foreach f ( f58r f99v ) cat .units-parags.dir \ | egrep "^${f}[:.]" \ | sed \ -e 's/:.*$//g' \ -e 's:^:L16-ecc-x/:g' \ > .tmp cat `cat .tmp` \ | find-and-show-occurrences .labels-rarest.dic \ > .label-rarest-occs2-$f end Let's now generate the same files from the occurrence lists. We will add to the latter the label's definition code. foreach f ( '' '-rare' '-rarest' ) cat .label${f}-occurrences.idx \ | sort -b +3 -4 \ > .occ cat .labels-first.def \ | sort -b +1 -2 \ | tr '<>' '{}' \ >.def join \ -a1 -e '{???}' \ -j1 4 -j2 2 \ -o1.1,1.2,1.3,0,2.3 \ .occ .def \ | sort -b +2 -3n +3 -4 \ > .label${f}-occ-def.idx end foreach f ( f58r f99v ) cat .units-parags.dir \ | egrep "^${f}[:.]" \ | sed \ -e 's/:.*$//g' \ -e 's:^:L16-ecc-x/:g' \ > .tmp foreach g ( '' '-rare' '-rarest' ) cat .label${g}-occ-def.idx \ | egrep "<${f}[.]" \ > .occ cat `cat .tmp` \ | show-occurrences .occ \ > .label${g}-occs-$f end end Now let's prepare a similar file in FSG format. We must use the FSG text, and we must replace each ECC label in the occurrence file by its correspondent FSG label. For the second part, let's first extract the labels in FSG notation, just as we did for ECC. Note that we must remove the "."s from the text, but keep the "-"s and "="s: cat .units-labels.dir \ | sed \ -e 's/:.*$//g' \ -e 's:^:L16/:g' \ > .tmp cat `cat .tmp` \ > .labels-m-fsg.evt cat .units-parags.dir \ | sed \ -e 's/:.*$//g' \ -e 's:^:L16/:g' \ > .tmp cat `cat .tmp` \ > .parags-m-fsg.evt /bin/rm -f .labels-fsg.def cat .labels-m-fsg.evt \ | remove-comments-from-evt \ | sed \ -e 's/ *//g' \ -e 's/;[A-Z]>/>/g' \ -e 's/[-=]//g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>/> /g' \ | egrep ' .' \ > .labels1.def cat .labels-m-fsg.evt \ | remove-comments-from-evt \ | /n/gnu/bin/sed \ -e 's/ *//g' \ -e 's/;[A-Z]>/>/g' \ -e 's/[=-]$//g' \ -e 's/^/@/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/>\(.*\)[.]/>\1/g' \ -e 's/@\(<[^>]*>\)\([^ @-][^ @-]*\)[ -][ -]*/@\1\2@\1/g' \ -e 's/@\(<[^>]*>\)\([^ @-][^ @-]*\)[ -][ -]*/@\1\2@\1/g' \ -e 's/@\(<[^>]*>\)\([^ @-][^ @-]*\)[ -][ -]*/@\1\2@\1/g' \ -e 's/@\(<[^>]*>\)\([^ @-][^ @-]*\)[ -][ -]*/@\1\2@\1/g' \ -e 's/@\(<[^>]*>\)\([^ @-][^ @-]*\)[ -][ -]*/@\1\2@\1/g' \ -e 's/>/> /g' \ | tr '@' '\012' \ | egrep ' .' \ > .labels2.def cat .labels1.def .labels2.def \ | sort | uniq \ | sed \ -e 's/<\(.*\)> \(.*\)/<\1> \2 {\1}/g' \ -e 's/\.[^>]*> */> /g' \ | panel-to-page \ | tr '{}' '<>' \ > .labels-fsg.def Now make file that lists each FSG label with the equivalent ECC label: cat .labels-fsg.def \ | tr '<>' '{}' \ | gawk 'BEGIN{n=0} /./ {n++; printf " %s %s\n",n,$2,$3}' \ > .foo-fsg cat .foo-fsg \ | fsg2ecc \ > .foo-ecc join \ -j1 1 -j2 1 \ -o1.2,2.2,2.3 \ .foo-ecc .foo-fsg \ | sort | uniq \ > .label-ecc-fsg.map Now let's use that table to "translate" the label occurrence index from ECC to FSG. foreach g ( '' '-rare' '-rarest' ) cat .label${g}-occ-def.idx \ | translate-occurrences-ecc-to-fsg \ .label-ecc-fsg.map \ > .label-fsg${g}-occ-def.idx end Finally, let's list the occurrences on pages f58r and f99v. We must "fake" entries with transcription code ";S" in order for show-occurrences to work. foreach f ( f58r f99v f95v2 f69r f9v ) cat .units-parags.dir \ | egrep "^${f}[:.]" \ | sed \ -e 's/:.*$//g' \ -e 's:^:L16/:g' \ > .tmp-$f end foreach f ( f58r f99v f95v2 f69r f9v ) foreach g ( '' '-rare' '-rarest' ) cat .label-fsg${g}-occ-def.idx \ | egrep "<${f}[.]" \ > .occ cat `cat .tmp-$f` \ | fake-S-transcription-codes \ | show-occurrences .occ \ | egrep -v '^<[^>]*;S> *$' \ > .label-fsg${g}-occs-$f end end Curiously, label f89v1.t.4 is identical to f89v1.b.4. There is a comment on the latter: #looks like 3 plants and 4 names. none of the other "vases" seem to be named. 97-10-19 stolfi =============== In order to make the results easier to describe, I should rename the textual unit files with standard panel numbers (like f77v) rather than modern page numbers (like 114).