# Last edited on 2026-07-03 16:31:36 by stolfi
050 Repeating token sequences in the SPS and the SBJ

SUMMARY

  In this note we convert the Mandarin pinyn text of the Shennong Bencaojing (SBJ)
  -- with and without tones, with and without noise -- to Voynichese-like texts.
  We also compare the result with Voynichese and "de-noised" Voynichese.
  
SETUP

  ln -s ../.. work
  ln -s work/convert_pinyn_to_numeric.py
  ln -s work/ivtff_format.py
  ln -s work/replace_hanzi_punct_by_ascii_punct.sed
  
  ln -s ../077
  ln -s 077/convert_starps_raw_to_lin_ivt.py
  ln -s 077/convert_starps_lin_to_par_ivt.py
  ln -s 077/note_077_funcs.py
  
  
CONVERTING THE SBJ TO VOYNICHESE-LIKE TEXT

  For the first part of this note,
  the main input file is in/bencao-pys.utf, a Mandarin pinyin reading of a
  version of the Shennong Bencaojing. Original SBJ files in Chinese
  characters were downloaded from the Chinese Texts Project site and
  from the Chinese wikisource. Those file were compatibilized and merged
  by hand, then several gross errors (like merged recipes) were fixed.
  Then the whole was converted to Mandarin pinyin with tone diacritics
  through Google Translate. Then multi-syllable compounds were split
  with some RE patterns.  Then all punctuation was deleted.
  
  The numbering of the recipes in this file ("<b{S}.{U}.{NNN}>") should mostly match 
  the numbering used in Notes/077.  However the SBJ files over there have evolved
  a lot since this pinyin file was created.

  The basic idea is to convert tone diacritics and the ü vowel to plain lowercase
  Latin letters, then optionally intriduce some random errors respecting the
  broad categories (consonants, vowels, tone marks), then permute vowels
  and consonants just to obfuscate things a little.  Optionally the tone
  marks are removed.
  
DE-NOISING VOYNICHESE TEXT

  For the second part of this note we convert the Starred Parags section (SPS)
  to a string of tokens in the similar format, without any punctualtion.  We
  then try to remove some of the spelling "noise" by some replacements
  (like @m -> @iin, @ir -> @iin, ) and guessing replacements for @?.
  
COUNTING REPEATED WORD SEQUENCES

  For each of the texts above we count repeated word sequences of length k
  and plot the counts as a function of k.
  
DO IT

  do_note_050.sh