sodupath.com Logo
sodupath.com Research Paper
Segmentation & Tokenization • October 2026 • 12 min read

Tokenization and Sandhi Decomposition for Low-Resource Indo-Aryan Languages: A Finite-State Hybrid Approach

Resolving boundary ambiguities, vowel elision, and run-together clitics in Sinhala digital texts.

SP
Sodupath Computational Linguistics Research Group
Digital Humanities & South Asian Language Technology
📌 Abstract

"Standard tokenizers trained on Western European languages assume that whitespace strictly delineates word boundaries. In South Asian languages, especially Sinhala and Sanskrit, morphophonemic euphonic combinations (Sandhi) merge adjacent word boundaries, whilst colloquial writing often erroneously breaks indivisible compound morphemes or conjoins independent words with clitic particles. This paper presents the hybrid tokenization and Sandhi decomposition engine utilized in Sodupath to guarantee syntactically sound segmentation."

1. The Word Boundary Problem in South Asian NLP

Tokenization is universally recognized as the foundational pipeline step in Natural Language Processing. However, in South Asian languages characterized by agglutinative postpositions, compound noun formation, and Sandhi processes, whitespace cannot be naively treated as an invariant delimiter.

In contemporary Sinhala typography, two diametrically opposed errors dominate:

  • Erroneous Conjoining (අනවශ්‍ය පද එකතු කිරීම): Independent clitics, emphatic particles (e.g., -ද, -නම්, -මිස, -ම, -ය), and case markers merged indiscriminately onto lexical stems (e.g., ඔහුදකියන්නේය* instead of ඔහු ද කියන්නේය).
  • Erroneous Disjunction (අනවශ්‍ය පද වෙන් කිරීම): Semantically unified lexical units, reduplications, and prefixes erroneously split across spaces (e.g., ප්‍රති පත්ති* instead of ප්‍රතිපත්ති, or අන්තර් ජාලය* instead of අන්තර්ජාලය).

2. Sandhi Typology in Sinhala

Traditional Sinhala grammar recognizes several classes of euphonic combination (සන්ධි ක්‍රම):

Lopa Sandhi (ලෝප සන්ධි - Elision)

The dropping of a terminal vowel when juxtaposed with an initial vowel:

නර + ඉඳු -> නරිඳු (Nara + Indu -> Narindu)

Adesha Sandhi (ආදේශ සන්ධි - Substitution)

The transmutation of vowels into semivowels (y, v) under glide formation:

ගුරු + උපදේශ -> ගුරූපදේශ (Guru + Upadesha -> Gurūpadesha)

3. The Sodupath Multi-Phase Segmentation Pipeline

To preserve genuine Sandhi whilst correcting colloquial conjoining and disjunction errors, Sodupath deploys a bidirectional sliding-window finite state automaton (FSA):

// Bidirectional Clitic Disaggregation Routine
INPUT: String token T
FOR suffix S in INDIVISIBLE_CLITICS: {"නම්", "මිස", "වුවද", "හෙයින්", "බැවින්"}:
IF T.endswith(S) AND Len(T) - Len(S) >= MIN_STEM_LEN:
Stem = T[:-len(S)]
IF Is_Valid_Lemma(Stem):
YIELD [Stem, S] // Decompose into independent tokens
ELSE:
RETAIN T // Avoid corrupting integrated roots (e.g., 'පැමිණියානම්' vs. 'ගමනම්')

Academic Citation & BibTeX

@article{sodupath_tokenization_sandhi_joining_south_asian_nlp,
  title={ Tokenization and Sandhi Decomposition for Low-Resource Indo-Aryan Languages: A Finite-State Hybrid Approach },
  author={ Sodupath Computational Linguistics Research Group },
  journal={Sodupath South Asian Computational Linguistics Hub},
  year={2026},
  url={https://www.sodupath.com/research/tokenization-sandhi-joining-south-asian-nlp}
}

Test These Computational Rules Live

Experience the Sodupath rule engine in action. Paste complex Sinhala prose and analyze subject-verb concord, orthography, and segmentation in real-time.

Launch Live Editor ➔