Member Avatar for merse

Are there ay freely available language library?

I need class of LETTER, WORD, SENTANCE, TEXT, ...

for example a letter can be more than one character, like sch in German
WORD category has to be stored, like noun/adjective etc.

I have to write my own package?

  • 2 Contributors
  • 2 Replies
  • 94 Views
  • 8 Hours Discussion Span
  • Latest Post Latest Post by merse

Dani AI

Generated

After the thread started with (and the off-topic reply by ), here are practical, modern options and a short integration plan for building reliable language primitives without reinventing everything.

Start with robust Unicode handling and segmentation. ICU (C/C++) implements Unicode normalization, collation and the boundary rules you need (grapheme clusters, word and sentence breaks). The Unicode text segmentation spec is TR29 and is what modern tokenizers follow. See ICU and the TR29 rules for details: ICU Unicode TR29.

For linguistic analysis (tokenization + POS + lemmas + morph features) use off-the-shelf analyzers and models rather than hand-coding rules. UDPipe offers C/C++ code plus pretrained models for many languages (tokenize, tag, lemmatize, parse). For dictionary-based morphology and spell/morph rules, Hunspell is widely used and embeddable. For language-specific needs there are mature analyzers (e.g., MeCab for Japanese). See: UDPipe Hunspell MeCab.

Design notes and workflow: normalize text first (consistent NFC/NFKC), use ICU to compute grapheme-clusters and sentence boundaries, then feed token sequences to a tagger/morph analyzer. Represent tokens with a small attribute bag (normalized form, lemma, POS tag, UD morph features) and keep raw offsets into the original string. Prefer a standard tagset like Universal Dependencies to keep categories portable: Universal Dependencies.

Troubleshooting tips: inconsistent normalization breaks lookups; combining marks and emoji need grapheme-aware logic; language-specific tokenizers/models are often essential for high quality. If a single C++ dependency is preferred, start with ICU + UDPipe/Hunspell; writing everything yourself is possible but requires significant linguistics and Unicode care.

Recommended Answers

All 2 Replies

Member Avatar for pseudorandom21

Windows Update?

Member Avatar for merse

He???

I need a C++ library.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.