|
There is a newer version available. For the latest stable version, please use dengjen-tashkeel 3.x! |
Model accuracy and limitations
dengjen-tashkeel restores diacritics with an ONNX neural model (from Hareef, trained mainly on Modern Standard Arabic data) — it does not run independent grammatical analysis. Where the correct diacritic depends on context the model wasn’t trained to resolve, it can guess wrong. That’s a model-accuracy limitation, not a bug in this library’s code (see Architecture for what this library is and isn’t responsible for).
Known case: proper nouns without resolvable context
Given the input:
عن أمير المؤمنين أبي حفص عمر بن الخطابthe model diacritizes the name حفص as حِفَصِ ("hifsi") instead of the grammatically correct
حَفْصٍ ("hafsin"). Per the project’s own README, the correct case ending for a proper noun like
this depends on context the model wasn’t trained to resolve. See
issue #28 for the full discrepancy and any
follow-up.
Mitigating uncertain diacritics: taskeen
Every binding exposes a taskeen option (taskeen_threshold in Rust and Python, the second
Optional<Float> argument in Java, --taskeen/--prob in the CLI — see
Per-binding usage). When enabled, the model substitutes a sukoon for any
predicted diacritic whose confidence is below the threshold, instead of guessing — trading a
possibly-wrong diacritic for an explicit "unresolved" marker. It applies at every character that
receives a diacritic, not only the final case ending (the CLI’s --taskeen help text says
"case-ending", which understates this).
Taskeen only helps where the model’s confidence is actually low. A confidently wrong prediction
is not replaced, so taskeen does not guarantee a fix for the حفص example above.
Scope
Per the project’s non-scope note: diacritization only, not a general-purpose Arabic NLP toolkit. For proper nouns, genitive chains, or any other case-ending decision that depends on context spanning multiple words, verify output against domain-relevant text before relying on it in a downstream product.
