There is a newer version available. For the latest stable version, please use dengjen-tashkeel 3.x!

Quick start

Rust

use dengjen_tashkeel::{create_inference_engine, do_tashkeel};

// None = use the bundled model.
let engine = create_inference_engine(None)?;
let diacritized = do_tashkeel(&engine, "بسم الله الرحمن الرحيم", None, false)?;

do_tashkeel’s third argument is an optional taskeen threshold (see below) and the fourth is `preprocessed — pass true only if text is already sentence-segmented, otherwise the library segments it for you. The library segments the text into sentences and caps each sentence at CHAR_LIMIT (12,000 characters); a single longer sentence returns Err(DengjenTashkeelError::InputTooLong(_)). With preprocessed = true the whole text is one sentence. The CLI does not truncate input: with an input file it passes each line to the library and exits with that error for an over-long sentence; from stdin it reads all of stdin.

Python

from dengjen_tashkeel_py import tashkeel

tashkeel("بسم الله الرحمن الرحيم")

The bundled model loads on the first call. To load it up front, or from your own ONNX file, call dengjen_tashkeel_py.init() or init("/path/to/model.onnx") before the first tashkeel call.

The Rust side logs via the log facade but installs no backend; if you want its diagnostics (e.g. a redundant-init warning) surfaced through Python’s own logging module, install pyo3-log in your embedding process.

C

The header exports four functions, spelled exactly as shown: dengjen_tashkeel_init, dengjenTashkeelTashkeel, dengjenTashkeelTashkeelWithOptions (the same call with its settings in a DengjenTashkeelOptions struct), and dengjen_tashkeel_free_string. Errors come back through the ExternError out-parameter (code == 0 is success). Release every returned string and every non-null err.message with dengjen_tashkeel_free_string. Without a prior dengjen_tashkeel_init, the first dengjenTashkeelTashkeel or dengjenTashkeelTashkeelWithOptions call initializes the bundled model lazily, after which dengjen_tashkeel_init returns DENGJEN_TASHKEEL_ALREADY_INITIALIZED (4) and keeps the existing engine. A null text argument returns DENGJEN_TASHKEEL_INVALID_ARGUMENT (5).

#include <stdio.h>
#include "dengjen_tashkeel.h"

int main(void) {
    ExternError err = {0};

    /* Optional: NULL loads the bundled model. Must precede the first tashkeel call. */
    dengjen_tashkeel_init(NULL, &err);
    if (err.code != 0) {
        fprintf(stderr, "%s\n", err.message);
        dengjen_tashkeel_free_string(err.message);
        return 1;
    }

    /* NULL threshold disables taskeen; false = let the library segment sentences. */
    char *out = dengjenTashkeelTashkeel("بسم الله الرحمن الرحيم", NULL, false, &err);
    if (err.code != 0) {
        fprintf(stderr, "%s\n", err.message);
        dengjen_tashkeel_free_string(err.message);
        return 1;
    }
    puts(out);
    dengjen_tashkeel_free_string(out);
    return 0;
}

See crates/capi/ffi_usage_example.py in the repository for the same flow via ctypes, and Per-binding usage for the error codes.

Java

import io.github.zirekhq.dengjen.tashkeel.Tashkeel;
import java.util.Optional;

Tashkeel tashkeel = Tashkeel.loadDefault();
String diacritized = tashkeel.diacritize("بسم الله الرحمن الرحيم", Optional.empty(), false);

diacritize’s second argument is an optional taskeen threshold (see below) and the third is `preprocessed — pass true only if the text is already sentence-segmented, otherwise the library segments it for you. Errors surface as a checked TashkeelException, whose reason() is an exhaustively switchable sealed type: InputTooLong, InferenceError, ModelLoadError for codes 1-3, and Unknown(code, message) for every other code in dengjen_tashkeel.h.

CLI

echo "بسم الله الرحمن الرحيم" > input.txt
dengjen-tashkeel -f input.txt
Usage: dengjen-tashkeel [OPTIONS]

Options:
  -f, --input-file <INPUT_FILE>    Input file (default `stdin`)
  -o, --output-file <OUTPUT_FILE>  Output file (default `stdout`)
  -i, --interactive                Use interactive mode (useful for testing)
  -t, --taskeen                    Use sukoon for case-ending diacritic if the model is uncertain
  -p, --prob <PROB>                Taskeen threshold probability [default: 0.95]
  -x, --onnx <ONNX_MODEL>          ONNX model (default: use bundled model if available)
  -h, --help                       Print help
  -V, --version                    Print version

The --taskeen help text mentions case-ending diacritics, but the threshold applies to every predicted diacritic (see below). With neither --input-file nor --output-file, the CLI runs in interactive mode by default. Set TASHKEEL_LOG (an env_logger filter such as debug; default info) to change log verbosity.

The taskeen option

When enabled (taskeen_threshold in Rust/Python, --taskeen/--prob in the CLI), the model substitutes a sukoon for the predicted diacritic wherever its confidence (the probability of its top prediction at that position) is below the threshold (0 to 1). This applies at every character that receives a diacritic, not only the final case ending.