dnn_Tokenizer

Deep neural networksclassOpenCV 5.0.0

import { dnn_Tokenizer } from '@banou/opencv-wasm'

Use after await initOpenCV(). See the initialization and named imports guide.

ARGUMENTSConstructor or factory

→

CLASSdnn_Tokenizer

→

RETURN TYPEOwned native handle

Call structure. A void return can still write to destination arguments. The parameter descriptions define inputs, outputs and ownership.

Native object: release it with using or delete(). Factories can return null; check before calling methods.

High-level tokenizer wrapper for DNN usage.

Provides a simple API to encode and decode tokens for LLMs. Models are loaded via Tokenizer::load().

using namespace cv::dnn;
Tokenizer tok = Tokenizer::load("/path/to/model/");
std::vector<int> ids = tok.encode("hello world");
std::string text = tok.decode(ids);

Upstream declaration ↗

Constructors and members

static load

Load a tokenizer from a model directory.

Expects the directory to contain:

• config.json with field model_type with value "gpt2" or "gpt4".
• tokenizer.json produced by the corresponding model family.

The argument is a path prefix; this function concatenates file names directly (e.g. model_dir + "config.json"), so model_dir must end with an appropriate path separator.

load(model_config: EmbindString): dnn_Tokenizer;

model_config

Path to config.json for model.

Returns

A Tokenizer ready for use. Throws cv::Exception if files are missing or model_type is unsupported.

Upstream declaration ↗

clone

Create another handle to the same native object. This retains the object without copying its pixels or algorithm state; dispose both handles separately.

clone(): this;

Returns

The this result.

encode

Encode UTF-8 text to token ids (special tokens currently disabled).

Calls the underlying CoreBPE::encode with an empty allowed-special set.

encode(text: EmbindString): IntVector;

text

UTF-8 input string.

Returns

Vector of token ids (32-bit ids narrowed to int for convenience). Release returned native handles with using or delete(), including handles nested in results.

Upstream declaration ↗

decode

dnn_Tokenizer.decode native operation from cv::dnn::Tokenizer::decode.

decode(tokens: IntVector): string;

tokens

tokens argument (IntVector).

Returns

The string result.

Upstream declaration ↗

These signatures describe this package. Upstream documentation can mention optional backends that are absent from this build. Check runtime compatibility before choosing a backend or file format.