Interface: BasicTokenizer

Interface for a basic tokenizer capable of converting text into tokens.

This serves as the base for tokenization functionalities where the focus is on the transformation of input text into a series of numeric tokens.

Hierarchy

• BasicTokenizer

  ↳ FullTokenizer

Implemented by

• LlamaCppTokenizer

Properties

tokenize

• tokenize: (text: string) => PromiseLike<number[]>

Asynchronously tokenize the given text into a sequence of numeric tokens.

Param

Input text string that needs to be tokenized.

Type declaration

▸ (text): PromiseLike<number[]>

Asynchronously tokenize the given text into a sequence of numeric tokens.

Parameters

Name	Type	Description
text	string	Input text string that needs to be tokenized.

Returns

PromiseLike<number[]>

A promise containing an array of numbers, where each number is a token representing a part or the whole of the input text.

Defined in

packages/modelfusion/src/model-function/tokenize-text/Tokenizer.ts:13