Class: LlamaCppTokenizer
Tokenizer for LlamaCpp.
Example
const tokenizer = new LlamaCppTokenizer();
const text = "At first, Nox didn't know what to do with the pup.";
const tokenCount = await countTokens(tokenizer, text);
const tokens = await tokenizer.tokenize(text);
const tokensAndTokenTexts = await tokenizer.tokenizeWithTexts(text);
const reconstructedText = await tokenizer.detokenize(tokens);
Implements
Constructors
constructor
• new LlamaCppTokenizer(api?): LlamaCppTokenizer
Parameters
| Name | Type |
|---|---|
api | ApiConfiguration |
Returns
Defined in
packages/modelfusion/src/model-provider/llamacpp/LlamaCppTokenizer.ts:30
Methods
callTokenizeAPI
▸ callTokenizeAPI(text, callOptions?): Promise<{ tokens: number[] }>
Parameters
| Name | Type |
|---|---|
text | string |
callOptions? | FunctionCallOptions |
Returns
Promise<{ tokens: number[] }>
Defined in
packages/modelfusion/src/model-provider/llamacpp/LlamaCppTokenizer.ts:34
tokenize
▸ tokenize(text): Promise<number[]>
Asynchronously tokenize the given text into a sequence of numeric tokens.
Parameters
| Name | Type | Description |
|---|---|---|
text | string | Input text string that needs to be tokenized. |
Returns
Promise<number[]>
A promise containing an array of numbers, where each number is a token representing a part or the whole of the input text.
Implementation of
Defined in
packages/modelfusion/src/model-provider/llamacpp/LlamaCppTokenizer.ts:65
Properties
api
• Readonly api: ApiConfiguration
Defined in
packages/modelfusion/src/model-provider/llamacpp/LlamaCppTokenizer.ts:28