Skip to main content

Ollama

Generate text and embeddings using Ollama. You can run the Ollama server locally or remote.

Setup​

  1. Install Ollama following the instructions in the jmorganca/ollama repository.
  2. Pull the model you want to use, e.g. Llama 2.
  3. Start Ollama in server mode: ollama serve

Model Functions​

Examples

Generate Text (Completion)​

Generate Text Docs | OllamaCompletionModel API

The OllamaCompletionModel uses the Ollama completion API to generate text.

import { ollama, generateText } from "modelfusion";

const text = await generateText({
  model: ollama
    .CompletionTextGenerator({
      model: "mistral:text", // mistral base model without instruct fine-tuning (no prompt template)
      temperature: 0.7,
      maxGenerationTokens: 120,
    })
    .withTextPrompt(), // use text prompt style

  prompt: "Write a short story about a robot learning to love:\n\n",
});

When using the Ollama completion API, you can use the raw mode and set a prompt template on the model. This enables you to use the withChatPrompt, withInstructionPrompt and withTextPrompt helpers.

import { generateText, ollama } from "modelfusion";

const text = await generateText({
  model: ollama
    .CompletionTextGenerator({
      model: "mistral",
      promptTemplate: ollama.prompt.Mistral,
      raw: true, // required when using custom prompt template
      maxGenerationTokens: 120,
    })
    .withTextPrompt(), // use text prompt style

  prompt: "Write a short story about a robot learning to love.",
});

Generate Text (Chat)​

Generate Text Docs | OllamaChatModel API

The OllamaChatModel uses the Ollama chat API to generate text.

import { ollama, generateText } from "modelfusion";

const text = await generateText({
  model: ollama
    .ChatTextGenerator({
      model: "llama2:chat",
      maxGenerationTokens: 500,
    })
    .withTextPrompt(),

  prompt: "Write a short story about a robot learning to love:",
});

The Ollama prompt also supports base64-encoded png and jpeg images, e.g. with an instruction prompt:

import { ollama, generateText } from "modelfusion";

const image = fs.readFileSync(path.join("data", "comic-mouse.png"));

const text = await generateText({
  model: openai
    .ChatTextGenerator({
      model: "gpt-4-vision-preview",
      maxGenerationTokens: 1000,
    })
    .withInstructionPrompt(),

  prompt: {
    instruction: [
      { type: "text", text: "Describe the image in detail:\n\n" },
      { type: "image", image, mimeType: "image/png" },
    ],
  },
});

Stream Text (Completion)​

Stream Text Docs | OllamaCompletionModel API

import { ollama, streamText } from "modelfusion";

const textStream = await streamText({
  model: ollama
    .CompletionTextGenerator({
      model: "mistral",
      promptTemplate: ollama.prompt.Mistral,
      raw: true, // required when using custom prompt template
      maxGenerationTokens: 500,
    })
    .withTextPrompt(),

  prompt: "Write a short story about a robot learning to love:",
});

for await (const textPart of textStream) {
  process.stdout.write(textPart);
}

Stream Text (Chat)​

Stream Text Docs | OllamaChatModel API

import { ollama, streamText } from "modelfusion";

const textStream = await streamText({
  model: ollama
    .ChatTextGenerator({
      model: "llama2:chat",
      maxGenerationTokens: 500,
    })
    .withTextPrompt(),

  prompt: "Write a short story about a robot learning to love:",
});

Generate Object (Chat)​

Generate Object Docs

Object generation is possible with capable open-source models like OpenHermes 2.5.

import { ollama, zodSchema, generateObject } from "modelfusion";
import { z } from "zod";

const sentiment = await generateObject({
  model: ollama
    .ChatTextGenerator({
      model: "openhermes2.5-mistral",
      maxGenerationTokens: 1024,
      temperature: 0,
    })
    .asObjectGenerationModel(jsonObjectPrompt.instruction()),

  schema: zodSchema(
    z.object({
      sentiment: z
        .enum(["positive", "neutral", "negative"])
        .describe("Sentiment."),
    })
  ),

  prompt: {
    system:
      "You are a sentiment evaluator. " +
      "Analyze the sentiment of the following product review:",
    instruction:
      "After I opened the package, I was met by a very unpleasant smell " +
      "that did not disappear even after washing. Never again!",
  },
});

Stream Object (Chat)​

Stream Object Docs

import { jsonObjectPrompt, ollama, streamObject, zodSchema } from "modelfusion";
import { z } from "zod";

const objectStream = await streamObject({
  model: ollama
    .ChatTextGenerator({
      model: "openhermes2.5-mistral",
      maxGenerationTokens: 1024,
      temperature: 0,
    })
    .asObjectGenerationModel(jsonObjectPrompt.text()),

  schema: zodSchema(
    z.object({
      characters: z.array(
        z.object({
          name: z.string(),
          class: z
            .string()
            .describe("Character class, e.g. warrior, mage, or thief."),
          description: z.string(),
        })
      ),
    })
  ),

  prompt: "Generate 3 character descriptions for a fantasy role playing game. ",
});

for await (const { partialObject } of objectStream) {
  console.clear();
  console.log(partialObject);
}

Embed Text​

Embed Value Docs | OllamaTextEmbeddingModel API

import { embedMany, ollama } from "modelfusion";

const embeddings = await embedMany({
  model: ollama.TextEmbedder({ model: "llama2" }),
  values: [
    "At first, Nox didn't know what to do with the pup.",
    "He keenly observed and absorbed everything around him, from the birds in the sky to the trees in the forest.",
  ],
});

Configuration​

API Configuration​

Ollama API Configuration

const api = ollama.Api({
  baseUrl: {
    port: "12345",
  },
  // ...
});

const model = ollama.CompletionTextGenerator({
  api,
  // ...
});

Prompt Templates​

Many models are trained on specific prompts. You can use prompt templates to use higher-level prompt templates such as instruction and chat prompts and map them to the correct format for your model. The prompt template that the model expected is usually described on the model card on HuggingFace.

Specific prompt templates for Ollama CompletionTextGenerator models are available under ollama.prompt:

Prompt TemplateOllama Prompt TemplateText PromptInstruction PromptChat Prompt
Alpacaollama.prompt.Alpaca✅✅❌
ChatMLollama.prompt.ChatML✅✅✅
Llama 2ollama.prompt.Llama2✅✅✅
Mistral Instructollama.prompt.Mistral✅✅✅
NeuralChatollama.prompt.NeuralChat✅✅✅
Synthiaollama.prompt.Synthia✅✅✅
Vicunaollama.prompt.Vicuna✅✅✅
Generic Text (default)ollama.prompt.Text✅✅✅

Links & Resources​