Generate Object

Generates a typed object that matches a schema. The object can be generated in one pass or streamed as a sequence of partial results.

First, a language model is invoked with a schema and a prompt. When possible and required by the prompt, the model output is restricted, for example, to JSON. Then the model output is parsed and type inference is executed. The result is a typed object that matches the schema.

You can use this for e.g. the following tasks:

• information generation, e.g. generating a list of characters for a role playing game
• information extraction, e.g. extracting structured information from websites
• classification, e.g. sentiment analysis.

Usage

First you need to create an object generation model. Such a model can be derived from a text chat model or a completion model, for example.

ObjectGenerationModel

ObjectGenerationModel API

OpenAI chat model with function calls

You can create an object generation model by using function calls on an OpenAI chat model. You need to pass the function name and optionally a function description to asFunctionCallObjectGenerationModel when deriving the object generation model.

import { openai } from "modelfusion";

const model = openai
  .ChatTextGenerator({
    model: "gpt-4-1106-preview",
    temperature: 0,
    maxGenerationTokens: 50,
  })
  .asFunctionCallObjectGenerationModel({ fnName: "sentiment" })
  .withInstructionPrompt(); // optional, required in example below

OpenAI chat model with JSON output

You can also use the JSON output of OpenAI chat models to generate an object. The jsonObjectPrompt automatically restricts the output to JSON.

import { jsonObjectPrompt, openai } from "modelfusion";

const model = openai
  .ChatTextGenerator({
    model: "gpt-4-1106-preview",
    maxGenerationTokens: 1024,
    temperature: 0,
  })
  .asObjectGenerationModel(jsonObjectPrompt.instruction());

Ollama chat model with JSON output

You can also use the JSON output of Ollama chat models to generate an object. The jsonObjectPrompt automatically restricts the output to JSON.

note

When using Ollama for object generation, it is important to choose a model that is capable of creating the object that you want. I had good results with openhermes2.5-mistral and mixtral, for example, but this depends on your use case.

import { jsonObjectPrompt, ollama } from "modelfusion";

const model = ollama
  .ChatTextGenerator({
    model: "openhermes2.5-mistral",
    maxGenerationTokens: 1024,
    temperature: 0,
  })
  .asObjectGenerationModel(jsonObjectPrompt.instruction());

Llama.cpp JSON grammar

You can generate objects with Llama.cpp models. The jsonObjectPrompt automatically restricts the output to your JSON schema using a GBNF grammar.

note

When using Llama.cpp for object generation, it is important to choose a model that is capable of creating the object that you want. I had good results with openhermes2.5-mistral and mixtral, for example, but this depends on your use case.

const model = llamacpp
  .CompletionTextGenerator({
    // run openhermes-2.5-mistral-7b.Q4_K_M.gguf in llama.cpp
    promptTemplate: llamacpp.prompt.ChatML,
    maxGenerationTokens: 1024,
    temperature: 0,
  })
  // automatically restrict the output to your schema using GBNF:
  .asObjectGenerationModel(jsonObjectPrompt.text());

jsonObjectPrompt

jsonObjectPrompt API

jsonObjectPrompt is a helper for getting language models to generate JSON output. It injects a JSON schema and instructions to generate JSON into your prompt. When possible, it restricts the model output to JSON.

It allows you to create text or instruction prompts. You can pass a custom schema prefix and suffix.

• jsonObjectPrompt.text()
• jsonObjectPrompt.instruction()

generateObject

generateObject API

Example: Sentiment analysis

import { openai, zodSchema, generateObject } from "modelfusion";
import { z } from "zod";

const sentiment = await generateObject({
  model,

  schema: zodSchema(
    z.object({
      sentiment: z
        .enum(["positive", "neutral", "negative"])
        .describe("Sentiment."),
    })
  ),

  prompt: {
    system:
      "You are a sentiment evaluator. " +
      "Analyze the sentiment of the following product review:",
    instruction:
      "After I opened the package, I was met by a very unpleasant smell " +
      "that did not disappear even after washing. Never again!",
  },
});

streamObject

streamObject API

You can stream partial object results with streamObject. This is useful for long-running tasks, e.g. when generating a large number of objects and streaming them to a UI.

streamObject returns an ObjectStream. The ObjectStream is an async iterable over partial results. It returns objects with the following properties:

• partialObject: the partial object result. It is typed, but not validated. You can use your own logic to validate partial objects, e.g. with Zod .deepPartial().
• partialText: the partial text that was used to generate the partial object.
• textDelta: the text that was received since the last partial result.

Example: RPG character generation

const objectStream = await streamObject({
  model,

  schema: zodSchema(
    z.object({
      characters: z.array(
        z.object({
          name: z.string(),
          class: z
            .string()
            .describe("Character class, e.g. warrior, mage, or thief."),
          description: z.string(),
        })
      ),
    })
  ),

  prompt: {
    instruction:
      "Generate 3 character descriptions for a fantasy role playing game.",
  },
});

for await (const { partialObject } of objectStream) {
  console.clear();
  console.log(partialObject);
}

note

With most models, you need to have an object at the top level. If you want to produce arrays, you need to use a property in that object. For example, you can use a characters property to generate an array of characters.

When using Llama.cpp with a JSON array grammar, you can generate a top-level array and do not need to use a property.

Example: Full response with object promise

You can use the fullResponse property to get a full response with an additional promise to the fully typed and validated object.

const { objectStream, objectPromise, metadata } = await streamObject({
  model: openai
    .ChatTextGenerator({
      model: "gpt-3.5-turbo",
      temperature: 0,
      maxGenerationTokens: 2000,
    })
    .asFunctionCallObjectGenerationModel({
      fnName: "generateCharacter",
      fnDescription: "Generate character descriptions.",
    })
    .withTextPrompt(),

  schema: zodSchema(
    z.object({
      characters: z.array(
        z.object({
          name: z.string(),
          class: z
            .string()
            .describe("Character class, e.g. warrior, mage, or thief."),
          description: z.string(),
        })
      ),
    })
  ),

  prompt: "Generate 3 character descriptions for a fantasy role playing game.",

  fullResponse: true,
});

for await (const { partialObject } of objectStream) {
  console.clear();
  console.log(partialObject);
}

const object = await objectPromise;

console.clear();
console.log("FINAL OBJECT");
console.log(object);

Forwarding Object Streams to the Browser

You can use the ObjectStream and ObjectStreamFromResponse to serialize and deserialize object streams. This allows you to forward object streams to the browser.

Example: Next.js 14 route & OpenAI

Source Code

Schema:

import { zodSchema } from "modelfusion";
import { z } from "zod";

export const itinerarySchema = zodSchema(
  z.object({
    days: z.array(
      z.object({
        theme: z.string(),
        activities: z.array(
          z.object({
            name: z.string(),
            description: z.string(),
            duration: z.number(),
          })
        ),
      })
    ),
  })
);

export type Itinerary = typeof itinerarySchema._partialType;

Server:

import {
  ObjectStreamResponse,
  jsonObjectPrompt,
  openai,
  streamObject,
} from "modelfusion";
import { itinerarySchema } from "../../stream-object-openai/itinerarySchema";

export const runtime = "edge";

export async function POST(req: Request) {
  const { destination, lengthOfStay } = await req.json();

  const objectStream = await streamObject({
    model: openai
      .ChatTextGenerator({
        model: "gpt-4-1106-preview",
        maxGenerationTokens: 2500,
      })
      .asObjectGenerationModel(jsonObjectPrompt.instruction()),

    schema: itinerarySchema,

    prompt: {
      system:
        "You help planning travel itineraries. " +
        "Respond to the users' request with a list of the best stops to make in their destination.",

      instruction: `I am planning a trip to ${destination} for ${lengthOfStay} days.`,
    },
  });

  return new ObjectStreamResponse(objectStream);
}

Client (React Hook):

import { ObjectStreamFromResponse } from "modelfusion";
import { useCallback, useState } from "react";
import { Itinerary, itinerarySchema } from "./itinerarySchema";

export function useItinerary() {
  const [isGenerating, setIsGenerating] = useState(false);
  const [itinerary, setItinerary] = useState<Itinerary>();

  const generateItinerary = useCallback(
    async ({
      destination,
      lengthOfStay,
    }: {
      destination: string;
      lengthOfStay: string;
    }) => {
      setItinerary(undefined);
      setIsGenerating(true);

      try {
        const response = await fetch("/api/stream-object-openai", {
          method: "POST",
          body: JSON.stringify({ destination, lengthOfStay }),
        });

        const stream = ObjectStreamFromResponse({
          schema: itinerarySchema,
          response,
        });

        for await (const { partialObject } of stream) {
          setItinerary(partialObject);
        }
      } finally {
        setIsGenerating(false);
      }
    },
    []
  );

  return {
    isGeneratingItinerary: isGenerating,
    generateItinerary,
    itinerary,
  };
}

Available Providers

• OpenAI
• Ollama
• Llama.cpp