Skip to main content

Retry Strategies

AI models are often accessed via APIs, and these APIs can fail for a variety of reasons. For example, the API might be temporarily unavailable, or the API provider might have rate limits in place that are exceeded by your application. ModelFusion provides retry strategies that you can configure to handle these situations.

Usage​

By default, API calls are not retried. You can configure different retry strategies on API configurations.

retryWithExponentialBackoff​

API

The retryWithExponentialBackoff strategy retries a failed API call with an exponential backoff. You can configure the maximum number of tries, the initial delay, and the backoff factor.

Example: retryWithExponentialBackoff in model constructor​

import { openai, api } from "modelfusion";

const model = openai.CompletionTextGenerator({
  api: openai.Api({
    // configure retries as part of the API configuration
    retry: api.retryWithExponentialBackoff({
      maxTries: 8,
      initialDelayInMs: 1000,
      backoffFactor: 2,
    }),
  }),
  model: "gpt-3.5-turbo-instruct",
});

retryNever​

API

The retryNever strategy never retries a failed API call.

Example: retryWithExponentialBackoff in function call​

import { openai, api } from "modelfusion";

const model = openai.CompletionTextGenerator({
  api: openai.Api({
    retry: api.retryNever(),
  }),
  model: "gpt-3.5-turbo-instruct",
});