generateStream()

Early Beta — The Web SDK is in early beta. APIs may change between releases.

Overview

RunAnywhere.generateStream() yields answer tokens as the model produces them. It takes the same options object as generate() and resolves to a handle carrying two iterables, a result promise, and a cancel function.

Basic usage

import { RunAnywhere } from '@runanywhere/web'

const { stream, result } = await RunAnywhere.generateStream({
  prompt: 'Write a short story about a robot',
  maxTokens: 200,
})

let fullResponse = ''
for await (const token of stream) {
  fullResponse += token
  document.getElementById('output')!.textContent = fullResponse
}

const final = await result
console.log(final.tokensPerSecond.toFixed(1), 'tok/s')

Signature

RunAnywhere.generateStream(
  options: TextGenerationOptions,
  extra?: { signal?: AbortSignal }
): Promise<LLMStreamingResult>

interface LLMStreamingResult {
  /** Answer tokens only. Thinking tokens are routed to `events`. */
  stream: AsyncIterable<string>
  /** Canonical typed events, including thinking, progress, and terminal state. */
  events?: AsyncIterable<LLMStreamEvent>
  result: Promise<LLMGenerationResult>
  cancel: () => void
}

Parameters and defaults are identical to generate(). See Generation options.

Typed events

stream gives you answer text only. When you need reasoning tokens, tool calls, or progress counters, iterate events instead and switch on eventKind.

import { RunAnywhere } from '@runanywhere/web'
import { LLMStreamEventKind } from '@runanywhere/proto-ts/llm_service'

const streaming = await RunAnywhere.generateStream({ prompt, maxTokens: 300 })

for await (const event of streaming.events ?? []) {
  switch (event.eventKind) {
    case LLMStreamEventKind.LLM_STREAM_EVENT_KIND_TOKEN:
      appendAnswer(event.token)
      break
    case LLMStreamEventKind.LLM_STREAM_EVENT_KIND_THINKING:
      appendReasoning(event.token)
      break
    case LLMStreamEventKind.LLM_STREAM_EVENT_KIND_COMPLETED:
      console.log('finish reason:', event.finishReason)
      break
    case LLMStreamEventKind.LLM_STREAM_EVENT_KIND_ERROR:
      throw new Error(event.errorMessage)
  }
}

LLMStreamEventKind covers UNSPECIFIED, STARTED, TOKEN, THINKING, TOOL_CALL, PROGRESS, COMPLETED, and ERROR. Every event also carries seq, timestampUs, isFinal, promptTokensProcessed, completionTokensGenerated, and elapsedMs. LLMStreamEventKind lives in @runanywhere/proto-ts, a published dependency of @runanywhere/web, because the root facade re-exports only the enums it uses in its own signatures.

Cancellable streaming

const { stream, result, cancel } = await RunAnywhere.generateStream({
  prompt: 'Write a very long story...',
  maxTokens: 1000,
})

const timeout = setTimeout(() => cancel(), 3000)

let text = ''
for await (const token of stream) {
  text += token
}
clearTimeout(timeout)

An AbortSignal works too, and is wired to cancel() internally:

const controller = new AbortController()
const streaming = await RunAnywhere.generateStream({ prompt }, { signal: controller.signal })
// later
controller.abort()

React hook

useStreamingGenerate.ts

import { useState, useCallback, useRef } from 'react'
import { RunAnywhere } from '@runanywhere/web'
import type { LLMGenerationResult, TextGenerationOptions } from '@runanywhere/web'

export function useStreamingGenerate() {
  const [text, setText] = useState('')
  const [isStreaming, setIsStreaming] = useState(false)
  const [error, setError] = useState<Error | null>(null)
  const [metrics, setMetrics] = useState<LLMGenerationResult | null>(null)
  const cancelRef = useRef<(() => void) | null>(null)

  const generate = useCallback(async (options: TextGenerationOptions) => {
    setText('')
    setIsStreaming(true)
    setError(null)
    setMetrics(null)

    try {
      const { stream, result, cancel } = await RunAnywhere.generateStream(options)
      cancelRef.current = cancel

      let accumulated = ''
      for await (const token of stream) {
        accumulated += token
        setText(accumulated)
      }

      const final = await result
      setMetrics(final)
      return final
    } catch (err) {
      const e = err instanceof Error ? err : new Error('Streaming failed')
      setError(e)
      throw e
    } finally {
      setIsStreaming(false)
      cancelRef.current = null
    }
  }, [])

  return { text, isStreaming, error, metrics, generate, cancel: () => cancelRef.current?.() }
}

Batching UI updates

For fast models, coalesce tokens into one paint per frame.

function createThrottledUpdater(element: HTMLElement) {
  let pending = ''
  let frameId: number | null = null

  return {
    append(token: string) {
      pending += token
      if (!frameId) {
        frameId = requestAnimationFrame(() => {
          element.textContent = pending
          frameId = null
        })
      }
    },
    reset() {
      pending = ''
      element.textContent = ''
      if (frameId) {
        cancelAnimationFrame(frameId)
        frameId = null
      }
    },
  }
}

const updater = createThrottledUpdater(document.getElementById('output')!)
const { stream } = await RunAnywhere.generateStream({ prompt: 'Tell me a story', maxTokens: 200 })

for await (const token of stream) {
  updater.append(token)
}

Performance notes

Streaming adds negligible overhead. Time to first token is the same as non-streaming, and total generation time is close to identical. Cancel streams when the user navigates away so the WASM heap is released promptly.

Simple generation

Prompt in, string out

Generate

Full generation with metrics

System prompts

Control model behavior

Configuration

SDK configuration

Getting Started

Swift SDK

Kotlin SDK

React Native SDK

Flutter SDK

Web SDK

Vibe Coding

Overview

Basic usage

Signature

Typed events

Cancellable streaming

React hook

Batching UI updates

Performance notes

Simple generation

Generate

System prompts

Configuration

​Overview

​Basic usage

​Signature

​Typed events

​Cancellable streaming

​React hook

​Batching UI updates

​Performance notes

​Related

Simple generation

Generate

System prompts

Configuration

Overview

Basic usage

Signature

Typed events

Cancellable streaming

React hook

Batching UI updates

Performance notes

Related