generateStream() - RunAnywhere Documentation

for await (const event of RunAnywhere.llm.generateStream('Write a haiku')) {
  if (event.type === 'textDelta') setText((t) => t + event.text)
}

Signature

generateStream(input: string | ChatMessage[], options?: LlmOptions): AsyncIterable<GenerationEvent>

Not a promise. You get the iterable back synchronously and consume it with for await.

The events

started, then token deltas, then completed.

for await (const event of RunAnywhere.llm.generateStream(prompt)) {
  switch (event.type) {
    case 'textDelta':
      setText((t) => t + event.text)
      break
    case 'completed':
      setSpeed(event.result.tokensPerSecond)
      break
  }
}

Where errors surface

try {
  for await (const event of RunAnywhere.llm.generateStream(prompt)) {
    if (event.type === 'textDelta') setText((t) => t + event.text)
  }
} catch (error) {
  showError(error)
}

Updating React state

Setting state on every token re-renders on every token. For long outputs, accumulate and flush on an interval:

const buffer = useRef('')

useEffect(() => {
  const id = setInterval(() => {
    if (buffer.current) {
      setText((t) => t + buffer.current)
      buffer.current = ''
    }
  }, 50)

  return () => clearInterval(id)
}, [])

// in the loop
if (event.type === 'textDelta') buffer.current += event.text

Cancelling

Break out of the loop. Track unmount so you do not set state on a gone component:

const cancelled = useRef(false)

useEffect(
  () => () => {
    cancelled.current = true
  },
  []
)

for await (const event of RunAnywhere.llm.generateStream(prompt)) {
  if (cancelled.current) break
  if (event.type === 'textDelta') setText((t) => t + event.text)
}