Vision Language Models

Early Beta — The Web SDK is in early beta. APIs may change between releases.

Overview

Vision language models take an image plus a text prompt and produce text: a description, an answer, or an analysis. The Web SDK runs them through llama.cpp’s mtmd multimodal path in the @runanywhere/web-llamacpp WASM artifact. Two entry points, both on the root facade:

RunAnywhere.processImage(
  image: VLMImage,
  options: VLMGenerationOptions,
  extra?: { signal?: AbortSignal }
): Promise<VLMResult>

RunAnywhere.processImageStream(
  image: VLMImage,
  options: VLMGenerationOptions
): Promise<AsyncIterable<VLMStreamEvent>>

RunAnywhere.processImageStream(
  image: VLMImage,
  prompt: string,
  options?: VLMGenerationOptions
): Promise<AsyncIterable<VLMStreamEvent>>

RunAnywhere.cancelVLMGeneration(): Promise<void>

Note that processImageStream returns a promise wrapping the iterable, so it needs two awaits: one for the handle, then for await over it.

Imports

Everything comes from the core package plus the browser helper entrypoint. The backend package root exposes only its registration facade.

import {
  RunAnywhere,
  ModelCategory,
  VLMModelFamily,
  VLMStreamEventKind,
  vlmImageFromRawRGB,
  type VLMGenerationOptions,
} from '@runanywhere/web'
import { VideoCapture } from '@runanywhere/web/browser'
import { LlamaCPP } from '@runanywhere/web-llamacpp'

Loading a model

Loading a multimodal model syncs the SDK’s vision-language provider automatically. There is no separate VLM initialization step.

await LlamaCPP.register()

await RunAnywhere.downloadModel('lfm2-vl-450m-q8_0')
await RunAnywhere.loadModel({
  modelId: 'lfm2-vl-450m-q8_0',
  category: ModelCategory.MODEL_CATEGORY_MULTIMODAL,
  forceReload: false,
  validateAvailability: true,
})

Building a VLMImage

VLMImage is a proto message with a format discriminator. Use the factory helpers rather than constructing it by hand.

Helper	Format	Notes
`vlmImageFromRawRGB(bytes, width, height)`	`RAW_RGB`	What `VideoCapture.captureFrame()` produces.
`vlmImageFromRawRGBA(bytes, width, height)`	`RAW_RGBA`
`vlmImageFromEncoded(bytes, format)`	`JPEG` / `PNG` / `WEBP`	Container bytes; dimensions read by the decoder.
`vlmImageFromBase64(text)`	`BASE64`
`vlmImageFromFilePath(path)`	`FILE_PATH`	A WASM MEMFS or OPFS path.

Width and height are required for the raw formats because there is no header to read them from.

Generation options

VLMGenerationOptions carries the prompt, unlike the LLM options object which keeps it separate.

Option	Type	Default
`prompt`	`string`	`''`
`maxTokens`	`number`	`2048`
`temperature`	`number`	`0.7`
`topP`	`number`	`0.9`
`topK`	`number`	`0`
`repetitionPenalty`	`number`	`1.1`
`minP`	`number`	`0.0`
`seed`	`number`	`0`
`stopSequences`	`string[]`	`[]`
`streamingEnabled`	`boolean`	`true`
`systemPrompt`	`string`	unset
`maxImageSize`	`number`	`0` (no cap)
`nThreads`	`number`	`0` (backend default)
`useGpu`	`boolean`	`true`
`modelFamily`	`VLMModelFamily`	`VLM_MODEL_FAMILY_UNSPECIFIED`
`customChatTemplate`	`VLMChatTemplate`	unset
`imageMarkerOverride`	`string`	unset
`emitImageEmbeddings`	`boolean`	`false`

VLMModelFamily values are UNSPECIFIED, AUTO, QWEN2_VL, SMOLVLM, LLAVA, and CUSTOM. Leave it unspecified unless a model needs a template the SDK cannot infer. 2048 default tokens is generous for a caption. Drop maxTokens to 30-60 for one-line descriptions; image encoding already dominates the latency budget in WASM.

Streaming a camera frame

import { RunAnywhere, VLMStreamEventKind, vlmImageFromRawRGB } from '@runanywhere/web'
import { VideoCapture } from '@runanywhere/web/browser'

const camera = new VideoCapture({ facingMode: 'environment' })
await camera.start()
document.getElementById('preview')!.appendChild(camera.videoElement)

const frame = camera.captureFrame(384)
if (!frame) throw new Error('Camera not ready')

const image = vlmImageFromRawRGB(frame.rgbPixels, frame.width, frame.height)

const stream = await RunAnywhere.processImageStream(image, {
  prompt: 'Describe what you see in this image.',
  maxTokens: 200,
  temperature: 0.7,
  topP: 0.9,
  topK: 40,
  stopSequences: [],
  streamingEnabled: true,
  maxImageSize: 384,
  nThreads: 0,
  useGpu: false,
  modelFamily: VLMModelFamily.VLM_MODEL_FAMILY_UNSPECIFIED,
  seed: 0,
  repetitionPenalty: 1.1,
  minP: 0.05,
  emitImageEmbeddings: false,
})

let text = ''
for await (const event of stream) {
  switch (event.kind) {
    case VLMStreamEventKind.VLM_STREAM_EVENT_KIND_TOKEN:
      text += event.token
      render(text)
      break
    case VLMStreamEventKind.VLM_STREAM_EVENT_KIND_COMPLETED: {
      const r = event.result
      if (r) console.log(`${r.completionTokens} tokens in ${Math.round(r.processingTimeMs)}ms`)
      break
    }
    case VLMStreamEventKind.VLM_STREAM_EVENT_KIND_ERROR:
      throw new Error(event.errorMessage || 'VLM stream failed')
  }
}

camera.stop()

Stream event kinds are UNSPECIFIED, STARTED, IMAGE_ENCODED, TOKEN, COMPLETED, and ERROR. Each event carries seq, timestampUs, requestId, token, tokenIndex, isFinal, tokensPerSecond, and, on terminal events, result. To cancel mid-generation, call RunAnywhere.cancelVLMGeneration() or pass an AbortSignal to the non-streaming processImage().

Result

interface VLMResult {
  text: string
  promptTokens: number
  imageTokens: number
  completionTokens: number
  totalTokens: number
  processingTimeMs: number
  timeToFirstTokenMs: number
  imageEncodeTimeMs: number
  tokensPerSecond: number
  imagesProcessed: number
  hardwareUsed?: string
  finishReason: string
  errorMessage?: string
  errorCode: number
}

VideoCapture

VideoCapture from @runanywhere/web/browser owns the getUserMedia stream, an internal <video> element, and an offscreen canvas for pixel extraction.

const camera = new VideoCapture({
  facingMode: 'environment', // or 'user'
  idealWidth: 640,
  idealHeight: 480,
})

await camera.start()
document.body.appendChild(camera.videoElement)

const frame = camera.captureFrame(384) // downscale so the longest side is 384px
// frame: { rgbPixels: Uint8Array, width: number, height: number } | null

console.log(camera.isCapturing, camera.videoWidth, camera.videoHeight)
camera.stop()

captureFrame(maxDimension = 512) returns null when the stream has no valid dimensions yet, so a null check is enough. If you need to know exactly when the camera is ready, videoWidth and videoHeight report the live stream dimensions.

Registering VLM models

A VLM needs two files: the primary GGUF and a multimodal projector. Register them with registerModelMultiFile and a ModelFileRole per entry.

import { RunAnywhere } from '@runanywhere/web'
import {
  InferenceFramework,
  ModelCategory,
  ModelFileRole,
  ModelFormat,
} from '@runanywhere/proto-ts/model_types'

RunAnywhere.registerModelMultiFile({
  id: 'lfm2-vl-450m-q8_0',
  name: 'LFM2-VL 450M',
  framework: InferenceFramework.INFERENCE_FRAMEWORK_LLAMA_CPP,
  description: 'LiquidAI LFM2-VL vision-language model.',
  format: ModelFormat.MODEL_FORMAT_GGUF,
  modality: ModelCategory.MODEL_CATEGORY_MULTIMODAL,
  contextLength: 2048,
  memoryRequirement: 650_000_000,
  files: [
    {
      url: 'https://huggingface.co/runanywhere/LFM2-VL-450M-GGUF/resolve/main/LFM2-VL-450M-Q8_0.gguf',
      filename: 'LFM2-VL-450M-Q8_0.gguf',
      role: ModelFileRole.MODEL_FILE_ROLE_PRIMARY_MODEL,
      sizeBytes: 379_215_264,
    },
    {
      url: 'https://huggingface.co/runanywhere/LFM2-VL-450M-GGUF/resolve/main/mmproj-LFM2-VL-450M-Q8_0.gguf',
      filename: 'mmproj-LFM2-VL-450M-Q8_0.gguf',
      role: ModelFileRole.MODEL_FILE_ROLE_VISION_PROJECTOR,
      sizeBytes: 103_890_016,
    },
  ],
})

The SDK assembles the MultiFileArtifact and ExpectedModelFiles protos from that list and drives the download orchestrator; the app never builds those messages itself.

Models in the example app

Model id	Architecture	Approx. memory
`smolvlm2-256m-video-instruct-q8_0`	SmolVLM2	420 MB
`lfm2-vl-450m-q8_0`	LiquidAI LFM2-VL	650 MB
`smolvlm-500m-instruct-q8_0`	SmolVLM	720 MB

Exact URLs and per-file sizes live in examples/web/RunAnywhereAI/src/services/model-catalog.ts. Larger VLMs are omitted deliberately. WASM32 caps the heap at 4 GB and a download must fit in memory before it lands in OPFS, so a multi-GB multimodal GGUF is not practical in a tab.

Performance notes

Capture at 256 to 384 pixels. Image encoding cost grows quickly with resolution and it usually dominates total latency. Keep maxTokens low for live captioning. 30 to 60 tokens is enough for a sentence. SmolVLM2 256M is the cheapest starting point at roughly 420 MB of working memory.

LLM generation

Text-only generation

Tool calling

Function calling with LLMs

Best practices

Performance optimization

Getting Started

Swift SDK

Kotlin SDK

React Native SDK

Flutter SDK

Web SDK

Vibe Coding

Overview

Imports

Loading a model

Building a VLMImage

Generation options

Streaming a camera frame

Result

VideoCapture

Registering VLM models

Models in the example app

Performance notes

LLM generation

Tool calling

Best practices

​Overview

​Imports

​Loading a model

​Building a VLMImage

​Generation options

​Streaming a camera frame

​Result

​VideoCapture

​Registering VLM models

​Models in the example app

​Performance notes

​Related

LLM generation

Tool calling

Best practices

Overview

Imports

Loading a model

Building a VLMImage

Generation options

Streaming a camera frame

Result

VideoCapture

Registering VLM models

Models in the example app

Performance notes

Related