Tool Calling & Structured Output

Early Beta — The Web SDK is in early beta. APIs may change between releases.

Overview

The Web SDK supports Tool Calling (function calling) and Structured Output (JSON schema-guided generation). These features enable LLMs to interact with external tools and produce structured, type-safe responses — all running on-device via WebAssembly.

Package Imports

Tool Calling and Structured Output are exported from @runanywhere/web-llamacpp (the LLM backend), not from @runanywhere/web:

import {
  ToolCalling,
  ToolCallFormat,
  StructuredOutput,
  TextGeneration,
  toToolValue,
  fromToolValue,
  getStringArg,
  getNumberArg,
} from '@runanywhere/web-llamacpp'

import type {
  ToolDefinition,
  ToolCall,
  ToolResult,
  ToolCallingOptions,
  ToolCallingResult,
  ToolExecutor,
  ToolValue,
  ToolParameter,
  StructuredOutputConfig,
  StructuredOutputValidation,
} from '@runanywhere/web-llamacpp'

Common mistake: importing ToolCalling or StructuredOutput from @runanywhere/web. These are NOT exported from the core package. Always import from @runanywhere/web-llamacpp.

Tool Calling

How It Works

The orchestration loop is: generate → parse tool call → execute → feed result back → repeat until done.

Basic Usage

import { ToolCalling, toToolValue, getStringArg } from '@runanywhere/web-llamacpp'

// 1. Register a tool
ToolCalling.registerTool(
  {
    name: 'get_weather',
    description: 'Gets the current weather for a location',
    parameters: [{ name: 'location', type: 'string', description: 'City name', required: true }],
  },
  async (args) => {
    const location = getStringArg(args, 'location') ?? 'NYC'
    return {
      temperature: toToolValue('72F'),
      condition: toToolValue('Sunny'),
      location: toToolValue(location),
    }
  }
)

// 2. Generate with tools (auto-execute is on by default)
const result = await ToolCalling.generateWithTools('What is the weather in San Francisco?', {
  maxToolCalls: 3,
  autoExecute: true,
  temperature: 0.3,
})

console.log(result.text) // "The weather in San Francisco is 72F and Sunny."
console.log(result.toolCalls) // [{ toolName: 'get_weather', arguments: { location: ... } }]
console.log(result.toolResults) // [{ toolName: 'get_weather', success: true, result: { ... } }]
console.log(result.isComplete) // true

Recommended Model

For best tool calling results, use the LFM2 1.2B Tool model which is specifically optimized for function calling:

{
  id: 'lfm2-1.2b-tool-q4_k_m',
  name: 'LFM2 1.2B Tool Q4_K_M',
  repo: 'LiquidAI/LFM2-1.2B-Tool-GGUF',
  files: ['LFM2-1.2B-Tool-Q4_K_M.gguf'],
  framework: LLMFramework.LlamaCpp,
  modality: ModelCategory.Language,
  memoryRequirement: 800_000_000,
}

The smaller LFM2 350M also works but may produce less reliable tool call formatting.

API Reference

ToolCalling (Singleton)

const ToolCalling: {
  /** Register a tool with its definition and executor */
  registerTool(definition: ToolDefinition, executor: ToolExecutor): void

  /** Remove a registered tool */
  unregisterTool(name: string): void

  /** Get all registered tool definitions */
  getRegisteredTools(): ToolDefinition[]

  /** Remove all registered tools */
  clearTools(): void

  /** Execute a single tool call */
  executeTool(toolCall: ToolCall): Promise<ToolResult>

  /** Full orchestration: generate -> parse -> execute -> loop */
  generateWithTools(prompt: string, options?: ToolCallingOptions): Promise<ToolCallingResult>

  /** Continue after manual execution (when autoExecute: false) */
  continueWithToolResult(
    previousPrompt: string,
    toolCall: ToolCall,
    toolResult: ToolResult,
    options?: ToolCallingOptions
  ): Promise<ToolCallingResult>

  cleanup(): void
}

Types

interface ToolDefinition {
  name: string
  description: string
  parameters: ToolParameter[]
  category?: string
}

interface ToolParameter {
  name: string
  type: 'string' | 'number' | 'boolean' | 'object' | 'array'
  description: string
  required?: boolean
  enumValues?: string[]
}

interface ToolCallingOptions {
  tools?: ToolDefinition[] // Override registered tools for this call
  maxToolCalls?: number // Max tool calls per generation (default: 5)
  autoExecute?: boolean // Auto-execute tool calls (default: true)
  temperature?: number
  maxTokens?: number
  systemPrompt?: string
  replaceSystemPrompt?: boolean
  keepToolsAvailable?: boolean
  format?: ToolCallFormat // 'default' | 'lfm2'
}

interface ToolCallingResult {
  text: string // Final response text
  toolCalls: ToolCall[] // All tool calls made
  toolResults: ToolResult[] // All tool results
  isComplete: boolean // True if generation is done
}

interface ToolCall {
  toolName: string
  arguments: Record<string, ToolValue>
  callId?: string
}

interface ToolResult {
  toolName: string
  success: boolean
  result?: Record<string, ToolValue>
  error?: string
  callId?: string
}

ToolExecutor

The executor function receives parsed arguments as Record<string, ToolValue> and must return Record<string, ToolValue>:

type ToolExecutor = (args: Record<string, ToolValue>) => Promise<Record<string, ToolValue>>

TypeScript gotcha: If your executor has try/catch branches that return different keys, TypeScript may infer a union type where some properties are undefined, which is not assignable to ToolValue. Fix this by explicitly annotating the return type:

// WRONG — TypeScript infers { result: ToolValue; error?: undefined } | { error: ToolValue; result?: undefined }
;async (args) => {
  try {
    return { result: toToolValue(42) }
  } catch {
    return { error: toToolValue('failed') }
  }
}

// CORRECT — explicit return type annotation
;async (args): Promise<Record<string, ToolValue>> => {
  try {
    return { result: toToolValue(42) }
  } catch {
    return { error: toToolValue('failed') }
  }
}

ToolValue & Helpers

ToolValue is a tagged union that wraps JavaScript values for type-safe serialization:

type ToolValue =
  | { type: 'string'; value: string }
  | { type: 'number'; value: number }
  | { type: 'boolean'; value: boolean }
  | { type: 'array'; value: ToolValue[] }
  | { type: 'object'; value: Record<string, ToolValue> }
  | { type: 'null' }

Helper functions for working with ToolValue:

import { toToolValue, fromToolValue, getStringArg, getNumberArg } from '@runanywhere/web-llamacpp'

// Convert plain JS values → ToolValue
toToolValue('hello') // { type: 'string', value: 'hello' }
toToolValue(42) // { type: 'number', value: 42 }
toToolValue(true) // { type: 'boolean', value: true }
toToolValue(null) // { type: 'null' }
toToolValue([1, 'a']) // { type: 'array', value: [...] }
toToolValue({ k: 'v' }) // { type: 'object', value: { k: { type: 'string', value: 'v' } } }

// Convert ToolValue → plain JS values
fromToolValue({ type: 'string', value: 'hello' }) // 'hello'

// Extract typed args from tool call arguments
const city = getStringArg(args, 'location') // string | undefined
const count = getNumberArg(args, 'limit') // number | undefined

ToolCallFormat

enum ToolCallFormat {
  Default = 'default', // XML: <tool_call>{"tool":"name","arguments":{...}}</tool_call>
  LFM2 = 'lfm2', // Pythonic: <|tool_call_start|>[func(arg="val")]<|tool_call_end|>
}

Format	Style	Best For
`default`	`<tool_call>{"tool":"name","arguments":{...}}</tool_call>`	Most models
`lfm2`	`<\|tool_call_start\|>[func_name(arg="val")]<\|tool_call_end\|>`	Liquid AI LFM2 models

The format is auto-detected based on the loaded model, or you can override it:

const result = await ToolCalling.generateWithTools(prompt, {
  format: ToolCallFormat.LFM2,
})

Working Examples

Weather Tool (Fake Data)

A simple tool that returns randomized weather data — useful for demos:

import { ToolCalling, toToolValue, getStringArg, type ToolValue } from '@runanywhere/web-llamacpp'

ToolCalling.registerTool(
  {
    name: 'get_weather',
    description:
      'Gets the current weather for a city. Returns temperature in Fahrenheit and a short condition.',
    parameters: [
      {
        name: 'location',
        type: 'string',
        description: 'City name (e.g. "San Francisco")',
        required: true,
      },
    ],
    category: 'Utility',
  },
  async (args) => {
    const city = getStringArg(args, 'location') ?? 'Unknown'
    const conditions = ['Sunny', 'Partly Cloudy', 'Overcast', 'Rainy', 'Windy', 'Foggy']
    const temp = Math.round(45 + Math.random() * 50)
    const condition = conditions[Math.floor(Math.random() * conditions.length)]
    return {
      location: toToolValue(city),
      temperature_f: toToolValue(temp),
      condition: toToolValue(condition),
      humidity_pct: toToolValue(Math.round(30 + Math.random() * 60)),
    }
  }
)

Weather Tool (Real API)

A tool that calls the Open-Meteo API for real weather data:

import { ToolCalling, toToolValue, getStringArg, type ToolValue } from '@runanywhere/web-llamacpp'

ToolCalling.registerTool(
  {
    name: 'get_weather',
    description: 'Gets current weather for a city using the Open-Meteo API',
    parameters: [{ name: 'location', type: 'string', description: 'City name', required: true }],
    category: 'Utility',
  },
  async (args): Promise<Record<string, ToolValue>> => {
    const city = getStringArg(args, 'location') ?? 'New York'

    const geo = await fetch(
      `https://geocoding-api.open-meteo.com/v1/search?name=${encodeURIComponent(city)}&count=1`
    ).then((r) => r.json())

    if (!geo.results?.length) {
      return { error: toToolValue(`City "${city}" not found`) }
    }

    const { latitude, longitude, name } = geo.results[0]
    const weather = await fetch(
      `https://api.open-meteo.com/v1/forecast?latitude=${latitude}&longitude=${longitude}&current=temperature_2m,weather_code`
    ).then((r) => r.json())

    return {
      location: toToolValue(name),
      temperature_celsius: toToolValue(weather.current.temperature_2m),
      weather_code: toToolValue(weather.current.weather_code),
    }
  }
)

Calculator Tool

Evaluates mathematical expressions using JavaScript:

import { ToolCalling, toToolValue, getStringArg, type ToolValue } from '@runanywhere/web-llamacpp'

ToolCalling.registerTool(
  {
    name: 'calculate',
    description: 'Evaluates a mathematical expression and returns the numeric result.',
    parameters: [
      {
        name: 'expression',
        type: 'string',
        description: 'Math expression (e.g. "2 + 3 * 4")',
        required: true,
      },
    ],
    category: 'Math',
  },
  async (args): Promise<Record<string, ToolValue>> => {
    const expr = getStringArg(args, 'expression') ?? '0'
    try {
      const sanitized = expr.replace(/[^0-9+\-*/().%\s^]/g, '')
      const result = Function(`"use strict"; return (${sanitized})`)()
      return { result: toToolValue(Number(result)), expression: toToolValue(expr) }
    } catch {
      return { error: toToolValue(`Invalid expression: ${expr}`) }
    }
  }
)

Time / Timezone Tool

Returns the current time for any IANA timezone:

import { ToolCalling, toToolValue, getStringArg, type ToolValue } from '@runanywhere/web-llamacpp'

ToolCalling.registerTool(
  {
    name: 'get_time',
    description: 'Returns the current date and time, optionally for a specific timezone.',
    parameters: [
      {
        name: 'timezone',
        type: 'string',
        description: 'IANA timezone (e.g. "America/New_York"). Defaults to UTC.',
        required: false,
      },
    ],
    category: 'Utility',
  },
  async (args): Promise<Record<string, ToolValue>> => {
    const tz = getStringArg(args, 'timezone') ?? 'UTC'
    try {
      const now = new Date()
      const formatted = now.toLocaleString('en-US', {
        timeZone: tz,
        dateStyle: 'full',
        timeStyle: 'long',
      })
      return { datetime: toToolValue(formatted), timezone: toToolValue(tz) }
    } catch {
      return {
        datetime: toToolValue(new Date().toISOString()),
        timezone: toToolValue('UTC'),
        note: toToolValue('Fell back to UTC — invalid timezone'),
      }
    }
  }
)

Random Number Tool

Generates a random integer in a range — demonstrates getNumberArg:

import { ToolCalling, toToolValue, getNumberArg } from '@runanywhere/web-llamacpp'

ToolCalling.registerTool(
  {
    name: 'random_number',
    description: 'Generates a random integer between min and max (inclusive).',
    parameters: [
      { name: 'min', type: 'number', description: 'Minimum value', required: true },
      { name: 'max', type: 'number', description: 'Maximum value', required: true },
    ],
    category: 'Math',
  },
  async (args) => {
    const min = getNumberArg(args, 'min') ?? 1
    const max = getNumberArg(args, 'max') ?? 100
    const value = Math.floor(Math.random() * (max - min + 1)) + min
    return { value: toToolValue(value), min: toToolValue(min), max: toToolValue(max) }
  }
)

Manual Execution Mode

When autoExecute: false, the SDK returns tool calls without executing them. You handle execution and feed results back:

import { ToolCalling } from '@runanywhere/web-llamacpp'

const result = await ToolCalling.generateWithTools('What time is it in Tokyo?', {
  autoExecute: false,
})

if (!result.isComplete && result.toolCalls.length > 0) {
  const toolCall = result.toolCalls[0]
  console.log('LLM wants to call:', toolCall.toolName, toolCall.arguments)

  // Execute manually (or apply approval logic, rate limiting, etc.)
  const toolResult = await ToolCalling.executeTool(toolCall)

  // Feed the result back to continue generation
  const finalResult = await ToolCalling.continueWithToolResult(
    'What time is it in Tokyo?',
    toolCall,
    toolResult
  )

  console.log(finalResult.text)
}

Registering Custom Tools at Runtime

You can dynamically register tools — useful for plugin systems or user-configurable tools:

import {
  ToolCalling,
  toToolValue,
  type ToolDefinition,
  type ToolValue,
} from '@runanywhere/web-llamacpp'

function registerCustomTool(name: string, description: string, paramNames: string[]) {
  const def: ToolDefinition = {
    name,
    description,
    parameters: paramNames.map((p) => ({
      name: p,
      type: 'string' as const,
      description: p,
      required: true,
    })),
    category: 'Custom',
  }

  const executor = async (args: Record<string, ToolValue>): Promise<Record<string, ToolValue>> => {
    // Replace this with your real logic
    const result: Record<string, ToolValue> = {
      status: toToolValue('executed'),
      tool: toToolValue(name),
    }
    for (const [k, v] of Object.entries(args)) {
      result[`input_${k}`] = v
    }
    return result
  }

  ToolCalling.registerTool(def, executor)
}

// Register and use
registerCustomTool('search_web', 'Searches the web for a query', ['query'])
registerCustomTool('send_email', 'Sends an email', ['to', 'subject', 'body'])

// List registered tools
console.log(ToolCalling.getRegisteredTools().map((t) => t.name))

// Unregister when done
ToolCalling.unregisterTool('search_web')

// Clear all
ToolCalling.clearTools()

Complete React Component

A full React component with tool registration, execution trace, and UI:

ToolCallingDemo.tsx

import { ModelCategory } from '@runanywhere/web'
import {
  ToolCalling,
  ToolCallFormat,
  toToolValue,
  getStringArg,
  getNumberArg,
  type ToolCallingResult,
  type ToolValue,
} from '@runanywhere/web-llamacpp'
import { useState, useEffect, useCallback } from 'react'

interface TraceStep {
  type: 'user' | 'tool_call' | 'tool_result' | 'response'
  content: string
}

export function ToolCallingDemo() {
  const [input, setInput] = useState('')
  const [generating, setGenerating] = useState(false)
  const [trace, setTrace] = useState<TraceStep[]>([])

  // Register demo tools on mount
  useEffect(() => {
    ToolCalling.clearTools()

    ToolCalling.registerTool(
      {
        name: 'get_weather',
        description: 'Gets the current weather for a city',
        parameters: [
          { name: 'location', type: 'string', description: 'City name', required: true },
        ],
      },
      async (args) => {
        const city = getStringArg(args, 'location') ?? 'Unknown'
        return {
          location: toToolValue(city),
          temperature_f: toToolValue(Math.round(45 + Math.random() * 50)),
          condition: toToolValue(['Sunny', 'Cloudy', 'Rainy'][Math.floor(Math.random() * 3)]),
        }
      }
    )

    ToolCalling.registerTool(
      {
        name: 'calculate',
        description: 'Evaluates a math expression',
        parameters: [
          { name: 'expression', type: 'string', description: 'Math expression', required: true },
        ],
      },
      async (args): Promise<Record<string, ToolValue>> => {
        const expr = getStringArg(args, 'expression') ?? '0'
        try {
          const val = Function(`"use strict"; return (${expr.replace(/[^0-9+\-*/().%\s]/g, '')})`)()
          return { result: toToolValue(Number(val)) }
        } catch {
          return { error: toToolValue('Invalid expression') }
        }
      }
    )

    return () => {
      ToolCalling.clearTools()
    }
  }, [])

  const send = useCallback(async () => {
    const text = input.trim()
    if (!text || generating) return

    setInput('')
    setGenerating(true)
    setTrace([{ type: 'user', content: text }])

    try {
      const result: ToolCallingResult = await ToolCalling.generateWithTools(text, {
        autoExecute: true,
        maxToolCalls: 5,
        temperature: 0.3,
        maxTokens: 512,
        format: ToolCallFormat.Default,
      })

      const steps: TraceStep[] = [{ type: 'user', content: text }]

      for (let i = 0; i < result.toolCalls.length; i++) {
        const call = result.toolCalls[i]
        const argStr = Object.entries(call.arguments)
          .map(([k, v]) => `${k}=${JSON.stringify('value' in v ? v.value : v)}`)
          .join(', ')
        steps.push({ type: 'tool_call', content: `${call.toolName}(${argStr})` })

        if (result.toolResults[i]) {
          const res = result.toolResults[i]
          steps.push({
            type: 'tool_result',
            content:
              res.success && res.result
                ? JSON.stringify(
                    Object.fromEntries(
                      Object.entries(res.result).map(([k, v]) => [k, 'value' in v ? v.value : v])
                    ),
                    null,
                    2
                  )
                : `Error: ${res.error ?? 'Unknown'}`,
          })
        }
      }

      if (result.text) {
        steps.push({ type: 'response', content: result.text })
      }
      setTrace(steps)
    } catch (err) {
      setTrace((prev) => [
        ...prev,
        { type: 'response', content: `Error: ${(err as Error).message}` },
      ])
    } finally {
      setGenerating(false)
    }
  }, [input, generating])

  return (
    <div>
      <div>
        {trace.map((step, i) => (
          <div
            key={i}
            style={{
              margin: 8,
              padding: 8,
              background: step.type === 'tool_call' ? '#1a2744' : '#1E293B',
              borderRadius: 8,
            }}
          >
            <strong>{step.type.toUpperCase()}</strong>
            <pre style={{ whiteSpace: 'pre-wrap', fontSize: 12 }}>{step.content}</pre>
          </div>
        ))}
      </div>
      <form
        onSubmit={(e) => {
          e.preventDefault()
          send()
        }}
      >
        <input
          value={input}
          onChange={(e) => setInput(e.target.value)}
          placeholder="Ask something..."
          disabled={generating}
        />
        <button type="submit" disabled={!input.trim() || generating}>
          {generating ? 'Running...' : 'Send'}
        </button>
      </form>
    </div>
  )
}

Lifecycle Management

import { ToolCalling } from '@runanywhere/web-llamacpp'

// Register tools early (e.g., in SDK init or component mount)
ToolCalling.registerTool(weatherDef, weatherExecutor)
ToolCalling.registerTool(calcDef, calcExecutor)

// Check registered tools
const tools = ToolCalling.getRegisteredTools()
console.log(
  `${tools.length} tools registered:`,
  tools.map((t) => t.name)
)

// Remove a specific tool
ToolCalling.unregisterTool('get_weather')

// Clear all tools (e.g., on component unmount)
ToolCalling.clearTools()

// cleanup() is an alias for clearTools()
ToolCalling.cleanup()

Always clean up tools on component unmount in React apps. If multiple components register tools with the same name, they will overwrite each other. Use ToolCalling.clearTools() in useEffect cleanup functions.

Structured Output

Generate type-safe JSON responses from LLMs using JSON schema validation.

Basic Usage

import { TextGeneration, StructuredOutput } from '@runanywhere/web-llamacpp'

// 1. Define a JSON schema
const schema = JSON.stringify({
  type: 'object',
  properties: {
    name: { type: 'string' },
    ingredients: { type: 'array', items: { type: 'string' } },
    prepTime: { type: 'number' },
  },
  required: ['name', 'ingredients', 'prepTime'],
})

// 2. Get a system prompt that instructs the LLM to produce JSON
const systemPrompt = StructuredOutput.getSystemPrompt(schema)

// 3. Generate with the system prompt
const result = await TextGeneration.generate('Suggest a quick pasta recipe', {
  systemPrompt,
  maxTokens: 300,
  temperature: 0.3,
})

// 4. Extract and validate the JSON
const validation = StructuredOutput.validate(result.text, { jsonSchema: schema })

if (validation.isValid) {
  const recipe = JSON.parse(validation.extractedJson!)
  console.log(recipe.name) // "Quick Garlic Pasta"
  console.log(recipe.ingredients) // ["pasta", "garlic", "olive oil", ...]
  console.log(recipe.prepTime) // 15
} else {
  console.error('Invalid output:', validation.errorMessage)
}

API Reference

const StructuredOutput: {
  /** Extract the first complete JSON object/array from raw LLM output */
  extractJson(text: string): string | null

  /** Augment a prompt with schema instructions */
  preparePrompt(originalPrompt: string, config: StructuredOutputConfig): string

  /** Get a system prompt for JSON schema compliance */
  getSystemPrompt(jsonSchema: string): string

  /** Validate LLM output against a JSON schema */
  validate(text: string, config: StructuredOutputConfig): StructuredOutputValidation

  /** Quick check if text contains complete JSON */
  hasCompleteJson(text: string): boolean
}

interface StructuredOutputConfig {
  jsonSchema: string
  includeSchemaInPrompt?: boolean // default: true
}

interface StructuredOutputValidation {
  isValid: boolean
  errorMessage?: string
  extractedJson?: string
}

Workflow

Define schema

Create a JSON Schema that describes the desired output format.

Prepare prompt

Use getSystemPrompt() or preparePrompt() to build schema-aware prompts.

Generate

Run TextGeneration.generate() with the prepared prompt.

Validate

Use validate() to extract and verify the JSON output.

All JSON extraction and schema validation runs in the C++ WASM layer for performance. The TypeScript API is a thin wrapper around the rac_structured_output_* C functions.

LLM Generation

Text generation

VLM

Vision Language Models

System Prompts

Control AI behavior

Best Practices

Performance optimization

Getting Started

Swift SDK

Kotlin SDK

React Native SDK

Flutter SDK

Web SDK

Vibe Coding

Overview

Package Imports

Tool Calling

How It Works

Basic Usage

Recommended Model

API Reference

ToolCalling (Singleton)

Types

ToolExecutor

ToolValue & Helpers

ToolCallFormat

Working Examples

Weather Tool (Fake Data)

Weather Tool (Real API)

Calculator Tool

Time / Timezone Tool

Random Number Tool

Manual Execution Mode

Registering Custom Tools at Runtime

Complete React Component

Lifecycle Management

Structured Output

Basic Usage

API Reference

Workflow

LLM Generation

VLM

System Prompts

Best Practices

​Overview

​Package Imports

​Tool Calling

​How It Works

​Basic Usage

​Recommended Model

​API Reference

​ToolCalling (Singleton)

​Types

​ToolExecutor

​ToolValue & Helpers

​ToolCallFormat

​Working Examples

​Weather Tool (Fake Data)

​Weather Tool (Real API)

​Calculator Tool

​Time / Timezone Tool

​Random Number Tool

​Manual Execution Mode

​Registering Custom Tools at Runtime

​Complete React Component

​Lifecycle Management

​Structured Output

​Basic Usage

​API Reference

​Workflow

​Related

LLM Generation

VLM

System Prompts

Best Practices

Overview

Package Imports

Tool Calling

How It Works

Basic Usage

Recommended Model

API Reference

ToolCalling (Singleton)

Types

ToolExecutor

ToolValue & Helpers

ToolCallFormat

Working Examples

Weather Tool (Fake Data)

Weather Tool (Real API)

Calculator Tool

Time / Timezone Tool

Random Number Tool

Manual Execution Mode

Registering Custom Tools at Runtime

Complete React Component

Lifecycle Management

Structured Output

Basic Usage

API Reference

Workflow

Related