Early Beta — The Web SDK is in early beta. APIs may change between releases.
Overview
The Web SDK supports Tool Calling (function calling) and Structured Output (JSON schema-guided generation). These features enable LLMs to interact with external tools and produce structured, type-safe responses — all running on-device via WebAssembly.Package Imports
Tool Calling and Structured Output are exported from@runanywhere/web-llamacpp (the LLM backend), not from @runanywhere/web:
import {
ToolCalling,
ToolCallFormat,
StructuredOutput,
TextGeneration,
toToolValue,
fromToolValue,
getStringArg,
getNumberArg,
} from '@runanywhere/web-llamacpp'
import type {
ToolDefinition,
ToolCall,
ToolResult,
ToolCallingOptions,
ToolCallingResult,
ToolExecutor,
ToolValue,
ToolParameter,
StructuredOutputConfig,
StructuredOutputValidation,
} from '@runanywhere/web-llamacpp'
Common mistake: importing
ToolCalling or StructuredOutput from @runanywhere/web. These
are NOT exported from the core package. Always import from @runanywhere/web-llamacpp.Tool Calling
How It Works
The orchestration loop is: generate → parse tool call → execute → feed result back → repeat until done.Basic Usage
import { ToolCalling, toToolValue, getStringArg } from '@runanywhere/web-llamacpp'
// 1. Register a tool
ToolCalling.registerTool(
{
name: 'get_weather',
description: 'Gets the current weather for a location',
parameters: [{ name: 'location', type: 'string', description: 'City name', required: true }],
},
async (args) => {
const location = getStringArg(args, 'location') ?? 'NYC'
return {
temperature: toToolValue('72F'),
condition: toToolValue('Sunny'),
location: toToolValue(location),
}
}
)
// 2. Generate with tools (auto-execute is on by default)
const result = await ToolCalling.generateWithTools('What is the weather in San Francisco?', {
maxToolCalls: 3,
autoExecute: true,
temperature: 0.3,
})
console.log(result.text) // "The weather in San Francisco is 72F and Sunny."
console.log(result.toolCalls) // [{ toolName: 'get_weather', arguments: { location: ... } }]
console.log(result.toolResults) // [{ toolName: 'get_weather', success: true, result: { ... } }]
console.log(result.isComplete) // true
Recommended Model
For best tool calling results, use the LFM2 1.2B Tool model which is specifically optimized
for function calling:The smaller LFM2 350M also works but may produce less reliable tool call formatting.
{
id: 'lfm2-1.2b-tool-q4_k_m',
name: 'LFM2 1.2B Tool Q4_K_M',
repo: 'LiquidAI/LFM2-1.2B-Tool-GGUF',
files: ['LFM2-1.2B-Tool-Q4_K_M.gguf'],
framework: LLMFramework.LlamaCpp,
modality: ModelCategory.Language,
memoryRequirement: 800_000_000,
}
API Reference
ToolCalling (Singleton)
const ToolCalling: {
/** Register a tool with its definition and executor */
registerTool(definition: ToolDefinition, executor: ToolExecutor): void
/** Remove a registered tool */
unregisterTool(name: string): void
/** Get all registered tool definitions */
getRegisteredTools(): ToolDefinition[]
/** Remove all registered tools */
clearTools(): void
/** Execute a single tool call */
executeTool(toolCall: ToolCall): Promise<ToolResult>
/** Full orchestration: generate -> parse -> execute -> loop */
generateWithTools(prompt: string, options?: ToolCallingOptions): Promise<ToolCallingResult>
/** Continue after manual execution (when autoExecute: false) */
continueWithToolResult(
previousPrompt: string,
toolCall: ToolCall,
toolResult: ToolResult,
options?: ToolCallingOptions
): Promise<ToolCallingResult>
cleanup(): void
}
Types
interface ToolDefinition {
name: string
description: string
parameters: ToolParameter[]
category?: string
}
interface ToolParameter {
name: string
type: 'string' | 'number' | 'boolean' | 'object' | 'array'
description: string
required?: boolean
enumValues?: string[]
}
interface ToolCallingOptions {
tools?: ToolDefinition[] // Override registered tools for this call
maxToolCalls?: number // Max tool calls per generation (default: 5)
autoExecute?: boolean // Auto-execute tool calls (default: true)
temperature?: number
maxTokens?: number
systemPrompt?: string
replaceSystemPrompt?: boolean
keepToolsAvailable?: boolean
format?: ToolCallFormat // 'default' | 'lfm2'
}
interface ToolCallingResult {
text: string // Final response text
toolCalls: ToolCall[] // All tool calls made
toolResults: ToolResult[] // All tool results
isComplete: boolean // True if generation is done
}
interface ToolCall {
toolName: string
arguments: Record<string, ToolValue>
callId?: string
}
interface ToolResult {
toolName: string
success: boolean
result?: Record<string, ToolValue>
error?: string
callId?: string
}
ToolExecutor
The executor function receives parsed arguments asRecord<string, ToolValue> and must return Record<string, ToolValue>:
type ToolExecutor = (args: Record<string, ToolValue>) => Promise<Record<string, ToolValue>>
TypeScript gotcha: If your executor has
try/catch branches that return different keys, TypeScript
may infer a union type where some properties are undefined, which is not assignable to ToolValue.
Fix this by explicitly annotating the return type:// WRONG — TypeScript infers { result: ToolValue; error?: undefined } | { error: ToolValue; result?: undefined }
;async (args) => {
try {
return { result: toToolValue(42) }
} catch {
return { error: toToolValue('failed') }
}
}
// CORRECT — explicit return type annotation
;async (args): Promise<Record<string, ToolValue>> => {
try {
return { result: toToolValue(42) }
} catch {
return { error: toToolValue('failed') }
}
}
ToolValue & Helpers
ToolValue is a tagged union that wraps JavaScript values for type-safe serialization:
type ToolValue =
| { type: 'string'; value: string }
| { type: 'number'; value: number }
| { type: 'boolean'; value: boolean }
| { type: 'array'; value: ToolValue[] }
| { type: 'object'; value: Record<string, ToolValue> }
| { type: 'null' }
ToolValue:
import { toToolValue, fromToolValue, getStringArg, getNumberArg } from '@runanywhere/web-llamacpp'
// Convert plain JS values → ToolValue
toToolValue('hello') // { type: 'string', value: 'hello' }
toToolValue(42) // { type: 'number', value: 42 }
toToolValue(true) // { type: 'boolean', value: true }
toToolValue(null) // { type: 'null' }
toToolValue([1, 'a']) // { type: 'array', value: [...] }
toToolValue({ k: 'v' }) // { type: 'object', value: { k: { type: 'string', value: 'v' } } }
// Convert ToolValue → plain JS values
fromToolValue({ type: 'string', value: 'hello' }) // 'hello'
// Extract typed args from tool call arguments
const city = getStringArg(args, 'location') // string | undefined
const count = getNumberArg(args, 'limit') // number | undefined
ToolCallFormat
enum ToolCallFormat {
Default = 'default', // XML: <tool_call>{"tool":"name","arguments":{...}}</tool_call>
LFM2 = 'lfm2', // Pythonic: <|tool_call_start|>[func(arg="val")]<|tool_call_end|>
}
| Format | Style | Best For |
|---|---|---|
default | <tool_call>{"tool":"name","arguments":{...}}</tool_call> | Most models |
lfm2 | <|tool_call_start|>[func_name(arg="val")]<|tool_call_end|> | Liquid AI LFM2 models |
const result = await ToolCalling.generateWithTools(prompt, {
format: ToolCallFormat.LFM2,
})
Working Examples
Weather Tool (Fake Data)
A simple tool that returns randomized weather data — useful for demos:import { ToolCalling, toToolValue, getStringArg, type ToolValue } from '@runanywhere/web-llamacpp'
ToolCalling.registerTool(
{
name: 'get_weather',
description:
'Gets the current weather for a city. Returns temperature in Fahrenheit and a short condition.',
parameters: [
{
name: 'location',
type: 'string',
description: 'City name (e.g. "San Francisco")',
required: true,
},
],
category: 'Utility',
},
async (args) => {
const city = getStringArg(args, 'location') ?? 'Unknown'
const conditions = ['Sunny', 'Partly Cloudy', 'Overcast', 'Rainy', 'Windy', 'Foggy']
const temp = Math.round(45 + Math.random() * 50)
const condition = conditions[Math.floor(Math.random() * conditions.length)]
return {
location: toToolValue(city),
temperature_f: toToolValue(temp),
condition: toToolValue(condition),
humidity_pct: toToolValue(Math.round(30 + Math.random() * 60)),
}
}
)
Weather Tool (Real API)
A tool that calls the Open-Meteo API for real weather data:import { ToolCalling, toToolValue, getStringArg, type ToolValue } from '@runanywhere/web-llamacpp'
ToolCalling.registerTool(
{
name: 'get_weather',
description: 'Gets current weather for a city using the Open-Meteo API',
parameters: [{ name: 'location', type: 'string', description: 'City name', required: true }],
category: 'Utility',
},
async (args): Promise<Record<string, ToolValue>> => {
const city = getStringArg(args, 'location') ?? 'New York'
const geo = await fetch(
`https://geocoding-api.open-meteo.com/v1/search?name=${encodeURIComponent(city)}&count=1`
).then((r) => r.json())
if (!geo.results?.length) {
return { error: toToolValue(`City "${city}" not found`) }
}
const { latitude, longitude, name } = geo.results[0]
const weather = await fetch(
`https://api.open-meteo.com/v1/forecast?latitude=${latitude}&longitude=${longitude}¤t=temperature_2m,weather_code`
).then((r) => r.json())
return {
location: toToolValue(name),
temperature_celsius: toToolValue(weather.current.temperature_2m),
weather_code: toToolValue(weather.current.weather_code),
}
}
)
Calculator Tool
Evaluates mathematical expressions using JavaScript:import { ToolCalling, toToolValue, getStringArg, type ToolValue } from '@runanywhere/web-llamacpp'
ToolCalling.registerTool(
{
name: 'calculate',
description: 'Evaluates a mathematical expression and returns the numeric result.',
parameters: [
{
name: 'expression',
type: 'string',
description: 'Math expression (e.g. "2 + 3 * 4")',
required: true,
},
],
category: 'Math',
},
async (args): Promise<Record<string, ToolValue>> => {
const expr = getStringArg(args, 'expression') ?? '0'
try {
const sanitized = expr.replace(/[^0-9+\-*/().%\s^]/g, '')
const result = Function(`"use strict"; return (${sanitized})`)()
return { result: toToolValue(Number(result)), expression: toToolValue(expr) }
} catch {
return { error: toToolValue(`Invalid expression: ${expr}`) }
}
}
)
Time / Timezone Tool
Returns the current time for any IANA timezone:import { ToolCalling, toToolValue, getStringArg, type ToolValue } from '@runanywhere/web-llamacpp'
ToolCalling.registerTool(
{
name: 'get_time',
description: 'Returns the current date and time, optionally for a specific timezone.',
parameters: [
{
name: 'timezone',
type: 'string',
description: 'IANA timezone (e.g. "America/New_York"). Defaults to UTC.',
required: false,
},
],
category: 'Utility',
},
async (args): Promise<Record<string, ToolValue>> => {
const tz = getStringArg(args, 'timezone') ?? 'UTC'
try {
const now = new Date()
const formatted = now.toLocaleString('en-US', {
timeZone: tz,
dateStyle: 'full',
timeStyle: 'long',
})
return { datetime: toToolValue(formatted), timezone: toToolValue(tz) }
} catch {
return {
datetime: toToolValue(new Date().toISOString()),
timezone: toToolValue('UTC'),
note: toToolValue('Fell back to UTC — invalid timezone'),
}
}
}
)
Random Number Tool
Generates a random integer in a range — demonstratesgetNumberArg:
import { ToolCalling, toToolValue, getNumberArg } from '@runanywhere/web-llamacpp'
ToolCalling.registerTool(
{
name: 'random_number',
description: 'Generates a random integer between min and max (inclusive).',
parameters: [
{ name: 'min', type: 'number', description: 'Minimum value', required: true },
{ name: 'max', type: 'number', description: 'Maximum value', required: true },
],
category: 'Math',
},
async (args) => {
const min = getNumberArg(args, 'min') ?? 1
const max = getNumberArg(args, 'max') ?? 100
const value = Math.floor(Math.random() * (max - min + 1)) + min
return { value: toToolValue(value), min: toToolValue(min), max: toToolValue(max) }
}
)
Manual Execution Mode
WhenautoExecute: false, the SDK returns tool calls without executing them. You handle execution and feed results back:
import { ToolCalling } from '@runanywhere/web-llamacpp'
const result = await ToolCalling.generateWithTools('What time is it in Tokyo?', {
autoExecute: false,
})
if (!result.isComplete && result.toolCalls.length > 0) {
const toolCall = result.toolCalls[0]
console.log('LLM wants to call:', toolCall.toolName, toolCall.arguments)
// Execute manually (or apply approval logic, rate limiting, etc.)
const toolResult = await ToolCalling.executeTool(toolCall)
// Feed the result back to continue generation
const finalResult = await ToolCalling.continueWithToolResult(
'What time is it in Tokyo?',
toolCall,
toolResult
)
console.log(finalResult.text)
}
Registering Custom Tools at Runtime
You can dynamically register tools — useful for plugin systems or user-configurable tools:import {
ToolCalling,
toToolValue,
type ToolDefinition,
type ToolValue,
} from '@runanywhere/web-llamacpp'
function registerCustomTool(name: string, description: string, paramNames: string[]) {
const def: ToolDefinition = {
name,
description,
parameters: paramNames.map((p) => ({
name: p,
type: 'string' as const,
description: p,
required: true,
})),
category: 'Custom',
}
const executor = async (args: Record<string, ToolValue>): Promise<Record<string, ToolValue>> => {
// Replace this with your real logic
const result: Record<string, ToolValue> = {
status: toToolValue('executed'),
tool: toToolValue(name),
}
for (const [k, v] of Object.entries(args)) {
result[`input_${k}`] = v
}
return result
}
ToolCalling.registerTool(def, executor)
}
// Register and use
registerCustomTool('search_web', 'Searches the web for a query', ['query'])
registerCustomTool('send_email', 'Sends an email', ['to', 'subject', 'body'])
// List registered tools
console.log(ToolCalling.getRegisteredTools().map((t) => t.name))
// Unregister when done
ToolCalling.unregisterTool('search_web')
// Clear all
ToolCalling.clearTools()
Complete React Component
A full React component with tool registration, execution trace, and UI:ToolCallingDemo.tsx
import { ModelCategory } from '@runanywhere/web'
import {
ToolCalling,
ToolCallFormat,
toToolValue,
getStringArg,
getNumberArg,
type ToolCallingResult,
type ToolValue,
} from '@runanywhere/web-llamacpp'
import { useState, useEffect, useCallback } from 'react'
interface TraceStep {
type: 'user' | 'tool_call' | 'tool_result' | 'response'
content: string
}
export function ToolCallingDemo() {
const [input, setInput] = useState('')
const [generating, setGenerating] = useState(false)
const [trace, setTrace] = useState<TraceStep[]>([])
// Register demo tools on mount
useEffect(() => {
ToolCalling.clearTools()
ToolCalling.registerTool(
{
name: 'get_weather',
description: 'Gets the current weather for a city',
parameters: [
{ name: 'location', type: 'string', description: 'City name', required: true },
],
},
async (args) => {
const city = getStringArg(args, 'location') ?? 'Unknown'
return {
location: toToolValue(city),
temperature_f: toToolValue(Math.round(45 + Math.random() * 50)),
condition: toToolValue(['Sunny', 'Cloudy', 'Rainy'][Math.floor(Math.random() * 3)]),
}
}
)
ToolCalling.registerTool(
{
name: 'calculate',
description: 'Evaluates a math expression',
parameters: [
{ name: 'expression', type: 'string', description: 'Math expression', required: true },
],
},
async (args): Promise<Record<string, ToolValue>> => {
const expr = getStringArg(args, 'expression') ?? '0'
try {
const val = Function(`"use strict"; return (${expr.replace(/[^0-9+\-*/().%\s]/g, '')})`)()
return { result: toToolValue(Number(val)) }
} catch {
return { error: toToolValue('Invalid expression') }
}
}
)
return () => {
ToolCalling.clearTools()
}
}, [])
const send = useCallback(async () => {
const text = input.trim()
if (!text || generating) return
setInput('')
setGenerating(true)
setTrace([{ type: 'user', content: text }])
try {
const result: ToolCallingResult = await ToolCalling.generateWithTools(text, {
autoExecute: true,
maxToolCalls: 5,
temperature: 0.3,
maxTokens: 512,
format: ToolCallFormat.Default,
})
const steps: TraceStep[] = [{ type: 'user', content: text }]
for (let i = 0; i < result.toolCalls.length; i++) {
const call = result.toolCalls[i]
const argStr = Object.entries(call.arguments)
.map(([k, v]) => `${k}=${JSON.stringify('value' in v ? v.value : v)}`)
.join(', ')
steps.push({ type: 'tool_call', content: `${call.toolName}(${argStr})` })
if (result.toolResults[i]) {
const res = result.toolResults[i]
steps.push({
type: 'tool_result',
content:
res.success && res.result
? JSON.stringify(
Object.fromEntries(
Object.entries(res.result).map(([k, v]) => [k, 'value' in v ? v.value : v])
),
null,
2
)
: `Error: ${res.error ?? 'Unknown'}`,
})
}
}
if (result.text) {
steps.push({ type: 'response', content: result.text })
}
setTrace(steps)
} catch (err) {
setTrace((prev) => [
...prev,
{ type: 'response', content: `Error: ${(err as Error).message}` },
])
} finally {
setGenerating(false)
}
}, [input, generating])
return (
<div>
<div>
{trace.map((step, i) => (
<div
key={i}
style={{
margin: 8,
padding: 8,
background: step.type === 'tool_call' ? '#1a2744' : '#1E293B',
borderRadius: 8,
}}
>
<strong>{step.type.toUpperCase()}</strong>
<pre style={{ whiteSpace: 'pre-wrap', fontSize: 12 }}>{step.content}</pre>
</div>
))}
</div>
<form
onSubmit={(e) => {
e.preventDefault()
send()
}}
>
<input
value={input}
onChange={(e) => setInput(e.target.value)}
placeholder="Ask something..."
disabled={generating}
/>
<button type="submit" disabled={!input.trim() || generating}>
{generating ? 'Running...' : 'Send'}
</button>
</form>
</div>
)
}
Lifecycle Management
import { ToolCalling } from '@runanywhere/web-llamacpp'
// Register tools early (e.g., in SDK init or component mount)
ToolCalling.registerTool(weatherDef, weatherExecutor)
ToolCalling.registerTool(calcDef, calcExecutor)
// Check registered tools
const tools = ToolCalling.getRegisteredTools()
console.log(
`${tools.length} tools registered:`,
tools.map((t) => t.name)
)
// Remove a specific tool
ToolCalling.unregisterTool('get_weather')
// Clear all tools (e.g., on component unmount)
ToolCalling.clearTools()
// cleanup() is an alias for clearTools()
ToolCalling.cleanup()
Always clean up tools on component unmount in React apps. If multiple components register
tools with the same name, they will overwrite each other. Use
ToolCalling.clearTools() in
useEffect cleanup functions.Structured Output
Generate type-safe JSON responses from LLMs using JSON schema validation.Basic Usage
import { TextGeneration, StructuredOutput } from '@runanywhere/web-llamacpp'
// 1. Define a JSON schema
const schema = JSON.stringify({
type: 'object',
properties: {
name: { type: 'string' },
ingredients: { type: 'array', items: { type: 'string' } },
prepTime: { type: 'number' },
},
required: ['name', 'ingredients', 'prepTime'],
})
// 2. Get a system prompt that instructs the LLM to produce JSON
const systemPrompt = StructuredOutput.getSystemPrompt(schema)
// 3. Generate with the system prompt
const result = await TextGeneration.generate('Suggest a quick pasta recipe', {
systemPrompt,
maxTokens: 300,
temperature: 0.3,
})
// 4. Extract and validate the JSON
const validation = StructuredOutput.validate(result.text, { jsonSchema: schema })
if (validation.isValid) {
const recipe = JSON.parse(validation.extractedJson!)
console.log(recipe.name) // "Quick Garlic Pasta"
console.log(recipe.ingredients) // ["pasta", "garlic", "olive oil", ...]
console.log(recipe.prepTime) // 15
} else {
console.error('Invalid output:', validation.errorMessage)
}
API Reference
const StructuredOutput: {
/** Extract the first complete JSON object/array from raw LLM output */
extractJson(text: string): string | null
/** Augment a prompt with schema instructions */
preparePrompt(originalPrompt: string, config: StructuredOutputConfig): string
/** Get a system prompt for JSON schema compliance */
getSystemPrompt(jsonSchema: string): string
/** Validate LLM output against a JSON schema */
validate(text: string, config: StructuredOutputConfig): StructuredOutputValidation
/** Quick check if text contains complete JSON */
hasCompleteJson(text: string): boolean
}
interface StructuredOutputConfig {
jsonSchema: string
includeSchemaInPrompt?: boolean // default: true
}
interface StructuredOutputValidation {
isValid: boolean
errorMessage?: string
extractedJson?: string
}
Workflow
All JSON extraction and schema validation runs in the C++ WASM layer for performance. The
TypeScript API is a thin wrapper around the
rac_structured_output_* C functions.Related
LLM Generation
Text generation
VLM
Vision Language Models
System Prompts
Control AI behavior
Best Practices
Performance optimization