Early Beta — The Web SDK is in early beta. APIs may change between releases.
Overview
The Web SDK supports Tool Calling (function calling) and Structured Output (JSON schema-guided generation). These features enable LLMs to interact with external tools and produce structured, type-safe responses — all running on-device via WebAssembly.Package Imports
Tool Calling and Structured Output are exported from@runanywhere/web-llamacpp (the LLM backend), not from @runanywhere/web:
Copy
Ask AI
import {
ToolCalling,
ToolCallFormat,
StructuredOutput,
TextGeneration,
toToolValue,
fromToolValue,
getStringArg,
getNumberArg,
} from '@runanywhere/web-llamacpp'
import type {
ToolDefinition,
ToolCall,
ToolResult,
ToolCallingOptions,
ToolCallingResult,
ToolExecutor,
ToolValue,
ToolParameter,
StructuredOutputConfig,
StructuredOutputValidation,
} from '@runanywhere/web-llamacpp'
Common mistake: importing
ToolCalling or StructuredOutput from @runanywhere/web. These
are NOT exported from the core package. Always import from @runanywhere/web-llamacpp.Tool Calling
How It Works
The orchestration loop is: generate → parse tool call → execute → feed result back → repeat until done.Basic Usage
Copy
Ask AI
import { ToolCalling, toToolValue, getStringArg } from '@runanywhere/web-llamacpp'
// 1. Register a tool
ToolCalling.registerTool(
{
name: 'get_weather',
description: 'Gets the current weather for a location',
parameters: [{ name: 'location', type: 'string', description: 'City name', required: true }],
},
async (args) => {
const location = getStringArg(args, 'location') ?? 'NYC'
return {
temperature: toToolValue('72F'),
condition: toToolValue('Sunny'),
location: toToolValue(location),
}
}
)
// 2. Generate with tools (auto-execute is on by default)
const result = await ToolCalling.generateWithTools('What is the weather in San Francisco?', {
maxToolCalls: 3,
autoExecute: true,
temperature: 0.3,
})
console.log(result.text) // "The weather in San Francisco is 72F and Sunny."
console.log(result.toolCalls) // [{ toolName: 'get_weather', arguments: { location: ... } }]
console.log(result.toolResults) // [{ toolName: 'get_weather', success: true, result: { ... } }]
console.log(result.isComplete) // true
Recommended Model
For best tool calling results, use the LFM2 1.2B Tool model which is specifically optimized
for function calling:The smaller LFM2 350M also works but may produce less reliable tool call formatting.
Copy
Ask AI
{
id: 'lfm2-1.2b-tool-q4_k_m',
name: 'LFM2 1.2B Tool Q4_K_M',
repo: 'LiquidAI/LFM2-1.2B-Tool-GGUF',
files: ['LFM2-1.2B-Tool-Q4_K_M.gguf'],
framework: LLMFramework.LlamaCpp,
modality: ModelCategory.Language,
memoryRequirement: 800_000_000,
}
API Reference
ToolCalling (Singleton)
Copy
Ask AI
const ToolCalling: {
/** Register a tool with its definition and executor */
registerTool(definition: ToolDefinition, executor: ToolExecutor): void
/** Remove a registered tool */
unregisterTool(name: string): void
/** Get all registered tool definitions */
getRegisteredTools(): ToolDefinition[]
/** Remove all registered tools */
clearTools(): void
/** Execute a single tool call */
executeTool(toolCall: ToolCall): Promise<ToolResult>
/** Full orchestration: generate -> parse -> execute -> loop */
generateWithTools(prompt: string, options?: ToolCallingOptions): Promise<ToolCallingResult>
/** Continue after manual execution (when autoExecute: false) */
continueWithToolResult(
previousPrompt: string,
toolCall: ToolCall,
toolResult: ToolResult,
options?: ToolCallingOptions
): Promise<ToolCallingResult>
cleanup(): void
}
Types
Copy
Ask AI
interface ToolDefinition {
name: string
description: string
parameters: ToolParameter[]
category?: string
}
interface ToolParameter {
name: string
type: 'string' | 'number' | 'boolean' | 'object' | 'array'
description: string
required?: boolean
enumValues?: string[]
}
interface ToolCallingOptions {
tools?: ToolDefinition[] // Override registered tools for this call
maxToolCalls?: number // Max tool calls per generation (default: 5)
autoExecute?: boolean // Auto-execute tool calls (default: true)
temperature?: number
maxTokens?: number
systemPrompt?: string
replaceSystemPrompt?: boolean
keepToolsAvailable?: boolean
format?: ToolCallFormat // 'default' | 'lfm2'
}
interface ToolCallingResult {
text: string // Final response text
toolCalls: ToolCall[] // All tool calls made
toolResults: ToolResult[] // All tool results
isComplete: boolean // True if generation is done
}
interface ToolCall {
toolName: string
arguments: Record<string, ToolValue>
callId?: string
}
interface ToolResult {
toolName: string
success: boolean
result?: Record<string, ToolValue>
error?: string
callId?: string
}
ToolExecutor
The executor function receives parsed arguments asRecord<string, ToolValue> and must return Record<string, ToolValue>:
Copy
Ask AI
type ToolExecutor = (args: Record<string, ToolValue>) => Promise<Record<string, ToolValue>>
TypeScript gotcha: If your executor has
try/catch branches that return different keys, TypeScript
may infer a union type where some properties are undefined, which is not assignable to ToolValue.
Fix this by explicitly annotating the return type:Copy
Ask AI
// WRONG — TypeScript infers { result: ToolValue; error?: undefined } | { error: ToolValue; result?: undefined }
;async (args) => {
try {
return { result: toToolValue(42) }
} catch {
return { error: toToolValue('failed') }
}
}
// CORRECT — explicit return type annotation
;async (args): Promise<Record<string, ToolValue>> => {
try {
return { result: toToolValue(42) }
} catch {
return { error: toToolValue('failed') }
}
}
ToolValue & Helpers
ToolValue is a tagged union that wraps JavaScript values for type-safe serialization:
Copy
Ask AI
type ToolValue =
| { type: 'string'; value: string }
| { type: 'number'; value: number }
| { type: 'boolean'; value: boolean }
| { type: 'array'; value: ToolValue[] }
| { type: 'object'; value: Record<string, ToolValue> }
| { type: 'null' }
ToolValue:
Copy
Ask AI
import { toToolValue, fromToolValue, getStringArg, getNumberArg } from '@runanywhere/web-llamacpp'
// Convert plain JS values → ToolValue
toToolValue('hello') // { type: 'string', value: 'hello' }
toToolValue(42) // { type: 'number', value: 42 }
toToolValue(true) // { type: 'boolean', value: true }
toToolValue(null) // { type: 'null' }
toToolValue([1, 'a']) // { type: 'array', value: [...] }
toToolValue({ k: 'v' }) // { type: 'object', value: { k: { type: 'string', value: 'v' } } }
// Convert ToolValue → plain JS values
fromToolValue({ type: 'string', value: 'hello' }) // 'hello'
// Extract typed args from tool call arguments
const city = getStringArg(args, 'location') // string | undefined
const count = getNumberArg(args, 'limit') // number | undefined
ToolCallFormat
Copy
Ask AI
enum ToolCallFormat {
Default = 'default', // XML: <tool_call>{"tool":"name","arguments":{...}}</tool_call>
LFM2 = 'lfm2', // Pythonic: <|tool_call_start|>[func(arg="val")]<|tool_call_end|>
}
| Format | Style | Best For |
|---|---|---|
default | <tool_call>{"tool":"name","arguments":{...}}</tool_call> | Most models |
lfm2 | <|tool_call_start|>[func_name(arg="val")]<|tool_call_end|> | Liquid AI LFM2 models |
Copy
Ask AI
const result = await ToolCalling.generateWithTools(prompt, {
format: ToolCallFormat.LFM2,
})
Working Examples
Weather Tool (Fake Data)
A simple tool that returns randomized weather data — useful for demos:Copy
Ask AI
import { ToolCalling, toToolValue, getStringArg, type ToolValue } from '@runanywhere/web-llamacpp'
ToolCalling.registerTool(
{
name: 'get_weather',
description:
'Gets the current weather for a city. Returns temperature in Fahrenheit and a short condition.',
parameters: [
{
name: 'location',
type: 'string',
description: 'City name (e.g. "San Francisco")',
required: true,
},
],
category: 'Utility',
},
async (args) => {
const city = getStringArg(args, 'location') ?? 'Unknown'
const conditions = ['Sunny', 'Partly Cloudy', 'Overcast', 'Rainy', 'Windy', 'Foggy']
const temp = Math.round(45 + Math.random() * 50)
const condition = conditions[Math.floor(Math.random() * conditions.length)]
return {
location: toToolValue(city),
temperature_f: toToolValue(temp),
condition: toToolValue(condition),
humidity_pct: toToolValue(Math.round(30 + Math.random() * 60)),
}
}
)
Weather Tool (Real API)
A tool that calls the Open-Meteo API for real weather data:Copy
Ask AI
import { ToolCalling, toToolValue, getStringArg, type ToolValue } from '@runanywhere/web-llamacpp'
ToolCalling.registerTool(
{
name: 'get_weather',
description: 'Gets current weather for a city using the Open-Meteo API',
parameters: [{ name: 'location', type: 'string', description: 'City name', required: true }],
category: 'Utility',
},
async (args): Promise<Record<string, ToolValue>> => {
const city = getStringArg(args, 'location') ?? 'New York'
const geo = await fetch(
`https://geocoding-api.open-meteo.com/v1/search?name=${encodeURIComponent(city)}&count=1`
).then((r) => r.json())
if (!geo.results?.length) {
return { error: toToolValue(`City "${city}" not found`) }
}
const { latitude, longitude, name } = geo.results[0]
const weather = await fetch(
`https://api.open-meteo.com/v1/forecast?latitude=${latitude}&longitude=${longitude}¤t=temperature_2m,weather_code`
).then((r) => r.json())
return {
location: toToolValue(name),
temperature_celsius: toToolValue(weather.current.temperature_2m),
weather_code: toToolValue(weather.current.weather_code),
}
}
)
Calculator Tool
Evaluates mathematical expressions using JavaScript:Copy
Ask AI
import { ToolCalling, toToolValue, getStringArg, type ToolValue } from '@runanywhere/web-llamacpp'
ToolCalling.registerTool(
{
name: 'calculate',
description: 'Evaluates a mathematical expression and returns the numeric result.',
parameters: [
{
name: 'expression',
type: 'string',
description: 'Math expression (e.g. "2 + 3 * 4")',
required: true,
},
],
category: 'Math',
},
async (args): Promise<Record<string, ToolValue>> => {
const expr = getStringArg(args, 'expression') ?? '0'
try {
const sanitized = expr.replace(/[^0-9+\-*/().%\s^]/g, '')
const result = Function(`"use strict"; return (${sanitized})`)()
return { result: toToolValue(Number(result)), expression: toToolValue(expr) }
} catch {
return { error: toToolValue(`Invalid expression: ${expr}`) }
}
}
)
Time / Timezone Tool
Returns the current time for any IANA timezone:Copy
Ask AI
import { ToolCalling, toToolValue, getStringArg, type ToolValue } from '@runanywhere/web-llamacpp'
ToolCalling.registerTool(
{
name: 'get_time',
description: 'Returns the current date and time, optionally for a specific timezone.',
parameters: [
{
name: 'timezone',
type: 'string',
description: 'IANA timezone (e.g. "America/New_York"). Defaults to UTC.',
required: false,
},
],
category: 'Utility',
},
async (args): Promise<Record<string, ToolValue>> => {
const tz = getStringArg(args, 'timezone') ?? 'UTC'
try {
const now = new Date()
const formatted = now.toLocaleString('en-US', {
timeZone: tz,
dateStyle: 'full',
timeStyle: 'long',
})
return { datetime: toToolValue(formatted), timezone: toToolValue(tz) }
} catch {
return {
datetime: toToolValue(new Date().toISOString()),
timezone: toToolValue('UTC'),
note: toToolValue('Fell back to UTC — invalid timezone'),
}
}
}
)
Random Number Tool
Generates a random integer in a range — demonstratesgetNumberArg:
Copy
Ask AI
import { ToolCalling, toToolValue, getNumberArg } from '@runanywhere/web-llamacpp'
ToolCalling.registerTool(
{
name: 'random_number',
description: 'Generates a random integer between min and max (inclusive).',
parameters: [
{ name: 'min', type: 'number', description: 'Minimum value', required: true },
{ name: 'max', type: 'number', description: 'Maximum value', required: true },
],
category: 'Math',
},
async (args) => {
const min = getNumberArg(args, 'min') ?? 1
const max = getNumberArg(args, 'max') ?? 100
const value = Math.floor(Math.random() * (max - min + 1)) + min
return { value: toToolValue(value), min: toToolValue(min), max: toToolValue(max) }
}
)
Manual Execution Mode
WhenautoExecute: false, the SDK returns tool calls without executing them. You handle execution and feed results back:
Copy
Ask AI
import { ToolCalling } from '@runanywhere/web-llamacpp'
const result = await ToolCalling.generateWithTools('What time is it in Tokyo?', {
autoExecute: false,
})
if (!result.isComplete && result.toolCalls.length > 0) {
const toolCall = result.toolCalls[0]
console.log('LLM wants to call:', toolCall.toolName, toolCall.arguments)
// Execute manually (or apply approval logic, rate limiting, etc.)
const toolResult = await ToolCalling.executeTool(toolCall)
// Feed the result back to continue generation
const finalResult = await ToolCalling.continueWithToolResult(
'What time is it in Tokyo?',
toolCall,
toolResult
)
console.log(finalResult.text)
}
Registering Custom Tools at Runtime
You can dynamically register tools — useful for plugin systems or user-configurable tools:Copy
Ask AI
import {
ToolCalling,
toToolValue,
type ToolDefinition,
type ToolValue,
} from '@runanywhere/web-llamacpp'
function registerCustomTool(name: string, description: string, paramNames: string[]) {
const def: ToolDefinition = {
name,
description,
parameters: paramNames.map((p) => ({
name: p,
type: 'string' as const,
description: p,
required: true,
})),
category: 'Custom',
}
const executor = async (args: Record<string, ToolValue>): Promise<Record<string, ToolValue>> => {
// Replace this with your real logic
const result: Record<string, ToolValue> = {
status: toToolValue('executed'),
tool: toToolValue(name),
}
for (const [k, v] of Object.entries(args)) {
result[`input_${k}`] = v
}
return result
}
ToolCalling.registerTool(def, executor)
}
// Register and use
registerCustomTool('search_web', 'Searches the web for a query', ['query'])
registerCustomTool('send_email', 'Sends an email', ['to', 'subject', 'body'])
// List registered tools
console.log(ToolCalling.getRegisteredTools().map((t) => t.name))
// Unregister when done
ToolCalling.unregisterTool('search_web')
// Clear all
ToolCalling.clearTools()
Complete React Component
A full React component with tool registration, execution trace, and UI:ToolCallingDemo.tsx
Copy
Ask AI
import { ModelCategory } from '@runanywhere/web'
import {
ToolCalling,
ToolCallFormat,
toToolValue,
getStringArg,
getNumberArg,
type ToolCallingResult,
type ToolValue,
} from '@runanywhere/web-llamacpp'
import { useState, useEffect, useCallback } from 'react'
interface TraceStep {
type: 'user' | 'tool_call' | 'tool_result' | 'response'
content: string
}
export function ToolCallingDemo() {
const [input, setInput] = useState('')
const [generating, setGenerating] = useState(false)
const [trace, setTrace] = useState<TraceStep[]>([])
// Register demo tools on mount
useEffect(() => {
ToolCalling.clearTools()
ToolCalling.registerTool(
{
name: 'get_weather',
description: 'Gets the current weather for a city',
parameters: [
{ name: 'location', type: 'string', description: 'City name', required: true },
],
},
async (args) => {
const city = getStringArg(args, 'location') ?? 'Unknown'
return {
location: toToolValue(city),
temperature_f: toToolValue(Math.round(45 + Math.random() * 50)),
condition: toToolValue(['Sunny', 'Cloudy', 'Rainy'][Math.floor(Math.random() * 3)]),
}
}
)
ToolCalling.registerTool(
{
name: 'calculate',
description: 'Evaluates a math expression',
parameters: [
{ name: 'expression', type: 'string', description: 'Math expression', required: true },
],
},
async (args): Promise<Record<string, ToolValue>> => {
const expr = getStringArg(args, 'expression') ?? '0'
try {
const val = Function(`"use strict"; return (${expr.replace(/[^0-9+\-*/().%\s]/g, '')})`)()
return { result: toToolValue(Number(val)) }
} catch {
return { error: toToolValue('Invalid expression') }
}
}
)
return () => {
ToolCalling.clearTools()
}
}, [])
const send = useCallback(async () => {
const text = input.trim()
if (!text || generating) return
setInput('')
setGenerating(true)
setTrace([{ type: 'user', content: text }])
try {
const result: ToolCallingResult = await ToolCalling.generateWithTools(text, {
autoExecute: true,
maxToolCalls: 5,
temperature: 0.3,
maxTokens: 512,
format: ToolCallFormat.Default,
})
const steps: TraceStep[] = [{ type: 'user', content: text }]
for (let i = 0; i < result.toolCalls.length; i++) {
const call = result.toolCalls[i]
const argStr = Object.entries(call.arguments)
.map(([k, v]) => `${k}=${JSON.stringify('value' in v ? v.value : v)}`)
.join(', ')
steps.push({ type: 'tool_call', content: `${call.toolName}(${argStr})` })
if (result.toolResults[i]) {
const res = result.toolResults[i]
steps.push({
type: 'tool_result',
content:
res.success && res.result
? JSON.stringify(
Object.fromEntries(
Object.entries(res.result).map(([k, v]) => [k, 'value' in v ? v.value : v])
),
null,
2
)
: `Error: ${res.error ?? 'Unknown'}`,
})
}
}
if (result.text) {
steps.push({ type: 'response', content: result.text })
}
setTrace(steps)
} catch (err) {
setTrace((prev) => [
...prev,
{ type: 'response', content: `Error: ${(err as Error).message}` },
])
} finally {
setGenerating(false)
}
}, [input, generating])
return (
<div>
<div>
{trace.map((step, i) => (
<div
key={i}
style={{
margin: 8,
padding: 8,
background: step.type === 'tool_call' ? '#1a2744' : '#1E293B',
borderRadius: 8,
}}
>
<strong>{step.type.toUpperCase()}</strong>
<pre style={{ whiteSpace: 'pre-wrap', fontSize: 12 }}>{step.content}</pre>
</div>
))}
</div>
<form
onSubmit={(e) => {
e.preventDefault()
send()
}}
>
<input
value={input}
onChange={(e) => setInput(e.target.value)}
placeholder="Ask something..."
disabled={generating}
/>
<button type="submit" disabled={!input.trim() || generating}>
{generating ? 'Running...' : 'Send'}
</button>
</form>
</div>
)
}
Lifecycle Management
Copy
Ask AI
import { ToolCalling } from '@runanywhere/web-llamacpp'
// Register tools early (e.g., in SDK init or component mount)
ToolCalling.registerTool(weatherDef, weatherExecutor)
ToolCalling.registerTool(calcDef, calcExecutor)
// Check registered tools
const tools = ToolCalling.getRegisteredTools()
console.log(
`${tools.length} tools registered:`,
tools.map((t) => t.name)
)
// Remove a specific tool
ToolCalling.unregisterTool('get_weather')
// Clear all tools (e.g., on component unmount)
ToolCalling.clearTools()
// cleanup() is an alias for clearTools()
ToolCalling.cleanup()
Always clean up tools on component unmount in React apps. If multiple components register
tools with the same name, they will overwrite each other. Use
ToolCalling.clearTools() in
useEffect cleanup functions.Structured Output
Generate type-safe JSON responses from LLMs using JSON schema validation.Basic Usage
Copy
Ask AI
import { TextGeneration, StructuredOutput } from '@runanywhere/web-llamacpp'
// 1. Define a JSON schema
const schema = JSON.stringify({
type: 'object',
properties: {
name: { type: 'string' },
ingredients: { type: 'array', items: { type: 'string' } },
prepTime: { type: 'number' },
},
required: ['name', 'ingredients', 'prepTime'],
})
// 2. Get a system prompt that instructs the LLM to produce JSON
const systemPrompt = StructuredOutput.getSystemPrompt(schema)
// 3. Generate with the system prompt
const result = await TextGeneration.generate('Suggest a quick pasta recipe', {
systemPrompt,
maxTokens: 300,
temperature: 0.3,
})
// 4. Extract and validate the JSON
const validation = StructuredOutput.validate(result.text, { jsonSchema: schema })
if (validation.isValid) {
const recipe = JSON.parse(validation.extractedJson!)
console.log(recipe.name) // "Quick Garlic Pasta"
console.log(recipe.ingredients) // ["pasta", "garlic", "olive oil", ...]
console.log(recipe.prepTime) // 15
} else {
console.error('Invalid output:', validation.errorMessage)
}
API Reference
Copy
Ask AI
const StructuredOutput: {
/** Extract the first complete JSON object/array from raw LLM output */
extractJson(text: string): string | null
/** Augment a prompt with schema instructions */
preparePrompt(originalPrompt: string, config: StructuredOutputConfig): string
/** Get a system prompt for JSON schema compliance */
getSystemPrompt(jsonSchema: string): string
/** Validate LLM output against a JSON schema */
validate(text: string, config: StructuredOutputConfig): StructuredOutputValidation
/** Quick check if text contains complete JSON */
hasCompleteJson(text: string): boolean
}
interface StructuredOutputConfig {
jsonSchema: string
includeSchemaInPrompt?: boolean // default: true
}
interface StructuredOutputValidation {
isValid: boolean
errorMessage?: string
extractedJson?: string
}
Workflow
All JSON extraction and schema validation runs in the C++ WASM layer for performance. The
TypeScript API is a thin wrapper around the
rac_structured_output_* C functions.