Best practices

Memory

On-device models are the largest allocation your app will make. Most crash reports from SDK integrations trace back to holding two of them at once.

Load one model per modality

Model lifecycle is per category. Loading a second language model replaces the first; loading an STT model does not touch the LLM. That means a chat model and a recognizer can coexist, but a 3B and a 0.5B language model cannot.

import com.runanywhere.sdk.public.extensions.loadModel
import com.runanywhere.sdk.public.types.RAModelLoadRequest

RunAnywhere.loadModel(RAModelLoadRequest(model_id = "smollm2-360m-instruct-q8_0"))

loadModel() is the single entry point for every modality. loadLLMModel(), loadSTTModel(), loadTTSVoice(), and loadVLMModel() do not exist.

Unload before a large allocation

import ai.runanywhere.proto.v1.ModelUnloadRequest
import com.runanywhere.sdk.public.extensions.unloadModel

RunAnywhere.unloadModel(ModelUnloadRequest(unload_all = true))

Free resident weights before starting a large download, so the download’s own memory preflight can pass while another model is still loaded. ModelUnloadRequest also accepts model_id, category, and framework for narrower unloads.

Check compatibility before committing

import com.runanywhere.sdk.public.extensions.checkModelCompatibility

val compatibility = RunAnywhere.checkModelCompatibility(modelId)

ModelInfo.memory_required_bytes is the runtime headroom estimate and is deliberately separate from download_size_bytes, which is transport size. Compare against available RAM, not against free storage. downloadModel() already runs this preflight for registered models.

Performance

Quantization

Smaller quantizations trade output quality for memory and speed. Q4 variants are the usual starting point on phones; Q8 is worth trying when the model is small enough that its full size still fits comfortably.

Stream anything the user reads

RunAnywhere.generateStream(prompt).collect { event ->
    if (event.token.isNotEmpty()) append(event.token)
}

generateStream() emits LLMStreamEvent, not String, and is not a suspend function. See Streaming.

Cap output tokens against the model, not against a constant

import ai.runanywhere.proto.v1.LLMGenerationOptions

LLMGenerationOptions(max_tokens = 96)   // voice replies
LLMGenerationOptions(max_tokens = 200)  // chat
LLMGenerationOptions(max_tokens = 500)  // long form

For VLM this is not a preference. Image tokens share the context with the text prompt, so on a 512-token vision model an unconstrained output budget leaves no room for the image. Cap output at roughly a quarter of the model’s declared context_length. See VLM.

Keep blocking calls off the main thread

loadModel(), processImage(), and the first call into a freshly registered backend all block for a long time in JNI. Dispatch them:

withContext(Dispatchers.IO) { RunAnywhere.loadModel(request) }
withContext(Dispatchers.Default) { RunAnywhere.processImage(image, options) }

The SDK already dispatches its own streaming paths (transcribeStream, synthesizeStream, streamVoiceAgent) onto Dispatchers.IO, so collecting those from the main thread is fine.

Lifecycle

Initialize once, in Application

Registration order matters. CPU backends register before initialize(); QHexRT registers after. See Configuration.

class MyApplication : Application() {
    private val scope = CoroutineScope(SupervisorJob() + Dispatchers.IO)

    override fun onCreate() {
        super.onCreate()
        scope.launch { setupSdk() }
    }

    private suspend fun setupSdk() {
        runCatching { LlamaCPP.register() }
        runCatching { ONNX.register() }
        RunAnywhere.initialize(context = this@MyApplication, environment = environment)
        QHexRT.register()
    }
}

Guard setup with a flag so a retry after a failed init cannot run it twice concurrently, and expose a retry rather than leaving the app in a permanently broken state.

Cancel streams when a screen goes away

A retained collector keeps feeding a shared model after the user has navigated on. Cancel the job and join it before tearing anything down:

fun stop() {
    val session = job
    session?.cancel()
    job = null
    cleanupJob = viewModelScope.launch(Dispatchers.IO) {
        session?.join()
        runCatching { RunAnywhere.cleanupVoiceAgent() }
    }
}

Joining matters because cancelling a coroutine blocked in JNI is cooperative. The flow completes only after the native call returns and its driver has stopped capture, so cleanup that does not wait can race an in-flight turn.

Bound live audio buffers

Live transcription reads from an upstream flow of microphone chunks. Bound it and drop the oldest under pressure, otherwise navigating away leaves minutes of audio queued behind one blocking recognizer call.

val channel = Channel<ByteArray>(capacity = 32, onBufferOverflow = BufferOverflow.DROP_OLDEST)

Preload at launch

lifecycleScope.launch {
    val model = RunAnywhere.getModel(ModelGetRequest(model_id = primaryModelId))
    if (model.found && model.model?.is_downloaded != true) {
        RunAnywhere.downloadModelStream(model.model!!).collect { progress -> update(progress) }
    }
    RunAnywhere.loadModel(RAModelLoadRequest(model_id = primaryModelId))
}

downloadModelStream(model: RAModelInfo) returns Flow<DownloadProgress>. The suspend form is downloadModel(model, onProgress), which returns the terminal DownloadProgress. Both take a ModelInfo, not a model id string. Cancelling a download propagates to the native worker and preserves resume bytes for a later retry.

Errors

Branch on category

import ai.runanywhere.proto.v1.ErrorCategory
import com.runanywhere.sdk.foundation.errors.SDKException

try {
    val result = RunAnywhere.generate(prompt)
    show(result.text)
} catch (e: SDKException) {
    when (e.category) {
        ErrorCategory.ERROR_CATEGORY_MODEL -> promptModelDownload()
        ErrorCategory.ERROR_CATEGORY_IO -> promptStorageCleanup()
        ErrorCategory.ERROR_CATEGORY_NETWORK -> showRetry()
        else -> show(e.recoverySuggestion ?: e.errorDescription ?: e.message)
    }
}

ErrorCode has roughly 280 values and grows with the proto. ErrorCategory has nine and is stable. Full names are ERROR_CATEGORY_*, not bare MODEL or STORAGE. See Error handling.

Rethrow CancellationException

try {
    doWork()
} catch (e: CancellationException) {
    throw e
} catch (e: Exception) {
    error = e.message
}

Swallowing CancellationException in a broad catch (e: Exception) breaks structured concurrency and turns a normal user-driven stop into a spurious error banner.

Check result flags, not just exceptions

loadModel, unloadModel, listModels, queryModels, getModel, and getStorageInfo return a result object with a success flag rather than throwing.

val result = RunAnywhere.loadModel(request)
if (!result.success) show(result.error_message)

Show progress that reflects reality

RunAnywhere.downloadModelStream(model).collect { progress ->
    val percent = if (progress.total_bytes > 0) {
        (progress.bytes_downloaded * 100 / progress.total_bytes).toInt()
    } else {
        (progress.stage_progress.coerceIn(0f, 1f) * 100).toInt()
    }
    updateProgress(percent)
}

total_bytes is 0 until the planner has sized the artifact, so a naive division shows nothing during the plan stage.

Testing

Test on hardware. An emulator does not reproduce the memory ceiling, the CPU and NPU performance, or the thermal throttling that determine whether a model is usable. Measure with the fields the result actually carries:

val result = RunAnywhere.generate(prompt, options)

Log.d("Perf", """
    model: ${result.model_used}
    output tokens: ${result.tokens_generated}
    tok/s: ${result.tokens_per_second}
    total: ${result.generation_time_ms}ms
    ttft: ${result.ttft_ms}ms
""".trimIndent())

ttft_ms is set in streaming mode only.

Security

Keep the api key out of source:

RunAnywhere.initialize(context = app, apiKey = BuildConfig.RUNANYWHERE_API_KEY, ...)

The same applies to a HuggingFace token. Read it from user-provided storage and pass it with RunAnywhere.setHfToken(token); never embed it in the APK. The SDK attaches it only to huggingface.co and hf.co over https and never logs it. RunAnywhere.reset() is a suspend function that clears SDK state.

Android specifics

Runtime permissions

STT and the voice agent need RECORD_AUDIO granted at runtime through ActivityResultContracts.RequestPermission(), not just declared in the manifest.

Audio formats

STT expects 16 kHz mono 16-bit PCM. TTS output sample rate varies by voice: read TTSOutput.sample_rate rather than assuming one. speak() handles the PCM-to-WAV conversion and playback for you.

Content URIs

VLMImage with VLM_IMAGE_FORMAT_FILE_PATH needs a filesystem path. Images from the photo picker arrive as content URIs and have to be copied to a cache file first, then deleted in a finally block. See VLM.

Backends may be absent

Wrap LlamaCPP.register() and ONNX.register() in runCatching. A build that ships only some native libraries will throw on the missing one, and an unguarded call aborts SDK setup entirely. Record which registrations succeeded so model pickers can hide rows whose backend cannot serve them.

JitPack repository

The SDK has transitive dependencies hosted on JitPack. Add maven { url = uri("https://jitpack.io") } to settings.gradle.kts. See Installation.

Qualcomm Hexagon NPU

The NPU has a single slot. Two NPU models cannot be resident at once, so loading one evicts the other. This has consequences that do not exist on CPU backends. The voice agent requires STT, LLM, and TTS to be co-resident, so it cannot run when both the recognizer and the chat model are QHexRT. Drive the turn manually, loading one model per phase. See Voice agent. A CPU or system TTS can play while an NPU LLM keeps generating, because they do not share the slot. An NPU TTS can only load after generation finishes. Branch on the framework, not on the model name:

val isNpu = model.framework == InferenceFramework.INFERENCE_FRAMEWORK_QHEXRT

NPU vision bundles often declare a small context_length (512 is common), which makes the VLM output cap mandatory. Gated NPU model repos need a HuggingFace token before they will download.

Checklist

Load one model per modality and unload before a large allocation. Cap max_tokens against the model’s context, especially for VLM. Stream anything the user reads. Dispatch load and image calls off the main thread. Cancel and join stream collectors on teardown. Branch on ErrorCategory and rethrow CancellationException. Check success flags on result-returning calls. Test on hardware.

Getting Started

Swift SDK

Kotlin SDK

React Native SDK

Flutter SDK

Web SDK

Vibe Coding

Memory

Load one model per modality

Unload before a large allocation

Check compatibility before committing

Performance

Quantization

Stream anything the user reads

Cap output tokens against the model, not against a constant

Keep blocking calls off the main thread

Lifecycle

Initialize once, in Application

Cancel streams when a screen goes away

Bound live audio buffers

Preload at launch

Errors

Branch on category

Rethrow CancellationException

Check result flags, not just exceptions

Show progress that reflects reality

Testing

Security

Android specifics

Runtime permissions

Audio formats

Content URIs

Backends may be absent

JitPack repository

Qualcomm Hexagon NPU

Checklist

​Memory

​Load one model per modality

​Unload before a large allocation

​Check compatibility before committing

​Performance

​Quantization

​Stream anything the user reads

​Cap output tokens against the model, not against a constant

​Keep blocking calls off the main thread

​Lifecycle

​Initialize once, in Application

​Cancel streams when a screen goes away

​Bound live audio buffers

​Preload at launch

​Errors

​Branch on category

​Rethrow CancellationException

​Check result flags, not just exceptions

​Show progress that reflects reality

​Testing

​Security

​Android specifics

​Runtime permissions

​Audio formats

​Content URIs

​Backends may be absent

​JitPack repository

​Qualcomm Hexagon NPU

​Checklist

Memory

Load one model per modality

Unload before a large allocation

Check compatibility before committing

Performance

Quantization

Stream anything the user reads

Cap output tokens against the model, not against a constant

Keep blocking calls off the main thread

Lifecycle

Initialize once, in Application

Cancel streams when a screen goes away

Bound live audio buffers

Preload at launch

Errors

Branch on category

Rethrow CancellationException

Check result flags, not just exceptions

Show progress that reflects reality

Testing

Security

Android specifics

Runtime permissions

Audio formats

Content URIs

Backends may be absent

JitPack repository

Qualcomm Hexagon NPU

Checklist