RAG sessions

Overview

A RAG session owns one corpus. You ingest documents into it, and it chunks, embeds, and indexes them. Asking a question retrieves the closest chunks and generates an answer grounded in them. Everything runs on the device.

Dependencies

RAG ships inside the core SDK, so there is no separate RAG artifact. You do need a backend that provides embeddings: on Android that is runanywhere-onnx, which is what the MiniLM embedding models run on.

dependencies {
    implementation("io.github.sanchitmonga22:runanywhere-sdk:0.20.12")
    implementation("io.github.sanchitmonga22:runanywhere-onnx:0.20.12")
}

rag.open() loads rac_backend_onnx itself on first use, so its embedding operations are registered before the session exists. You do not load it by hand.

Basic usage

import com.runanywhere.sdk.public.RunAnywhere
import com.runanywhere.sdk.public.api.ModelRef
import com.runanywhere.sdk.public.api.RagDocument
import com.runanywhere.sdk.public.api.rag

val session = RunAnywhere.rag.open(
    embeddingModel = ModelRef("all-minilm-l6-v2"),
    llmModel = ModelRef("smollm2-360m-instruct-q8_0"),
)

session.ingest(RagDocument("RunAnywhere runs AI models on the device."))
session.ingest(RagDocument("The SDK supports LLM, STT, TTS, VAD, VLM, and RAG."))

val result = session.query("What AI features does RunAnywhere support?")
println(result.answer)
println("${result.sources.size} chunks used")

session.close()

open() downloads and loads both models before returning the session, so there is no separate prepare step. Passing a null llmModel opens a retrieval-only session, where search() works and query() throws. Sessions are independent. Two sessions over different corpora can be open at the same time.

rag.open()

suspend fun open(
    embeddingModel: ModelRef,
    llmModel: ModelRef? = null,
    config: RagConfig? = null,
): RagSession

ModelRef(id) points at a registered model. It throws SDKException when either model cannot be loaded.

RagSession

suspend fun ingest(document: RagDocument)
suspend fun ingest(documents: List<RagDocument>)
suspend fun search(query: String, topK: Int? = null): List<Match>
suspend fun query(question: String, options: LlmOptions? = null): RagResult
fun queryStream(question: String, options: LlmOptions? = null): Flow<RagEvent>
suspend fun stats(): RagStats
suspend fun clear()
suspend fun close()

close() releases the native session and is safe to call twice. clear() drops every indexed document but keeps the session open. Every verb throws SDKException once the session is closed.

Close the session when you are done with it. It holds an embedding model, an LLM, and a vector index resident at the same time.

Ingestion

session.ingest(RagDocument("Your document text here."))

session.ingest(
    RagDocument(
        text = "Patient records indicate...",
        metadata = mapOf("source" to "medical-docs", "page" to "42"),
    ),
)

session.ingest(listOf(docA, docB, docC))

val fromDisk = RagDocument.file("/path/to/notes.txt")

RagDocument(text, metadata) takes plain text and a typed Map<String, String>, not a JSON string. RagDocument.file(path) reads a UTF-8 text file and throws when it is missing or unreadable. The metadata rides along onto every chunk and comes back on each Match.

Retrieval without generation

val matches = session.search("privacy architecture", topK = 5)
matches.forEach { println("${it.score} ${it.text.take(100)}") }

Match carries text, score (cosine similarity), and metadata. topK defaults to the session’s configured value.

Querying

import com.runanywhere.sdk.public.api.LlmOptions

val result = session.query(
    question = "How does the privacy architecture work?",
    options = LlmOptions(maxOutputTokens = 192, temperature = 0f),
)

println(result.answer)
result.sources.forEach { println("${it.score}: ${it.text.take(100)}") }
println("${result.outputTokens} tokens at ${result.tokensPerSecond}/s")

RagResult carries answer, sources: List<Match>, and the same metrics block as GenerationResult: inputTokens, outputTokens, timeToFirstTokenMs, tokensPerSecond, requestId, and model. The answer is generated with the same LlmOptions the llm namespace uses, so systemPrompt, reasoning, and the sampling knobs all apply. See generate().

Streaming

import com.runanywhere.sdk.public.api.RagEvent
import com.runanywhere.sdk.public.api.TokenKind

val streamed = StringBuilder()

session.queryStream(question, options).collect { event ->
    when (event) {
        is RagEvent.Retrieved -> showSources(event.matches)
        is RagEvent.Token -> {
            streamed.append(event.text)
            updateAnswer(streamed.toString())
        }
        is RagEvent.Completed -> {
            showAnswer(event.result.answer)
            showSources(event.result.sources)
        }
    }
}

RagEvent has three arms: Retrieved(matches) with the chunks that will ground the answer, then Token(text, kind), then Completed(result). A failure is thrown into the collector rather than delivered as an event. Cancelling the collector cancels that one native request. Concurrent collectors and unary queries on the same session serialize instead of killing each other.

Statistics

val stats = session.stats()
stats.documentCount   // Long
stats.chunkCount      // Long
stats.indexSizeBytes  // Long

RagConfig

Chunking and retrieval knobs are fixed for the life of the session. Defaults come from the rac_default annotations in idl/rag.proto.

Field	Type	Default	Description
`topK`	`Int`	`5`	Chunks retrieved per query
`chunkSize`	`Int`	`512`	Tokens per chunk
`chunkOverlap`	`Int`	`64`	Overlap tokens, must be less than `chunkSize`
`similarityThreshold`	`Float?`	`null`	Minimum cosine similarity. Null accepts everything
`persistPath`	`String?`	`null`	On-disk index path. Null keeps the index in memory
`rerank`	`Boolean`	`false`	Rerank retrieved chunks with the loaded cross-encoder
`multiQuery`	`Boolean`	`false`	Expand each question into several retrieval queries

val session = RunAnywhere.rag.open(
    embeddingModel = ModelRef("all-minilm-l6-v2"),
    llmModel = ModelRef("smollm2-360m-instruct-q8_0"),
    config = RagConfig(chunkSize = 256, chunkOverlap = 32, rerank = true),
)

Leave similarityThreshold null unless you have measured that a floor helps with your embedding model. MiniLM-class sentence embeddings rarely exceed a cosine similarity of about 0.5 even for genuinely relevant text, and chunking lowers each chunk’s score further, so a positive floor filters out real matches and the answer model reports that it has no information. topK bounds the result count instead. Chunk size and overlap apply at ingestion, so changing them means opening a new session and re-ingesting. Smaller chunks retrieve more precisely; larger ones carry more surrounding context. RagConfig.DEFAULT_TOP_K, DEFAULT_CHUNK_SIZE, and DEFAULT_CHUNK_OVERLAP expose the three numeric defaults.

ViewModel

class DocumentQAViewModel : ViewModel() {
    var answer by mutableStateOf("")
        private set
    var sources by mutableStateOf<List<Match>>(emptyList())
        private set
    var chunkCount by mutableStateOf(0L)
        private set

    private var session: RagSession? = null
    private var job: Job? = null

    fun open(embeddingModelId: String, llmModelId: String) {
        viewModelScope.launch {
            session?.close()
            session = RunAnywhere.rag.open(
                embeddingModel = ModelRef(embeddingModelId),
                llmModel = ModelRef(llmModelId),
                config = RagConfig(chunkSize = 256, chunkOverlap = 32),
            )
        }
    }

    fun ingest(text: String, source: String) {
        viewModelScope.launch {
            val open = session ?: return@launch
            open.ingest(RagDocument(text, mapOf("source" to source)))
            chunkCount = open.stats().chunkCount
        }
    }

    fun ask(question: String) {
        job?.cancel()
        job = viewModelScope.launch {
            val open = session ?: return@launch
            val streamed = StringBuilder()
            open.queryStream(question, options).collect { event ->
                when (event) {
                    is RagEvent.Retrieved -> sources = event.matches
                    is RagEvent.Token -> {
                        streamed.append(event.text)
                        answer = streamed.toString()
                    }
                    is RagEvent.Completed -> {
                        answer = event.result.answer
                        sources = event.result.sources
                    }
                }
            }
        }
    }

    @OptIn(DelicateCoroutinesApi::class)
    override fun onCleared() {
        job?.cancel()
        val open = session
        session = null
        if (open != null) GlobalScope.launch { runCatching { open.close() } }
    }
}

Closing the session outlives the ViewModel scope, which is why the teardown runs outside viewModelScope. Cancelling the scope first would cancel the close too.

Embeddings on their own

The embeddings namespace exposes the same embedding model directly, for your own similarity search or clustering.

import com.runanywhere.sdk.public.api.EmbedOptions
import com.runanywhere.sdk.public.api.embeddings
import com.runanywhere.sdk.public.api.models

RunAnywhere.models.load("all-minilm-l6-v2")

val vectors = RunAnywhere.embeddings.embed(listOf("hello", "goodbye"))
println(vectors.first().vector.size)

embed(texts, options: EmbedOptions? = null) returns one Embedding per input in input order, each carrying its index and a FloatArray vector. EmbedOptions has normalize (NormalizeMode.L2 by default, or NONE) and pooling (PoolingMode.MEAN by default, or CLS or LAST). It throws when no embedding model is loaded.

Reranking

rerank scores documents against a query with a cross-encoder, which is more accurate than the cosine similarity retrieval uses and much slower. It needs a rerank model loaded.

import com.runanywhere.sdk.public.api.rerank

val ranked = RunAnywhere.rerank.rerank(
    query = "battery life",
    documents = passages,
    topN = 3,
)

ranked.forEach { println("${it.relevanceScore}: ${passages[it.index]}") }

RankedResult carries index, pointing back at the input list, and relevanceScore. Results are sorted best first. topN is optional and null returns everything. Set RagConfig(rerank = true) to have a session rerank its retrieved chunks before generating, instead of calling this yourself.

Errors

import ai.runanywhere.proto.v1.ErrorCategory
import com.runanywhere.sdk.foundation.errors.SDKException

try {
    val result = session.query("What is RunAnywhere?")
    show(result.answer)
} catch (e: CancellationException) {
    throw e
} catch (e: SDKException) {
    when (e.category) {
        ErrorCategory.ERROR_CATEGORY_MODEL -> promptToLoadModels()
        ErrorCategory.ERROR_CATEGORY_COMPONENT -> promptToOpenSession()
        else -> show(e.recoverySuggestion ?: e.errorDescription ?: e.message)
    }
}

Branch on category, not on message text. queryStream throws into the collector rather than emitting an error event. See Error handling.

LLM generation

Text generation API

LoRA adapters

Adapt model behavior

Configuration

Best practices

Memory and lifecycle

Getting Started

Swift SDK

Kotlin SDK

React Native SDK

Flutter SDK

Web SDK

Vibe Coding

Overview

Dependencies

Basic usage

rag.open()

RagSession

Ingestion

Retrieval without generation

Querying

Streaming

Statistics

RagConfig

ViewModel

Embeddings on their own

Reranking

Errors

LLM generation

LoRA adapters

Configuration

Best practices

​Overview

​Dependencies

​Basic usage

​rag.open()

​RagSession

​Ingestion

​Retrieval without generation

​Querying

​Streaming

​Statistics

​RagConfig

​ViewModel

​Embeddings on their own

​Reranking

​Errors

​Related

LLM generation

LoRA adapters

Configuration

Best practices

Overview

Dependencies

Basic usage

rag.open()

RagSession

Ingestion

Retrieval without generation

Querying

Streaming

Statistics

RagConfig

ViewModel

Embeddings on their own

Reranking

Errors

Related