RAG, Embeddings, and Reranking

rag.open returns a RagSession: an open corpus you ingest into, search, and query. Each session owns its own native index, so two sessions over different documents can be live at once.

Opening a session

func open(
    embeddingModel: ModelRef,
    llmModel: ModelRef? = nil,
    config: RagConfig? = nil
) async throws -> RagSession

let session = try await RunAnywhere.rag.open(
    embeddingModel: ModelRef(id: "all-minilm-l6-v2"),
    llmModel: ModelRef(id: "lfm2-350m-q4_k_m")
)

open loads both models, downloading them if needed. Passing nil for llmModel gives a retrieval-only session: search works, and query throws .modelNotLoaded. ModelRef is ExpressibleByStringLiteral, so embeddingModel: "all-minilm-l6-v2" also works.

RagSession

RagSession is an actor, so every call is await.

func ingest(document: RagDocument) async throws
func ingest(documents: [RagDocument]) async throws
func search(query: String, topK: Int? = nil) async throws -> [Match]
func query(question: String, options: LlmOptions? = nil) async throws -> RagResult
func queryStream(question: String, options: LlmOptions? = nil) async throws
    -> AsyncThrowingStream<RagEvent, Error>
func stats() async throws -> RagStats
func clear() async throws
func close() async

Ingesting documents

try await session.ingest(document: RagDocument(
    text: noteText,
    metadata: ["source": "meeting-notes.md", "date": "2026-04-11"]
))

Or read from disk and let the backend do the parsing:

try await session.ingest(document: .file(url.path, metadata: ["source": url.lastPathComponent]))

Batch ingest takes an array:

try await session.ingest(documents: notes.map { RagDocument(text: $0.body) })

Chunking happens inside the session, controlled by RagConfig.

Retrieval without generation

let matches = try await session.search(query: "deployment checklist", topK: 5)

for match in matches {
    print("\(match.score): \(match.text)")
    print(match.metadata["source"] ?? "")
}

Match carries text, score, and metadata. topK defaults to whatever the session was opened with.

Grounded answers

let result = try await session.query(question: "What did we decide about the release date?")

print(result.answer)
for source in result.sources {
    print("cited: \(source.metadata["source"] ?? "unknown")")
}
print("\(result.outputTokens) tokens at \(result.tokensPerSecond) tok/s")

RagResult carries answer, sources, and the same metrics block generation results use: inputTokens, outputTokens, timeToFirstTokenMs, tokensPerSecond, requestId, model.

Streaming

let events = try await session.queryStream(question: question)

for try await event in events {
    switch event {
    case .retrieved(let matches):
        sources = matches
    case .token(let text, let kind):
        if kind == .text { answer += text }
    case .completed(let result):
        print("Done, \(result.sources.count) sources")
    }
}

.retrieved fires once, before the first token, so the UI can show sources while the answer is still being written.

RagConfig

var config = RagConfig()
config.topK = 5
config.chunkSize = 512
config.chunkOverlap = 64
config.similarityThreshold = 0.35
config.persistPath = indexURL.path

let session = try await RunAnywhere.rag.open(
    embeddingModel: "all-minilm-l6-v2",
    llmModel: "lfm2-350m-q4_k_m",
    config: config
)

Field	Type	Default	Meaning
`topK`	`Int`	5	Chunks retrieved per query
`chunkSize`	`Int`	512	Characters per chunk
`chunkOverlap`	`Int`	64	Overlap between adjacent chunks
`similarityThreshold`	`Float?`	`nil`	Drop matches below this score
`persistPath`	`String?`	`nil`	Where to persist the index. `nil` is in memory

Setting persistPath turns index persistence on, so the corpus survives a relaunch.

Stats and cleanup

let stats = try await session.stats()
print("\(stats.documentCount) documents, \(stats.chunkCount) chunks, \(stats.indexSizeBytes) bytes")

try await session.clear()   // drop every document, keep the session
await session.close()       // release the session and its native index

Calling anything after close() throws .invalidState.

Managing session lifetime

Sessions are cheap to hold and expensive to rebuild, so keep one per corpus and close it when the corpus goes away.

@MainActor
@Observable
final class DocumentChat {
    private var session: RagSession?
    private var openDocumentID: UUID?

    func prepare(document: Document) async throws -> RagSession {
        if openDocumentID == document.id, let session { return session }

        await session?.close()
        session = nil
        openDocumentID = nil

        let opened = try await RunAnywhere.rag.open(
            embeddingModel: "all-minilm-l6-v2",
            llmModel: "lfm2-350m-q4_k_m"
        )
        try await opened.ingest(document: RagDocument(
            text: document.text,
            metadata: ["source": document.filename]
        ))

        session = opened
        openDocumentID = document.id
        return opened
    }

    func close() async {
        await session?.close()
        session = nil
        openDocumentID = nil
    }
}

Embeddings

When you want vectors rather than a session, embeddings.embed returns one per input, in input order.

let vectors = try await RunAnywhere.embeddings.embed(["hello", "world"])

for vector in vectors {
    print("\(vector.index): \(vector.vector.count) dimensions")
}

Embedding carries index and vector. EmbedOptions controls post-processing:

Field	Type	Default	Cases
`normalize`	`NormalizeMode`	`.l2`	`.none`, `.l2`
`pooling`	`PoolingMode`	`.mean`	`.mean`, `.cls`, `.last`

let raw = try await RunAnywhere.embeddings.embed(
    texts,
    options: EmbedOptions(normalize: .none, pooling: .cls)
)

embed auto-loads an embedding model the way generation does.

Reranking

A cross-encoder scores each document against the query directly, which is more accurate than cosine similarity over embeddings and slower. The usual shape is to retrieve widely with embeddings, then rerank the shortlist.

let matches = try await session.search(query: question, topK: 20)

let ranked = try await RunAnywhere.rerank.rerank(
    query: question,
    documents: matches.map(\.text),
    topN: 5
)

let best = ranked.map { matches[$0.index] }

RankedResult carries index, pointing back into the documents you passed, and relevanceScore. Results are sorted best first. topN of nil returns all of them. Rerank has no model category, so the lifecycle cannot auto-load it. A rerank model must already be resident under the rerank component or the call throws .modelNotLoaded.

Error handling

do {
    let result = try await session.query(question: question)
    print(result.answer)
} catch let error as SDKException {
    switch error.code {
    case .invalidState:
        print("The session is closed")
    case .modelNotLoaded:
        print("This session is retrieval-only, or the model was unloaded")
    case .processingFailed:
        print("Query failed: \(error.message)")
    default:
        print("Error: \(error.localizedDescription)")
    }
}

LLM Generation

Generation options the query path shares

Configuration

Model registry and lifecycle

Getting Started

Swift SDK

Kotlin SDK

React Native SDK

Flutter SDK

Web SDK

Electron SDK

Vibe Coding

Opening a session

RagSession

Ingesting documents

Retrieval without generation

Grounded answers

Streaming

RagConfig

Stats and cleanup

Managing session lifetime

Embeddings

Reranking

Error handling

LLM Generation

Configuration

​Opening a session

​RagSession

​Ingesting documents

​Retrieval without generation

​Grounded answers

​Streaming

​RagConfig

​Stats and cleanup

​Managing session lifetime

​Embeddings

​Reranking

​Error handling

LLM Generation

Configuration

Opening a session

RagSession

Ingesting documents

Retrieval without generation

Grounded answers

Streaming

RagConfig

Stats and cleanup

Managing session lifetime

Embeddings

Reranking

Error handling