Chat - RunAnywhere Documentation

Pass [ChatMessage] instead of a prompt to hold a conversation.

let messages = [
    ChatMessage(role: .system, content: "You are a concise assistant."),
    ChatMessage(role: .user, content: "What is on-device inference?"),
    ChatMessage(role: .assistant, content: "Running the model on your own hardware."),
    ChatMessage(role: .user, content: "Why does that matter?"),
]

let reply = try await RunAnywhere.llm.generate(messages: messages)
print(reply.text)

How messages are split

The SDK does not send the array verbatim. It splits it:

System turns become options.systemPrompt.
The trailing user turn becomes the prompt.
Everything between travels as history.

A conversation with no user turn throws SDKException.

Streaming a conversation

let stream = try await RunAnywhere.llm.generateStream(messages: messages)

for try await event in stream {
    if case .token(let token) = event { print(token.text, terminator: "") }
}

Keeping a transcript

@MainActor
@Observable
final class Conversation {
    private(set) var messages: [ChatMessage] = [
        ChatMessage(role: .system, content: "You are a helpful assistant.")
    ]

    func send(_ text: String) async throws {
        messages.append(ChatMessage(role: .user, content: text))

        var reply = ""
        let stream = try await RunAnywhere.llm.generateStream(messages: messages)
        for try await event in stream {
            if case .token(let token) = event { reply += token.text }
        }

        messages.append(ChatMessage(role: .assistant, content: reply))
    }
}

ChatMessage

Field	Type
`role`	`.system`, `.user`, `.assistant`, `.tool`
`content`	`String`
`toolCallId`	`String?`

Context length

A conversation grows until it exceeds the model’s context window. Trim old turns yourself, keeping the system message:

if messages.count > 21 {
    messages = [messages[0]] + messages.suffix(20)
}