Chat - RunAnywhere Documentation

Pass a List<ChatMessage> instead of a prompt.

val messages = listOf(
    ChatMessage(ChatRole.System, "You are a concise assistant."),
    ChatMessage(ChatRole.User, "What is on-device inference?"),
    ChatMessage(ChatRole.Assistant, "Running the model on your own hardware."),
    ChatMessage(ChatRole.User, "Why does that matter?"),
)

val reply = RunAnywhere.llm.generate(messages)
println(reply.text)

How messages are split

The list is not sent verbatim:

System turns become options.systemPrompt.
The trailing user turn becomes the prompt.
Everything between travels as history.

A conversation with no user turn throws SDKException.

Streaming a conversation

RunAnywhere.llm.generateStream(messages).collect { event ->
    if (event is GenerationEvent.Token) print(event.text)
}

Keeping a transcript

class Conversation : ViewModel() {
    private val messages = mutableListOf(
        ChatMessage(ChatRole.System, "You are a helpful assistant.")
    )

    private val _transcript = MutableStateFlow<List<ChatMessage>>(emptyList())
    val transcript = _transcript.asStateFlow()

    fun send(text: String) {
        viewModelScope.launch {
            messages += ChatMessage(ChatRole.User, text)
            _transcript.value = messages.toList()

            val reply = StringBuilder()
            RunAnywhere.llm.generateStream(messages).collect { event ->
                if (event is GenerationEvent.Token) reply.append(event.text)
            }

            messages += ChatMessage(ChatRole.Assistant, reply.toString())
            _transcript.value = messages.toList()
        }
    }
}

ChatMessage

Field	Type
`role`	`ChatRole.System`, `User`, `Assistant`, `Tool`
`content`	`String`
`toolCallId`	`String?`

Context length

Trim old turns before the conversation outgrows the model’s context window, keeping the system message:

if (messages.size > 21) {
    val system = messages.first()
    val recent = messages.takeLast(20)
    messages.clear()
    messages += system
    messages += recent
}