Chat - RunAnywhere Documentation

Conversations use generateChat, a separate method from generate.

final reply = await RunAnywhere.llm.generateChat([
  ChatMessage(role: ChatRole.system, content: 'You are a concise assistant.'),
  ChatMessage(role: ChatRole.user, content: 'What is on-device inference?'),
  ChatMessage(role: ChatRole.assistant, content: 'Running the model on your own hardware.'),
  ChatMessage(role: ChatRole.user, content: 'Why does that matter?'),
]);

print(reply.text);

Dart has no method overloading. generate takes a String only. Passing a list of messages to it does not compile.

How messages are split

System turns become options.systemPrompt.
The trailing user turn becomes the prompt.
Everything between travels as history.

A conversation with no user turn throws.

Streaming a conversation

await for (final event in RunAnywhere.llm.generateChatStream(messages)) {
  if (event is GenerationEventToken) {
    setState(() => _reply += event.text);
  }
}

Keeping a transcript

class Conversation extends ChangeNotifier {
  final _messages = <ChatMessage>[
    ChatMessage(role: ChatRole.system, content: 'You are a helpful assistant.'),
  ];

  List<ChatMessage> get messages => List.unmodifiable(_messages);

  Future<void> send(String text) async {
    _messages.add(ChatMessage(role: ChatRole.user, content: text));
    notifyListeners();

    final reply = StringBuffer();
    await for (final event in RunAnywhere.llm.generateChatStream(_messages)) {
      if (event is GenerationEventToken) {
        reply.write(event.text);
        notifyListeners();
      }
    }

    _messages.add(ChatMessage(role: ChatRole.assistant, content: reply.toString()));
    notifyListeners();
  }
}

ChatMessage

Field	Type
`role`	`ChatRole.system`, `user`, `assistant`, `tool`
`content`	`String`
`toolCallId`	`String?`

Context length

Trim before the conversation outgrows the model’s window, keeping the system message:

if (_messages.length > 21) {
  final system = _messages.first;
  final recent = _messages.sublist(_messages.length - 20);
  _messages
    ..clear()
    ..add(system)
    ..addAll(recent);
}