Working with images and LLMs

All providers (Ollama, Anthropic, Gemini, and OpenAI) have vision capabilities [1]. As such, chats chats work with images through the UI or programmatically.

To attach an image through the UI, simply attach them using the Attach icon at the bottom of the message editor.

In the following, we illustrate attaching a screenshot of GT programmatically for illustration purposes.

First, we create a chat using the default provider:

chat := GtLChat new
  

We then create a screenshot of the current window of Glamorous Toolkit.

fileReference := BlExporter png
		element: (GtWorldElement allInstances detect: [:aWorldElement | aWorldElement space isFocused]);
		export
  

Finally, we can send this screenshot to the chat.

chat sendFiles: {fileReference} withMessage: 'What do you see?'
  

[1] Not all models provided by Ollama are equipped with Vision capabilities, but models such as llava or gemma3 come with such capabilities.