Moldable Chat: Working with objects

Glamorous Toolkit is a live environment. Generating code (Moldable Chat: Generating code) or documentation (Moldable Chat: Changing Lepiter pages) is useful, but to leverage the environment’s full potential beyond what is possible elsewhere, we should get the LLM to work with objects.

Let's start with a basic case in which we ask the LLM to produce a script that builds an icon. We also ask it to run the script and then evaluate the result by looking at the view:

c := GtL gt
	sendMarkdown: 'Produce a script that builds an 16x16 element with icon that shows a simple home. Take as blueprint for the script, the icon defined in {{gtMethod:BrGlamorousVectorIcons class>>chat}} which uses {{gtClass:BlPolygonGeometry}}. Execute the code. Evaluate the result by viewing the Live view of the resulting object. Give a grade of the result.'
  

To check the Live view of the resulting element, it needs a means to pass the reference from resulting from the code execution tool to the tool that gets the view. This is achieved through an object storage (see GtLChat>>#objectStorage objectStorage ^ objectStorage ) that associates ids to objects and stores them in a chat. Tools like GtLMagritteToolForSmalltalkCodeEvaluation GtLAbstractMagritteFunctionTool << #GtLMagritteToolForSmalltalkCodeEvaluation slots: {}; tag: 'Tools'; package: 'Gt4LlmCore' use this mechanism to store results, and other tools such as GtLToolForObjectView GtLAbstractPlainSchemaFunctionTool << #GtLToolForObjectView slots: {}; tag: 'Tools'; package: 'Gt4LlmCore' can interpret ids passed as arguments.

If you want to explore this use case further, try this:

c sendMarkdown: 'Improve it to get a higher grade.'
  

Inspecting the result of a script and viewing it is equivalent to working in a playground and an inspector. To complete the picture and allow the LLM to reach and inspect arbitrary objects, we also need the equivalent of a contextual playground. To this end, we need to be able to bind objects to variables in a script.

The case below exemplifies this idea:

GtL gt
	sendMarkdown: 'Execute {{gtMethod:BlGridLayoutExamples>>#exampleCellSpacingHorizontal1}}. What are the exact bounds of the two blue squares?'
  

The example method produces an element. To obtain details about that element, the LLM has to execute a script with a variable bound to the element. The example method tool stores the resulting object in the object storage, and the LLM has the option of binding a name to an ID as an argument in GtLMagritteToolForSmalltalkCodeEvaluation>>#bindingsDescription bindingsDescription <magritteDescription> ^ MAToManyRelationDescription new label: 'Bindings'; accessor: #bindings; classes: { GtLSmalltalkCodeEvaluationBinding }; beOptional . Inspect the Smalltalk execution message in the chat above to see this in action.

The above scenario is equivalent to inspecting the result of an example method and running a script in a contextual playground on that result. Sometimes, however, we want to start the conversation from an existing object in the image.

We can achieve this by storing an object in the object storage from outside the chat.

container := BrVerticalPane new matchParent; background: Color paleOrange.
c := GtL gt.
c objectStorage addNewObject: container.
c sendMarkdown: 'You have an input object in the object storage. Execute {{gtMethod:BlGridLayoutExamples>>#exampleCellSpacingHorizontal1}}. Add the result to the original input object' 
  

In this example, the LLM must find the ID of the input object. It can do that with GtLToolForObjectStorageList GtLAbstractMagritteFunctionTool << #GtLToolForObjectStorageList slots: {}; tag: 'Tools'; package: 'Gt4LlmCore' . It can then run the example to get another object’s ID, and eventually execute a script that relies on both IDs.

If the chat is successful, we should see the side effect in our variable external to the chat:

container