Report infrastructure for collecting data in GT
Some problems in a GT image only show while they happen: a slow UI, a process that hangs, many garbage collections, memory that grows. A report captures such a moment as plain data, so that it can be looked at later, possibly in another image than the one where it happened.
The infrastructure keeps four things apart, because they change for different reasons: what is recorded (the report), how and when it is recorded (the recorder), where it is kept (the store), and how it is found and read again (the metadata). Recording reports is on GT reports available for data collection, analysing them on Analysing stored GT reports. The classes are in GToolkit-Utility-SpaceTally, GToolkit-Utility-GarbageCollectAnalysis and GToolkit-Utility-Basic.
Piece Classes Responsibility Why it is separate Report GtProcessesReport
Object << #GtProcessesReport
slots: { #processReports . #imageData . #reportUid };
package: 'GToolkit-Utility-SpaceTally'
, GtMessageTallyReport
Object << #GtMessageTallyReport
slots: {
#textualContent .
#imageData .
#reportUid .
#startTime .
#duration .
#pollPeriodMilliseconds .
#numberOfSamples .
#measuredMilliseconds .
#vmParametersDiff .
#messageTally };
package: 'GToolkit-Utility-SpaceTally'
, GtGarbageCollectsReport
Object << #GtGarbageCollectsReport
slots: { #events . #overallParametersDiffReport . #imageData . #reportUid };
package: 'GToolkit-Utility-GarbageCollectAnalysis'
, GtSpaceTallyReport
Object << #GtSpaceTallyReport
slots: { #imageData . #reportUid . #timestamp . #spaceTally . #vmStatisticsReport };
package: 'GToolkit-Utility-SpaceTally'
the recorded data, its image data and its report uid; writes its own files (#selector
writeFilesTo: anEntry
anEntry writeJsonOf: self
) and reads itself back (#selector
fromStoredFiles: aStoredReportFiles
^ aStoredReportFiles readJson
) a report is a plain object: create it, inspect it, and only then decide whether to store it Image data GtReportImageData
Object << #GtReportImageData
slots: { #gtVersion . #gtReleaseTimestamp . #gtVmVersion . #pharoVmVersion . #pharoImageVersion };
package: 'GToolkit-Utility-Basic'
the GT, GT VM, Pharo VM and Pharo image versions every report says which image it came from, without identifying the user or the machine Recorder GtReportRecorder
Object << #GtReportRecorder
slots: { #imageData };
package: 'GToolkit-Utility-SpaceTally'
and its subclasses how one kind of report is recorded: #selector
recordReport
^ self subclassResponsibility
(the time-sensitive part) and #selector
completeReport: aReport
^ aReport
(the slow part) the same report can be recorded with other settings or other image data Recording GtReportRecording
Object << #GtReportRecording
slots: { #recorder . #store . #announcer . #process . #startedAt . #report . #entry . #error };
package: 'GToolkit-Utility-SpaceTally'
one background run of a recorder into a store; announces its start and end (GtReportRecordingStarted
GtReportRecordingChanged << #GtReportRecordingStarted
slots: {};
package: 'GToolkit-Utility-SpaceTally'
, GtReportRecordingEnded
GtReportRecordingChanged << #GtReportRecordingEnded
slots: {};
package: 'GToolkit-Utility-SpaceTally'
); holds the report and the entry, or the error recording must not block the UI, and a Message Tally must see what the UI does Store GtReportsStore
Object << #GtReportsStore
slots: {};
package: 'GToolkit-Utility-SpaceTally'
: GtReportsLocalStore
GtReportsStore << #GtReportsLocalStore
slots: { #rootFolder };
package: 'GToolkit-Utility-SpaceTally'
, GtReportsCompositeStore
GtReportsStore << #GtReportsCompositeStore
slots: { #stores };
package: 'GToolkit-Utility-SpaceTally'
, GtNoReportsStore
GtReportsStore << #GtNoReportsStore
slots: {};
package: 'GToolkit-Utility-SpaceTally'
where reports go: the folder layout, the storage id, writing text and bytes (#selector
write: aString to: aFileReference
aFileReference parent ensureCreateDirectory.
aFileReference writeStreamDo: [ :aStream | aStream nextPutAll: aString ].
^ aFileReference size
, #selector
writeBytes: aByteArray to: aFileReference
aFileReference parent ensureCreateDirectory.
aFileReference binaryWriteStreamDo: [ :aStream | aStream nextPutAll: aByteArray ].
^ aFileReference size
) a report does not know where it is kept Entry GtReportsStoreEntry
Object << #GtReportsStoreEntry
slots: { #store . #folder . #reportType . #storageId . #machineName . #files . #metadata };
package: 'GToolkit-Utility-SpaceTally'
with GtReportsStoreDataFile
GtReportsStoreFile << #GtReportsStoreDataFile
slots: {};
package: 'GToolkit-Utility-SpaceTally'
and GtReportsStoreMetadataFile
GtReportsStoreFile << #GtReportsStoreMetadataFile
slots: { #metadata };
package: 'GToolkit-Utility-SpaceTally'
one stored copy: its folder, the files written (size and write time) and the metadata shows what was written, without keeping the contents in memory Metadata GtReportMetadata
Object << #GtReportMetadata
slots: {
#formatVersion .
#reportType .
#reportClassName .
#reportAlias .
#reportUid .
#storageId .
#storedAt .
#machineName .
#imageData .
#files };
package: 'GToolkit-Utility-SpaceTally'
, GtReportFileDescription
Object << #GtReportFileDescription
slots: { #name . #numberOfBytes };
package: 'GToolkit-Utility-SpaceTally'
meta.json: report type, class and alias, report uid, storage id, machine name, image data, files with their sizes, format version reports can be found, grouped and sized without reading their data Reading back GtStoredReportFiles
Object << #GtStoredReportFiles
slots: { #folder . #metadata };
package: 'GToolkit-Utility-SpaceTally'
and the report's class-side #selector
fromStoredFiles: aStoredReportFiles
^ aStoredReportFiles readJson
turns a stored folder back into the report object an analysis works on the same objects that were recorded
Recording. #selector
recordInBackgroundInto: aStore
^ (self recordingInto: aStore) start
on a recorder starts a recording (#selector
start
startedAt := DateAndTime now.
self announceStarted.
process := [ self record ]
forkAt: recorder recordingPriority
named: 'Record ' , recorder recordingName.
). The recording runs #selector
recordReport
^ self subclassResponsibility
at the recording priority, above the UI (#selector
recordingPriority
^ Processor userInterruptPriority
); then it lowers its own priority to background and runs #selector
completeReport: aReport
^ aReport
and #selector
storeIn: aStore
^ aStore store: self
(#selector
record
[ report := recorder recordReport.
"The report is recorded at the recording priority, above the UI, so that it is not delayed by a busy UI.
Completing it (e.g. building the Message Tally text) and storing it (serializing, writing or uploading
files) are slow but not urgent: lower the priority first so that the UI stays responsive meanwhile."
Processor activeProcess priority: Processor userBackgroundPriority.
recorder completeReport: report.
entry := report storeIn: store.
self announceEnded ]
on: Error
do: [ :anError |
error := anError.
self announceEnded.
anError pass ]
). #selector
createReport
^ self completeReport: self recordReport
does the same synchronously, without storing.
Storing. #selector
storeIn: aStore
^ aStore store: self
on a report asks the store to store it (#selector
store: aReport
| entry |
entry := self entryFor: aReport storageId: self newStorageId.
aReport writeFilesTo: entry.
entry writeMetadataOf: aReport.
^ entry
): the store creates an entry with a new storage id (#selector
entryFor: aReport storageId: aStorageId
| aMachineName |
aMachineName := self machineNameFor: aReport.
^ GtReportsStoreEntry
store: self
folder: (self folderFor: aReport reportType storageId: aStorageId machineName: aMachineName)
reportType: aReport reportType
storageId: aStorageId
machineName: aMachineName
), the report writes its files into the entry (#selector
writeFilesTo: anEntry
anEntry writeJsonOf: self
), and the entry writes meta.json last (#selector
writeMetadataOf: aReport
| json |
metadata := self metadataFor: aReport.
json := LeJsonV4 uniqueInstance serializePretty: metadata.
^ self
record: (GtReportsStoreMetadataFile
metadata: metadata
destination: folder / GtReportMetadata fileName)
writing: [ :aDestination | store write: json to: aDestination ]
). Every write goes through the entry, which records the file with the number of bytes the store read back from the written file (#selector
record: aStoreFile writing: aBlock
aStoreFile writeWith: aBlock.
files add: aStoreFile.
^ aStoreFile
). A composite store creates one entry per store with the same storage id (#selector
entryFor: aReport storageId: aStorageId
^ GtReportsCompositeStoreEntry entries: (stores collect: [ :each |
each entryFor: aReport storageId: aStorageId ])
), so every store gets the same folder name.
Reading back. GtStoredReportFiles
Object << #GtStoredReportFiles
slots: { #folder . #metadata };
package: 'GToolkit-Utility-SpaceTally'
#selector
folder: aFolder
^ self new setFolder: aFolder
reads meta.json; #selector
report
^ self reportClass fromStoredFiles: self
takes the report class from the metadata (#selector
reportClass
^ Smalltalk globals at: metadata reportClassName asSymbol
) and asks it for #selector
fromStoredFiles: aStoredReportFiles
^ aStoredReportFiles readJson
. The folder layout and the content of meta.json are described under
Where reports are stored
on GT reports available for data collection.
- No common report superclass. Reports have little in common beyond a small protocol: #selector
reportType
^ 'processes'
, #selector
imageData
^ imageData
, #selector
reportUid
^ reportUid
, #selector
writeFilesTo: anEntry
anEntry writeJsonOf: self
, #selector
storeIn: aStore
^ aStore store: self
and the class-side #selector
fromStoredFiles: aStoredReportFiles
^ aStoredReportFiles readJson
. Each report implements it, so a new report does not have to fit a hierarchy.
- Recording in two steps. #selector
recordReport
^ self subclassResponsibility
must be quick and runs high, so that a busy UI does not delay it; completing and storing are slow and run at background priority, so that the UI stays usable meanwhile (#selector
record
[ report := recorder recordReport.
"The report is recorded at the recording priority, above the UI, so that it is not delayed by a busy UI.
Completing it (e.g. building the Message Tally text) and storing it (serializing, writing or uploading
files) are slow but not urgent: lower the priority first so that the UI stays responsive meanwhile."
Processor activeProcess priority: Processor userBackgroundPriority.
recorder completeReport: report.
entry := report storeIn: store.
self announceEnded ]
on: Error
do: [ :anError |
error := anError.
self announceEnded.
anError pass ]
). Only one Message Tally can run at a time.
- meta.json is written last (#selector
store: aReport
| entry |
entry := self entryFor: aReport storageId: self newStorageId.
aReport writeFilesTo: entry.
entry writeMetadataOf: aReport.
^ entry
). A folder without it is an interrupted write, never a report.
- Report type first in the layout (#selector
folderFor: aReportType storageId: aStorageId machineName: aMachineName
^ (self folderForType: aReportType) / (self dateOf: aStorageId) / aStorageId
). Most questions start from a kind of report. A store that keeps the reports of several machines can add a machine level to the layout; it gets the machine name for every report (#selector
machineNameFor: aReport
^ (aReport imageData machineName ifEmpty: [ self machineName ]) asLowercase
). Ids and date folders are in UTC, so that images in different time zones sort together.
- No identifying data in GT. GtReportImageData
Object << #GtReportImageData
slots: { #gtVersion . #gtReleaseTimestamp . #gtVmVersion . #pharoVmVersion . #pharoImageVersion };
package: 'GToolkit-Utility-Basic'
holds only versions and an empty machine name (#selector
machineName
^ ''
); a subclass can add more where that is wanted, and a recorder decides which image data its reports get (#selector
imageData: anImageData
^ self new setImageData: anImageData
).
- Two ids. The storage id names one stored copy (#selector
newStorageId
^ self utcTimestampString , '-' , (UUID new asString first: 8)
). The report uid is a UUID
Object << #UUID
slots: { #uuidData };
tag: 'Base';
package: 'Network-UUID'
given in the report's #selector
initialize
super initialize.
processReports := #().
imageData := GtReportImageData new.
reportUid := UUID new
and kept by every copy, so copies can be recognised and analysed once.
- Class name plus alias. The metadata names the report class (#selector
reportClassName
^ reportClassName
), to read the report back, and its #selector
leJsonV4Name
^ #gtProcessesReport
as a stable alias (#selector
reportAlias
^ reportAlias
), to read it back after the class is renamed.
- Nothing large in memory. Entries keep where and how much was written, not what (GtReportsStoreFile
Object << #GtReportsStoreFile
slots: { #destination . #numberOfBytes . #writtenAt . #writeDuration };
package: 'GToolkit-Utility-SpaceTally'
); the sizes come from the written files.
- The format version is a label. Every meta.json carries #selector
formatVersion
^ formatVersion
(#selector
currentFormatVersion
^ 1
); no code branches on it yet. When the format changes, read the version from the plain JSON first and pick a reader for it, because LeJsonV4
Object << #LeJsonV4
traits: {TGtUniqueInstance};
slots: { #newReader . #newWriter . #mutex };
tag: 'JsonV4';
package: 'Lepiter-Store'
reads the file into the current class before the version is known.
1. Write the report class, without a superclass, following GtProcessesReport
Object << #GtProcessesReport
slots: { #processReports . #imageData . #reportUid };
package: 'GToolkit-Utility-SpaceTally'
:
- slots for the data, plus imageData and reportUid; in #selector
initialize
super initialize.
processReports := #().
imageData := GtReportImageData new.
reportUid := UUID new
set imageData := GtReportImageData new (GtReportImageData
Object << #GtReportImageData
slots: { #gtVersion . #gtReleaseTimestamp . #gtVmVersion . #pharoVmVersion . #pharoImageVersion };
package: 'GToolkit-Utility-Basic'
) and reportUid := UUID new;
- #selector
reportType
^ 'processes'
answers the folder name, lowercase with dashes;
- #selector
writeFilesTo: anEntry
anEntry writeJsonOf: self
writes the files: #selector
writeJsonOf: anObject
^ self writeJsonOf: anObject named: reportType
for most reports, #selector
writeText: aString extension: anExtension
^ self writeText: aString named: reportType extension: anExtension
for text, #selector
writeGzippedStonOf: aReport
^ self writeBytes: aReport gzippedSton extension: 'ston.gz'
for large object graphs, #selector
writeBytes: aByteArray named: aName extension: anExtension
^ self
record: (GtReportsStoreDataFile destination: folder / (aName , '.' , anExtension))
writing: [ :aDestination | store writeBytes: aByteArray to: aDestination ]
for several files of one kind;
- the class-side #selector
fromStoredFiles: aStoredReportFiles
^ aStoredReportFiles readJson
reads them back (#selector
readJson
^ self readJsonNamed: metadata reportType
, #selector
readText: anExtension
^ self readTextNamed: metadata reportType extension: anExtension
, #selector
readGzippedSton
^ STON fromString: (GZipReadStream on:
(self readBytesNamed: metadata reportType extension: 'ston.gz')) upToEnd utf8Decoded
, #selector
readBytesNamed: aName extension: anExtension
^ (self fileNamed: aName extension: anExtension)
binaryReadStreamDo: [ :aStream | aStream upToEnd ]
);
- the class-side #selector
leJsonV4Name
^ #gtProcessesReport
gives a stable name; run LeJsonV4
Object << #LeJsonV4
traits: {TGtUniqueInstance};
slots: { #newReader . #newWriter . #mutex };
tag: 'JsonV4';
package: 'Lepiter-Store'
#selector
cleanUniqueInstance
uniqueInstance ifNotNil: [ :anInstance |
self onRemoveUniqueInstance: anInstance ].
uniqueInstance := nil.
after adding it or after changing slots. The JSON reader sets every written slot through a setter of the same name, so each one needs a setter (e.g. #selector
reportUid: aUUID
reportUid := aUUID
); a class-side leJsonV4AttributeMapping leaves out slots that must not be written (like the live tally of GtMessageTallyReport>>#leJsonV4AttributeMapping
leJsonV4AttributeMapping
^ #(#imageData #reportUid #startTime #duration #pollPeriodMilliseconds #numberOfSamples #measuredMilliseconds #vmParametersDiff)
collect: [ :each | each -> each ]
);
- #selector
storeIn: aStore
^ aStore store: self
answers aStore store: self; add #selector
printOn: aStream
super printOn: aStream.
aStream parenthesize: [
aStream
print: processReports size;
nextPutAll: ' processes, ';
print: (processReports select: [ :each | each isQuiescent ]) size;
nextPutAll: ' quiescent' ]
and a summary view, so that the report is useful in the inspector before it is stored.
2. Write a recorder only when recording needs settings or must run in the background: subclass GtReportRecorder
Object << #GtReportRecorder
slots: { #imageData };
package: 'GToolkit-Utility-SpaceTally'
and implement #selector
recordReport
^ self subclassResponsibility
, plus #selector
completeReport: aReport
^ aReport
for slow work (e.g. GtMessageTallyReportRecorder
GtReportRecorder << #GtMessageTallyReportRecorder
slots: { #duration . #pollPeriodMilliseconds };
package: 'GToolkit-Utility-SpaceTally'
).
3. Check it with examples that store the report in a memory store (GtReportsLocalStore
GtReportsStore << #GtReportsLocalStore
slots: { #rootFolder };
package: 'GToolkit-Utility-SpaceTally'
#selector
rootFolder: aFileReference
^ self new setRootFolder: aFileReference
with FileSystem memory root / 'gt-reports') and read it back (see GtReportsStoreExamples
Object << #GtReportsStoreExamples
slots: {};
package: 'GToolkit-Utility-SpaceTally'
and GtStoredReportFilesExamples
Object << #GtStoredReportFilesExamples
slots: {};
package: 'GToolkit-Utility-SpaceTally'
).
4. Document it: add a section to GT reports available for data collection, with one sentence on what the report shows, a snippet that creates it and one that stores it.
The smallest report, as a sketch (accessors and views left out):
Object << #GtExampleDataReport
slots: { #values . #imageData . #reportUid };
package: 'GToolkit-Utility-SpaceTally'
GtExampleDataReport >> initialize
super initialize.
values := #().
imageData := GtReportImageData new.
reportUid := UUID new
GtExampleDataReport >> reportType
^ 'example-data'
GtExampleDataReport >> writeFilesTo: anEntry
anEntry writeJsonOf: self
GtExampleDataReport >> storeIn: aStore
^ aStore store: self
GtExampleDataReport class >> leJsonV4Name
^ #gtExampleDataReport
GtExampleDataReport class >> fromStoredFiles: aStoredReportFiles
^ aStoredReportFiles readJson
A store decides where reports go, for example a remote file service. Subclass GtReportsStore
Object << #GtReportsStore
slots: {};
package: 'GToolkit-Utility-SpaceTally'
and implement:
- #selector
rootFolder
^ self subclassResponsibility
: the root of the layout, any object that answers / like a file reference;
- #selector
write: aString to: aDestination
^ self subclassResponsibility
and #selector
writeBytes: aByteArray to: aDestination
^ self subclassResponsibility
: write the text or bytes and answer the number of bytes written, read back from the written file (as #selector
write: aString to: aFileReference
aFileReference parent ensureCreateDirectory.
aFileReference writeStreamDo: [ :aStream | aStream nextPutAll: aString ].
^ aFileReference size
does), because that number goes into the metadata;
- #selector
description
^ self subclassResponsibility
: a short name shown in the entries and recordings;
- optionally #selector
machineName
^ ''
and #selector
folderFor: aReportType storageId: aStorageId machineName: aMachineName
^ (self folderForType: aReportType) / (self dateOf: aStorageId) / aStorageId
, to add a machine level to the layout.
The entries, file objects, metadata and composite stores then work without changes.