Recording What an Agent Does: Interaction Tracking with Socratic Learning
Socratic Learning is part of an earlier research prototype. It is not maintained as a product. This article describes what the code does. It reports no results from using it, and the earlier version of this article, which gave improvement figures, has been withdrawn because those figures were not measured.
An agent that never records what it did cannot be improved on evidence. If a model gives a poor answer on Tuesday, you need to know the input, the output, the model, the cost and, if you have it, what the user thought. Without that, "the agent got worse" is a feeling.
socratic-learning is a small Python library for keeping that record and reading it back. It does not retrain a model by itself. It stores the data and gives you simple tools to look at it.
What it does
According to its README:
• Interaction tracking: each call is stored with its input, output, model, provider, token counts and duration, inside a session.
• User feedback: a rating and a comment can be attached to an interaction.
• Metrics: success rate, average response time, average rating and cost, per agent.
• Pattern detection: recurring patterns in behaviour or errors.
• Recommendations: suggestions derived from the patterns and metrics.
• Export for fine-tuning: interactions above a chosen rating can be written to a JSONL file in OpenAI's format.
• Storage: an abstract interface with an SQLite backend.
Example
from socratic_learning import LearningManager
learning = LearningManager(storage="sqlite", db_path="learning.db")
session_id = learning.create_session(user_id="user123", context={})
interaction = learning.track_interaction(
session_id=session_id,
agent_name="MyAgent",
input_data={"topic": "recursion"},
output_data=result,
model_name="a-model",
provider="a-provider",
input_tokens=150,
output_tokens=500,
duration_ms=1200.0,
)
learning.add_feedback(interaction_id=interaction.interaction_id, rating=5, feedback="Clear")
metrics = learning.get_metrics(agent_name="MyAgent")
patterns = learning.detect_patterns(agent_name="MyAgent")
recommendations = learning.get_recommendations(agent_name="MyAgent")
Questions to ask before relying on a record like this
• Does the feedback measure what you care about? A click is not agreement, and a rating is not correctness. Decide what a good outcome is before you collect data on it.
• Can you attribute a result to a change? If you change a prompt and the model on the same day, a better rating tells you little.
• Do you keep user text? Interaction records can contain personal data. Store only what you need and say so in a privacy policy. In my current products, nothing the user writes goes into a log at all.
• Is the recommendation a hypothesis? Treat what the library suggests as something to test, not a finding.
I have not measured whether using this loop improves an agent. The library gives the loop its first part, the record. The judgement about what to change stays with a person.
The code is open source: github.com/Nireus79/Socratic-learning, and the package is on PyPI.