← All projects
LLM evaluation tooling

insideLLMs

A Python library and CLI for comparing model behaviour through shared probes, recorded runs and inspectable differences.

insideLLMs terminal example comparing baseline and candidate runs and identifying two regressions in dummy-model outputs.
A frame from the repository’s CLI demonstration using dummy-model data.

The idea

A model score can hide changes that matter to a product. insideLLMs provides a way to run shared probes across models and retain the resulting inputs, outputs and run artifacts for comparison.

The approach

The emphasis is on inspectable evidence: run a defined experiment, preserve its artifacts and compare the result with a baseline. Deterministic artifact handling makes identical recorded inputs and responses comparable; it does not make live model generation deterministic.

  • A shared interface for model and probe comparisons
  • Recorded run artifacts and behavioural diffs
  • An offline dummy-model path for exploring the workflow

Project details and current setup instructions: GitHub repository .