AI Plot-by-Code example (v1 design)¶
Status: implemented
Date: 2026-04-24
Scope: a first-class panel subpackage at src/panelini/panels/ai/plot/ plus a runnable example at examples/panels/ai/plot_by_code.py. The subpackage exposes a PlotPanel + two tool-factory functions (make_plot_tools, make_osw_tools) that plug into the existing AiChat unchanged.
Goals¶
Migrate the legacy
migration/agent.py+migration/osw_tools.pyintegration (built on a bespokeHistoryToolAgent+AgentExecutor) onto panelini’s existing AiChat backend. No new chat framework.Ship a self-contained example at examples/panels/ai/plot_by_code.py that lets a user ask for a matplotlib plot, runs the generated code in a
python:3.12-slimDocker sandbox viallm_sandbox, and renders the resulting PNG in a panel to the right of the chat.Register up to 5 plotting tools (
plot_by_code,run_code,load_data_from_csv,attach_current_plot_to_osw_page,document_current_evaluation) plus 8 OSW connector tools (get_page_html,download_osl_file,sparql_search,find_out_everything_about,get_topic_taxonomy,get_instances,get_file_header,get_website_html). All 13 are optional depending on which extras are installed and which env vars are set.Graceful degradation: the example must run end-to-end when Docker + an LLM API key are the only prerequisites. Missing
oswpackage → drop the two OSW-plot tools. Missing OSW env vars → drop the eight stateless OSW tools. No ImportError onpip install 'panelini[ai,ai-llm-sandbox]'alone.Honest typing story: the upstream
oswpackage ships nopy.typedmarker. Solve the mypy side of this without a global[[tool.mypy.overrides]] ignore_missing_imports = trueoverride — the trade-off must be visible in source.Reuse
AiChatend-to-end (sidebar provider/model picker, chat history, streaming, tool bind viamodel.bind_tools). No changes tosrc/panelini/panels/ai/frontend.pyorbackend.py.
Non-goals (v1)¶
Not a replacement for
migration/osw_chatbotcoupling —OswFrontendPanel,ChatFrontendWidget,call_client_side_toolare dropped. Any future OSW-side widget integration comes from the OSW project, not panelini.No port of
HistoryToolAgent/AgentExecutor.AiBackendalready owns history + dispatch.No OSW file-upload UI widget in the example — plots are uploaded to OSW only when the chat model decides to call
attach_current_plot_to_osw_page/document_current_evaluation.No
.drawio.svgsupport (that is a different example, see drawai-beautify spec).No in-browser Docker visualization — sandbox logs stay server-side; the user sees only the resulting PNG.
No port of langchain v1
pydantic.v1schemas — input schemas are migrated to pydantic v2 to match basic_tools.py.
User flow¶
User runs
python examples/panels/ai/plot_by_code.py.Browser opens the Panelini page. Sidebar shows the standard
AiChatcontrols (provider, model, tools, chat management). The tool list includes three or five plotting tools and zero or eight OSW tools depending on env.User types “plot y = sin(x) for x in 0..2π, save to /sandbox/output.png”.
Chat model picks
plot_by_code, passes a Python snippet. The tool delegates toPlotPanel.plot_by_code(code, libraries=[...]), which:starts a
SandboxSession(lang="python", image="python:3.12-slim");copies any listed files into
/sandbox/<BASENAME>, pairing MICRESS.geofwith.conc1binaries;runs the code, copies
/sandbox/output.pngback todata/outputs/output.png;validates with
PIL.Image.open(catches non-PNG output);points
pn.pane.Imageat the path.
PNG appears in the right pane. Chat answers with the sandbox stdout.
User iterates (“make it dashed red”). Each successful call overwrites
output.png.Optional: if OSW env vars +
osware available, chat can upload the current plot and attach it to an OSW page viaattach_current_plot_to_osw_page.
Architecture¶
Layout¶
┌─ Panelini header ─────────────────────────────────────────────────┐
├─ Sidebar ──────┬─ Main ────────────────────────────────────────── ┤
│ (from AiChat: │ ┌─ Chat (left) ──┐ ┌─ Plot (right) ───────────┐ │
│ provider, │ │ │ │ │ │
│ model, │ │ ChatInterface │ │ pn.pane.Image │ │
│ tools, │ │ (streaming, │ │ (rendered from │ │
│ chat mgmt) │ │ history) │ │ data/outputs/ │ │
│ │ │ │ │ output.png) │ │
│ │ │ │ │ │ │
│ │ └────────────────┘ └──────────────────────────┘ │
└────────────────┴──────────────────────────────────────────────────┘
Outer:
Panelini(title="AI + Plot by Code", sidebar_enabled=True).Sidebar:
AiChat.sidebar_objectspassed through unchanged.Main:
pn.Row(chat.main_objects[0], plot_panel.plot_panel)— no custom widgets, no extra cards. ThePlotPanelIS the right column.
Package layout¶
src/panelini/panels/ai/plot/
├── __init__.py # re-exports PlotPanel, make_plot_tools, make_osw_tools
├── panel.py # PlotPanel class (sandbox + pn.pane.Image)
├── utils/
│ ├── __init__.py
│ ├── sandbox.py # resolve_file_path, copy_files_to_sandbox, MICRESS pairing
│ └── osw_env.py # OSW_ENV_VARS, osw_env_present()
└── tools/
├── __init__.py
├── plot_tools.py # 3 delegation tools + make_plot_tools() factory
├── osw_plot_tools.py # 2 plot-bound OSW tools (lazy-imported by plot_tools)
└── osw_tools.py # 8 stateless OSW tools + make_osw_tools() factory
Component boundaries¶
Each unit has a single purpose, a narrow interface, and deliberate dependencies on upstream packages:
Unit |
Purpose |
Interface |
Depends on |
|---|---|---|---|
|
Holds mutable plot state; runs code inside |
|
|
|
Copy host files into |
|
stdlib only |
|
Resolve a bare filename against |
|
stdlib only |
|
Gate the 8 stateless OSW tools behind |
|
stdlib |
|
|
langchain |
|
|
Factory returning |
|
|
|
Read |
|
|
8 stateless OSW tools ( |
|
|
|
|
Factory — returns |
|
|
PlotPanel state¶
class PlotPanel:
data_path: Path # base data dir (default: cwd / "data")
download_dir: Path # downloaded OSW files land here
docker_image: str # "python:3.12-slim"
df: pandas.DataFrame | None # last CSV loaded via load_data_from_csv
current_python_code: str | None # last successful plot_by_code snippet
current_input_osw_id: str | None # "File:<basename>" from the last plot's inputs
output_file_path: Path | None # data/outputs/output.png
image_panel: pn.pane.Image
plot_panel: pn.Row # the servable widget
State mutates through plot_by_code / run_code / load_data_from_csv. The two OSW-plot tools (attach_current_plot_to_osw_page, document_current_evaluation) read — but never mutate — current_python_code and current_input_osw_id.
The mypy story¶
osw upstream ships no py.typed marker, so mypy emits “missing library stubs” for every from osw.* import .... The knee-jerk fix is a global [[tool.mypy.overrides]] module = ["osw.*"] ignore_missing_imports = true. That hides the cost of the dependency and is easy to forget when the package eventually ships types.
Instead, every module that imports from osw uses the TYPE_CHECKING: Any boundary pattern locally:
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
OswExpress: Any = Any
model: Any = Any
# ... etc
else:
from osw.express import OswExpress
from osw.core import model
The trade-off (mypy sees Any, runtime sees the real types) is visible at the boundary where the osw import happens. When osw ships py.typed, the TYPE_CHECKING branch becomes a real import and the Any aliases are deleted.
Combined with splitting the tools into plot_tools.py (no osw imports — importable in any environment) + osw_plot_tools.py / osw_tools.py (both import osw at module top), this keeps mypy happy without a global override.
The graceful-degradation story¶
Two independent axes:
Axis |
Gated by |
Tools affected |
Mechanism |
|---|---|---|---|
|
|
|
|
OSW env vars set? |
|
All 8 stateless OSW tools |
|
The tools — input schemas (pydantic v2)¶
All three input models live in plot_tools.py:
class PlotByCodeInput(BaseModel):
code: str = Field(..., description="Python that saves a PNG to /sandbox/output.png.")
file_paths: list[str] | None = None
libraries: list[str] | None = None # defaults to numpy, pandas, matplotlib, scipy
class RunCodeInput(BaseModel):
code: str
lang: str = "python"
file_paths: list[str] | None = None
libraries: list[str] | None = None
class LoadCsvInput(BaseModel):
file_path: str
delimiter: str = "\t"
skip_rows: int = 0
osw_plot_tools.py and osw_tools.py define their own input schemas (AttachPlotInput, DocumentEvaluationInput, GetPageHtmlInput, DownloadOslFileInput, SparqlSearchInput, FindOutEverythingAboutInput, GetTopicTaxonomyInput, GetInstancesInput, GetFileHeaderInput, GetWebsiteHtmlInput). All use pydantic v2; BaseTool subclasses pass arbitrary_types_allowed=True via ConfigDict so the panel: PlotPanel field type-checks.
The plot_by_code sandbox contract¶
Documented in the example’s system message and enforced by convention:
plot_by_code MUST save its figure to '/sandbox/output.png'.
Files passed via file_paths are available at '/sandbox/<BASENAME>'.
If the snippet saves elsewhere, session.copy_from_runtime("/sandbox/output.png", ...) fails, the panel returns "Exception during plotting: ...", and the model retries.
MICRESS/micpy guidance is also included in the system message — it is the one piece of prompt from migration/agent.py:636-656 worth preserving.
Dependencies¶
Three optional extras, two existing and one new:
[project.optional-dependencies]
ai = [ "langchain>=0.3.27", "langchain-anthropic>=0.3.0",
"langchain-community>=0.3.27", "langchain-openai>=0.3.32",
"python-dotenv>=1.0.0", "pyyaml>=6.0" ] # unchanged
ai-llm-sandbox = [ "llm-sandbox>=0.2", "pillow>=10.0" ] # unchanged
ai-osw = [ # NEW
"osw>=1.0",
"SPARQLWrapper>=2.0",
"pandas>=2.0.0",
"python-dotenv>=1.0.0",
]
osw is also added to the [dependency-groups.dev] list so mypy and the test suite can resolve OSW symbols even when a contributor hasn’t installed the [ai-osw] extra.
README entry: pip install panelini[ai,ai-llm-sandbox] for the basic case; add ai-osw for the OSW tools.
Error handling¶
All user-facing errors are surfaced as chat tool-result strings, never raised. The AI model decides whether to retry or inform the user.
Failure |
Where caught |
Handling |
|---|---|---|
Docker daemon unreachable |
|
Propagates — the caller (chat backend) catches and reports. Example’s system message instructs the user to start Docker. |
Snippet produces no |
|
Returned as |
Snippet saves corrupt PNG |
|
Same wrapper as above — caller sees the PIL error. |
|
|
Returned as |
OSW entity not found for |
|
Returns |
OSW / SPARQL network failure |
Per-tool |
Returns |
OSW env vars missing at factory time |
|
Returns |
|
|
The two OSW-plot tools are silently omitted; the three core tools stay. |
Nothing silently swallows an error; every caught exception is surfaced to the chat model as a string containing e.
Testing¶
Tests live at tests/panels/ai/plot/ and are fully self-contained — no Docker, no OSW endpoint, no network.
File |
Scope |
Key mocks |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
The three delegation tools call through to panel methods with the expected kwargs. |
|
|
|
|
|
Pure helpers ( |
|
Total: 6 test files, ~900 LOC, 80+ tests — all pass in <2 s without Docker. Coverage goal: every public function on PlotPanel, every _run method on every tool, every happy path + at least one error path.
pytest-asyncio is already a dev dep. _arun tests reuse the ThreadPoolExecutor escape from test_tools.py only if needed — most _arun methods here simply delegate to _run and test the sync path instead.
File layout summary¶
New files in src/:
src/panelini/panels/ai/plot/__init__.pysrc/panelini/panels/ai/plot/panel.pysrc/panelini/panels/ai/plot/utils/__init__.pysrc/panelini/panels/ai/plot/utils/sandbox.pysrc/panelini/panels/ai/plot/utils/osw_env.pysrc/panelini/panels/ai/plot/tools/__init__.pysrc/panelini/panels/ai/plot/tools/plot_tools.pysrc/panelini/panels/ai/plot/tools/osw_plot_tools.pysrc/panelini/panels/ai/plot/tools/osw_tools.py
New files in tests/:
tests/panels/ai/plot/__init__.pytests/panels/ai/plot/test_plot_panel.pytests/panels/ai/plot/test_plot_tools.pytests/panels/ai/plot/test_osw_plot_tools.pytests/panels/ai/plot/test_osw_tools.pytests/panels/ai/plot/test_sandbox_utils.pytests/panels/ai/plot/test_osw_env.py
New example:
examples/panels/ai/plot_by_code.py
Modified:
pyproject.toml— add[ai-osw]optional extra; addosw>=1.0.1to[dependency-groups.dev].examples/panels/ai/README.md— one-line entry pointing at the new example.uv.lock— regenerated.
No changes to src/panelini/panels/ai/{frontend,backend,config,utils}.py.
v1.2 — OSW credential handling via environment variables¶
Motivation¶
The upstream osw.express.OswExpress.__init__ calls input() / getpass.getpass() on a fresh machine (no osw_files/accounts.pwd.yaml yet), and writes the entered credentials to that yaml on first use. Both behaviours are hostile to a Panel server: the example would block on stdin instead of serving, and the resulting yaml would ship credentials to disk outside the .env file the user explicitly chose as their credential store.
Design¶
A new helper module src/panelini/panels/ai/plot/utils/osw_env.py centralises credential wiring:
OSW_AUTH_ENV_VARS = ("OSW_DOMAIN", "OSW_USER", "OSW_PASSWORD")— three vars required together.OSW_ENV_VARSnow includes those three plus the threeBLAZEGRAPH_*vars (was three before).check_osw_auth_env()raisesRuntimeErrorlisting every missing auth var — crafted to be actionable inside a.envfile.class EnvCredentialManager(CredentialManager)overrides onlyiri_in_file(iri)toreturn self.iri_in_credentials(iri) or super().iri_in_file(iri). This tellsOswExpress.__init__“you already have creds on disk for this domain” even though they live only in memory — bypassing both the prompt and the yaml write in one change.build_osw_express(domain=None)validates auth, reads env, constructs anEnvCredentialManagerpre-loaded with aCredentialManager.UserPwdCredential, and passes it toOswExpress(domain=..., cred_mngr=...).
All five OswExpress(domain=os.environ.get("OSW_DOMAIN")) callsites (2 in osw_plot_tools.py, 1 in osw_tools.py, plus 1 module-level factory path in osw_tools.DownloadOslFileTool which used the standalone osw_download_file function) route through build_osw_express(). DownloadOslFileTool now uses osw_obj.download_file(...) rather than the module-level helper to avoid the “no cred_mngr parameter” escape hatch.
Contract additions¶
Aspect |
Before |
After |
|---|---|---|
OSW auth env vars |
Undocumented; tools ran with |
|
OSW env gate ( |
4 vars ( |
6 vars (3 auth + 3 Blazegraph) |
Credential file on disk |
Written on first use to |
Never written — |
CLI prompt on fresh machine |
|
Never prompted — same short-circuit blocks |
Test contract¶
A direct integration test (TestNoCredentialsFileWritten) exercises the real OswExpress.__init__ with only requests.get and WtSite patched. It asserts that neither EnvCredentialManager.save_credentials_to_file nor EnvCredentialManager.get_credential is invoked, and that no osw_files/accounts.pwd.yaml appears in a tmp_path cwd after build_osw_express() returns. This catches any future regression that removes the iri_in_file override.
Files touched in v1.2¶
New:
src/panelini/panels/ai/plot/utils/osw_env.pyNew:
tests/panels/ai/plot/test_osw_env.py(18 tests)Edited:
src/panelini/panels/ai/plot/tools/osw_plot_tools.py(route throughbuild_osw_express, dropos, dropOswExpressimport)Edited:
src/panelini/panels/ai/plot/tools/osw_tools.py(same rewire;DownloadOslFileToolnow callsosw_obj.download_file)Edited:
tests/panels/ai/plot/test_osw_tools.py+test_osw_plot_tools.py(patchbuild_osw_expressinstead ofOswExpress/osw_download_file)Edited:
examples/panels/ai/plot_by_code.py(env-var docstring updated with auth vars + credential-handling note)Edited:
examples/panels/ai/README.md(one-liner updated for six-var requirement and in-memory credentials)
Explicit trade-offs¶
Subclass
iri_in_file, don’t monkey-patch. The override is a two-line method on a dedicated subclass. A module-level monkey-patch would affect any third-party code sharing the process; the subclass only lies about its own state.OSW_USER/OSW_PASSWORDare required hard, not optional. If the user has anaccounts.pwd.yamlalready, we still require env vars — the prior unified path avoids divergent behaviour between dev machines and CI. A graceful “fall back to yaml if present” would re-open the prompt path.No yaml writes is a first-class contract, not an implementation detail. Hence the dedicated
TestNoCredentialsFileWrittentest — if a contributor ever “helpfully” removes theiri_in_fileoverride thinking it’s dead code, the test fails immediately.
Follow-ups (explicitly out of scope for v1)¶
osw_tools.pytop-levelfrom osw.express import ...meansfrom panelini.panels.ai.plot import make_osw_toolsraises ImportError when theoswpackage is absent. The cleaner alternative is to lazy-import insidemake_osw_tools()(mirroringmake_plot_tools’s try/except forosw_plot_tools). Deferred to a follow-up — the[ai-osw]extra covers the runnable path and contributors install it via the dev dep.Promote MICRESS/micpy guidance out of the example system message into a reusable
PLOT_SYSTEM_MESSAGEconstant once a second consumer appears.Render a live “sandbox log” tab next to the plot for debugging long-running snippets.
Port the OSW
call_client_side_toolhook frommigration/if panelini ever needs to drive a page-side widget from the chat.Sphinx docs page at
docs/panels/ai.mdcovering this panel + layout diagram.Drop the
TYPE_CHECKING: Anyboundary when upstreamoswshipspy.typed.