[ZEPPELIN-6434] Add MCP server for operating notebooks from AI agents

[kkalyan]

What is this PR for?

Adds zeppelin-mcp, a Model Context Protocol (MCP) server that lets AI coding
agents — Claude Code, Kiro, Codex, Cursor, and any other MCP client — operate an
Apache Zeppelin instance: select or create a notebook, run paragraphs in any
interpreter, read results, delete paragraphs, clear output, restart a stuck
interpreter, and write a summary back into the notebook.

The server is a thin client over the existing Zeppelin REST API, so it works
against any reachable Zeppelin server without server-side changes. It is
implemented in Python on the official MCP SDK (FastMCP), mirroring the existing
python/ module's precedent for Python-in-the-reactor.

This PR also adds zeppelin-mcp/GETTING_STARTED.md, a build-from-source guide
for the current experimental phase: prerequisites, venv install, a stdio smoke
test, the env-var configuration table, MCP client wiring for Claude Code / Kiro
/ Codex / generic clients, the tool reference, and troubleshooting.

What type of PR is it?

Feature

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-6434

How should this be tested?

mvn test -pl zeppelin-mcp creates a venv under target/venv and runs the
pytest suite (61 tests, 97% line coverage) against an in-memory fake Zeppelin —
no live server required. RAT passes (0 unapproved licenses).

For a manual end-to-end check, point an MCP client at a local Zeppelin:

claude mcp add zeppelin --env ZEPPELIN_URL=http://localhost:8080 -- zeppelin-mcp

Open items (tracked for follow-up, not blocking)

• Binary distribution. Whether to bundle the zeppelin-mcp console script
  inside the Zeppelin binary distribution vs. publishing to PyPI for
  pip install. Currently build-from-source only.
• Standard distribution channel — PyPI release cadence and versioning.
• Semantic search as agent knowledge. Optionally index notebook/paragraph
  text plus interpreter metadata so an agent can discover relevant notebooks by
  meaning (e.g. a search_notebooks tool) and build richer context, rather than
  only operating on an explicitly selected note. Not implemented today.

Questions:

• Does the license files need to update? No
• Is there breaking changes for older versions? No
• Does this needs documentation? Module README + GETTING_STARTED.md added; a
  docs page under docs/ can follow.

[kkalyan]

@jongyoul gentle ping to get feedback on this.

[jongyoul]

@kkalyan Hello, I think we'd better provide a remote MCP server in the instance. Zeppelin runs as a web service so I don't believe that we have to have separate instances for MCP server. WDYT? Moreover, I recommend writing some documents for this feature and need to discuss with community first.

[kkalyan]

Thanks @jongyoul, this is great feedback — agreed on all three points.

I've added a design doc at zeppelin-mcp/DESIGN.md (in the latest commit) that works through the architecture question in detail. Short version:

On a remote in-instance MCP server: I agree that's the right long-term home. Since Zeppelin is already a web service, a remote MCP endpoint hosted by zeppelin-server over Streamable HTTP means no per-user install, one shared auth surface (reuse Shiro), multi-user by construction, and access to in-process state the REST API doesn't publish. The doc captures this as the Phase 2 target.

The reason this PR starts with a standalone stdio server is phasing, not disagreement: it ships with no server-side changes, works against any existing Zeppelin (including already-deployed older versions), uses the mature Python MCP SDK, and lets us stabilize the tool surface before committing it to the server's release cadence. Crucially the tool definitions + orchestration logic are transport-agnostic — the same tools move from stdio to HTTP without redesign — so Phase 1 is a stepping stone, not throwaway work.

One nuance worth surfacing: a useful MCP server isn't a 1:1 re-serialization of the REST API — the value is the orchestration layer (synchronous run semantics, stuck-interpreter detection, actionable errors, output shaping, cross-system diagnostics). That layer is identical regardless of where the server is hosted.

On docs + community discussion: agreed — I'll start a [DISCUSS] thread on dev@ linking the doc, JIRA, and this PR, and add a user-facing page under docs/. Will hold merge until that conversation lands.