Designing a conversational data-platform assistant

Problem

Platform users need to find datasets, understand lineage and ownership, and investigate application and job status. The evidence is spread across catalog, documentation, data storage, and operational systems.

My contribution

I designed a conversational entry point for those questions and mapped the sources it would consult. The proposal connected catalog and lineage, documentation, Spark execution history, and infrastructure evidence through a governed agent harness.

Proposed workflow

Google Chat or a chatbot embedded in the platform console would provide the entry point. An ingress layer and Pub/Sub would connect requests to a Pi-based harness with runbooks and workflows.

MCP connectors would expose OpenMetadata, Confluence, and Spark History Server. Tools for BigQuery, Google Cloud Storage, Kubernetes, logging, and monitoring would supply additional evidence. Documentation would be retrieved when a question was asked.

The assistant would use read-only tools, scoped permissions, auditable calls, and escalation when the available evidence could not answer the question. The proposal does not authorize the assistant to modify workloads.

Prerequisites and open decisions

Status: architecture proposal documented in September 2026. Implementation prerequisites remained pending.

• Complete catalog and lineage ingestion.
• Update the platform documentation.
• Deploy and configure Spark History Server with event logs.

Pilot scope, channel rollout, identity handling, and runtime choices were still open. These prerequisites and decisions must be resolved before the design can be evaluated as an operating service.

Evidence boundary

Production results and adoption metrics have not been established for this assistant. The scale and cost reductions in the data-platform case study describe the underlying platform, not results of this proposal.

Team credit

The proposal builds on an existing data platform operated by engineering teams. It describes my solution design and does not claim a deployed assistant or implementation by those teams.