I’ve been working on an open-source MVP, a web UI for DLT that brings low/no-code and AI-assisted data pipeline management to life!
The idea is to have for DLT (dltHub) what Databricks is for Spark/PySpark, a UI/X layer focused on designing/diagraming, scheduling, managing and monitoring data pipelines.
The MPV1 is already live in the cloud for testing and validation, check it out and see some of the features in action in the videos below!
This MVP already lets you:
- Design/Create pipelines through a low/no-code
- Query ingested data on DuckDB with an AI Agent with a full SQL editor
- Schedule pipeline
- Pipelines versioning (to support safe iteration/rollback in the future)
- Monitor pipeline run
- App-level DuckDB lock management for safe concurrency
Built with Still.js on DLT’s (dltHub) powerful foundation, this MVP sets a solid base for a great product ahead.
I’d love to get the community insights, you can test it in the cloud or deploy it locally (see github repo readme for instructions), explore the code, open issues, and contribute new ideas, coding and new features. Let’s build this together!
Check the code: Github Repo
Videos with some features show case:
- Creating a basic pipeline
- Scheduling a job for pipeline
- Pipeline versioning, Data transformation and File system sourceing