Publish pipelines from Git

A team keeps its pipelines as files in Git and publishes them to a stage with one command. This page walks one example end to end: a bundle with one SQL pipeline that you describe, try in your own development stage, check in a pull request and publish to production, and keep in step with Git afterwards. Everything on this page is built; the last section says what is not.

You need the stardelt package (install and sign in) and, for everything except init and validate, a host and a sign-in: stardelt login, or STARDELT_TOKEN in a CI job.

Your group also needs a pipeline identity: the service identity pipeline-<group> that its pipelines run as, which an administrator creates once per group: a ServicePrincipal named pipeline-<group> with spec.ownerGroup: <group>. Until it is ready, plan and deploy refuse with No principal for this group yet and name the command to start with (make sync-principal NAME=pipeline-<group> GROUP=<group>). In the platform checkout, make demo-access does all of it for the group demo (GROUP= names another one) and also checks that the Catalogs bronze, silver and gold exist.

1. Describe what you publish

Every key and every command is listed on Bundle reference; this section shows the ones you need first.

A bundle is a directory in Git that says what a team publishes: the pipeline files, the notebooks and SQL they run, and one target per stage (dev, test, prod). init writes a small one that passes validate; neither command needs a host or a sign-in, so a pull request can run them:

stardelt bundle init shop --name shop-etl --group shop       # writes bundle.yaml, resources/daily.yaml, sql/daily.sql
stardelt bundle validate shop                                # checks every target
stardelt bundle validate shop -t prod                        # checks one

dev (development): daily
prod (production): daily

bundle.yaml names the bundle and its owner group, the variables (${var.catalog} in a pipeline file is replaced per target), the pipeline files (include), the files they read (files), the connections the stage must have (names only, never values) and the targets (shown here in short form; init writes the same in longer form):

bundle: {name: shop-etl, ownerGroup: shop}
variables: {catalog: {default: warehouse}}
include: [resources/*.yaml]       # one pipeline per file, the shape `stardelt apply -f` reads
files: [sql/*.sql]
targets:
  dev:  {mode: development, variables: {catalog: warehouse}}
  prod: {mode: production,  variables: {catalog: warehouse}, runAs: pipeline-shop}

resources/daily.yaml is one pipeline. Its statement lives in a file of the bundle and is read in when you deploy:

name: daily
sqlFile: sql/daily.sql
params:
  catalog: "${var.catalog}"
schedule:
  cron: "0 3 * * *"
  timezone: UTC

A pipeline can run a notebook instead of SQL: give notebook: {file: notebooks/explore.ipynb} (a file named under files), and deploy publishes it and pins the version; Bundle reference has the details.

validate prints one sentence per problem and exits 2: an unknown key, a variable without a value in some target, a pipeline file that is not a pipeline or has none (or two) of sql, sqlFile and notebook, a sqlFile that is not among the bundle's files, a schedule without both cron and timezone, a production target without runAs: pipeline-<group>, a path that leaves the bundle, a symbolic link, a file over 5 MiB. It checks what needs no cluster; whether the group's pipeline identity is ready is for Nova to say when you plan or deploy.

2. Try it in your own development stage

plan asks Nova what deploying one target would do, and writes nothing. deploy does it:

stardelt bundle plan shop -t dev
stardelt bundle deploy shop -t dev

{"action": "would-create", "name": "dev-alice-daily"}
{"action": "created", "name": "dev-alice-daily"}

Each pipeline file gets the target's variables, its sqlFile is read into sql, and the bundle's group is its ownerGroup when the file names none. Then Nova checks every pipeline with the rules of a create (your group, the group's pipeline identity, the statement, the schedule). A development target deploys under dev-<your sign-in name>-<pipeline> and without a schedule, so two people can deploy one bundle side by side and nothing runs on its own. Start it yourself and wait for it:

stardelt bundle run shop -t dev

{"action": "ran", "name": "dev-alice-daily", "runId": "manual__2026-11-01T08:00:00.000000+00:00", "state": "success"}

bundle run starts every pipeline the deploy left in your stage, one after the other, and stops at the first that does not succeed (exit code 1, with the failed tasks in the line); --only daily runs one, written without the dev-alice- prefix. To start any single pipeline by its namespace and name, see Run a pipeline and wait for it.

3. Check a change in a pull request

A pull request job needs two commands. The first needs nothing from the cluster; the second asks Nova and writes nothing:

stardelt bundle validate pipelines/            # no host, no sign-in
stardelt bundle plan pipelines/ -t prod        # STARDELT_NOVA_URL and STARDELT_TOKEN set

plan prints one line per pipeline, the same lines as stardelt apply --dry-run: would-create or would-update, with changes (which of statement, schedule, params, options and deployment would change) and, for a scheduled pipeline, the next runs. A pipeline someone changed by hand after its last deploy also carries "modified": true, with one warning on standard error, so the reviewer sees that the deploy would overwrite that change. If Nova refuses any pipeline, nothing would be written: the sentences are shown (prod: daily: you are not a member of the group shop), what passed is still listed, and the command exits 2, so the job fails. The token is one Nova accepts for a member of the bundle's owner group; keep it in the CI system's secret store, never in a file of the repository.

plan names the field that changes; add --diff to see the change itself. For every update it prints, on standard error after the JSON lines, a unified diff of each changed field (the statement line by line, a schedule or parameters as YAML), so the pull request log shows what a reviewer reviews. The JSON lines on standard output stay as they are.

4. Publish to production

stardelt bundle deploy shop -t prod

{"action": "created", "name": "daily"}

deploy runs the same check as plan first. Only when Nova accepts every pipeline does it create or update them; a refusal anywhere means nothing is written. A production target deploys the pipelines under the names the files give, with their schedules.

Every pipeline is sent with where it came from: the bundle's name, the target and the Git commit of the bundle directory (-dirty is appended when files are uncommitted; outside a Git checkout there is no commit). Nova stamps that on the pipeline, and the pipeline page shows it under Deployed from, with who deployed it and when. A pipeline made in the interface or with stardelt apply has no such stamp. To go back to an earlier state, check out the earlier commit and deploy again.

To read it for the whole bundle at once, stardelt bundle summary shop -t prod prints one line per pipeline the target left on the stage: its name, the Git revision it was deployed from, deployedBy, deployedAt and "modified": true when someone changed it after the deploy. A release job can compare the revision with the commit it just deployed. The command only reads, and a development target shows only your own pipelines.

The same view is in Nova under Build, Deployments: one block per bundle and stage with the commit, who deployed each pipeline and when, and a Changed since deploy badge on a pipeline someone changed afterwards. A pipeline deployed before that check existed shows a dash there, not "unchanged".

5. Keep Git and the stage in step

Remove what the bundle dropped. --prune (on plan and deploy) also handles the pipelines an earlier deploy of this bundle target stamped and the bundle no longer names: plan lists them as would-delete, and deploy removes them (deleted) only after every create and update succeeded. A pipeline made by hand, or stamped by another bundle or target, is never touched, and a development target removes only your own dev-<your name>- pipelines.

Empty a stage. stardelt bundle destroy shop -t dev lists, as would-delete, every pipeline an earlier deploy of this bundle target put on the stage, and --yes removes them (deleted). In a development target it removes only your own dev-<your name>- pipelines, so you can clean up after a finished branch without touching a colleague's; in prod it removes what the bundle published there. Pipelines made by hand or stamped by another bundle or target stay, and so do the files of the bundle.

Bring in what was made in the interface. stardelt bundle generate NAMESPACE NAME shop writes a pipeline made on the Pipelines page into the bundle in shop: resources/<name>.yaml, and for a statement sql/<name>.sql with sqlFile pointing at it (the bundle's group stands in for ownerGroup). It never overwrites a file, and when it writes nothing it says why in one sentence: the pipeline belongs to another group than the bundle, cannot be changed from a file, or the bundle's include or files would not pick the new file up. Run stardelt bundle validate shop, commit, and the next deploy -t prod updates that pipeline and stamps it, so Deployed from appears and the next change goes through Git. To look at a pipeline without putting it in a bundle, stardelt export NAMESPACE NAME prints it as a file that stardelt apply -f accepts (see Python, command line and agents).

See what was changed by hand. When someone changes a deployed pipeline after its deploy, with the form, the YAML editor or kubectl, its page says Changed since the last deploy under Deployed from. The next deploy of the bundle overwrites that change, so the page tells you whether a deploy would lose work; bundle plan and bundle deploy say the same with "modified": true and a warning on standard error, and deploy still goes ahead. It compares the statement or notebook, the schedule, the parameters and the options; the pause state and the group are not part of it. A pipeline that was deployed before this check existed shows nothing until its next deploy.

For a production stage you usually want more than a warning. Put drift: block on the target in bundle.yaml (prod: {mode: production, runAs: pipeline-shop, drift: block}; the default is drift: warn): then bundle plan and bundle deploy exit with 2 and name every pipeline that was changed after its deploy, a deploy writes nothing, and the plan still lists everything it would do, so a pull request check shows the whole picture before it fails. Look at the change, keep it in the bundle if it is right, and pass --accept-drift to overwrite what is left.

What is not there yet

• A record of what was deployed when, and a rollback button: the stamp on each pipeline says which commit it runs; to go back, deploy the earlier commit.
• A temporary stage per pull request.