Pipeline Configuration

Reference for all configuration that controls pipeline orchestration. For the practical how-to, see Running Multi-Entity Pipelines.

Entity list

Defined in roche-data.toml under [entities]. This is the authoritative list for multi-entity pipeline runs and the GitHub Actions matrix.

[entities]
list = ["waste-tracking", "organization-site"]

Each entry must:

• Be a valid entity name (lowercase, alphanumeric, hyphens, underscores)
• Have a corresponding [rtis.entities.<name>] section with RTiS IDs
• Produce a models/<name>/model.json after first rdt-model-pull run

Environment variables

Required

Variable	Description	Example
RDT_TARGET	Target environment	dev, test, prod

Workspace control

Variable	Description	Default
RDT_WORKSPACE_DIR	Base directory for pipeline workspaces	$TMPDIR or /tmp
RDT_RUN_ID	Correlation ID for the pipeline run (UUIDv7)	Auto-generated

CI-specific

Variable	Set by	Description
VAULT_ADDR	GitHub Environment	Vault URL for secret loading
VAULT_NAMESPACE	GitHub Environment	Vault namespace (rdt-model-prd)
All SNOWFLAKE_*	Vault action	Snowflake credentials per environment
All RTIS_*	Vault action	RTiS API credentials

CLI flags

rdt-model-compile run

Flag	Short	Description
--target <env>	-t	Required. Target environment: dev, test, prod
--entity <name>	-e	Required. Entity identifier
--workspace <path>	—	Enable workspace isolation mode
--phase <name>	—	Run only a single phase
--dry-run	-n	Skip all side-effects
--json	—	Emit PipelineResult to stdout
--run-id <id>	—	Correlation ID (generated if omitted)
--quiet	-q	Suppress tracing output
--verbose	-v	Increase verbosity (-v = DEBUG, -vv = TRACE)

rdt-model-compile promote

Flag	Short	Description
--target <env>	-t	Required. Target environment
--entity <name>	-e	Required. Entity to promote
--workspace <path>	—	Required. Workspace directory to promote from
--dry-run	-n	Show what would be promoted without copying
--json	—	Emit PromoteResult to stdout

Module --manifest flag

Every rdt-model-* binary accepts --manifest <path> as an alternative to --target + --entity. When provided, the module reads all invocation context from the manifest JSON.

rdt-model-store --manifest /tmp/rdt-waste-tracking-abc/store-manifest.json --json generate

Conflicts with --entity (cannot specify both).

Pipeline manifest schema

JSON Schema: cli/common/schemas/modules/pipeline-manifest.schema.json

Fields

Field	Type	Required	Description
entity_id	string	yes	Entity identifier (^[a-z0-9_-]+$)
workspace	string	yes	Absolute path to workspace root
target	string (enum)	yes	dev, test, or prod
run_id	string	yes	UUIDv7 correlation ID
dry_run	boolean	no	Default: false
inputs	object	no	Named input artifact paths (workspace-relative)
output_dir	string	yes	Phase subdirectory for outputs (workspace-relative)

Example

{
  "entity_id": "waste-tracking",
  "workspace": "/tmp/rdt-waste-tracking-019600ab-cdef",
  "target": "dev",
  "run_id": "019600ab-cdef-7000-89ab-0123456789ab",
  "dry_run": false,
  "inputs": {
    "model": "pull/model.json",
    "governance": "govern/governance.json"
  },
  "output_dir": "deploy"
}

Input keys by phase

The orchestrator populates the inputs map based on outputs from previous phases:

Phase	Module	Available input keys
1 Ingest	pull, profile	(none — entry point)
2 Enrich	govern, infer	model
3 Prepare	validate	model, governance
4 Deploy	store, policy, api, mcp, sdk, contract	model, governance, suggestions
5 Register	register, gupri, search	All Phase 4 outputs
6 Support	docs, cidb, event	All Phase 5 outputs

Result envelopes

PipelineResult (orchestrator output)

Emitted by rdt-model-compile run --json:

{
  "entity_id": "waste-tracking",
  "status": "ok",
  "timestamp": "2026-05-12T10:30:00Z",
  "duration_ms": 45000,
  "phases": [
    {
      "phase": "ingest",
      "status": "ok",
      "duration_ms": 5000,
      "modules": [ /* ModuleResult[] */ ]
    }
  ]
}

ModuleResult (per-module output)

Emitted by every rdt-model-* binary with --json:

{
  "module": "rdt-model-store",
  "version": "0.1.0",
  "status": "ok",
  "entity_id": "waste-tracking",
  "timestamp": "2026-05-12T10:30:05Z",
  "duration_ms": 1200,
  "outputs": {
    "snowflake/ddl/waste-tracking.sql": "wrote",
    "dbt/models/bronze/waste-tracking.sql": "updated"
  },
  "errors": []
}

PromoteResult (promote output)

Emitted by rdt-model-compile promote --json:

{
  "entity_id": "waste-tracking",
  "promoted": [
    {"source": "pull/model.json", "destination": "models/waste-tracking/model.json", "action": "wrote"}
  ],
  "skipped": ["infer/suggestions.json"]
}

Status values

Status	Meaning
ok	All required modules succeeded, at least one artifact produced
warning	All required succeeded, but all outputs were skipped (unchanged or dry-run)
partial	Some required modules failed, but some outputs were produced
error	Complete failure — no outputs (or all required modules failed)

Output actions

Action	Meaning
wrote	New file created
updated	Existing file overwritten with different content
skipped	Dry-run, or file already exists with identical content

Workspace directory structure

Created by scripts/pipeline.sh or WorkspaceConfig::create():

/tmp/rdt-{entity}-{run_id}/
├── pull/
├── govern/
├── infer/
├── compile/
│   └── artifacts/
├── validate/
├── deploy/
├── register/
├── support/
├── pipeline-result.json
└── pipeline-stderr.log

Pipeline script arguments

scripts/pipeline.sh <entity> [options]

Argument	Description
<entity>	Required. Entity identifier
--target <env>	Target environment (default: $RDT_TARGET or dev)
--keep-workspace	Do not delete workspace after completion

GitHub Actions workflow inputs

entity-pipeline.yml supports workflow_dispatch with:

Input	Required	Default	Description
entities	no	(all from roche-data.toml)	Comma-separated entity list
target	yes	dev	Target environment (choice: dev/test/prod)

Related pages

• Running Multi-Entity Pipelines — practical guide
• Pipeline Overview — architecture and design
• rdt-model-compile — orchestrator module reference
• Project Settings — URLs, Vault, system endpoints