🖥️ Wurzel CLI Reference¶
The Wurzel CLI provides a powerful command-line interface for managing and executing ETL pipelines for RAG systems.
Quick Start¶
# Install wurzel
pip install wurzel
# Run a step
wurzel run wurzel.steps.manual_markdown.ManualMarkdownStep --inputs ./data --output ./out
# Inspect a step
wurzel inspect wurzel.steps.manual_markdown.ManualMarkdownStep
# Generate a pipeline
wurzel generate examples.pipeline.pipelinedemo.pipeline
Glossary¶
PIPELINE¶
Module path to a chained pipeline (multiple steps combined with the >> operator). Example: examples.pipeline.pipelinedemo.pipeline
CLI Commands Reference¶
The following documentation is automatically generated from the Wurzel CLI code:
wurzel¶
Global settings, main.
Usage¶
wurzel [OPTIONS] COMMAND [ARGS]...
Arguments¶
No arguments available
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
--log-level TEXT | [default: INFO] | No | - |
--install-completion | Install completion for the current shell. | No | - |
--show-completion | Show completion for the current shell, to copy it or customize the installation. | No | - |
--help | Show this message and exit. | No | - |
Commands¶
| Name | Description |
|---|---|
run | Run a step |
inspect | Display information about a step |
env | Inspect or validate environment variables... |
generate | generate a pipeline |
Sub Commands¶
wurzel run¶
Run a step
Usage¶
wurzel run [OPTIONS] STEP
Arguments¶
| Name | Description | Required |
|---|---|---|
STEP | module path to step | Yes |
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
-o, --output DIRECTORY | Folder with outputs [default: | No | - |
-i, --inputs DIRECTORY | input folders | No | - |
-e, --executor TEXT | executor to use [default: BaseStepExecutor] | No | - |
--encapsulate-env / --no-encapsulate-env | [default: encapsulate-env] | No | - |
--help | Show this message and exit. | No | - |
wurzel inspect¶
Display information about a step
Usage¶
wurzel inspect [OPTIONS] STEP
Arguments¶
| Name | Description | Required |
|---|---|---|
STEP | module path to step | Yes |
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
--gen-env / --no-gen-env | [default: no-gen-env] | No | - |
--help | Show this message and exit. | No | - |
wurzel env¶
Inspect or validate environment variables for a pipeline
Usage¶
wurzel env [OPTIONS] PIPELINE
Arguments¶
| Name | Description | Required |
|---|---|---|
PIPELINE | module path to step or pipeline | Yes |
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
--include-optional / --only-required | display optional variables as well [default: include-optional] | No | - |
--gen-env | emit .env content instead of a table | No | - |
--check | validate that required env vars are set | No | - |
--allow-extra-fields | allow unknown settings when validating | No | - |
--help | Show this message and exit. | No | - |
wurzel generate¶
generate a pipeline
Usage¶
wurzel generate OPTIONS
Arguments¶
| Name | Description | Required |
|---|---|---|
[PIPELINE] | module path to step or pipeline(which is a chained step) | No |
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
-b, --backend TEXT | backend to use [default: DvcBackend] | No | - |
--list-backends | List all available backends and exit | No | - |
--help | Show this message and exit. | No | - |
Usage Examples¶
Running Steps¶
# Basic usage
wurzel run wurzel.steps.manual_markdown.ManualMarkdownStep \
--inputs ./markdown-files \
--output ./processed-output
# With custom executor
wurzel run wurzel.steps.manual_markdown.ManualMarkdownStep \
--inputs ./markdown-files \
--output ./processed-output \
--executor PrometheusStepExecutor
# Multiple input folders
wurzel run wurzel.steps.splitter.SimpleSplitterStep \
--inputs ./docs \
--inputs ./markdown \
--inputs ./pdfs \
--output ./split-output
Inspecting Steps¶
# Basic inspection
wurzel inspect wurzel.steps.manual_markdown.ManualMarkdownStep
# Generate environment file
wurzel inspect wurzel.steps.manual_markdown.ManualMarkdownStep --gen-env
Managing Environment Variables¶
Use the wurzel env helper to inspect or validate the variables your pipeline needs:
# Show required env vars (toggle optional ones via --only-required)
wurzel env examples.pipeline.pipelinedemo:pipeline --only-required
# Generate a .env snippet (prefers values already present in your shell)
wurzel env examples.pipeline.pipelinedemo:pipeline --gen-env > .env.sample
# Fail fast when something is missing
wurzel env examples.pipeline.pipelinedemo:pipeline --check
# Allow dynamically added settings (equivalent to setting ALLOW_EXTRA_SETTINGS)
wurzel env examples.pipeline.pipelinedemo:pipeline --check --allow-extra-fields
Generating Pipelines¶
The wurzel generate command creates backend-specific pipeline configurations from chained pipelines.
Arguments: - pipeline - Module path to a chained pipeline (multiple steps combined with >> operator)
Options: - -b, --backend - Backend to use (default: DvcBackend). Case-insensitive. - --list-backends - List all available backends and exit
# List all available backends
wurzel generate --list-backends
# Generate from a chained pipeline
wurzel generate examples.pipeline.pipelinedemo.pipeline
# Generate with explicit backend (case-insensitive)
wurzel generate myproject.pipelines.MyPipeline --backend DvcBackend
wurzel generate myproject.pipelines.MyPipeline -b dvcbackend
# Generate Argo Workflows pipeline (requires wurzel[argo])
wurzel generate myproject.pipelines.MyPipeline --backend ArgoBackend
wurzel generate myproject.pipelines.MyPipeline -b argobackend
Creating a Chained Pipeline:
A chained pipeline is created by combining multiple steps with the >> operator:
# In myproject/pipelines.py
from wurzel.steps.manual_markdown import ManualMarkdownStep
from wurzel.steps.splitter import SimpleSplitterStep
from wurzel.utils import WZ
# Wrap steps with WZ
source = WZ(ManualMarkdownStep)
splitter = WZ(SimpleSplitterStep)
# Chain steps together
source >> splitter
# Export the final step as the pipeline
pipeline = splitter
Then generate the pipeline configuration:
Step Auto-Discovery¶
The CLI supports intelligent auto-completion for step names using TAB completion:
wurzel run <TAB> # Shows all available steps
wurzel run wurzel.steps.<TAB> # Shows wurzel built-in steps
wurzel run mysteps.<TAB> # Shows your custom steps
The auto-completion discovers:
- Built-in Wurzel steps - Available in the
wurzel.steps.*namespace - User-defined steps - TypedStep classes in your current project