D. Configure

Set up your environment file, lay out the data directories, and rotate the config.yaml salt before you start the pipeline for the first time.

Requires: Install package (step C)

Step C clones the repository and pulls the container images. Complete C. Install package before continuing here.

1. Copy the environment file

cp .env.example .env

2. Lay out the data directories

Every host directory the pipeline bind-mounts derives from a single variable, PIPELINE_ROOT, in .env (default ./data). You set one path and storage, images, reference data, certificates, and submitter state all follow from it.

Windows: use Git Bash

All make commands must be run in Git Bash, not PowerShell or cmd.exe.

For a single-host install the default PIPELINE_ROOT=./data is fine — leave it as shipped; it is all you need for a test run. Create the directory tree it points to:

make dirs

make dirs reads the paths from your .env and creates exactly the directories the containers expect, printing one line per directory:

ok: PIPELINE_IMAGE_ROOT -> ./data/images
ok: PIPELINE_OUTBOX_DIR -> ./data/storage/storage
ok: PIPELINE_REFDATA_DIR -> ./data/refdata
ok: PIPELINE_SSL_DIR -> ./data/certs
ok: PIPELINE_STORAGE -> ./data/storage
ok: PIPELINE_SUBMITTER_CREDS_DIR_PRODUCTION -> ./data/certs/production
ok: PIPELINE_SUBMITTER_CREDS_DIR_STAGING -> ./data/certs/staging
ok: PIPELINE_SUBMITTER_STATE -> ./data/submitter-state

Staging and production BigPicture credentials live in two separate directories (certs/staging and certs/production) so make upload can reach either instance without swapping files — see J. Upload to BigPicture.

The repeated storage/storage is intentional, not a typo: PIPELINE_STORAGE is the storage root, and the export outbox (PIPELINE_OUTBOX_DIR) is a folder named storage inside it — slidetap always writes finished bundles to <storage root>/storage.

Mounted filesystems: own the mount before running make dirs

If PIPELINE_ROOT points to an absolute path on a mounted filesystem (e.g. /mnt/data), the mount must be owned by the container user (UID 1000) before you run make dirs. A root-owned mount causes make dirs to create root-owned directories that the containers cannot write to, silently breaking steps E, H, and J.

Transfer ownership first, then create the directories as a regular user:

sudo chown -R 1000:1000 /mnt/data   # once, as root
make dirs                            # as your regular user — no sudo

Do not run sudo make dirs — it produces 0755 root-owned directories that UID 1000 cannot write to.

make dirs is for the local layout only. To place a category on a network share, see Relocating individual categories below.

Relocating individual categories

Bundling everything under PIPELINE_ROOT suits a single host. In production you might want to place categories on different mounts by performance and sensitivity. Override only the categories that move, the rest stay derived from PIPELINE_ROOT.

Category (.env variable) Put it on Why
PIPELINE_IMAGE_ROOT Large, read-only NFS/CIFS share Terabytes of whole-slide images; read-only, no write-speed requirement
PIPELINE_STORAGE (download, conversion and the export outbox all live here) Large and reasonably fast volume Holds the entire converted dataset plus intermediate work. Conversion also writes here, so a slow network share hurts
PIPELINE_REFDATA_DIR Small shared NFS reference data that multiple people might want access to, like snomed conversions
PIPELINE_SSL_DIR, PIPELINE_SUBMITTER_CREDS_DIR_STAGING, PIPELINE_SUBMITTER_CREDS_DIR_PRODUCTION Local disk, not a shared mount TLS keys and BigPicture staging/production credentials — keep secrets off shared storage
PIPELINE_SUBMITTER_STATE Local disk Small read/write upload state for the bp-submitter

To relocate a category, set its variable to an absolute path in .env, for example:

PIPELINE_IMAGE_ROOT=/mnt/nas/wsi-images
PIPELINE_STORAGE=/srv/bp-storage

PIPELINE_OUTBOX_DIR always follows PIPELINE_STORAGE automatically — you never set it by hand, so the outbox can never drift out of storage.

When you relocate a category to a network share, mounting that share and giving the container user (UID 1000) write access is an administrator prerequisite — set it up via fstab/NFS/CIFS with uid=1000. The pipeline only verifies the result with make doctor; it never creates or mounts anything.

For the deeper discussion of which category belongs on which storage, NFS/CIFS mount options, and the throughput caveats of network shares, see Best Practices.

3. Fill in .env

Open .env in any text editor.

Windows: save with LF line endings

Docker reads .env inside a Linux container. Save with LF (Unix) line endings, not CRLF. VS Code does this automatically; other editors may need a setting change.

Must set

These variables have no working defaults — the pipeline will not start without them:

Variable What to set How
SLIDETAP_SECRET_KEY Random string, ≥ 32 characters Run openssl rand -hex 32 and paste the output
XLSX_MAPPER_GROUP_NAME Your institution’s short name e.g. karolinska — used as the mapper group label

To generate a value for SLIDETAP_SECRET_KEY:

openssl rand -hex 32

Paste the printed string as the value in .env.

Host paths — derived, nothing to set

You set PIPELINE_ROOT in step 2; these all follow from it automatically. You only touch them to relocate a category (see Relocating individual categories above) — never to wire up the default layout.

Variable Derived value
PIPELINE_STORAGE ${PIPELINE_ROOT}/storage
PIPELINE_OUTBOX_DIR ${PIPELINE_STORAGE}/storage — always inside storage
PIPELINE_IMAGE_ROOT ${PIPELINE_ROOT}/images
PIPELINE_REFDATA_DIR ${PIPELINE_ROOT}/refdata
PIPELINE_SSL_DIR ${PIPELINE_ROOT}/certs
PIPELINE_SUBMITTER_CREDS_DIR_STAGING ${PIPELINE_ROOT}/certs/staging
PIPELINE_SUBMITTER_CREDS_DIR_PRODUCTION ${PIPELINE_ROOT}/certs/production
PIPELINE_SUBMITTER_STATE ${PIPELINE_ROOT}/submitter-state

make upload picks the staging or production creds dir from its TARGET argument; you do not set the path it mounts. (The internal .env slot PIPELINE_SUBMITTER_CREDS_DIR defaults to the staging dir and is best left alone.)

Windows: forward-slash paths only

If you set an absolute PIPELINE_ROOT (or relocate a category), Docker’s bind-mount parser does not accept Windows backslashes. Write C:/bp-data, not C:\bp-data or .\bp-data.

Host paths vs container paths

Two different things get called a “path” in .env, and mixing them up fails silently, not with an error:

  • Host paths point at a location on your machine — everything in the table above, plus PIPELINE_CONFIG_FILE and PIPELINE_CREDENTIALS_FILE below. Docker uses these as a bind-mount source without checking whether the location makes sense: give it /refdata/mappers.xlsx (a container path) and Docker just creates an empty directory with that literal name on your host and mounts it. make dirs and make doctor still pass — it exists and is owned correctly, it’s just the wrong thing.
  • Container paths point at a location inside the container, past where a host path was already mounted — XLSX_MAPPER_FILE and BIGPICTURE_MODEL_PATH below are both under /refdata, where PIPELINE_REFDATA_DIR (a host path) gets mounted. A host-style value here (e.g. ./data/refdata/mappers.xlsx) means nothing to the container, since ./data doesn’t exist inside it.

Rule of thumb: a default that looks like ./data/... (or derives from PIPELINE_ROOT) is a host path you can relocate; a default like /refdata/... with no host counterpart is a container path you leave alone. PIPELINE_SSL_CERT / PIPELINE_SSL_KEY below are neither — bare filenames resolved against the host path PIPELINE_SSL_DIR. See F. TLS + credentials for what breaks if you give them a path.

May change

Variable Default When to change
PIPELINE_SERVERNAME localhost Set to your host’s DNS name or IP if other machines need to reach it
PIPELINE_PORT 443 Change if port 443 is already in use on the host
PIPELINE_SSL_CERT server.crt TLS certificate — filename only, not a path (resolved against PIPELINE_SSL_DIR)
PIPELINE_SSL_KEY server.key TLS private key — filename only, not a path (resolved against PIPELINE_SSL_DIR)
PIPELINE_CREDENTIALS_FILE ./credentials.json Path to the hashed credentials file (generated in step E)
PIPELINE_CONFIG_FILE ./config.yaml Path to the app config file — set up in the next step
XLSX_MAPPER_FILE /refdata/mappers.xlsx Container path to the mapper workbook — change only if you relocate it inside the container
BIGPICTURE_MODEL_PATH /refdata/model.json Container path to model.json (the attribute/observation model) — change only if you relocate it inside the container

Leave as-is

Everything in the Dev / advanced section of .env (PIPELINE_DB_*, XLSX_MAPPER_GROUP_UID, SLIDETAP_APP_CREATOR, PIPELINE_IMAGE_PREFIX, etc.) has working defaults. Do not change these unless you have a specific reason.

4. Set up config.yaml

Copy the example file:

cp config.yaml.example config.yaml

Open config.yaml and replace the salt value with a unique secret of at least 32 characters:

json_file_auth:
  credentials_file: /credentials.json
  salt: "replace-this-with-your-own-secret-at-least-32-characters"
Set the salt once — never change it afterwards

The salt is mixed into every hashed password. Changing it after you run make gen-credentials (step E) invalidates all existing passwords and locks everyone out.

Other settings (optional)

The rest of config.yaml controls application behaviour and DICOM conversion. The defaults are fine for most deployments; adjust only if you have a specific need.

Setting Default When to change
restore_projects false Set to true to resume in-progress submissions after a container restart
dicomization.threads 1 Increase for faster conversion on multi-core hosts (no effect on .vms slides)
dicomization.levels all Restrict (e.g. [0, 1, 2]) for smaller output; fewer levels reduce zoom range in viewers. Useful to reduce for quick tests.
dicomization.include_labels false Set to true to include the barcode/cassette label image
dicomization.include_overviews false Set to true to include the macro overview thumbnail
Verify

Confirm the salt has been changed:

grep salt config.yaml

The output should show your string, not change-this-salt-to-something-secret-in-production!!.

Already have the stack running? Both files need a restart to take effect

If you edit .env or config.yaml after make up, the running containers don’t see the change on their own — see When do I need to restart? for which command to run and why.

If something goes wrong

  • make dirs fails with Permission denied — your PIPELINE_ROOT is on a mount owned by root. Fix it with sudo chown -R 1000:1000 <mount>, then re-run make dirs as a regular user (see Lay out the data directories above).
  • make doctor reports the salt is unchanged — the salt in config.yaml still matches the example value. Open the file and replace it, then re-run make doctor.
  • Auth error after changing the salt — you changed the salt after running make gen-credentials. The credentials file was hashed with the old salt and is now invalid. Re-run make gen-credentials with the current salt.
  • make doctor rejects PIPELINE_SSL_CERT or PIPELINE_SSL_KEY — one of them holds a path instead of a bare filename. See F. TLS + credentials for the fix.
  • make commands fail on Windows — ensure you are running in Git Bash, not PowerShell or cmd.exe. Also confirm .env uses LF line endings (see Fill in .env above).