FAQ

Common questions and errors across the submission journey

Answers to the questions and error messages that come up most often while working through the bp-submission-pipeline manual. Each entry links back to the step it’s drawn from, so you can jump to the full context.

Can’t find what you’re looking for, or need to reach a person? See Support.

B. GitHub access

The browser doesn’t open when I run gh auth login

gh also prints a one-time code and a URL in the terminal. Open https://github.com/login/device manually, enter the code, and complete the sign-in there. See B. GitHub access.

C. Install package

make pull fails with “lacks the read:packages scope”

Your GitHub token doesn’t have the scope Docker needs to pull private images yet. Run gh auth refresh -s read:packages, then try again. See B. GitHub access and C. Install package.

Cloning fails with “repository not found” or “access denied”

Your GitHub account hasn’t been granted access to the private repository yet. Request access via Support before retrying. See C. Install package.

make pull fails with “no such file or directory” on /var/run/docker.sock

The Docker daemon isn’t running. Start Docker Desktop (Windows/macOS) or run systemctl start docker (Linux), then run make pull again. See C. Install package.

The submodule folders are empty after cloning

The repository was cloned without --recurse-submodules. Delete the cloned folder and re-run the full clone command. See C. Install package.

D. Configure

make dirs fails with “Permission denied”

PIPELINE_ROOT points at a mount owned by root. Transfer ownership with sudo chown -R 1000:1000 <mount>, then re-run make dirs as a regular user — never with sudo. See D. Configure.

make doctor reports the salt is unchanged

The salt value in config.yaml still matches the shipped example. Open the file, replace it with your own secret (≥ 32 characters), and re-run make doctor. See D. Configure.

I get an auth error after changing the salt

The salt was changed after make gen-credentials already hashed a password with the old one, invalidating it. Rotate the salt once, then re-run make gen-credentials with the current salt. See D. Configure and F. TLS + credentials.

make commands fail on Windows

They must be run in Git Bash, not PowerShell or cmd.exe. Also confirm .env was saved with LF (Unix) line endings, not CRLF. See D. Configure.

E1. model.json

make doctor (or setup) reports “model.json missing”

Run make setup-refdata again and confirm model.json appears in PIPELINE_REFDATA_DIR. See E1. model.json.

I see “Warning: model.json not found in bigpicture-slidetap/tests/test_data”

The pipeline’s Git submodules weren’t initialised. Run git submodule update --init --recursive, then retry make setup-refdata. See E1. model.json.

E2. mappers.xlsx

An operator’s input ends up as an empty attribute after export

No mapping key matched it. Check the relevant sheet in mappers.xlsx for regex anchoring, typos, or a case mismatch. The operator’s original input isn’t lost — fix the mapping row, re-upload mappers.xlsx, and re-trigger mapping from the curation UI; no workbook re-submission is needed. See E2. mappers.xlsx.

A mapper seems to be silently missing from the running pipeline

Check the pipeline logs for a logging.warning or logging.exception line at startup. Common causes: a wrong Sheet name in the master sheet, an invalid UUID in the UID column, or an unknown Type value. See E2. mappers.xlsx.

make setup-refdata warns “mappers.xlsx not found in …”

Same cause as the model.json version of this warning: the pipeline’s Git submodules weren’t initialised. Run git submodule update --init --recursive, then retry make setup-refdata. See E2. mappers.xlsx.

F. TLS + credentials

make doctor says the certificate or key is missing

Re-run make ssl, then check that PIPELINE_SSL_DIR, PIPELINE_SSL_CERT, and PIPELINE_SSL_KEY in .env match the files it created. See F. TLS + credentials.

make doctor rejects PIPELINE_SSL_CERT/PIPELINE_SSL_KEY as “not a path”, or make ssl fails with a doubled path like ./data/certs/./data/certs/

One of the two variables holds a path instead of a bare filename. Set it back to a filename, e.g. server.crt — see F. TLS + credentials for why.

make doctor says the credentials file is missing

Re-run make gen-credentials. It writes to the repo root by default — make sure PIPELINE_CREDENTIALS_FILE in .env points at that file. See F. TLS + credentials.

I can’t log in even though the certificate and credentials files exist

The password was almost certainly hashed with a different salt than the app is currently using. Set the salt in config.yaml first, then re-run make gen-credentials so the hash matches. See F. TLS + credentials and D. Configure.

G. Preflight + boot

make doctor says a cert, key, or credentials file is missing

You skipped or mis-set F. TLS + credentials. Re-run make ssl and make gen-credentials. See G. Preflight + boot.

make doctor says the config still matches the example / the salt is not rotated

Rotate the salt in D. Configure, then re-run make gen-credentials so your login still matches. See G. Preflight + boot.

make doctor says a host path is missing or not owned by UID 1000

Create the local data tree with make dirs (single-host setups), or fix the mount ownership — see D. Configure. See also G. Preflight + boot.

make doctor says gh is not logged in

Re-run the sign-in from B. GitHub access. See G. Preflight + boot.

The browser can’t reach the UI after make up

Give it a few seconds and refresh — first boot can take up to a minute. If it still fails, check make logs for a service that failed to start, and confirm nothing else is using the port in PIPELINE_PORT. See G. Preflight + boot.

H. Run a submission test

Parse produces no items after I upload the workbook

The workbook’s format doesn’t match what the pipeline expects. Check you’re uploading the correct .xlsx and that its sheet layout matches the import schema. See H. Run a submission test.

Images stay “failed” even after I click Retry

Check make logs for the image worker’s output. Common causes: the slide file isn’t reachable at the path referenced in the workbook, or DICOM conversion failed on that specific file format. See H. Run a submission test.

The Post-process Start button is greyed out

One or more items are still invalid. Return to Curate, fix the mapping for the flagged items in mappers.xlsx, and remap. See H. Run a submission test and E2. mappers.xlsx.

The Submit button is greyed out on the Export page

The project hasn’t reached Completed status yet. Confirm the batch was completed and check the batch status chip in the Batch sidebar. See H. Run a submission test.

Export never reaches “Export Complete” and reverts to “Completed” with no error

An observation is missing a required coded attribute (e.g. diagnose) that has no "optional": true flag in model.json. Check make logs for a ValueError naming the missing attribute’s tag — export stops at the first one it hits, so fixing one observation can just surface the next.

A green status in Curate does not rule this out, and it is not simply “stale.” For an attribute nested inside another attribute — a diagnosis or other pathology code inside an observation is the normal case — Remap recomputes the correct value and marks the item valid, but never writes that value back to the database: the validity flag and the mapping’s hit count both update, the value itself does not. The item is wrong from the moment you click Remap, and Remap will keep reporting success no matter how many times you run it.

Do not remap a nested attribute. Clear the affected mappers and re-parse instead — delete the batch (or start a fresh project) and re-upload the workbook. Re-parsing maps the value before the item is created, which is the code path that persists it correctly; remap only runs after the item already exists, which is the path that drops it.

For a plain (non-nested) attribute, re-checking the mapping in mappers.xlsx and remapping does work — this failure is specific to attributes nested inside another attribute.

Alternatively, fix by supplying the missing value from the source workbook data, or by marking the attribute optional in model.json and restarting the stack — only for attributes your site added, never for BigPicture’s MSMDAD attributes, and note this loses the value rather than recovering it. See H. Run a submission test and E1. model.json.

make export-check reports “FAIL outbox not found”

The PIPELINE_STORAGE path in .env doesn’t exist or isn’t mounted correctly. See D. Configure and H. Run a submission test.

make export-check reports “FAIL no project directories”

The export didn’t finish writing. Check make logs and confirm the project status shows Export Complete in the UI — if it shows Completed instead, see the entry above on a missing required attribute, which also prevents the bundle from being written. See H. Run a submission test.

I. Validate the bundle

validate-upload fails with “dataset directory not found”

The path passed to DATASET= doesn’t exist. Double-check the path and make sure the export step finished successfully. See I. Validate the bundle.

validate-upload fails with “missing required folder”

The export didn’t create the expected bundle structure. Re-export from the UI; if the folder is still missing, check make logs for export worker errors. See I. Validate the bundle.

validate-upload fails with a schema validation error

A metadata XML file doesn’t conform to the BigPicture XSD schema — usually a mismatch between your model.json or mappers.xlsx and the current metaflex schema. Check the named file and element, or pinpoint it by hand with xmllint — see Validate metadata. See also I. Validate the bundle.

staining.xml has a separate, near-identical STAINING entry for every slide, even when they’re all the same stain

This is expected, not a mapping bug. In the BigPicture metadata standard, a Slide’s STAINING_INFORMATION_REF points to exactly one Staining Entity — there’s no mechanism for multiple slides to share one. So a dataset where every slide is plain H&E still gets one STAINING block per slide, each with its own generated alias, all carrying identical content (same staining_method, same SNOMED code). Check sample.xml — each SLIDE element’s STAINING_INFORMATION_REF resolves to a different STAINING_PROCEDURE_* alias in staining.xml, one per slide. See Staining mapping and I. Validate the bundle.

J. Upload to BigPicture

make upload fails with “TARGET is required”

make upload never guesses the destination. Re-run with TARGET=staging (the A–J manual) or TARGET=production. See J. Upload to BigPicture.

make upload fails with “Invalid TARGET ‘’”

The TARGET value is mistyped. Use exactly staging or production. See J. Upload to BigPicture.

I get “Missing creds file” for s3cmd-inbox.conf or crypt4gh_bp_key.pub

One of the two NBIS-provided files is absent from the credentials directory for the TARGET you chose — certs/staging for TARGET=staging, certs/production for TARGET=production. Place the missing file there and retry. See J. Upload to BigPicture.

make upload fails with “DATASET not set”

Re-run the command with DATASET=<project_dir>/DATASET_<alias> — the bundle’s path relative to the outbox root, not just the project directory name from your make export-check output. See J. Upload to BigPicture.

make upload fails with “Dataset dir not found”

The <project_dir>/DATASET_<alias> value doesn’t match what’s under PIPELINE_OUTBOX_DIR. Confirm <project_dir> from the make export-check output in H. Run a submission test, then check the DATASET_<alias> subfolder actually exists inside it. See also J. Upload to BigPicture.

I get a connection or authentication error during upload

Your s3cmd-inbox.conf may be expired or misconfigured. Contact Support for an updated file. See J. Upload to BigPicture.

My upload stalls partway and doesn’t resume

Delete upload.db from your PIPELINE_SUBMITTER_STATE directory and retry from the beginning. See J. Upload to BigPicture.

make upload-persistent fails with “tmux not found”

Install tmux on the machine running the command, or use plain make upload instead — tmux is only needed for uploads run over an SSH connection that might drop. See J. Upload to BigPicture.

After submission

I submitted without a landing page

landing_page.xml is optional at submission time, so a missing one doesn’t block upload — you don’t need to resubmit the dataset to add it. See Add a landing page after submission for the current process.