J. Upload to BigPicture

Encrypt and submit your validated bundle to the NBIS staging inbox with bp-submitter. This is the final step of the operator journey.

This is a staging (test) upload

The A–J manual is a test submission, so this step uploads to BigPicture’s staging environment — the s3cmd-inbox.conf NBIS provides for the manual points there.

Precondition: NBIS receiver credentials

You need two files before you can upload:

File	Purpose
s3cmd-inbox.conf	S3 connection configuration for the NBIS inbox
crypt4gh_bp_key.pub	NBIS receiver public key for encryption

s3cmd-inbox.conf — log in at the BigPicture portal and click Download credentials to upload to the Inbox. Use staging-login.bp.nbis.se for the manual; login.bp.nbis.se for a real production submission.

crypt4gh_bp_key.pub — download the NBIS receiver public key:

Windows

curl -OL https://raw.githubusercontent.com/NBISweden/EGA-SE-user-docs/main/crypt4gh_bp_key.pub

macOS

curl -OL https://raw.githubusercontent.com/NBISweden/EGA-SE-user-docs/main/crypt4gh_bp_key.pub

Linux

curl -OL https://raw.githubusercontent.com/NBISweden/EGA-SE-user-docs/main/crypt4gh_bp_key.pub

Place both in the credentials directory for your target — certs/staging (PIPELINE_SUBMITTER_CREDS_DIR_STAGING) for the manual, certs/production (PIPELINE_SUBMITTER_CREDS_DIR_PRODUCTION) for production — under the certs/ directory from F. TLS + credentials, so the two instances never share credentials. Do not commit them to version control; they are already covered by the .gitignore.

No restart needed if NBIS sends you replacement files

Unlike the rest of the pipeline’s config, a replacement s3cmd-inbox.conf or crypt4gh_bp_key.pub takes effect on your very next make upload — see When do I need to restart? for why.

1. Dry run

Run a dry run first. It validates your credentials, bundle path, and encryption readiness without transferring any data.

Windows

In Git Bash, run:

make upload-dry-run DATASET=<project_dir>/DATASET_<alias>

macOS

make upload-dry-run DATASET=<project_dir>/DATASET_<alias>

Linux

make upload-dry-run DATASET=<project_dir>/DATASET_<alias>

DATASET= here is the bundle’s path relative to the outbox root ($PIPELINE_OUTBOX_DIR), with two segments:

• <project_dir> — the project directory name, the value after Project: in the make export-check output from H. Run a submission, for example test-submission.10494c10-0d6b-4103-89e0-4909ad6e23af.
• DATASET_<alias> — the bundle subfolder one level inside it (named after the Dataset Name you set in step H), for example DATASET_test-submission.

So for that example, the full value is DATASET=test-submission.10494c10-0d6b-4103-89e0-4909ad6e23af/DATASET_test-submission.

Different from the validate-upload value

This is a different value from the one make validate-upload takes — that target wants the full path to the DATASET_<alias> folder itself, not a path relative to the outbox. See I. Validate the bundle.

make upload-dry-run defaults to staging. To dry-run against production credentials instead, add TARGET=production.

Verify

A passing dry run starts with:

Dry-run upload of '<project_dir>/DATASET_<alias>' to staging (zero bytes) ...

and exits 0 with no ERROR lines. Fix any reported errors before uploading.

2. Upload

When the dry run passes, run the real upload. make upload requires a TARGET — it never guesses which instance to push to. The A–J manual is a staging test, so use TARGET=staging:

Windows

In Git Bash, run:

make upload DATASET=<project_dir>/DATASET_<alias> TARGET=staging

macOS

make upload DATASET=<project_dir>/DATASET_<alias> TARGET=staging

Linux

make upload DATASET=<project_dir>/DATASET_<alias> TARGET=staging

Use the same <project_dir>/DATASET_<alias> value as the dry run above.

For a real submission, use TARGET=production instead — it reads the separate PIPELINE_SUBMITTER_CREDS_DIR_PRODUCTION credentials:

make upload DATASET=<project_dir>/DATASET_<alias> TARGET=production

Use --threads 1 (the default)

NBIS has explicitly asked submitters not to use parallel upload threads. make upload enforces --threads 1 by default.

If NBIS has explicitly told you that more threads are allowed for your site, you can override:

make upload DATASET=<project_dir>/DATASET_<alias> TARGET=staging THREADS=4

3. Confirm it finalized

A finished make upload is not the same as a finalized submission. The submitter uploads all image files first and only then serializes and uploads the metadata XMLs; if that last phase fails, every image is still in the inbox and the run looks superficially complete.

upload.db cannot tell you whether it finalized

The resume database records data files only — the metadata phase writes no rows. A run that shipped all images and then died on metadata validation is byte-for-byte identical in upload.db to a fully finalized one: same row count, same checksums. Do not use it as evidence.

make upload also writes no log file, and the container is removed on exit, so once your terminal scrollback is gone there is no local record either way.

The only reliable check is to list what actually arrived in the inbox. From your bp-submission-pipeline checkout, with DATASET_<alias> substituted:

set -a && . ./.env && set +a
PIPELINE_SUBMITTER_CREDS_DIR="$PIPELINE_SUBMITTER_CREDS_DIR_STAGING" \
docker compose -f docker-compose.yml -f docker-compose.build.yml run --rm --no-deps \
  --entrypoint python bp-submitter -c '
from bigpicture_metadata_interface.sda.destination.s3 import S3Destination
ALIAS = "DATASET_<alias>"
with S3Destination("/creds/s3cmd-inbox.conf") as destination:
    prefix = f"{destination.bucket}/{destination.bucket}/{ALIAS}"
    for key in sorted(destination._fs.find(prefix)):
        print(key[len(prefix) + 1:])
'

Swap PIPELINE_SUBMITTER_CREDS_DIR_STAGING for PIPELINE_SUBMITTER_CREDS_DIR_PRODUCTION to check a production submission. This only lists objects — it uploads and changes nothing.

Verify

Every METADATA/*.xml.c4gh present means the metadata phase completed, which is what finalization means. If you see IMAGES/ but no METADATA/, the run stopped after the image phase and the submission is not finalized — fix what the metadata phase rejected and re-run make upload.

Also check that no object is 124 bytes or smaller. That is the size of an empty crypt4gh header, i.e. a file that encrypted to nothing.

Notify NBIS

Once your submission is completed, notify NBIS by sending an email. The subject of the email should be in the format of: submission - DATASET_{IDENTIFIER}, for example submission - DATASET_test-submission.

The email should contain:

• Name of the uploader
• Name of the dataset

Resumable uploads

bp-submitter tracks progress in a local SQLite database at ${PIPELINE_SUBMITTER_STATE}/upload.db (default: $PIPELINE_ROOT/submitter-state/upload.db). If an upload is interrupted, re-run the same make upload DATASET=<project_dir>/DATASET_<alias> TARGET=<staging|production> command — it resumes from where it left off.

To start over (for example, after replacing a corrected bundle), delete upload.db from your PIPELINE_SUBMITTER_STATE directory before re-running.

Persistent uploads (surviving a dropped connection)

If you’re running the upload from a remote server over SSH rather than your own machine, a dropped connection kills your view of make upload — but not the upload itself. The transfer runs inside a Docker container that keeps going in the background even after the client attached to it dies, so the upload usually finishes on its own. What you lose is the live terminal: there’s no supported way to reattach to a plain make upload, so you’re stuck manually hunting for the orphaned container (docker ps) to check on it or see when it’s done.

Because of that, don’t assume a dropped make upload failed — check docker ps for a still-running bp-submitter container before re-issuing the command, to avoid racing a duplicate upload against the original.

make upload-persistent runs the same upload inside a tmux session that keeps running even after your connection drops. It takes the same arguments as make upload:

make upload-persistent DATASET=<project_dir>/DATASET_<alias> TARGET=staging

The first run starts the upload and attaches you to it live — it looks identical to running make upload directly. If your connection drops, reconnect and run the exact same command again: it reattaches to the still-running upload instead of starting a duplicate.

A couple of tmux basics, once attached:

• Detach (leave it running in the background, return to your shell): press Ctrl-b, release, then press d.
• Reattach: re-run the same make upload-persistent ... command.

When the upload finishes, the pane stays open showing [exit 0] (or the nonzero exit code on failure) instead of vanishing — if you’re still attached, you’ll see it directly. If you reconnect after it already finished, running the same make upload-persistent ... command prints Previous run finished with exit N to your terminal right away, then starts a genuinely new upload — it does not reattach you to the old, finished pane.

make upload-persistent requires tmux on the machine running it. This is only relevant on a VM/server you access over SSH — not needed for a desktop walkthrough of this manual, and not checked by make doctor. If tmux isn’t installed, the command fails immediately with an install hint instead of silently running unwrapped.

If something goes wrong

• TARGET is required: ... — make upload will not run without a target. Re-run with TARGET=staging (the manual) or TARGET=production.
• Invalid TARGET '<x>': expected 'staging' or 'production' — the TARGET value is mistyped. Use exactly staging or production.
• Missing creds file: .../s3cmd-inbox.conf or Missing creds file: .../crypt4gh_bp_key.pub — one of the NBIS-provided files is absent from the credentials directory for the TARGET you chose (certs/staging for TARGET=staging, certs/production for TARGET=production). Place the missing file in that directory and retry.
• DATASET not set — re-run with DATASET=<project_dir>/DATASET_<alias> (see step 1 above for how to build this value).
• Dataset dir not found — the <project_dir>/DATASET_<alias> value doesn’t match what’s under PIPELINE_OUTBOX_DIR. Confirm <project_dir> from the make export-check output (step H), then check the DATASET_<alias> subfolder actually exists inside it.
• Connection or authentication errors — your s3cmd-inbox.conf may be expired or misconfigured. Contact NBIS for an updated file.
• Upload stalls and does not resume — delete upload.db from PIPELINE_SUBMITTER_STATE and retry from the beginning.
• tmux not found ... (only with make upload-persistent) — install tmux on the machine running the command, or use plain make upload instead.