Submit data to BigPicture

An operator’s manual — from a bare machine to a validated upload

There is more than one way to submit a dataset to BigPicture. This manual covers three paths, from a fully-orchestrated pipeline to a hands-on command-line upload. Choose the one that fits your context — you don’t have to use the bp-submission-pipeline, and it isn’t the only supported route.

All three paths end at the same place: a validated dataset uploaded to BigPicture. They differ in how much is wired up for you and how much control you keep.

Before you start: you need access

Every path assumes you have already been granted access to the private BigPicture repositories and container images. If you have not, the first steps will fail when you try to download the code. Request access before you begin — see Prerequisites and GitHub access.

Choose your path

Two questions decide which path is yours.

1. Do you need the curation UI?

The curation UI (bigpicture-slidetap) lets you review and fix your metadata interactively before submission — mapping values, catching invalid items, and correcting them in a browser.

  • No — I’ll prepare metadata by hand. Take the leanest route: Direct upload (sda-cli). No stack to run; you hand-prepare and validate the dataset, then encrypt and upload it with sda-cli. The most command-line-heavy path, with the fewest moving parts.
  • Yes — I want to curate metadata in the UI. You’ll be running Slidetap, so answer question 2.

2. Wired for you, or run it yourself?

Both of these give you the full curation UI — they differ only in who does the integration work.

  • Wire it for me. Use the bp-submission-pipeline — the step-by-step operator manual, written for a technically proficient user who is not a software developer. Make targets and a Docker Compose stack do the wiring; you move through ten steps, A to J, each telling you exactly what to paste and what to expect. This is the most guided, least-setup path. Start at A. Prerequisites.
  • I’ll run it myself. Take Direct components: run the individual BigPicture submodules directly, without the pipeline’s orchestration. You keep the curation UI but own the wiring — the most flexible path, and the most setup.
Path Curation UI Who does the wiring In short
bp-submission-pipeline Yes — guided The pipeline (Make + Compose) Most guided, least setup; a tested, pinned, step-by-step flow.
Direct components Yes — self-run You Most flexible; you run the four submodules and own the integration.
Direct upload (sda-cli) No — manual metadata You (leanest stack) Fewest moving parts; hand-prep the dataset and upload with sda-cli.

New here? Learn the vocabulary first

Whichever path you take, a handful of words recur — model, mappers, metadata, bundle. Key concepts is an alphabetical glossary — skim it before you start, and keep it open to look up any term you hit along the way.

How to read the pipeline manual (A–J)

If you take the pipeline path:

  • Step by step. A–G set up your data pipeline; H–J submit a test set. The « Prev / Next » buttons at the bottom of each page walk you through them in order, so you never have to guess what to do next.
  • Verify as you go. Most steps end with a Verify box: paste the command, compare against the expected output, and only move on once it matches.

Need help?