Upload (sda-cli)

This is the core of the direct path: encrypt your prepared dataset and upload it to your BigPicture inbox with sda-cli. Work through the steps in order.

Note

Make sure sda-cli is installed first — see Prerequisites (alternative path) — and that your dataset is laid out and validated per Prepare data and Validate metadata.

0. Test submission (recommended)

We strongly recommend a small test upload to the staging environment first, so you don’t have to re-upload terabytes if something is wrong. Upload two images with metadata generated per Prepare data — include all XML files, the PRIVATE folder (always required), and the LANDING_PAGE folder (if you want a landing page).

Follow the same steps below, but log in at the staging service instead of production.

1. Download the encryption key

For BigPicture to read your files, they must be encrypted with the correct public key. Download it:

Windows

curl -OL https://raw.githubusercontent.com/NBISweden/EGA-SE-user-docs/main/crypt4gh_bp_key.pub

macOS

curl -OL https://raw.githubusercontent.com/NBISweden/EGA-SE-user-docs/main/crypt4gh_bp_key.pub

Linux

curl -OL https://raw.githubusercontent.com/NBISweden/EGA-SE-user-docs/main/crypt4gh_bp_key.pub

Verify

ls crypt4gh_bp_key.pub

The file should be present in your working directory.

2. Encrypt the files

With the public key in place, encrypt your submission files. sda-cli creates its own encryption key automatically.

Windows

sda-cli encrypt --key crypt4gh_bp_key.pub <file_1_to_encrypt> <file_2_to_encrypt> ...

macOS

./sda-cli encrypt --key crypt4gh_bp_key.pub <file_1_to_encrypt> <file_2_to_encrypt> ...

Linux

./sda-cli encrypt --key crypt4gh_bp_key.pub <file_1_to_encrypt> <file_2_to_encrypt> ...

The tool also writes four checksum files — save these, you need them during submission:

checksum_encrypted.md5
checksum_encrypted.sha256
checksum_unencrypted.md5
checksum_unencrypted.sha256

Verify

Each input file should now have an encrypted .c4gh counterpart, and the four checksum files should exist:

ls *.c4gh checksum_*.md5 checksum_*.sha256

3. Download your configuration file and upload

1. Log in at the BigPicture login portal and follow the prompts to authenticate. Click Download credentials to upload to the Inbox to get your s3cmd-inbox.conf file. (For a test upload, use the staging service instead.)

2. Upload your encrypted files. By default the local folder structure is not preserved — all files land in the root of your inbox. Use --target-directory <path> to place them in a specific remote folder.

   To upload files individually:

   Windows

   sda-cli --config <configuration_file> upload <encrypted_file_1> <encrypted_file_2> ...

   macOS

   ./sda-cli --config <configuration_file> upload <encrypted_file_1> <encrypted_file_2> ...

   Linux

   ./sda-cli --config <configuration_file> upload <encrypted_file_1> <encrypted_file_2> ...

   To upload an entire directory — often more efficient — use the recursive -r flag, which preserves your local folder structure on the remote server:

   Windows

   sda-cli --config <configuration_file> upload -r <folder_to_upload>

   macOS

   ./sda-cli --config <configuration_file> upload -r <folder_to_upload>

   Linux

   ./sda-cli --config <configuration_file> upload -r <folder_to_upload>

Verify

sda-cli reports each file as it finishes uploading. For the full set of commands and flags — which may change between versions — consult the tool’s built-in help, the most reliable reference:

Windows

sda-cli help

macOS

./sda-cli help

Linux

./sda-cli help

4. Notify us

When the upload is complete, email bp-ops so we know to process it. Use this subject line:

submission - DATASET_{IDENTIFIER}

And include in the body:

• the name of the uploader, and
• the name of the dataset.