Prepare data

Before you upload anything, your dataset must be laid out in the exact folder structure BigPicture expects, and every data file must be encrypted with crypt4gh (the sda-cli encrypt step on the Upload page produces these .c4gh files for you).

Folder structure

The root of your submission is the dataset folder, which contains several subfolders:

DATASET_{IDENTIFIER}
|--- METADATA
|    |--- dataset.xml      (contains: Dataset)
|    |--- policy.xml       (contains: Policy)
|    |--- image.xml        (contains: Images)
|    |--- annotation.xml   (contains: Annotations)
|    |--- observation.xml  (contains: Observations)
|    |--- observer.xml     (contains: Observers)
|    |--- sample.xml        (contains: Biological Beings, Cases (if present), Specimens, Blocks and Slides)
|    |--- staining.xml      (contains: Stainings)
|--- IMAGES
|    |--- IMAGE_{IDENTIFIER}
|    |    |--- *.dcm        (DICOM files of an Image)
|    |--- IMAGE_{IDENTIFIER}
|    |    |--- *.dcm
|--- ANNOTATIONS
|    |--- *.geojson
|--- LANDING_PAGE
|    |--- landing_page.xml (contains: Landing Page)
|    |--- THUMBNAILS
|    |    |--- *.jpg
|--- PRIVATE               (not shared with users)
|    |--- rems.xml
|    |--- organisation.xml
|    |--- datacite.xml     (contains: DataCite, optional)
  • The root folder must be named DATASET_{IDENTIFIER}, where IDENTIFIER is either the accession ID generated by the repository (when data is downloaded) or the ALIAS you define at dataset creation/submission.
  • Each image folder must be named IMAGE_{IDENTIFIER}, following the same rule.
  • If the dataset has no annotations, the ANNOTATIONS folder and the annotation.xml file can be omitted.
Warning

Anything in the LANDING_PAGE folder should be treated as visible to the entire world. The PRIVATE folder, by contrast, holds metadata that is never shared with users who are granted access to a dataset.

File rules

  1. Every data file must be encrypted with crypt4gh and carry the .c4gh extension — for example image.xml.c4gh, image1.dcm.c4gh. (sda-cli encrypt does this on the Upload page.)

  2. Metadata is split between two subfolders: METADATA and PRIVATE.

  3. The only files allowed in METADATA are: dataset.xml, image.xml, observation.xml, observer.xml (optional), policy.xml, sample.xml, annotation.xml (optional), and staining.xml.

  4. The image.xml file must include the full path of each DICOM image plus the checksums of both the encrypted and unencrypted files:

    <FILES>
    <FILE filename="IMAGES/IMAGE_{IDENTIFIER}/*.dcm"
          checksum_method="SHA256"
          checksum="<encrypted_checksum>"
          unencrypted_checksum="<unencrypted_checksum>"
          filetype="dcm"/>
</FILES>
Verify

From inside your dataset folder, list the top-level layout and confirm the expected subfolders are present:

Get-ChildItem -Directory
ls -d */
ls -d */

You should see METADATA, IMAGES, and PRIVATE (plus ANNOTATIONS and LANDING_PAGE if your dataset uses them), and the folder name should start with DATASET_.

With the layout in place, continue to Validate metadata.