Description of feature
Check files for integrity (checksum verification) and copy them to destination.
Optionally read again from destination and check again (at least for existence)
- input:
- source directory
- list of files, relative to the source directory, including checksums (as csv, tsv, or space-separated files as created by
md5sum etc.)
- output:
Big questions: how can we guarantee that publishing files worked correctly? For AWS at least the API guarantees that a file is either stored correctly, or not at all, but "not at all" is not acceptable. Afaik failures to "publish" a file just create a warning in the nextflow log.
Description of feature
Check files for integrity (checksum verification) and copy them to destination.
Optionally read again from destination and check again (at least for existence)
md5sumetc.)Big questions: how can we guarantee that publishing files worked correctly? For AWS at least the API guarantees that a file is either stored correctly, or not at all, but "not at all" is not acceptable. Afaik failures to "publish" a file just create a warning in the nextflow log.