diff -Nru heudiconv-0.9.0/CHANGELOG.md heudiconv-0.10.0/CHANGELOG.md --- heudiconv-0.9.0/CHANGELOG.md 2020-12-23 15:35:32.000000000 +0000 +++ heudiconv-0.10.0/CHANGELOG.md 2021-09-16 18:13:22.000000000 +0000 @@ -4,6 +4,42 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). +## [0.10.0] - 2021-09-16 + +Various improvements and compatibility/support (dcm2niix, datalad) changes. + +### Added + +- Add "AcquisitionTime" to the seqinfo ([#487][]) +- Add support for saving the Phoenix Report in the sourcedata folder ([#489][]) + +### Changed + +- Python 3.5 EOLed, supported (tested) versions now: 3.6 - 3.9 +- In reprorin heuristic, allow for having multiple accessions since now there is + `-g all` groupping ([#508][]) +- For BIDS, produce a singular `scans.json` at the top level, and not one per + sub/ses (generates too many identical files) ([#507][]) + + +### Fixed + +- Compatibility with DataLad 0.15.0. Minimal version is 0.13.0 now. +- Try to open top level BIDS .json files a number of times for adjustment, + so in the case of competition across parallel processes, they just end up + with the last one "winning over" ([#523][]) +- Don't fail if etelemetry.get_project returns None ([#501][]) +- Consistently use `n/a` for age/sex, also handle ?M for months ([#500][]) +- To avoid crashing on unrelated derivatives files etc, make `find_files` to + take list of topdirs (excluding `derivatives/` etc), + and look for _bold only under sub-* directories ([#496][]) +- Ensure bvec/bval files are only created for dwi output ([#491][]) + +### Removed + +- In reproin heuristic, old hardcoded sequence renamings and filters ([#508][]) + + ## [0.9.0] - 2020-12-23 Various improvements and compatibility/support (dcm2niix, datalad, @@ -22,12 +58,12 @@ directory (with a `_heudiconv???` suffix, renamed into ultimate target name later on), which avoids hitting file size limits of /tmp ([#481][]) and helped to avoid a regression in dcm2nixx 1.0.20201102 -- #477 replaced `rec-` with `part-` now - that BIDS supports the part entity -- #473 made default for CogAtlasID to be a TODO URL -- #459 made AcquisitionTime used for acq_time scans file field -- #451 retained sub-second resolution in scans files -- #442 refactored code so there is now heudiconv.main.workflow for +- [#477][] replaced `rec-` with `part-` now + hat BIDSsupports the part entity +- [#473][] made default for CogAtlasID to be a TODO URL +- [#459][] made AcquisitionTime used for acq_time scans file field +- [#451][] retained sub-second resolution in scans files +- [#442][] refactored code so there is now heudiconv.main.workflow for more convenient use as a Python module ### Fixed @@ -350,6 +386,11 @@ [#368]: https://github.com/nipy/heudiconv/issues/368 [#373]: https://github.com/nipy/heudiconv/issues/373 [#485]: https://github.com/nipy/heudiconv/issues/485 +[#442]: https://github.com/nipy/heudiconv/issues/442 +[#451]: https://github.com/nipy/heudiconv/issues/451 +[#459]: https://github.com/nipy/heudiconv/issues/459 +[#473]: https://github.com/nipy/heudiconv/issues/473 +[#477]: https://github.com/nipy/heudiconv/issues/477 [#293]: https://github.com/nipy/heudiconv/issues/293 [#304]: https://github.com/nipy/heudiconv/issues/304 [#306]: https://github.com/nipy/heudiconv/issues/306 @@ -391,3 +432,12 @@ [#464]: https://github.com/nipy/heudiconv/issues/464 [#480]: https://github.com/nipy/heudiconv/issues/480 [#481]: https://github.com/nipy/heudiconv/issues/481 +[#487]: https://github.com/nipy/heudiconv/issues/487 +[#489]: https://github.com/nipy/heudiconv/issues/489 +[#491]: https://github.com/nipy/heudiconv/issues/491 +[#496]: https://github.com/nipy/heudiconv/issues/496 +[#500]: https://github.com/nipy/heudiconv/issues/500 +[#501]: https://github.com/nipy/heudiconv/issues/501 +[#507]: https://github.com/nipy/heudiconv/issues/507 +[#508]: https://github.com/nipy/heudiconv/issues/508 +[#523]: https://github.com/nipy/heudiconv/issues/523 diff -Nru heudiconv-0.9.0/debian/changelog heudiconv-0.10.0/debian/changelog --- heudiconv-0.9.0/debian/changelog 2021-11-17 09:42:36.000000000 +0000 +++ heudiconv-0.10.0/debian/changelog 2021-11-17 17:22:11.000000000 +0000 @@ -1,3 +1,18 @@ +heudiconv (0.10.0-1) unstable; urgency=medium + + * Team Upload. + * New upstream version 0.10.0 + * d/rules: copy test dir to build directory before + tests, since some files are not copied by default + * Drop merged patches, pull a upstream patch to fix tests + * Add python3:Depends to Depends field, use --with python3 to + use the python helper + * Set PYTHONPATH to properly generate manpage + * Fix copyright years + * Migrate to secure URI + + -- Nilesh Patra Wed, 17 Nov 2021 22:52:11 +0530 + heudiconv (0.9.0-3) unstable; urgency=medium * Team upload. diff -Nru heudiconv-0.9.0/debian/control heudiconv-0.10.0/debian/control --- heudiconv-0.9.0/debian/control 2021-11-17 09:42:36.000000000 +0000 +++ heudiconv-0.10.0/debian/control 2021-11-17 17:22:11.000000000 +0000 @@ -26,7 +26,8 @@ Package: heudiconv Architecture: all -Depends: ${misc:Depends}, +Depends: ${python3:Depends}, + ${misc:Depends}, dcm2niix, python3, python3-dcmstack, diff -Nru heudiconv-0.9.0/debian/copyright heudiconv-0.10.0/debian/copyright --- heudiconv-0.9.0/debian/copyright 2021-11-17 09:42:36.000000000 +0000 +++ heudiconv-0.10.0/debian/copyright 2021-11-17 17:22:11.000000000 +0000 @@ -4,18 +4,18 @@ Files: * Copyright: 2014-2015 Satrajit Ghosh , - 2015 Michael Hanke - 2016- HeuDiConv Team + 2015-2021 Michael Hanke + 2016-2021 HeuDiConv Team License: Apache-2.0 Files: debian/* -Copyright: 2015-2017 Michael Hanke - 2017- Yaroslav Halchenko +Copyright: 2015-2021 Michael Hanke + 2017-2021 Yaroslav Halchenko License: Apache-2.0 License: Apache-2.0 This file is licensed under the terms of the Apache License Version 2.0 - http://www.apache.org/licenses. This notice must appear in modified or not + https://www.apache.org/licenses. This notice must appear in modified or not redistributions of this file. . Redistributions of this Software, with or without modification, must diff -Nru heudiconv-0.9.0/debian/patches/0001-BF-datalad-use-public-call_git-added-in-0.12.0-not-d.patch heudiconv-0.10.0/debian/patches/0001-BF-datalad-use-public-call_git-added-in-0.12.0-not-d.patch --- heudiconv-0.9.0/debian/patches/0001-BF-datalad-use-public-call_git-added-in-0.12.0-not-d.patch 2021-11-17 09:42:36.000000000 +0000 +++ heudiconv-0.10.0/debian/patches/0001-BF-datalad-use-public-call_git-added-in-0.12.0-not-d.patch 1970-01-01 00:00:00.000000000 +0000 @@ -1,26 +0,0 @@ -From 9dfafb340600684265a0af50f2738f25260838c1 Mon Sep 17 00:00:00 2001 -From: Yaroslav Halchenko -Date: Sat, 6 Feb 2021 12:41:20 -0500 -Subject: [PATCH] BF: datalad - use public call_git (added in 0.12.0), not - deprected now protected method - ---- - heudiconv/tests/test_main.py | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -diff --git a/heudiconv/tests/test_main.py b/heudiconv/tests/test_main.py -index 89fde8b..31c0142 100644 ---- a/heudiconv/tests/test_main.py -+++ b/heudiconv/tests/test_main.py -@@ -155,7 +155,7 @@ def test_prepare_for_datalad(tmpdir): - assert '.heudiconv/dummy.nii.gz' in ds.repo.get_files() - - # Let's now roll back and make it a proper submodule -- ds.repo._git_custom_command([], ['git', 'reset', '--hard', old_hexsha]) -+ ds.repo.call_git(['reset', '--hard', old_hexsha]) - # now we do not add dummy to git - create_file_if_missing(dummy_path, '') - add_to_datalad(str(tmpdir), studydir_, None, False) --- -2.29.2 - diff -Nru heudiconv-0.9.0/debian/patches/0001-BF-use-caplog-instead-of-capfd-for-testing-if-we-log-a-warning.patch heudiconv-0.10.0/debian/patches/0001-BF-use-caplog-instead-of-capfd-for-testing-if-we-log-a-warning.patch --- heudiconv-0.9.0/debian/patches/0001-BF-use-caplog-instead-of-capfd-for-testing-if-we-log-a-warning.patch 1970-01-01 00:00:00.000000000 +0000 +++ heudiconv-0.10.0/debian/patches/0001-BF-use-caplog-instead-of-capfd-for-testing-if-we-log-a-warning.patch 2021-11-17 17:22:11.000000000 +0000 @@ -0,0 +1,45 @@ +From 144fbfce789d8fbd08623aecb459dd7616e8c740 Mon Sep 17 00:00:00 2001 +From: Yaroslav Halchenko +Date: Thu, 28 Oct 2021 15:24:47 -0400 +Subject: [PATCH] BF(TST): use caplog instead of capfd for testing if we log a + warning + +for some reason on my laptop capfd failed to provide desired effect. May be because +I am running pytest with -s ;) will not investigate -- using caplog is the more +kosher way for this purpose, and allows for control of the logging level we would like +to have to ensure that the message is output +--- + heudiconv/tests/test_convert.py | 7 ++++--- + 1 file changed, 4 insertions(+), 3 deletions(-) + +diff --git a/heudiconv/tests/test_convert.py b/heudiconv/tests/test_convert.py +index 14082484..6a29895b 100644 +--- a/heudiconv/tests/test_convert.py ++++ b/heudiconv/tests/test_convert.py +@@ -85,12 +85,14 @@ def test_update_uncombined_name(): + assert out_fn_test == out_fn_true + + +-def test_b0dwi_for_fmap(tmpdir, capfd): ++def test_b0dwi_for_fmap(tmpdir, caplog): + """Make sure we raise a warning when .bvec and .bval files + are present but the modality is not dwi. + We check it by extracting a few DICOMs from a series with + bvals: 5 5 1500 + """ ++ import logging ++ caplog.set_level(logging.WARNING) + tmppath = tmpdir.strpath + subID = 'b0dwiForFmap' + args = ( +@@ -101,9 +103,8 @@ def test_b0dwi_for_fmap(tmpdir, capfd): + + # assert that it raised a warning that the fmap directory will contain + # bvec and bval files. +- output = capfd.readouterr().err.split('\n') + expected_msg = DW_IMAGE_IN_FMAP_FOLDER_WARNING.format(folder=op.join(tmppath, 'sub-%s', 'fmap') % subID) +- assert [o for o in output if expected_msg in o] ++ assert any(expected_msg in c.message for c in caplog.records) + + # check that both 'fmap' and 'dwi' directories have been extracted and they contain + # *.bvec and a *.bval files diff -Nru heudiconv-0.9.0/debian/patches/0002-BF-account-for-datalad.create-no_annex-removal-deprecated-in-0.13.0 heudiconv-0.10.0/debian/patches/0002-BF-account-for-datalad.create-no_annex-removal-deprecated-in-0.13.0 --- heudiconv-0.9.0/debian/patches/0002-BF-account-for-datalad.create-no_annex-removal-deprecated-in-0.13.0 2021-11-17 09:42:36.000000000 +0000 +++ heudiconv-0.10.0/debian/patches/0002-BF-account-for-datalad.create-no_annex-removal-deprecated-in-0.13.0 1970-01-01 00:00:00.000000000 +0000 @@ -1,38 +0,0 @@ -From d09cec11b975ad4642a6978d4660b9d6bde45e7a Mon Sep 17 00:00:00 2001 -From: Yaroslav Halchenko -Date: Wed, 15 Sep 2021 12:00:47 -0400 -Subject: [PATCH] BF: account for datalad.create no_annex removal, deprecated - in 0.13.0 - -(now minimal supported version) ---- - heudiconv/external/dlad.py | 2 +- - heudiconv/info.py | 2 +- - 2 files changed, 2 insertions(+), 2 deletions(-) - -diff --git a/heudiconv/external/dlad.py b/heudiconv/external/dlad.py -index 01c229b6..db8907f1 100644 ---- a/heudiconv/external/dlad.py -+++ b/heudiconv/external/dlad.py -@@ -58,7 +58,7 @@ def add_to_datalad(topdir, studydir, msg, bids): - ds_ = dl.create(curdir_, dataset=superds, - force=True, - # initiate annex only at the bottom repository -- no_annex=isubdir<(len(subdirs)-1), -+ annex=isubdir==(len(subdirs)-1), - fake_dates=True, - # shared_access='all', - ) -diff --git a/heudiconv/info.py b/heudiconv/info.py -index 24e4c32f..2b065847 100644 ---- a/heudiconv/info.py -+++ b/heudiconv/info.py -@@ -38,7 +38,7 @@ - 'inotify', - ] - --MIN_DATALAD_VERSION = '0.12.4' -+MIN_DATALAD_VERSION = '0.13.0' - EXTRA_REQUIRES = { - 'tests': TESTS_REQUIRES, - 'extras': [ diff -Nru heudiconv-0.9.0/debian/patches/deb-no-demand-on-etelemetry heudiconv-0.10.0/debian/patches/deb-no-demand-on-etelemetry --- heudiconv-0.9.0/debian/patches/deb-no-demand-on-etelemetry 2021-11-17 09:42:36.000000000 +0000 +++ heudiconv-0.10.0/debian/patches/deb-no-demand-on-etelemetry 2021-11-17 17:22:11.000000000 +0000 @@ -8,7 +8,7 @@ --- a/heudiconv/info.py +++ b/heudiconv/info.py -@@ -25,7 +25,7 @@ REQUIRES = [ +@@ -26,7 +26,7 @@ 'pydicom', 'nipype >=1.2.3', 'dcmstack>=0.8', diff -Nru heudiconv-0.9.0/debian/patches/series heudiconv-0.10.0/debian/patches/series --- heudiconv-0.9.0/debian/patches/series 2021-11-17 09:42:36.000000000 +0000 +++ heudiconv-0.10.0/debian/patches/series 2021-11-17 17:22:11.000000000 +0000 @@ -1,3 +1,2 @@ deb-no-demand-on-etelemetry -0001-BF-datalad-use-public-call_git-added-in-0.12.0-not-d.patch -0002-BF-account-for-datalad.create-no_annex-removal-deprecated-in-0.13.0 +0001-BF-use-caplog-instead-of-capfd-for-testing-if-we-log-a-warning.patch diff -Nru heudiconv-0.9.0/debian/rules heudiconv-0.10.0/debian/rules --- heudiconv-0.9.0/debian/rules 2021-11-17 09:42:36.000000000 +0000 +++ heudiconv-0.10.0/debian/rules 2021-11-17 17:22:11.000000000 +0000 @@ -10,14 +10,15 @@ export EMAIL=debian@example.com export GIT_AUTHOR_NAME="The Name" export GIT_COMMITTER_NAME=$(GIT_AUTHOR_NAME) +export PYBUILD_BEFORE_TEST := cp -a heudiconv/tests {build_dir}/heudiconv/ %: - dh $@ --buildsystem=pybuild + dh $@ --with python3 --buildsystem=pybuild override_dh_auto_install: dh_auto_install mkdir -p build - PYTHONPATH=$$(/bin/ls -d ./debian/heudiconv/usr/lib/python*/*-packages | head -n 1) + PYTHONPATH=$$(/bin/ls -d ./debian/heudiconv/usr/lib/python*/*-packages | head -n 1) \ help2man -n 'DICOM converter for organizing brain imaging data into structured directory layouts' \ -N --no-discard-stderr \ "python3 -W ignore ./debian/heudiconv/usr/bin/heudiconv" >| build/heudiconv.1 diff -Nru heudiconv-0.9.0/debian/watch heudiconv-0.10.0/debian/watch --- heudiconv-0.9.0/debian/watch 2021-11-17 09:42:36.000000000 +0000 +++ heudiconv-0.10.0/debian/watch 2021-11-17 17:22:11.000000000 +0000 @@ -1,3 +1,3 @@ version=4 opts="filenamemangle=s/.*\/v(\d)\.(\d).*/heudiconv-$1.$2\.tar\.gz/" \ - http://github.com/nipy/heudiconv/tags .*archive/.*/v(\d[\d.]+).tar.gz + https://github.com/nipy/heudiconv/tags .*archive/.*/v(\d[\d.]+).tar.gz diff -Nru heudiconv-0.9.0/docs/conf.py heudiconv-0.10.0/docs/conf.py --- heudiconv-0.9.0/docs/conf.py 2020-12-23 15:35:32.000000000 +0000 +++ heudiconv-0.10.0/docs/conf.py 2021-09-16 18:13:22.000000000 +0000 @@ -26,7 +26,7 @@ # The short X.Y version version = '' # The full version, including alpha/beta/rc tags -release = '0.9.0' +release = '0.10.0' # -- General configuration --------------------------------------------------- diff -Nru heudiconv-0.9.0/docs/installation.rst heudiconv-0.10.0/docs/installation.rst --- heudiconv-0.9.0/docs/installation.rst 2020-12-23 15:35:32.000000000 +0000 +++ heudiconv-0.10.0/docs/installation.rst 2021-09-16 18:13:22.000000000 +0000 @@ -26,7 +26,7 @@ can visit `our page on Docker Hub `_ to view available releases. To pull the latest release, run:: - $ docker pull nipy/heudiconv:0.9.0 + $ docker pull nipy/heudiconv:0.10.0 Singularity @@ -35,4 +35,4 @@ you can use it to pull and convert our Docker images! For example, to pull and build the latest release, you can run:: - $ singularity pull docker://nipy/heudiconv:0.9.0 + $ singularity pull docker://nipy/heudiconv:0.10.0 diff -Nru heudiconv-0.9.0/docs/usage.rst heudiconv-0.10.0/docs/usage.rst --- heudiconv-0.9.0/docs/usage.rst 2020-12-23 15:35:32.000000000 +0000 +++ heudiconv-0.10.0/docs/usage.rst 2021-09-16 18:13:22.000000000 +0000 @@ -82,7 +82,7 @@ DCMDIR=${DCMDIRS[${SLURM_ARRAY_TASK_ID}]} echo Submitted directory: ${DCMDIR} - IMG="/singularity-images/heudiconv-0.9.0-dev.sif" + IMG="/singularity-images/heudiconv-0.10.0-dev.sif" CMD="singularity run -B ${DCMDIR}:/dicoms:ro -B ${OUTDIR}:/output -e ${IMG} --files /dicoms/ -o /output -f reproin -c dcm2niix -b notop --minmeta -l ." printf "Command:\n${CMD}\n" @@ -97,7 +97,7 @@ set -eu OUTDIR=${1} - IMG="/singularity-images/heudiconv-0.9.0-dev.sif" + IMG="/singularity-images/heudiconv-0.10.0-dev.sif" CMD="singularity run -B ${OUTDIR}:/output -e ${IMG} --files /output -f reproin --command populate-templates" printf "Command:\n${CMD}\n" diff -Nru heudiconv-0.9.0/heudiconv/bids.py heudiconv-0.10.0/heudiconv/bids.py --- heudiconv-0.9.0/heudiconv/bids.py 2020-12-23 15:13:14.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/bids.py 2021-09-16 18:13:22.000000000 +0000 @@ -18,7 +18,7 @@ load_json, save_json, create_file_if_missing, - json_dumps_pretty, + json_dumps, set_readonly, is_readonly, get_datetime, @@ -47,6 +47,36 @@ BIDS_VERSION = "1.4.1" +def maybe_na(val): + """Return 'n/a' if non-None value represented as str is not empty + + Primarily for the consistent use of lower case 'n/a' so 'N/A' and 'NA' + are also treated as 'n/a' + """ + if val is not None: + val = str(val) + val = val.strip() + return 'n/a' if (not val or val in ('N/A', 'NA')) else val + + +def treat_age(age): + """Age might encounter 'Y' suffix or be a float""" + age = str(age) + if age.endswith('M'): + age = age.rstrip('M') + age = float(age) / 12 + age = ('%.2f' if age != int(age) else '%d') % age + else: + age = age.rstrip('Y') + if age: + # strip all leading 0s but allow to scan a newborn (age 0Y) + age = '0' if not age.lstrip('0') else age.lstrip('0') + if age.startswith('.'): + # we had float point value, let's prepend 0 + age = '0' + age + return age + + def populate_bids_templates(path, defaults={}): """Premake BIDS text files with templates""" @@ -90,6 +120,9 @@ create_file_if_missing(op.join(path, 'README'), "TODO: Provide description for the dataset -- basic details about the " "study, possibly pointing to pre-registration (if public or embargoed)") + create_file_if_missing(op.join(path, 'scans.json'), + json_dumps(SCANS_FILE_FIELDS, sort_keys=False) + ) populate_aggregated_jsons(path) @@ -114,7 +147,8 @@ # way too many -- let's just collect all which are the same! # FIELDS_TO_TRACK = {'RepetitionTime', 'FlipAngle', 'EchoTime', # 'Manufacturer', 'SliceTiming', ''} - for fpath in find_files('.*_task-.*\_bold\.json', topdir=path, + for fpath in find_files('.*_task-.*\_bold\.json', + topdir=glob(op.join(path, 'sub-*')), exclude_vcs=True, exclude="/\.(datalad|heudiconv)/"): # @@ -123,7 +157,7 @@ # TODO: if we are to fix it, then old ones (without _acq) should be # removed first task = re.sub('.*_(task-[^_\.]*(_acq-[^_\.]*)?)_.*', r'\1', fpath) - json_ = load_json(fpath) + json_ = load_json(fpath, retry=100) if task not in tasks: tasks[task] = json_ else: @@ -178,7 +212,7 @@ "CogAtlasID": "http://www.cognitiveatlas.org/task/id/TODO", } if op.lexists(task_file): - j = load_json(task_file) + j = load_json(task_file, retry=100) # Retain possibly modified placeholder fields for f in placeholders: if f in j: @@ -277,12 +311,13 @@ "control group)")])), ]), sort_keys=False) + # Add a new participant with open(participants_tsv, 'a') as f: f.write( '\t'.join(map(str, [participant_id, - age.lstrip('0').rstrip('Y') if age else 'N/A', - sex, + maybe_na(treat_age(age)), + maybe_na(sex), 'control'])) + '\n') @@ -372,11 +407,6 @@ os.unlink(fn) else: fnames2info = newrows - # Populate _scans.json (an optional file to describe column names in - # _scans.tsv). This auto generation will make BIDS-validator happy. - scans_json = '.'.join(fn.split('.')[:-1] + ['json']) - if not op.lexists(scans_json): - save_json(scans_json, SCANS_FILE_FIELDS, sort_keys=False) header = SCANS_FILE_FIELDS # prepare all the data rows diff -Nru heudiconv-0.9.0/heudiconv/convert.py heudiconv-0.10.0/heudiconv/convert.py --- heudiconv-0.9.0/heudiconv/convert.py 2020-12-23 15:13:15.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/convert.py 2021-09-16 18:13:22.000000000 +0000 @@ -41,6 +41,7 @@ ) LOCKFILE = 'heudiconv.lock' +DW_IMAGE_IN_FMAP_FOLDER_WARNING = 'Diffusion-weighted image saved in non dwi folder ({folder})' lgr = logging.getLogger(__name__) @@ -674,9 +675,22 @@ return if isdefined(res.outputs.bvecs) and isdefined(res.outputs.bvals): - outname_bvecs, outname_bvals = prefix + '.bvec', prefix + '.bval' - safe_movefile(res.outputs.bvecs, outname_bvecs, overwrite) - safe_movefile(res.outputs.bvals, outname_bvals, overwrite) + if prefix_dirname.endswith('dwi'): + outname_bvecs, outname_bvals = prefix + '.bvec', prefix + '.bval' + safe_movefile(res.outputs.bvecs, outname_bvecs, overwrite) + safe_movefile(res.outputs.bvals, outname_bvals, overwrite) + else: + if bvals_are_zero(res.outputs.bvals): + os.remove(res.outputs.bvecs) + os.remove(res.outputs.bvals) + lgr.debug("%s and %s were removed since not dwi", res.outputs.bvecs, res.outputs.bvals) + else: + lgr.warning(DW_IMAGE_IN_FMAP_FOLDER_WARNING.format(folder= prefix_dirname)) + lgr.warning(".bvec and .bval files will be generated. This is NOT BIDS compliant") + outname_bvecs, outname_bvals = prefix + '.bvec', prefix + '.bval' + safe_movefile(res.outputs.bvecs, outname_bvecs, overwrite) + safe_movefile(res.outputs.bvals, outname_bvals, overwrite) + if isinstance(res_files, list): res_files = sorted(res_files) @@ -806,3 +820,23 @@ # write to outfile save_json(infofile, meta_info) + + +def bvals_are_zero(bval_file): + """Checks if all entries in a bvals file are zero (or 5, for Siemens files). + Returns True if that is the case, otherwise returns False + + Parameters + ---------- + bval_file : file with the bvals + + Returns + ------- + True if all are zero; False otherwise. + """ + + with open(bval_file) as f: + bvals = f.read().split() + + bvals_unique = set(float(b) for b in bvals) + return bvals_unique == {0.} or bvals_unique == {5.} diff -Nru heudiconv-0.9.0/heudiconv/dicoms.py heudiconv-0.10.0/heudiconv/dicoms.py --- heudiconv-0.9.0/heudiconv/dicoms.py 2020-07-29 16:27:07.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/dicoms.py 2021-09-16 18:13:22.000000000 +0000 @@ -89,7 +89,8 @@ patient_age=dcminfo.get('PatientAge'), patient_sex=dcminfo.get('PatientSex'), date=dcminfo.get('AcquisitionDate'), - series_uid=dcminfo.get('SeriesInstanceUID') + series_uid=dcminfo.get('SeriesInstanceUID'), + time=dcminfo.get('AcquisitionTime'), ) return seqinfo @@ -265,8 +266,13 @@ series_id = '-'.join(map(str, series_id)) if mw.image_shape is None: # this whole thing has no image data (maybe just PSg DICOMs) - # nothing to see here, just move on - continue + # If this is a Siemens PhoenixZipReport or PhysioLog, keep it: + if mw.dcm_data.SeriesDescription == 'PhoenixZIPReport': + # give it a dummy shape, so that we can continue: + mw.image_shape = (0, 0, 0) + else: + # nothing to see here, just move on + continue seqinfo = create_seqinfo(mw, series_files, series_id) if per_studyUID: diff -Nru heudiconv-0.9.0/heudiconv/external/dlad.py heudiconv-0.10.0/heudiconv/external/dlad.py --- heudiconv-0.9.0/heudiconv/external/dlad.py 2020-12-22 22:19:56.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/external/dlad.py 2021-09-16 18:13:22.000000000 +0000 @@ -58,7 +58,7 @@ ds_ = dl.create(curdir_, dataset=superds, force=True, # initiate annex only at the bottom repository - no_annex=isubdir<(len(subdirs)-1), + annex=isubdir==(len(subdirs)-1), fake_dates=True, # shared_access='all', ) diff -Nru heudiconv-0.9.0/heudiconv/heuristics/bids_PhoenixReport.py heudiconv-0.10.0/heudiconv/heuristics/bids_PhoenixReport.py --- heudiconv-0.9.0/heudiconv/heuristics/bids_PhoenixReport.py 1970-01-01 00:00:00.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/heuristics/bids_PhoenixReport.py 2021-09-16 18:13:22.000000000 +0000 @@ -0,0 +1,38 @@ +"""Heuristic demonstrating conversion of the PhoenixZIPReport from Siemens. + +It only cares about converting a series with have PhoenixZIPReport in their +series_description and outputs **only to sourcedata**. +""" + + +def create_key(template, outtype=('nii.gz',), annotation_classes=None): + if template is None or not template: + raise ValueError('Template must be a valid format string') + return template, outtype, annotation_classes + + +def infotodict(seqinfo): + """Heuristic evaluator for determining which runs belong where + + allowed template fields - follow python string module: + + item: index within category + subject: participant id + seqitem: run number during scanning + subindex: sub index within group + """ + sbref = create_key('sub-{subject}/func/sub-{subject}_task-QA_sbref', outtype=('nii.gz', 'dicom',)) + scout = create_key('sub-{subject}/anat/sub-{subject}_T1w', outtype=('nii.gz', 'dicom',)) + phoenix_doc = create_key('sub-{subject}/misc/sub-{subject}_phoenix', outtype=('dicom',)) + + info = {sbref: [], scout: [], phoenix_doc: []} + for s in seqinfo: + if ( + 'PhoenixZIPReport' in s.series_description + and s.image_type[3] == 'CSA REPORT' + ): + info[phoenix_doc].append({'item': s.series_id}) + if 'scout' in s.series_description.lower(): + info[scout].append({'item': s.series_id}) + + return info diff -Nru heudiconv-0.9.0/heudiconv/heuristics/reproin.py heudiconv-0.10.0/heudiconv/heuristics/reproin.py --- heudiconv-0.9.0/heudiconv/heuristics/reproin.py 2020-12-22 22:19:56.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/heuristics/reproin.py 2021-09-16 18:13:22.000000000 +0000 @@ -148,150 +148,55 @@ # NOTE: even if filename has number that is 0-padded, internally no padding # is done fix_accession2run = { - 'A000005': ['^1-'], - 'A000035': ['^8-', '^9-'], - 'A000067': ['^9-'], - 'A000072': ['^5-'], - 'A000081': ['^5-'], - 'A000082': ['^5-'], - 'A000088': ['^9-'], - 'A000090': ['^5-'], - 'A000127': ['^21-'], - 'A000130': ['^15-'], - 'A000137': ['^9-', '^11-'], - 'A000297': ['^12-'], - 'A000326': ['^15-'], - 'A000376': ['^15-'], - 'A000384': ['^8-', '^11-'], - 'A000467': ['^15-'], - 'A000490': ['^15-'], - 'A000511': ['^15-'], - 'A000797': ['^[1-7]-'], + # e.g.: + # 'A000035': ['^8-', '^9-'], } -# dictionary containing fixes, keys are md5sum of study_description from -# dicoms, in the form of PI-Experimenter^protocolname -# values are list of tuples in the form (regex_pattern, substitution) +# A dictionary containing fixes/remapping for sequence names per study. +# Keys are md5sum of study_description from DICOMs, in the form of PI-Experimenter^protocolname +# You can use `heudiconv -f reproin --command ls --files PATH +# to list the "study hash". +# Values are list of tuples in the form (regex_pattern, substitution). +# If the key is an empty string`''''`, it would apply to any study. protocols2fix = { - # QA - '43b67d9139e8c7274578b7451ab21123': - [ - # ('anat-scout.*', 'anat-scout_ses-{date}'), - # do not change it so we retain _ses-{date} - # ('anat-scout.*', 'anat-scout'), - ('BOLD_p2_s4_3\.5mm', 'func_task-rest_acq-p2-s4-3.5mm'), - ('BOLD_p2_s4', 'func_task-rest_acq-p2-s4'), - ('BOLD_p2_noprescannormalize', 'func-bold_task-rest_acq-p2noprescannormalize'), - ('BOLD_p2', 'func-bold_task-rest_acq-p2'), - ('BOLD_', 'func_task-rest'), - ('DTI_30_p2_s4_3\.5mm', 'dwi_acq-DTI-30-p2-s4-3.5mm'), - ('DTI_30_p2_s4', 'dwi_acq-DTI-30-p2-s4'), - ('DTI_30_p2', 'dwi_acq-DTI-30-p2'), - ('_p2_s4_3\.5mm', '_acq-p2-s4-3.5mm'), - ('_p2_s4', '_acq-p2-s4'), - ('_p2', '_acq-p2'), - ], - '9d148e2a05f782273f6343507733309d': - [('anat_', 'anat-'), - ('run-life[0-9]', 'run+_task-life'), - ('scout_run\+', 'scout'), - ('T2w', 'T2w_run+'), - # substitutions for old protocol names - ('AAHead_Scout_32ch-head-coil', 'anat-scout'), - ('MPRAGE', 'anat-T1w_acq-MPRAGE_run+'), - ('gre_field_mapping_2mm', 'fmap_run+_acq-2mm'), - ('gre_field_mapping_3mm', 'fmap_run+_acq-3mm'), - ('epi_bold_sms_p2_s4_2mm_life1_748', - 'func_run+_task-life_acq-2mm748'), - ('epi_bold_sms_p2_s4_2mm_life2_692', - 'func_run+_task-life_acq-2mm692'), - ('epi_bold_sms_p2_s4_2mm_life3_754', - 'func_run+_task-life_acq-2mm754'), - ('epi_bold_sms_p2_s4_2mm_life4_824', - 'func_run+_task-life_acq-2mm824'), - ('epi_bold_p2_3mm_nofs_life1_374', - 'func_run+_task-life_acq-3mmnofs374'), - ('epi_bold_p2_3mm_nofs_life2_346', - 'func_run+_task-life_acq-3mmnofs346'), - ('epi_bold_p2_3mm_nofs_life3_377', - 'func_run+_task-life_acq-3mmnofs377'), - ('epi_bold_p2_3mm_nofs_life4_412', - 'func_run+_task-life_acq-3mmnofs412'), - ('t2_space_sag_p4_iso', 'anat-T2w_run+'), - ('gre_field_mapping_2.4mm', 'fmap_run+_acq-2.4mm'), - ('rest_p2_sms4_2.4mm_64sl_1000tr_32te_600dyn', - 'func_run+_task-rest_acq-2.4mm64sl1000tr32te600dyn'), - ('DTI_30', 'dwi_run+_acq-30'), - ('t1_space_sag_p2_iso', 'anat-T1w_acq-060mm_run+')], - '76b36c80231b0afaf509e2d52046e964': - [('fmap_run\+_2mm', 'fmap_run+_acq-2mm')], - 'c6d8fbccc72990bee61d28e73b2618a4': - [('run=', 'run+')], - 'a751cc977f1e354fcafcb0ea2de123bd': - [ - ('_unlabeled', '_task-unlabeled'), - ('_mSense', '_acq-mSense'), - ('_p1_sms4_2.5mm', '_acq-p1-sms4-2.5mm'), - ('_p1_sms4_3mm', '_acq-p1-sms4-3mm'), - ], - 'd160113cf5ea8c5d0cbbbe14ef625e76': - [ - ('_run0', '_run-0'), - ], - '1bd62e10672fe0b435a9aa8d75b45425': - [ - # need to add incrementing session -- study should have 2 - # and no need for run+ for the scout! - ('scout(_run\+)?$', 'scout_ses+'), - ], - 'da218a66de902adb3ad9407d514e3639': - [ - # those sequences renamed later to include DTI- in their acq- - # so fot consistency - ('hardi_64', 'dwi_acq-DTI-hardi64'), - ('acq-hardi', 'acq-DTI-hardi'), - ], - 'ed20c1ad4a0861b2b65768e159258eec': - [ - ('fmap_acq-discorr-dti-', 'fmap_acq-dwi_dir-'), - ('_test', ''), - ], - '1996f745c30c1df1d3851844e56d294f': - [ - ('fmap_acq-discorr-dti-', 'fmap_acq-dwi_dir-'), - ], - # '022969bfde39c2940c114edf1db3fabc': - # [ # should be applied only for ses-03! - # ('_acq-MPRAGE_ses-02', '_acq-MPRAGE_ses-03'), - # ], - # to be used only once for one interrupted accession but we cannot - # fix per accession yet - # '23763823d2b9b4b09dafcadc8e8edf21': - # [ - # ('anat-T1w_acq-MPRAGE', 'anat-T1w_acq-MPRAGE_run-06'), - # ('anat_T2w', 'anat_T2w_run-06'), - # ('fmap_acq-3mm', 'fmap_acq-3mm_run-06'), - # ], + # e.g., QA: + # '43b67d9139e8c7274578b7451ab21123': + # [ + # ('BOLD_p2_s4_3\.5mm', 'func_task-rest_acq-p2-s4-3.5mm'), + # ('BOLD_', 'func_task-rest'), + # ('_p2_s4', '_acq-p2-s4'), + # ('_p2', '_acq-p2'), + # ], + # '': # for any study example with regexes used + # [ + # ('AAHead_Scout_.*', 'anat-scout'), + # ('^dti_.*', 'dwi'), + # ('^.*_distortion_corr.*_([ap]+)_([12])', r'fmap-epi_dir-\1_run-\2'), + # ('^(.+)_ap.*_r(0[0-9])', r'func_task-\1_run-\2'), + # ('^t1w_.*', 'anat-T1w'), + # # problematic case -- multiple identically named pepolar fieldmap runs + # # I guess we will just sacrifice ability to detect canceled runs here. + # # And we cannot just use _run+ since it would increment independently + # # for ap and then for pa. We will rely on having ap preceding pa. + # # Added _acq-mb8 so they match the one in funcs + # ('func_task-discorr_acq-ap', r'fmap-epi_dir-ap_acq-mb8_run+'), + # ('func_task-discorr_acq-pa', r'fmap-epi_dir-pa_acq-mb8_run='), + # ] } -# there was also screw up in the locator specification -# so we need to fix in both -# protocols2fix['67ae5e641ea9d487b6fdf56fb91aeb93'] = protocols2fix['022969bfde39c2940c114edf1db3fabc'] # list containing StudyInstanceUID to skip -- hopefully doesn't happen too often dicoms2skip = [ - '1.3.12.2.1107.5.2.43.66112.30000016110117002435700000001', - '1.3.12.2.1107.5.2.43.66112.30000016102813152550600000004', # double scout + # e.g. + # '1.3.12.2.1107.5.2.43.66112.30000016110117002435700000001', ] DEFAULT_FIELDS = { # Let it just be in each json file extracted - # 'Manufacturer': "Siemens", - # 'ManufacturersModelName': "Prisma", "Acknowledgements": "We thank Terry Sacket and the rest of the DBIC (Dartmouth Brain Imaging " "Center) personnel for assistance in data collection, and " - "Yaroslav Halchenko and Matteo Visconti for preparing BIDS dataset. " - "TODO: more", + "Yaroslav O. Halchenko for preparing BIDS dataset. " + "TODO: adjust to your case.", } @@ -311,38 +216,10 @@ def filter_files(fn): """Return True if a file should be kept, else False. - We're using it to filter out files that do not start with a number.""" - # do not check for these accession numbers because they haven't been - # recopied with the initial number - donotfilter = ['A000012', 'A000013', 'A000020', 'A000041'] - - split = os.path.split(fn) - split2 = os.path.split(split[0]) - sequence_dir = split2[1] - split3 = os.path.split(split2[0]) - accession_number = split3[1] + ATM reproin does not do any filtering. Override if you need to add some + """ return True - if accession_number == 'A000043': - # crazy one that got copied for some runs but not for others, - # so we are going to discard those that got copied and let heudiconv - # figure out the rest - return False if re.match('^[0-9]+-', sequence_dir) else True - elif accession_number == 'unknown': - # this one had some stuff without study description, filter stuff before - # collecting info, so it doesn't crash completely - return False if re.match('^[34][07-9]-sn', sequence_dir) else True - elif accession_number in donotfilter: - return True - elif accession_number.startswith('phantom-'): - # Accessions on phantoms, e.g. in dartmouth-phantoms/bids_test4-20161014 - return True - elif accession_number.startswith('heudiconvdcm'): - # we were given some tarball with dicoms which was extracted so we - # better obey - return True - else: - return True if re.match('^[0-9]+-', sequence_dir) else False def create_key(subdir, file_suffix, outtype=('nii.gz', 'dicom'), @@ -381,13 +258,17 @@ def fix_canceled_runs(seqinfo): """Function that adds cancelme_ to known bad runs which were forgotten """ - accession_number = get_unique(seqinfo, 'accession_number') - if accession_number in fix_accession2run: - lgr.info("Considering some runs possibly marked to be " - "canceled for accession %s", accession_number) - badruns = fix_accession2run[accession_number] - badruns_pattern = '|'.join(badruns) - for i, s in enumerate(seqinfo): + if not fix_accession2run: + return seqinfo # nothing to do + for i, s in enumerate(seqinfo): + accession_number = getattr(s, 'accession_number') + if accession_number and accession_number in fix_accession2run: + lgr.info("Considering some runs possibly marked to be " + "canceled for accession %s", accession_number) + # This code is reminiscent of prior logic when operating on + # a single accession, but left as is for now + badruns = fix_accession2run[accession_number] + badruns_pattern = '|'.join(badruns) if re.match(badruns_pattern, s.series_id): lgr.info('Fixing bad run {0}'.format(s.series_id)) fixedkwargs = dict() diff -Nru heudiconv-0.9.0/heudiconv/heuristics/test_b0dwi_for_fmap.py heudiconv-0.10.0/heudiconv/heuristics/test_b0dwi_for_fmap.py --- heudiconv-0.9.0/heudiconv/heuristics/test_b0dwi_for_fmap.py 1970-01-01 00:00:00.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/heuristics/test_b0dwi_for_fmap.py 2021-09-16 18:13:22.000000000 +0000 @@ -0,0 +1,33 @@ +"""Heuristic to extract a b-value=0 DWI image (basically, a SE-EPI) +both as a fmap and as dwi + +It is used just to test that a 'DIFFUSION' image that the user +chooses to extract as fmap (pepolar case) doesn't produce _bvecs/ +_bvals json files, while it does for dwi images +""" + + +def create_key(template, outtype=('nii.gz',), annotation_classes=None): + if template is None or not template: + raise ValueError('Template must be a valid format string') + return template, outtype, annotation_classes + +def infotodict(seqinfo): + """Heuristic evaluator for determining which runs belong where + + allowed template fields - follow python string module: + + item: index within category + subject: participant id + seqitem: run number during scanning + subindex: sub index within group + """ + fmap = create_key('sub-{subject}/fmap/sub-{subject}_acq-b0dwi_epi') + dwi = create_key('sub-{subject}/dwi/sub-{subject}_acq-b0dwi_dwi') + + info = {fmap: [], dwi: []} + for s in seqinfo: + if 'DIFFUSION' in s.image_type: + info[fmap].append(s.series_id) + info[dwi].append(s.series_id) + return info diff -Nru heudiconv-0.9.0/heudiconv/info.py heudiconv-0.10.0/heudiconv/info.py --- heudiconv-0.9.0/heudiconv/info.py 2020-12-23 15:35:32.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/info.py 2021-09-16 18:13:22.000000000 +0000 @@ -1,4 +1,4 @@ -__version__ = "0.9.0" +__version__ = "0.10.0" __author__ = "HeuDiConv team and contributors" __url__ = "https://github.com/nipy/heudiconv" __packagename__ = 'heudiconv' @@ -12,13 +12,14 @@ 'Environment :: Console', 'Intended Audience :: Science/Research', 'License :: OSI Approved :: Apache Software License', - 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', + 'Programming Language :: Python :: 3.9', 'Topic :: Scientific/Engineering' ] -PYTHON_REQUIRES = ">=3.5" +PYTHON_REQUIRES = ">=3.6" REQUIRES = [ 'nibabel', @@ -37,7 +38,7 @@ 'inotify', ] -MIN_DATALAD_VERSION = '0.12.4' +MIN_DATALAD_VERSION = '0.13.0' EXTRA_REQUIRES = { 'tests': TESTS_REQUIRES, 'extras': [ diff -Nru heudiconv-0.9.0/heudiconv/main.py heudiconv-0.10.0/heudiconv/main.py --- heudiconv-0.9.0/heudiconv/main.py 2020-12-23 15:13:14.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/main.py 2021-09-16 18:13:22.000000000 +0000 @@ -252,16 +252,16 @@ outdir = op.abspath(outdir) + latest = None try: import etelemetry latest = etelemetry.get_project("nipy/heudiconv") except Exception as e: lgr.warning("Could not check for version updates: %s", str(e)) - latest = {"version": 'Unknown'} lgr.info(INIT_MSG(packname=__packagename__, version=__version__, - latest=latest["version"])) + latest=(latest or {}).get("version", "Unknown"))) if command: process_extra_commands(outdir, command, files, dicom_dir_template, diff -Nru heudiconv-0.9.0/heudiconv/parser.py heudiconv-0.10.0/heudiconv/parser.py --- heudiconv-0.9.0/heudiconv/parser.py 2020-12-23 15:13:15.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/parser.py 2021-09-16 18:13:22.000000000 +0000 @@ -24,6 +24,7 @@ _VCS_REGEX = '%s\.(?:git|gitattributes|svn|bzr|hg)(?:%s|$)' % (op.sep, op.sep) + @docstring_parameter(_VCS_REGEX) def find_files(regex, topdir=op.curdir, exclude=None, exclude_vcs=True, dirs=False): @@ -36,12 +37,16 @@ exclude_vcs: If True, excludes commonly known VCS subdirectories. If string, used as regex to exclude those files (regex: `{}`) - topdir: basestring, optional + topdir: basestring or list, optional Directory where to search dirs: bool, optional Either to match directories as well as files """ - + if isinstance(topdir, (list, tuple)): + for topdir_ in topdir: + yield from find_files( + regex, topdir=topdir_, exclude=exclude, exclude_vcs=exclude_vcs, dirs=dirs) + return for dirpath, dirnames, filenames in os.walk(topdir): names = (dirnames + filenames) if dirs else filenames paths = (op.join(dirpath, name) for name in names) Binary files /tmp/tmprra4g7hs/Nd7oYj0ErJ/heudiconv-0.9.0/heudiconv/tests/data/b0dwiForFmap/b0dwi_for_fmap+00001.dcm and /tmp/tmprra4g7hs/Z0zHwUjvAd/heudiconv-0.10.0/heudiconv/tests/data/b0dwiForFmap/b0dwi_for_fmap+00001.dcm differ Binary files /tmp/tmprra4g7hs/Nd7oYj0ErJ/heudiconv-0.9.0/heudiconv/tests/data/b0dwiForFmap/b0dwi_for_fmap+00002.dcm and /tmp/tmprra4g7hs/Z0zHwUjvAd/heudiconv-0.10.0/heudiconv/tests/data/b0dwiForFmap/b0dwi_for_fmap+00002.dcm differ Binary files /tmp/tmprra4g7hs/Nd7oYj0ErJ/heudiconv-0.9.0/heudiconv/tests/data/b0dwiForFmap/b0dwi_for_fmap+00003.dcm and /tmp/tmprra4g7hs/Z0zHwUjvAd/heudiconv-0.10.0/heudiconv/tests/data/b0dwiForFmap/b0dwi_for_fmap+00003.dcm differ Binary files /tmp/tmprra4g7hs/Nd7oYj0ErJ/heudiconv-0.9.0/heudiconv/tests/data/Phoenix/01+AA/01+AA+00001.dcm and /tmp/tmprra4g7hs/Z0zHwUjvAd/heudiconv-0.10.0/heudiconv/tests/data/Phoenix/01+AA/01+AA+00001.dcm differ Binary files /tmp/tmprra4g7hs/Nd7oYj0ErJ/heudiconv-0.9.0/heudiconv/tests/data/Phoenix/99+PhoenixDocument/99+PhoenixDocument+00001.dcm and /tmp/tmprra4g7hs/Z0zHwUjvAd/heudiconv-0.10.0/heudiconv/tests/data/Phoenix/99+PhoenixDocument/99+PhoenixDocument+00001.dcm differ diff -Nru heudiconv-0.9.0/heudiconv/tests/test_bids.py heudiconv-0.10.0/heudiconv/tests/test_bids.py --- heudiconv-0.9.0/heudiconv/tests/test_bids.py 1970-01-01 00:00:00.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/tests/test_bids.py 2021-09-16 18:13:22.000000000 +0000 @@ -0,0 +1,26 @@ +"""Test functions in heudiconv.bids module. +""" + +from heudiconv.bids import ( + maybe_na, + treat_age, +) + + +def test_maybe_na(): + for na in '', ' ', None, 'n/a', 'N/A', 'NA': + assert maybe_na(na) == 'n/a' + for notna in 0, 1, False, True, 'value': + assert maybe_na(notna) == str(notna) + + +def test_treat_age(): + assert treat_age(0) == '0' + assert treat_age('0') == '0' + assert treat_age('0000') == '0' + assert treat_age('0000Y') == '0' + assert treat_age('000.1Y') == '0.1' + assert treat_age('1M') == '0.08' + assert treat_age('12M') == '1' + assert treat_age('0000.1') == '0.1' + assert treat_age(0000.1) == '0.1' \ No newline at end of file diff -Nru heudiconv-0.9.0/heudiconv/tests/test_convert.py heudiconv-0.10.0/heudiconv/tests/test_convert.py --- heudiconv-0.9.0/heudiconv/tests/test_convert.py 2020-12-22 22:19:56.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/tests/test_convert.py 2021-09-16 18:13:22.000000000 +0000 @@ -1,11 +1,18 @@ """Test functions in heudiconv.convert module. """ +import os.path as op +from glob import glob + import pytest +from .utils import TESTS_DATA_PATH from heudiconv.convert import (update_complex_name, update_multiecho_name, - update_uncombined_name) + update_uncombined_name, + DW_IMAGE_IN_FMAP_FOLDER_WARNING, + ) from heudiconv.bids import BIDSError +from heudiconv.cli.run import main as runner def test_update_complex_name(): @@ -76,3 +83,31 @@ out_fn_true = 'sub-X_ses-Y_task-Z_run-01_ch-04_bold' out_fn_test = update_uncombined_name(metadata, fn, channel_names) assert out_fn_test == out_fn_true + + +def test_b0dwi_for_fmap(tmpdir, capfd): + """Make sure we raise a warning when .bvec and .bval files + are present but the modality is not dwi. + We check it by extracting a few DICOMs from a series with + bvals: 5 5 1500 + """ + tmppath = tmpdir.strpath + subID = 'b0dwiForFmap' + args = ( + "-c dcm2niix -o %s -b -f test_b0dwi_for_fmap --files %s -s %s" + % (tmpdir, op.join(TESTS_DATA_PATH, 'b0dwiForFmap'), subID) + ).split(' ') + runner(args) + + # assert that it raised a warning that the fmap directory will contain + # bvec and bval files. + output = capfd.readouterr().err.split('\n') + expected_msg = DW_IMAGE_IN_FMAP_FOLDER_WARNING.format(folder=op.join(tmppath, 'sub-%s', 'fmap') % subID) + assert [o for o in output if expected_msg in o] + + # check that both 'fmap' and 'dwi' directories have been extracted and they contain + # *.bvec and a *.bval files + for mod in ['fmap', 'dwi']: + assert op.isdir(op.join(tmppath, 'sub-%s', mod) % (subID)) + for ext in ['bval', 'bvec']: + assert glob(op.join(tmppath, 'sub-%s', mod, 'sub-%s_*.%s') % (subID, subID, ext)) diff -Nru heudiconv-0.9.0/heudiconv/tests/test_dicoms.py heudiconv-0.10.0/heudiconv/tests/test_dicoms.py --- heudiconv-0.9.0/heudiconv/tests/test_dicoms.py 2020-07-29 16:27:07.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/tests/test_dicoms.py 2021-09-16 18:13:22.000000000 +0000 @@ -1,12 +1,18 @@ import os.path as op import json +from glob import glob import pytest from heudiconv.external.pydicom import dcm from heudiconv.cli.run import main as runner from heudiconv.convert import nipype_convert -from heudiconv.dicoms import parse_private_csa_header, embed_dicom_and_nifti_metadata +from heudiconv.dicoms import ( + OrderedDict, + embed_dicom_and_nifti_metadata, + group_dicoms_into_seqinfos, + parse_private_csa_header, +) from .utils import ( assert_cwd_unchanged, TESTS_DATA_PATH, @@ -64,3 +70,18 @@ assert out3.pop("existing") == "data" assert out3 == out2 + + +def test_group_dicoms_into_seqinfos(tmpdir): + """Tests for group_dicoms_into_seqinfos""" + + # 1) Check that it works for PhoenixDocuments: + # set up testing files + dcmfolder = op.join(TESTS_DATA_PATH, 'Phoenix') + dcmfiles = glob(op.join(dcmfolder, '*', '*.dcm')) + + seqinfo = group_dicoms_into_seqinfos(dcmfiles, 'studyUID', flatten=True) + + assert type(seqinfo) is OrderedDict + assert len(seqinfo) == len(dcmfiles) + assert [s.series_description for s in seqinfo] == ['AAHead_Scout_32ch-head-coil', 'PhoenixZIPReport'] diff -Nru heudiconv-0.9.0/heudiconv/tests/test_heuristics.py heudiconv-0.10.0/heudiconv/tests/test_heuristics.py --- heudiconv-0.9.0/heudiconv/tests/test_heuristics.py 2020-07-29 16:27:07.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/tests/test_heuristics.py 2021-09-16 18:13:22.000000000 +0000 @@ -176,3 +176,20 @@ assert not op.exists(pjoin(tmppath, 'Halchenko/Yarik/950_bids_test4', fname)) else: assert op.exists(pjoin(tmppath, 'Halchenko/Yarik/950_bids_test4', fname)) + + +def test_phoenix_doc_conversion(tmpdir): + tmppath = tmpdir.strpath + subID = 'Phoenix' + args = ( + "-c dcm2niix -o %s -b -f bids_PhoenixReport --files %s -s %s" + % (tmpdir, pjoin(TESTS_DATA_PATH, 'Phoenix'), subID) + ).split(' ') + runner(args) + + # check that the Phoenix document has been extracted (as gzipped dicom) in + # the sourcedata/misc folder: + assert op.exists(pjoin(tmppath, 'sourcedata', 'sub-%s', 'misc', 'sub-%s_phoenix.dicom.tgz') % (subID, subID)) + # check that no "sub-/misc" folder has been created in the BIDS + # structure: + assert not op.exists(pjoin(tmppath, 'sub-%s', 'misc') % subID) diff -Nru heudiconv-0.9.0/heudiconv/tests/test_main.py heudiconv-0.10.0/heudiconv/tests/test_main.py --- heudiconv-0.9.0/heudiconv/tests/test_main.py 2020-12-23 15:13:14.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/tests/test_main.py 2021-09-16 18:13:22.000000000 +0000 @@ -4,13 +4,16 @@ from heudiconv.main import workflow from heudiconv import __version__ from heudiconv.utils import (create_file_if_missing, + load_json, set_readonly, is_readonly) from heudiconv.bids import (populate_bids_templates, add_participant_record, get_formatted_scans_key_row, add_rows_to_scans_keys_file, - find_subj_ses) + find_subj_ses, + SCANS_FILE_FIELDS, + ) from heudiconv.external.dlad import MIN_VERSION, add_to_datalad from .utils import TESTS_DATA_PATH @@ -81,6 +84,8 @@ assert "something" not in description_file.read() assert "TODO" in description_file.read() + assert load_json(tmpdir / "scans.json") == SCANS_FILE_FIELDS + def test_add_participant_record(tmpdir): tf = tmpdir.join('participants.tsv') @@ -127,6 +132,7 @@ '.gitattributes', '.datalad/config', '.datalad/.gitattributes', 'dataset_description.json', + 'scans.json', 'CHANGES', 'README'} assert set(ds.repo.get_indexed_files()) == target_files # and all are under git @@ -155,7 +161,7 @@ assert '.heudiconv/dummy.nii.gz' in ds.repo.get_files() # Let's now roll back and make it a proper submodule - ds.repo._git_custom_command([], ['git', 'reset', '--hard', old_hexsha]) + ds.repo.call_git(['reset', '--hard', old_hexsha]) # now we do not add dummy to git create_file_if_missing(dummy_path, '') add_to_datalad(str(tmpdir), studydir_, None, False) @@ -217,7 +223,9 @@ assert dates == sorted(dates) _check_rows(fn, rows) - assert op.exists(opj(tmpdir.strpath, 'file.json')) + # we no longer produce a sidecar .json file there and only generate + # it while populating templates for BIDS + assert not op.exists(opj(tmpdir.strpath, 'file.json')) # add a new one extra_rows = { 'a_new_file.nii.gz': ['2016adsfasd23', '', 'fasadfasdf'], @@ -283,6 +291,11 @@ assert (cachedir / 'S01.auto.txt').exists() assert (cachedir / 'S01.edit.txt').exists() + # check dicominfo has "time" as last column: + with open(str(cachedir / 'dicominfo.tsv'), 'r') as f: + cols = f.readline().split() + assert cols[26] == "time" + def test_no_etelemetry(): # smoke test at large - just verifying that no crash if no etelemetry diff -Nru heudiconv-0.9.0/heudiconv/tests/test_utils.py heudiconv-0.10.0/heudiconv/tests/test_utils.py --- heudiconv-0.9.0/heudiconv/tests/test_utils.py 2020-12-23 15:13:14.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/tests/test_utils.py 2021-09-16 18:13:22.000000000 +0000 @@ -2,6 +2,8 @@ import os import os.path as op +import mock + from heudiconv.utils import ( get_known_heuristics_with_descriptions, get_heuristic_description, @@ -77,6 +79,13 @@ with pytest.raises(JSONDecodeError): load_json(str(invalid_json_file)) + # and even if we ask to retry a few times -- should be the same + with pytest.raises(JSONDecodeError): + load_json(str(invalid_json_file), retry=3) + + with pytest.raises(FileNotFoundError): + load_json("absent123not.there", retry=3) + assert ifname in caplog.text # test valid json @@ -87,6 +96,23 @@ assert load_json(valid_json_file) == vcontent + calls = [0] + json_load = json.load + + def json_load_patched(fp): + calls[0] += 1 + if calls[0] == 1: + # just reuse bad file + load_json(str(invalid_json_file)) + elif calls[0] == 2: + raise FileNotFoundError() + else: + return json_load(fp) + + with mock.patch.object(json, 'load', json_load_patched): + assert load_json(valid_json_file, retry=3) == vcontent + + def test_get_datetime(): """ diff -Nru heudiconv-0.9.0/heudiconv/utils.py heudiconv-0.10.0/heudiconv/utils.py --- heudiconv-0.9.0/heudiconv/utils.py 2020-12-23 15:13:15.000000000 +0000 +++ heudiconv-0.10.0/heudiconv/utils.py 2021-09-16 18:13:22.000000000 +0000 @@ -14,6 +14,7 @@ from glob import glob from subprocess import check_output from datetime import datetime +from time import sleep from nipype.utils.filemanip import which @@ -46,7 +47,8 @@ 'patient_sex', # 23 'date', # 24 'series_uid', # 25 - ] + 'time', # 26 +] SeqInfo = namedtuple('SeqInfo', seqinfo_fields) @@ -146,39 +148,40 @@ fp.writelines(PrettyPrinter().pformat(info)) -def _canonical_dumps(json_obj, **kwargs): - """ Dump `json_obj` to string, allowing for Python newline bug - - Runs ``json.dumps(json_obj, \*\*kwargs), then removes trailing whitespaces - added when doing indent in some Python versions. See - https://bugs.python.org/issue16333. Bug seems to be fixed in 3.4, for now - fixing manually not only for aestetics but also to guarantee the same - result across versions of Python. - """ - out = json.dumps(json_obj, **kwargs) - if 'indent' in kwargs: - out = out.replace(' \n', '\n') - return out - - -def load_json(filename): +def load_json(filename, retry=0): """Load data from a json file Parameters ---------- filename : str Filename to load data from. + retry: int, optional + Number of times to retry opening/loading the file in case of + failure. Code will sleep for 0.1 seconds between retries. + Could be used in code which is not sensitive to order effects + (e.g. like populating bids templates where the last one to + do it, would make sure it would be the correct/final state). Returns ------- data : dict """ - try: - with open(filename, 'r') as fp: - data = json.load(fp) - except JSONDecodeError: - lgr.error("{fname} is not a valid json file".format(fname=filename)) - raise + assert retry >= 0 + for i in range(retry + 1): # >= 10 sec wait + try: + try: + with open(filename, 'r') as fp: + data = json.load(fp) + break + except JSONDecodeError: + lgr.error("{fname} is not a valid json file".format(fname=filename)) + raise + except (JSONDecodeError, FileNotFoundError) as exc: + if i >= retry: + raise + lgr.warning("Caught %s. Will retry again", exc) + sleep(0.1) + continue return data @@ -219,19 +222,25 @@ % (str(exc), filename) ) if not pretty: - j = _canonical_dumps(data, **dumps_kw) + j = json_dumps(data, **dumps_kw) assert j is not None # one way or another it should have been set to a str with open(filename, 'w') as fp: fp.write(j) +def json_dumps(json_obj, indent=2, sort_keys=True): + """Unified (default indent and sort_keys) invocation of json.dumps + """ + return json.dumps(json_obj, indent=indent, sort_keys=sort_keys) + + def json_dumps_pretty(j, indent=2, sort_keys=True): """Given a json structure, pretty print it by colliding numeric arrays into a line. If resultant structure differs from original -- throws exception """ - js = _canonical_dumps(j, indent=indent, sort_keys=sort_keys) + js = json_dumps(j, indent=indent, sort_keys=sort_keys) # trim away \n and spaces between entries of numbers js_ = re.sub( '[\n ]+("?[-+.0-9e]+"?,?) *\n(?= *"?[-+.0-9e]+"?)', r' \1', diff -Nru heudiconv-0.9.0/heudiconv.egg-info/dependency_links.txt heudiconv-0.10.0/heudiconv.egg-info/dependency_links.txt --- heudiconv-0.9.0/heudiconv.egg-info/dependency_links.txt 2020-12-23 15:35:33.000000000 +0000 +++ heudiconv-0.10.0/heudiconv.egg-info/dependency_links.txt 1970-01-01 00:00:00.000000000 +0000 @@ -1 +0,0 @@ - diff -Nru heudiconv-0.9.0/heudiconv.egg-info/entry_points.txt heudiconv-0.10.0/heudiconv.egg-info/entry_points.txt --- heudiconv-0.9.0/heudiconv.egg-info/entry_points.txt 2020-12-23 15:35:33.000000000 +0000 +++ heudiconv-0.10.0/heudiconv.egg-info/entry_points.txt 1970-01-01 00:00:00.000000000 +0000 @@ -1,4 +0,0 @@ -[console_scripts] -heudiconv = heudiconv.cli.run:main -heudiconv_monitor = heudiconv.cli.monitor:main - diff -Nru heudiconv-0.9.0/heudiconv.egg-info/PKG-INFO heudiconv-0.10.0/heudiconv.egg-info/PKG-INFO --- heudiconv-0.9.0/heudiconv.egg-info/PKG-INFO 2020-12-23 15:35:33.000000000 +0000 +++ heudiconv-0.10.0/heudiconv.egg-info/PKG-INFO 1970-01-01 00:00:00.000000000 +0000 @@ -1,23 +0,0 @@ -Metadata-Version: 2.1 -Name: heudiconv -Version: 0.9.0 -Summary: Heuristic DICOM Converter -Home-page: UNKNOWN -Author: HeuDiConv team and contributors -License: Apache 2.0 -Description: Convert DICOM dirs based on heuristic info - HeuDiConv - uses the dcmstack package and dcm2niix tool to convert DICOM directories or - tarballs into collections of NIfTI files following pre-defined heuristic(s). -Platform: UNKNOWN -Classifier: Environment :: Console -Classifier: Intended Audience :: Science/Research -Classifier: License :: OSI Approved :: Apache Software License -Classifier: Programming Language :: Python :: 3.5 -Classifier: Programming Language :: Python :: 3.6 -Classifier: Programming Language :: Python :: 3.7 -Classifier: Topic :: Scientific/Engineering -Requires-Python: >=3.5 -Provides-Extra: all -Provides-Extra: datalad -Provides-Extra: extras -Provides-Extra: tests diff -Nru heudiconv-0.9.0/heudiconv.egg-info/requires.txt heudiconv-0.10.0/heudiconv.egg-info/requires.txt --- heudiconv-0.9.0/heudiconv.egg-info/requires.txt 2020-12-23 15:35:33.000000000 +0000 +++ heudiconv-0.10.0/heudiconv.egg-info/requires.txt 1970-01-01 00:00:00.000000000 +0000 @@ -1,28 +0,0 @@ -dcmstack>=0.8 -etelemetry -filelock>=3.0.12 -nibabel -nipype>=1.2.3 -pydicom - -[all] -datalad>=0.12.4 -duecredit -inotify -mock -pytest -six -tinydb - -[datalad] -datalad>=0.12.4 - -[extras] -duecredit - -[tests] -inotify -mock -pytest -six -tinydb diff -Nru heudiconv-0.9.0/heudiconv.egg-info/SOURCES.txt heudiconv-0.10.0/heudiconv.egg-info/SOURCES.txt --- heudiconv-0.9.0/heudiconv.egg-info/SOURCES.txt 2020-12-23 15:35:33.000000000 +0000 +++ heudiconv-0.10.0/heudiconv.egg-info/SOURCES.txt 1970-01-01 00:00:00.000000000 +0000 @@ -1,94 +0,0 @@ -.coveragerc -.dockerignore -.gitignore -.mailmap -.travis.yml -CHANGELOG.md -Dockerfile -LICENSE -Makefile -NOTES -README.rst -dev-requirements.txt -pytest.ini -requirements.txt -setup.py -tox.ini -.github/ISSUE_TEMPLATE.md -custom/dbic/README -custom/dbic/singularity-env.def -docs/Makefile -docs/api.rst -docs/changes.rst -docs/conf.py -docs/heuristics.rst -docs/index.rst -docs/installation.rst -docs/requirements.txt -docs/tutorials.rst -docs/usage.rst -docs/api/bids.rst -docs/api/convert.rst -docs/api/dicoms.rst -docs/api/parser.rst -docs/api/queue.rst -docs/api/utils.rst -heudiconv/__init__.py -heudiconv/bids.py -heudiconv/convert.py -heudiconv/dicoms.py -heudiconv/due.py -heudiconv/info.py -heudiconv/main.py -heudiconv/parser.py -heudiconv/queue.py -heudiconv/utils.py -heudiconv.egg-info/PKG-INFO -heudiconv.egg-info/SOURCES.txt -heudiconv.egg-info/dependency_links.txt -heudiconv.egg-info/entry_points.txt -heudiconv.egg-info/requires.txt -heudiconv.egg-info/top_level.txt -heudiconv/cli/__init__.py -heudiconv/cli/monitor.py -heudiconv/cli/run.py -heudiconv/external/__init__.py -heudiconv/external/dcmstack.py -heudiconv/external/dlad.py -heudiconv/external/pydicom.py -heudiconv/external/tests/__init__.py -heudiconv/external/tests/test_dlad.py -heudiconv/heuristics/__init__.py -heudiconv/heuristics/banda-bids.py -heudiconv/heuristics/bids_ME.py -heudiconv/heuristics/bids_with_ses.py -heudiconv/heuristics/cmrr_heuristic.py -heudiconv/heuristics/convertall.py -heudiconv/heuristics/example.py -heudiconv/heuristics/multires_7Tbold.py -heudiconv/heuristics/reproin.py -heudiconv/heuristics/reproin_validator.cfg -heudiconv/heuristics/studyforrest_phase2.py -heudiconv/heuristics/test_reproin.py -heudiconv/heuristics/uc_bids.py -heudiconv/tests/__init__.py -heudiconv/tests/anonymize_script.py -heudiconv/tests/test_convert.py -heudiconv/tests/test_dicoms.py -heudiconv/tests/test_heuristics.py -heudiconv/tests/test_main.py -heudiconv/tests/test_monitor.py -heudiconv/tests/test_queue.py -heudiconv/tests/test_regression.py -heudiconv/tests/test_tarballs.py -heudiconv/tests/test_utils.py -heudiconv/tests/utils.py -heudiconv/tests/data/axasc35.dcm -heudiconv/tests/data/phantom.dcm -heudiconv/tests/data/01-anat-scout/0001.dcm -heudiconv/tests/data/01-fmap_acq-3mm/1.3.12.2.1107.5.2.43.66112.2016101409263663466202201.dcm -utils/gen-docker-image.sh -utils/link_issues_CHANGELOG -utils/prep_release -utils/test-compare-two-versions.sh -utils/update_changes.sh \ No newline at end of file diff -Nru heudiconv-0.9.0/heudiconv.egg-info/top_level.txt heudiconv-0.10.0/heudiconv.egg-info/top_level.txt --- heudiconv-0.9.0/heudiconv.egg-info/top_level.txt 2020-12-23 15:35:33.000000000 +0000 +++ heudiconv-0.10.0/heudiconv.egg-info/top_level.txt 1970-01-01 00:00:00.000000000 +0000 @@ -1 +0,0 @@ -heudiconv diff -Nru heudiconv-0.9.0/PKG-INFO heudiconv-0.10.0/PKG-INFO --- heudiconv-0.9.0/PKG-INFO 2020-12-23 15:35:33.484483700 +0000 +++ heudiconv-0.10.0/PKG-INFO 1970-01-01 00:00:00.000000000 +0000 @@ -1,23 +0,0 @@ -Metadata-Version: 2.1 -Name: heudiconv -Version: 0.9.0 -Summary: Heuristic DICOM Converter -Home-page: UNKNOWN -Author: HeuDiConv team and contributors -License: Apache 2.0 -Description: Convert DICOM dirs based on heuristic info - HeuDiConv - uses the dcmstack package and dcm2niix tool to convert DICOM directories or - tarballs into collections of NIfTI files following pre-defined heuristic(s). -Platform: UNKNOWN -Classifier: Environment :: Console -Classifier: Intended Audience :: Science/Research -Classifier: License :: OSI Approved :: Apache Software License -Classifier: Programming Language :: Python :: 3.5 -Classifier: Programming Language :: Python :: 3.6 -Classifier: Programming Language :: Python :: 3.7 -Classifier: Topic :: Scientific/Engineering -Requires-Python: >=3.5 -Provides-Extra: all -Provides-Extra: datalad -Provides-Extra: extras -Provides-Extra: tests diff -Nru heudiconv-0.9.0/setup.cfg heudiconv-0.10.0/setup.cfg --- heudiconv-0.9.0/setup.cfg 2020-12-23 15:35:33.488483700 +0000 +++ heudiconv-0.10.0/setup.cfg 1970-01-01 00:00:00.000000000 +0000 @@ -1,4 +0,0 @@ -[egg_info] -tag_build = -tag_date = 0 - diff -Nru heudiconv-0.9.0/.travis.yml heudiconv-0.10.0/.travis.yml --- heudiconv-0.9.0/.travis.yml 2020-07-29 16:27:07.000000000 +0000 +++ heudiconv-0.10.0/.travis.yml 1970-01-01 00:00:00.000000000 +0000 @@ -1,43 +0,0 @@ -# vim ft=yaml -language: python -python: - - 3.5 - - 3.6 - - 3.7 - - 3.8 - -cache: - - apt - -env: - global: - # will be used in the matrix, where neither other variable is used - - BOTO_CONFIG=/tmp/nowhere - - DATALAD_TESTS_SSH=1 - -before_install: - # The ultimate one-liner setup for NeuroDebian repository - - bash <(wget -q -O- http://neuro.debian.net/_files/neurodebian-travis.sh) - - travis_retry sudo apt-get update -qq - - travis_retry sudo apt-get install git-annex-standalone dcm2niix - # Install in our own virtualenv - - python -m pip install --upgrade pip - - pip install --upgrade virtualenv - - virtualenv --python=python venv - - source venv/bin/activate - - pip --version # check again since seems that python_requires='>=3.5' in secretstorage is not in effect - - python --version # just to check - - pip install -r dev-requirements.txt - - pip install requests # below installs pyld but that assumes we have requests already - - pip install datalad - - pip install codecov pytest - -install: - - git config --global user.email "test@travis.land" - - git config --global user.name "Travis Almighty" - -script: - - coverage run `which py.test` -s -v heudiconv - -after_success: - - codecov