data files crawler and data distribution

 DataLad is a data distribution providing access to a wide range of
 data resources already available online (initially aiming at
 neuroscience domain). Using git-annex as its backend for data
 logistics it provides following facilities
 .
  - crawling of web sites to automatically prepare and update git-annex
    repositories with content from online websites, S3, etc
  - command line interface for manipulation of collections of datasets
    (install, uninstall, update, publish, save, etc.) and
 separate files/directories (add, get), as well as search within
 aggregated meta-data
 .
 This package provides the command line tool. Install without
 Recommends if you need only core functionality.