Data handling

class flyingpigeon.processes.FetchProcess[source]

fetch_resources Fetch Resources (v0.10)

Fetch data resources (limited to 50GB) to the local filesystem of the birdhouse compute provider.

Paramètres:resource (application/x-netcdf, application/x-tar, application/zip) – NetCDF Files or archive (tar/zip) containing netCDF files.
Renvoie:
  • output (text/plain) – File containing the local pathes to downloades files.
  • output_log (text/plain) – Collected logs during process run.

References

class malleefowl.processes.ESGSearchProcess[source]

esgsearch ESGF Search (v0.6)

Search ESGF datasets, files and aggreations.

Paramètres:
  • url (string) – URL of ESGF Search Index. Example: http://esgf-data.dkrz.de/esg-search
  • distrib (boolean, optional) – If flag is set then a distributed search will be run.
  • replica (boolean, optional) – If flag is set then search will include replicated datasets.
  • latest (boolean, optional) – If flag is set then search will include only latest datasets.
  • temporal (boolean, optional) – If flag is set then search will use temporal filter.
  • search_type ({'Dataset', 'File', 'Aggregation'}, optional) – Search on Datasets, Files or Aggregations.
  • constraints (string) – Constraints as list of key/value pairs.Example: project:CORDEX, time_frequency:mon, variable:tas
  • query (string, optional) – Freetext query. For Example: temperatue
  • start (dateTime, optional) – Startime: 2000-01-11T12:00:00Z
  • end (dateTime, optional) – Endtime: 2005-12-31T12:00:00Z
  • limit ({'0', '1', '2', '5', '10', '20', '50', '100', '200'}, optional) – Maximum number of datasets in search result
  • offset (integer, optional) – Start search of datasets at offset.
Renvoie:

  • output (application/json) – JSON document with search result
  • summary (application/json) – JSON document with search result summary
  • facet_counts (application/json) – JSON document with facet counts for constraints.

References

class malleefowl.processes.Download[source]

download Download files (v0.8)

Downloads files and provides file list as json document.

Paramètres:resource (string) – URL of your resource.
Renvoie:output – Json document with list of downloaded files with file url.
Type renvoyé:application/json

References

class malleefowl.processes.ThreddsDownload[source]

thredds_download Download files from Thredds Catalog (v0.5)

Downloads files from Thredds Catalog and provides file list as JSON Document.

Paramètres:url (string) – URL of the catalog.
Renvoie:output – JSON document with list of downloaded files with file url.
Type renvoyé:application/json

References

class flyingpigeon.processes.NCMergeProcess[source]

ncmerge NetCDF merge (v0.1)

Merge NetCDF files in the time dimension.

Paramètres:resource (application/x-netcdf, application/x-tar, application/zip) – NetCDF files, can be OPEnDAP urls.
Renvoie:output – Temporally merged NetCDF files.
Type renvoyé:application/x-netcdf