Contributing

Benchmark development environment

For debugging and editing purposes, PAVICS should be set up on a virtual machine. We recommend installing the Oracle VM VirtualBox with Extensions and creating a VM with the following base specifications:

  • AMD64 Ubuntu Linux 16.04 Long-Term-Support (LTS) (via Ubuntu Downloads)
  • > 8 GB RAM
  • > 70 GB Storage
  • > 2 CPUs
  • Network bridge access
  • Install VBoxGuestAdditions within the Ubuntu guest for corresponding VirtualBox version. This can be done via the Devices tab of the VM via the Insert Guest Additions ISO image… and following the install instruction from the autorun.sh script.

Required VM packages for various PAVICS components (most can be installed via `apt-get`with root privileges):

  • python-dev
  • curl or wget
  • git
  • docker.io
  • docker-compose

Setting up PyCharm

Note

For missing python modules: https://www.jetbrains.com/help/pycharm/installing-uninstalling-and-upgrading-packages.html

The PAVICS back-end relies on developmental builds based on the Open Geospatial Consortium Web Processing Services. To install these libraries in your working environment, run the following within your PyCharm console:

import pip
pip.main(['install', 'https://github.com/geopython/pywps/archive/7cab3866e34ce24d3df56e3c1c546739b1cda2d7.zip'])
pip.main(['install', '--upgrade', '--force-reinstall', 'https://github.com/bird-house/OWSLib/archive/pingudev.zip'])

Warning

Some packages are not happy with Python wheel, try uninstalling it if all else fails.

Launching individual local components

Solr

docker pull pavics/solr
docker run --name my_solr -d -p 8983:8983 -t pavics/solr

Check that Solr is running at http://localhost:8983/solr/#/birdhouse

Public THREDDS

git clone https://github.com/Ouranosinc/PAVICS.git
cp PAVICS/birdhouse/templates/docker-compose.override.public_thredds.yml PAVICS/birdhouse/docker-compose.override.yml

In this new docker-compose.override.yml change ${PATH_TO_LOCAL_NETCDF_FILES} to an actual path on disk with NetCDF files.

docker-compose up -d thredds

Check that thredds is running at http://localhost:8083/thredds/

Secure THREDDS (HTTPS)

First you will need a self-signed certificate:

openssl req -x509 -nodes -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365

This will ask for various inputs that may be left blank.

cat key.pem >> cert.pem

cert.pm is your self-signed certificate.

git clone https://github.com/Ouranosinc/PAVICS.git

Switch to PAVICS/birdhouse directory.

./set_hostname.sh dummy

Add this dummy hostname to /etc/hosts

127.0.0.1       dummy.crim.ca

Either set SSL_CERTIFICATE to the location of cert.pem and HOSTNAME to dummy.crim.ca or use the template docker-compose_shorcut.sh and set those values in it (here renamed to mycompose.sh).

cp PAVICS/birdhouse/templates/docker-compose.override.local_https_thredds.yml PAVICS/birdhouse/docker-compose.override.yml

In this override file specify the path to local netcdf files and set localhost IP.

./mycompose.sh up -d thredds proxy magpie twitcher

It may take a minute for twitcher to get online. If it does not respond after a while, try to restart everything once or twice.

./mycompose down
./mycompose.sh up -d thredds proxy magpie twitcher

Check that twitcher is running at https://localhost/twitcher/ (returns hello)

Check that magpie is running at https://localhost/magpie/

Check that thredds is running at https://localhost/twitcher/ows/proxy/thredds/

Play around with magpie permissions to check that the security is working

HTTPS custom WPS service

Follow all the steps of the HTTPS THREDDS setup above up to the template copy, instead use:

cp PAVICS/birdhouse/templates/docker-compose.override.local_https_wps.yml PAVICS/birdhouse/docker-compose.override.yml

In this override file, set the localhost IP, then you can switch the wpsandbox image for the wps service image of your choice and assign it an available port of your choice. Then assign a corresponding port to the proxy. A new proxy configuration file need to be added to PAVICS/birdhouse/config/proxy/conf.d/wpsandbox.conf for this service, e.g.:

server {
    listen 8081;
    location / {
        proxy_pass http://wpsandbox;
    }
}

./mycompose.sh up -d proxy magpie twitcher wpsandbox

It may take a minute for twitcher to get online. If it does not respond after a while, try to restart everything once or twice.

./mycompose down
./mycompose.sh up -d thredds proxy magpie twitcher

Check that twitcher is running at https://localhost/twitcher/ (returns hello)

Check that magpie is running at https://localhost/magpie/

Register the new wps service in magpie: In Home > Edit Services > wps > Add Service. In our case the name is wpsandbox and the public url is https://dummy.crim.ca/twitcher/ows/proxy/wpsandbox with the wps service type. Then edit this new service Protected URL to http://dummy.crim.ca:8081

Alternatively, this can be entered in PAVICS/birdhouse/config/magpie/providers.cfg

Check that the wps is running at https://localhost/twitcher/ows/proxy/wpsandbox/pywps?service=WPS&version=1.0.0&request=GetCapabilities

Play around with magpie permissions to check that the security is working

PAVICS-DataCatalog development

git clone https://github.com/Ouranosinc/PAVICS-DataCatalog.git
cd PAVICS-DataCatalog
cp catalog.cfg ~/catalog.cfg

Edit ~/catalog.cfg with Solr address. Note that within docker, localhost is not the same as the workstation localhost, so the address must use the ip of the local machine (retrieve with, e.g., ifconfig). Also point to a valid thredds server.

# Docker requires root privileges
sudo su
docker build -t pavics-datacatalog .
docker run --name pavics-datacatalog1 -d -v ~/catalog.cfg:/home/catalog.cfg -p 8009:80 pavics-datacatalog

Check that the wps is running at http://localhost:8009/pywps?service=WPS&request=GetCapabilities&version=1.0.0

Note that when running with magpie, the root directory of thredds does not properly redirect to the proxy. To crawl a thredds server with threddsclient, provide the birdhouse root directory of thredds (e.g. in catalog.cfg):

https://localhost/twitcher/ows/proxy/thredds/catalog/birdhouse

Flyingpigeon development

git clone https://github.com/Ouranosinc/flyingpigeon.git
cd flyingpigeon
git checkout pavics

Need to either add a custom.cfg or modify profiles/base.cfg with:

[settings]
geoserver = http://host:port/geoserver/wfs

Then proceed with installation:

make clean install
make test
make start

The WPS will be running at:

http://localhost:8093/wps?service=WPS&version=1.0.0&request=GetCapabilities

To restart flyingpigeon (e.g. after modifications):

make stop
make start

Malleefowl development

git clone https://github.com/Ouranosinc/malleefowl.git
cd malleefowl
git checkout pavics

Need to either add a custom.cfg or modify profiles/base.cfg with:

[settings]
persist-path = /tmp
archive-root = /

Then proceed with installation:

make clean install
make test
make start

The WPS will be running at:

http://localhost:8091/wps?service=WPS&version=1.0.0&request=GetCapabilities