.. _contrib: ============ Contributing ============ Benchmark development environment ================================= For debugging and editing purposes, PAVICS should be set up on a virtual machine. We recommend installing the `Oracle VM VirtualBox with Extensions `_ and creating a VM with the following base specifications: - AMD64 Ubuntu Linux 16.04 Long-Term-Support (LTS) (via `Ubuntu Downloads `_) - > 8 GB RAM - > 70 GB Storage - > 2 CPUs - Network bridge access - Install VBoxGuestAdditions within the Ubuntu guest for corresponding VirtualBox version. This can be done via the `Devices` tab of the VM via the `Insert Guest Additions ISO image...` and following the install instruction from the ``autorun.sh`` script. Required VM packages for various PAVICS components (most can be installed via `apt-get`with root privileges): - python-dev - curl `or` wget - git - docker.io - docker-compose Setting up PyCharm ================== .. note:: For missing python modules: https://www.jetbrains.com/help/pycharm/installing-uninstalling-and-upgrading-packages.html The PAVICS back-end relies on developmental builds based on the Open Geospatial Consortium Web Processing Services. To install these libraries in your working environment, run the following within your PyCharm console:: import pip pip.main(['install', 'https://github.com/geopython/pywps/archive/7cab3866e34ce24d3df56e3c1c546739b1cda2d7.zip']) pip.main(['install', '--upgrade', '--force-reinstall', 'https://github.com/bird-house/OWSLib/archive/pingudev.zip']) .. warning:: Some packages are not happy with Python ``wheel``, try uninstalling it if all else fails. Launching individual local components ===================================== Solr ---- :: docker pull pavics/solr docker run --name my_solr -d -p 8983:8983 -t pavics/solr Check that Solr is running at http://localhost:8983/solr/#/birdhouse Public THREDDS -------------- :: git clone https://github.com/Ouranosinc/PAVICS.git cp PAVICS/birdhouse/templates/docker-compose.override.public_thredds.yml PAVICS/birdhouse/docker-compose.override.yml In this new docker-compose.override.yml change ${PATH_TO_LOCAL_NETCDF_FILES} to an actual path on disk with NetCDF files. :: docker-compose up -d thredds Check that thredds is running at http://localhost:8083/thredds/ Secure THREDDS (HTTPS) ---------------------- First you will need a self-signed certificate: :: openssl req -x509 -nodes -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 This will ask for various inputs that may be left blank. :: cat key.pem >> cert.pem cert.pm is your self-signed certificate. :: git clone https://github.com/Ouranosinc/PAVICS.git Switch to PAVICS/birdhouse directory. :: ./set_hostname.sh dummy Add this dummy hostname to /etc/hosts :: 127.0.0.1 dummy.crim.ca Either set SSL_CERTIFICATE to the location of cert.pem and HOSTNAME to dummy.crim.ca or use the template docker-compose_shorcut.sh and set those values in it (here renamed to mycompose.sh). :: cp PAVICS/birdhouse/templates/docker-compose.override.local_https_thredds.yml PAVICS/birdhouse/docker-compose.override.yml In this override file specify the path to local netcdf files and set localhost IP. :: ./mycompose.sh up -d thredds proxy magpie twitcher It may take a minute for twitcher to get online. If it does not respond after a while, try to restart everything once or twice. :: ./mycompose down ./mycompose.sh up -d thredds proxy magpie twitcher Check that twitcher is running at https://localhost/twitcher/ (returns hello) Check that magpie is running at https://localhost/magpie/ Check that thredds is running at https://localhost/twitcher/ows/proxy/thredds/ Play around with magpie permissions to check that the security is working HTTPS custom WPS service ------------------------ Follow all the steps of the HTTPS THREDDS setup above up to the template copy, instead use:: cp PAVICS/birdhouse/templates/docker-compose.override.local_https_wps.yml PAVICS/birdhouse/docker-compose.override.yml In this override file, set the localhost IP, then you can switch the wpsandbox image for the wps service image of your choice and assign it an available port of your choice. Then assign a corresponding port to the proxy. A new proxy configuration file need to be added to PAVICS/birdhouse/config/proxy/conf.d/wpsandbox.conf for this service, e.g.:: server { listen 8081; location / { proxy_pass http://wpsandbox; } } :: ./mycompose.sh up -d proxy magpie twitcher wpsandbox It may take a minute for twitcher to get online. If it does not respond after a while, try to restart everything once or twice. :: ./mycompose down ./mycompose.sh up -d thredds proxy magpie twitcher Check that twitcher is running at https://localhost/twitcher/ (returns hello) Check that magpie is running at https://localhost/magpie/ Register the new wps service in magpie: In Home > Edit Services > wps > Add Service. In our case the name is wpsandbox and the public url is https://dummy.crim.ca/twitcher/ows/proxy/wpsandbox with the wps service type. Then edit this new service Protected URL to http://dummy.crim.ca:8081 Alternatively, this can be entered in PAVICS/birdhouse/config/magpie/providers.cfg Check that the wps is running at https://localhost/twitcher/ows/proxy/wpsandbox/pywps?service=WPS&version=1.0.0&request=GetCapabilities Play around with magpie permissions to check that the security is working PAVICS-DataCatalog development ------------------------------ :: git clone https://github.com/Ouranosinc/PAVICS-DataCatalog.git cd PAVICS-DataCatalog cp catalog.cfg ~/catalog.cfg Edit ~/catalog.cfg with Solr address. Note that within docker, localhost is not the same as the workstation localhost, so the address must use the ip of the local machine (retrieve with, e.g., ifconfig). Also point to a valid thredds server. :: # Docker requires root privileges sudo su docker build -t pavics-datacatalog . docker run --name pavics-datacatalog1 -d -v ~/catalog.cfg:/home/catalog.cfg -p 8009:80 pavics-datacatalog Check that the wps is running at http://localhost:8009/pywps?service=WPS&request=GetCapabilities&version=1.0.0 Note that when running with magpie, the root directory of thredds does not properly redirect to the proxy. To crawl a thredds server with threddsclient, provide the birdhouse root directory of thredds (e.g. in catalog.cfg):: https://localhost/twitcher/ows/proxy/thredds/catalog/birdhouse Flyingpigeon development ------------------------ :: git clone https://github.com/Ouranosinc/flyingpigeon.git cd flyingpigeon git checkout pavics Need to either add a custom.cfg or modify profiles/base.cfg with:: [settings] geoserver = http://host:port/geoserver/wfs Then proceed with installation:: make clean install make test make start The WPS will be running at:: http://localhost:8093/wps?service=WPS&version=1.0.0&request=GetCapabilities To restart flyingpigeon (e.g. after modifications):: make stop make start Malleefowl development ---------------------- :: git clone https://github.com/Ouranosinc/malleefowl.git cd malleefowl git checkout pavics Need to either add a custom.cfg or modify profiles/base.cfg with:: [settings] persist-path = /tmp archive-root = / Then proceed with installation:: make clean install make test make start The WPS will be running at:: http://localhost:8091/wps?service=WPS&version=1.0.0&request=GetCapabilities