Plasmopy v1.0

Infection prediction modeling of Plasmopara viticola life cycle stages in vineyards using weather data. This project represents a revisited and customizable Python version of the VitiMeteo-Plasmopara model in use at Agroscope (Switzerland) for downy mildew infection forecasting.

Quick setup & run

Setup with Bash console

1. Make sure to have Python3 in your Unix system. Install Poetry, a Python packaging and dependency manager:

curl -sSL https://install.python-poetry.org | python3 - --version 2.1.1

Note: if you see an error such as Command '['/home/user/.pyenv/shims/python', ...] returned non-zero exit status 127 during make install or make run, it means Poetry is trying to use a broken pyenv shim. Run these steps inside the WSL/Bash shell (not PowerShell) and ensure a working Python is available (e.g. install python3 via sudo apt install python3 or configure pyenv with pyenv install 3.x and pyenv local 3.x). You can also force Poetry to use the system interpreter with:

poetry env use /usr/bin/python3

Adjust the path as appropriate and add poetry to your PATH per the instructions below.

N.B. Do not forget to add poetry's path to your .bashrc file.

E.g. on Linux machines, modify the ".bashrc" file for permanent inclusion to path, by adding the following line at the end:

export PATH="home/YOUR_USERNAME/.local/bin/poetry:$PATH"

2. Make sure to have Git installed on your computer (https://github.com/git-guides/install-git), then clone Plasmopy with the following command in your terminal:

git clone https://github.com/agroscope-ch/plasmopy.git

3. Move into the plasmopy folder where the Makefile is located (replace user_path_to_plasmopy with your folder path):

cd user_path_to_plasmopy/plasmopy

4. Install all dependencies and activate the virtual environment for this project:

make setup

N.B. To separately install the required packages and activate the virtual environment, the respective specific commands are:

make install

and

make activate

Data processing, model configuration and run from terminal

1. Load your raw timeseries input data in the folder data/input/

2. Choose your data processing and model parameters configurations by customizing the main.yaml config file in the config/ folder. You may also specify an explicit output.run_name (and output.directory) in this file if you wish to control the name of the results folder and files instead of using the input filename.

3. Run the infection prediction model with the command:

make run

Data processing, model configuration and run from the streamlit web-app

Launch the web-app from your browser with the command:

make app

Operational deployment

Scheduled runs with run_cron.sh

For continuous monitoring, run_cron.sh runs the model automatically every 3 hours (at 00:00, 03:00, 06:00, … local time) and publishes the result.

Start in the background:

nohup ./run_cron.sh &

Stop it:

kill <PID>   # PID is printed to the console and logged on startup

Logs are written to logs/cron.log. The script runs make run at each interval, so it picks up the current config/main.yaml and config/secrets.yaml on every execution.

FTP publishing

After each successful run, the script uploads *.html (the mobile-optimised combined view) to a remote FTP server. The destination host, username, and password are configured at the top of run_cron.sh:

FTP_HOST="hostname/path"
FTP_USER="username"
FTP_PASS="password"

The target filename on the server matches the local basename (e.g. my_run.html). Upload status is logged to logs/cron.log. If the output file is not found or the upload fails, the model run is still considered successful and the error is logged without aborting the schedule.

Security note: store credentials directly in run_cron.sh only on a private server. For shared environments, use a .netrc file or environment variables instead.

Season window filtering for spore data

When spore count data is loaded (from a file or API), only records whose date falls within the current weather data date range are used for algorithmic shortcuts. Records from a previous season present in the same file are automatically excluded and logged. This prevents historical surge events from generating spurious infection events in the current run.

Developer Tools

Tools used in this project

• Poetry: Dependency management - article
• hydra: Manage configuration files - article
• pre-commit plugins: Automate code reviewing formatting
• DVC: Data version control - article
• pdoc: Automatically create an API documentation for your project

Project Structure

.
├── config
│   ├── main.yaml                   # Main configuration file (committed; secrets left null)
│   ├── secrets.yaml                # Site credentials and API keys (gitignored — do not commit)
│   └── secrets.example.yaml        # Template for secrets.yaml (committed)
├── data
│   ├── input                       # input data
│   ├── output                      # output from model run, including logs and graphs
│   ├── tmp                         # temporary processing data needed for running
│   ├── raw                         # raw data
│   └── raw.dvc                     # DVC file of data/raw
├── docs                            # documentation for your project
├── .gitignore                      # ignore files that cannot commit to Git
├── Makefile                        # store useful commands to set up the environment and run the model
├── models                          # store models
├── notebooks                       # store notebooks
├── .pre-commit-config.yaml         # configurations for pre-commit
├── pyproject.toml                  # dependencies for poetry
├── README.md                       # describe your project
├── src                             # store source code
│   ├── __init__.py                 # make src a Python module
│   ├── main.py                     # orchestrates the execution of the scripts
│   ├── load_data.py                # functions for loading raw data
│   ├── process_data.py             # functions for formatting and processing weather raw data
│   ├── infection_model.py          # launches the specific modeling algorithm at each infection stage
│   ├── infection_functions         # store the algorithms for each infection stage
│   ├── plots.py                    # all plotting functions (PDF, HTML, risk heatmap, combined view)
│   ├── utils.py                    # utility functions (output filenames, sun times, daily stats)
│   ├── decision_support_tool.py    # spore count analysis and API fetch for the spore-driven model
│   └── automated_weather_pull.py   # background weather data fetch and merge from Meteoblue API
└── tests                           # store tests
    ├── __init__.py                 # make tests a Python module
    ├── test_process.py             # test functions for process_data.py
    └── test_model.py               # test functions for infection_model.py

Secrets setup: Copy config/secrets.example.yaml to config/secrets.yaml and fill in your site coordinates, timezone, and API keys. secrets.yaml is listed in .gitignore and will never be committed.

Recreate the project folder template

This project development guide is inspired from Khuyen Tran’s blog.

1. Install cookiecutter (allows for a structured / organized creation of a new python project https://github.com/cookiecutter/cookiecutter) via pip:

pip install cookiecutter

2. Add local installation binaries to $PATH temporarily on the Bash console (replace #### with your user folder):

export PATH="/home/###/.local/bin:$PATH"

Or permanently by adding the same command line at the end of the .bashrc file in the users’s home folder.

If the change was made in a permanent fashion, make sure to apply the changes by sourcing again the .bashrc file, by first moving to your personal home folder:

cd ~

And reloading the .bashrc profile:

source .bashrc

3. We can now call the cookiecutter software and download a default project template to start our project specifications:

cookiecutter https://github.com/khuyentran1401/data-science-template --checkout dvc-poetry

This command will start a prompt asking for the project folder name, the author’s name, and the compatible python version. At the time of writing, the Python version installed in RStudio Server “Agsad” is Python 3.10.12.

Virtual environment plasmopy

Creating a Python project environment for the Plasmopy model implementation in Python (vitimeteo-plasmopara-py).

1. Install poetry for optimized package version management:

curl -sSL https://install.python-poetry.org | python3 - --version 2.1.1

2. Download dependencies and create the project’s specific virtual environment with the following command line code launched from within the project’s home folder:

poetry install

To add a new library, run:

poetry add <library-name>

To remove an existing library, run:

poetry remove <library-name>

Initialize a git repository

git init

Check code before committing

This template uses the following hooks, called in the hidden file .pre-commit-config.yaml:

• Ruff: An extremely fast Python linter, written in Rust. It supports 500 lint rules, many of which are inspired by popular tools like Flake8, isort, pyupgrade, and others.
• black is a code formatter in Python.
• interrogate: Checks your code base for missing docstrings.

To add pre-commit to git hooks, type:

pre-commit install

Now, whenever you run git commit, your code will be automatically checked and reformatted before being committed.

Version your data

To track changes to the data directory, type:

dvc add data

This command will create the data.dvc file, which contains a unique identifier and the location of the data directory in the file system.

To keep track of the data associated with a particular version, commit the data.dvc file to Git:

git add data.dvc
git commit -m "add data"

To push the data to remote storage, type:

dvc push 

Auto-generate API documentation

To auto-generate API document for your project, run:

make docs