Open Source Software Catalog

The Open Source Software Catalogue provides an overview of free and open-source software developed by or for the Swiss Confederation. It is based on the Public Code standard, which describes software in a machine-readable way, making it discoverable and understandable for both developers and non-technical users.

Context

flowchart TB
  subgraph OSS_CATALOG[OSS Catalog]
    direction LR
    db[(PostgreSQL)]
    api[Developers Italia API]
    crawler[Publiccode Crawler]
    web[OSS Catalog Client]

    api --> db
    crawler -- "Read publishers<br>Create softwares<br>(REST)" --> api
    web <-- Static build<br>(REST) --> api
  end

  user((User))
  github[GitHub, GitLab, others]
  action[Scheduled Github Action]
  swissindex[https://github.com/swiss/index/blob/main/README.md]
  user -- "(HTTP)" --> web
  crawler -- "Scan repositories<br>Parse publiccode.yml" --> github

  swissindex -- Reads publishers --> action
  action -- creates publishers (REST) --> api
  action -- triggers--> crawler

Usage

Overview

What you will do:

• Clone this repo
• Generate PASETO key & create GitHub API token
• Start API service (and DB)
• Add publisher(s) using curl
• Run crawler
• Build and run the client

Preparation

Clone this repository:

git clone git@github.com:swiss/oss-catalog.git
cd oss-catalog/

The API uses a paseto key for authentication. Generate it with:

./paseto/generate-paseto-key.sh

Create GitHub API Token with the public_repo Permission under https://github.com/settings/tokens and add it to the .env file. The crawler needs this token to access GitHub repositories.

echo "GITHUB_TOKEN=<your access token>" >> .env

Install and enable pnpm (see pnpm docs):

curl -fsSL https://get.pnpm.io/install.sh | sh -

API Service

Start API with database:

./start-api

Use API via Curl (optional)

The API uses PASETO for authentication. Generate a PASETO token (valid for 24h):

source .env
cd paseto/go
PASETO_TOKEN="$(go run paseto-generate.go $PASETO_KEY)"

or, using just:

export PASETO_TOKEN="$(just pt)"

List publishers (no authentication needed):

curl http://localhost:3000/v1/publishers

Create some publishers:

curl -X POST -H "Authorization: Bearer $PASETO_TOKEN" -H "Content-Type: application/json" -d '{"codeHosting": [{"url": "https://github.com/swiss/", "group": true}], "description": "Swiss Government"}' http://localhost:3000/v1/publishers

curl -X POST -H "Authorization: Bearer $PASETO_TOKEN" -H "Content-Type: application/json" -d '{"codeHosting": [{"url": "https://github.com/DCC-BS", "group": true}], "description": "Basel-Stadt"}' http://localhost:3000/v1/publishers

curl -X POST -H "Authorization: Bearer $PASETO_TOKEN" -H "Content-Type: application/json" -d '{"codeHosting": [{"url": "https://github.com/agridata-ch", "group": true}], "description": "agridata-ch"}' http://localhost:3000/v1/publishers

Crawler

Run crawler - this will crawl all repositories in the API, checks for publiccode.yml and add them to the database if available and valid.

./start-crawler

Catalog Client Application

Start the catalog client application at http://localhost:8080:

./start-client

Or for development mode:

pnpm install
pnpm scope:client dev

Then visit http://localhost:4321

Known Issues

• The actual deletion of a publisher or software in the database takes a while (as if it is done asynchronously in the API).

Resources

• publiccode.yml Standard
• publiccode.yml crawler for the software catalog of Developers Italia
  • Fetches registered publishers from the Developers Italia API, crawles all their repositories & feeds the publiccode.yml results into the Developers Italia API.
• publiccode.yml parser for Go by Developers Italia – Used by the publiccode.yml crawler
• Developers Italia API – Stores the results of the publiccode.yml crawler in a PostgreSQL db, runs at https://api.developers.italia.it/v1/software
• publiccode yml Editor by Developers Italia – Web UI to conveniently edit publiccode.yml files
• Developers Italia website – Italy's OSS catalog (Jekyll site)
  • Downloads crawled softwares from Developers Italia API: https://github.com/italia/developers.italia.it/blob/main/scripts/get-software.js
• More publicode.yml components by Developers Italia
• List of the organisations of the Swiss Confederation