Docker Compose#
The quickest way to setup piveau metrics is with a Docker Compose file you can get here. This will start all metrics services and in addition a configured piveau-hub-repo and all external databases. You can use this as a blueprint for a production deployment.
Prerequisites#
- Docker & Docker Compose
- Git
- curl
Run images#
Build and start everything with Docker Compose:
This may take a couple of minutes. Upon completion, you can check if all services are running, e.g piveau-hup-repo at http://localhost:8081 or piveau-metrics-cache at http://localhost:8185.
For usage refer to the User Guide.
Manual setup#
For a manual setup without the provided docker compose or to enable different behaviour, you can configure the services as follows:
piveau-hub-repo#
To run the metrics pipeline after piveau-hub-repo stores a resource, register the metrics pipe as a system pipe in the
environment variable PIVEAU_HUB_SYSTEM_PIPES. It holds a JSON array with one object per system pipe:
PIVEAU_HUB_SYSTEM_PIPES=[{"name":"metrics","runOnHidden":false,"config":{"filters":{"resourceTypes":["http://www.w3.org/ns/dcat#Dataset"]}}}]
| Variable | Description | Default Value |
|---|---|---|
| PIVEAU_HUB_SYSTEM_PIPES[].name | Name of the pipe to launch. Must match the header.name of the pipe descriptor, i.e. metrics |
|
| PIVEAU_HUB_SYSTEM_PIPES[].runOnHidden | Also launch the pipe for resources in hidden catalogues | false |
| PIVEAU_HUB_SYSTEM_PIPES[].config.filters.resourceTypes | Resource types (URIs) the pipe is launched for. If not set or empty, it is launched for all types | [] |
Note
hub-repo 4.4.3 and earlier read the filter on the resource endpoints from config.filter (singular) and ignore
config.filters, so the pipe runs for every resource type. Later versions read config.filters and still
accept config.filter.
The pipe itself must be available to piveau-hub-repo, either from a pipe repository in PIVEAU_CLUSTER_CONFIG or from
a mounted pipes folder (see How to trigger the Metric Pipeline).
PIVEAU_HUB_VALIDATOR is a JSON object with further metrics settings:
| Variable | Description | Default Value |
|---|---|---|
| PIVEAU_HUB_VALIDATOR.enabled | Delete the metrics graph and metrics history of a dataset together with the dataset | false |
| PIVEAU_HUB_VALIDATOR.history | If the old metrics metadata should be saved as history or overwritten | false |
PIVEAU_HUB_VALIDATOR.metricsPipeName is deprecated
Older setups configure the metrics pipe with PIVEAU_HUB_VALIDATOR.metricsPipeName. In hub-repo 4.x
this key is only honoured by the dataset endpoints (/catalogues/{catalogueId}/datasets). The resource endpoints
(/catalogues/{catalogueId}/resources) launch only the pipes listed in PIVEAU_HUB_SYSTEM_PIPES. A deployment
that still sets only metricsPipeName therefore stops producing metrics for resources stored through the resource
endpoints, without an error in the log. Configure PIVEAU_HUB_SYSTEM_PIPES as shown above instead.
piveau-metrics-reporter#
| Variable | Description | Default Value |
|---|---|---|
| API_KEY | The APi key of the reporter | |
| QUICKCHART_ADDRESS | The adress of the quickchart service the reporter uses to create charts | http://quickchart:3400/chart |
| CORS_DOMAINS | Domains from which a frontend is allowed to access the reporter, if they are different than the one the piveau-metrics-reporter is accessed on. | [] |
| SCHEDULES | Array of languages the reports are generated in and the schedule of their generation | [] |
piveau-metrics-cache#
| Variable | Description | Default Value |
|---|---|---|
| CACHE_APIKEY | The APi key of the piveau-metrics-cache | |
| MONGODB_CONNECTION | The adress of the mongodb service piveau-metrics-cache uses to store aggregations | mongodb://mongodb:27017 |
| CORS_DOMAINS | Domains from which a frontend is allowed to access piveau-metrics-cache, if they are different than the one the piveau-metrics-cache is accessed on. | [] |
| PIVEAU_TRIPLESTORE_CONFIG | JSON object with configuration on how to access the triplestore | |
| PIVEAU_TRIPLESTORE_CONFIG.address | URL of the triplestore | string |
| PIVEAU_TRIPLESTORE_CONFIG.data_endpoint | Relative CRUD endpoint of the triplestore | string |
| PIVEAU_TRIPLESTORE_CONFIG.query_endpoint | Relative query endpoint of the triplestore | string |
| PIVEAU_TRIPLESTORE_CONFIG.username | Username for the triplestore | string |
| PIVEAU_TRIPLESTORE_CONFIG.password | Password for the triplestore | string |
| PIVEAU_TRIPLESTORE_CONFIG.transferFormat | Format to send to the triplestore | string |