Scoring methodology

Unreleased preview — metrics v2

This section describes metrics v2, which is not yet released. Current piveau deployments run metrics v1 (documented in the pages above). The content here is published in advance so operators and publishers can prepare; details may still change until the release.

• Scoring methodology
  • The property → layer → weight → result model
  • The property catalogue
    • Dataset (maximum 7.5)
    • Distribution (maximum 7.5)
    • Data Service (maximum 7.5)
  • Check kinds
  • Entity scoring
  • Dataset roll-up
  • Standalone data services
  • Catalogue score
  • Per-metric coverage
  • Deprecation
  • FAIR dimensions
  • DCAT-AP compliance (SHACL)

The property → layer → weight → result model

Every property that metrics v2 checks on a Dataset, Distribution, or Data Service produces:

1. A layer, derived from how important the property is considered: Mandatory → L1, Recommended → L2, Optional → L3.
2. A weight, derived from the layer: L1 = 1.0, L2 = 0.5, L3 = 0.25.
3. A binary result, 0 or 1 — the property either satisfies its check or it does not (see Check kinds below).
4. A property score = weight × result.

There is no partial credit, and no extra penalty for failing a mandatory property beyond its (larger) weight — importance is expressed purely through the weight.

Each entity type's property set is designed so its weights sum to a maximum of 7.5. A single RDF property can back several distinct checks — for example, dct:format on a Distribution backs four separate checks: whether it is set at all, whether its value is in the format vocabulary, whether that format is non-proprietary, and whether it is machine-readable.

The property catalogue

The tables below list every property check in the default metrics v2 configuration, grouped by entity type. The Dimension column is one of the four FAIR dimensions (Findability, Accessibility, Interoperability, Reusability) — this is carried through as metadata on each metric, but v2 never aggregates or scores by dimension (see FAIR dimensions below).

Dataset (maximum 7.5)

Metric	Property	Importance	Weight	Dimension
Title	dct:title	Mandatory	1.0	Findability
Description	dct:description	Mandatory	1.0	Findability
Keyword usage	dcat:keyword	Recommended	0.5	Findability
Categories	dcat:theme	Recommended	0.5	Findability
Geo search	dct:spatial	Recommended	0.5	Findability
Temporal coverage	dct:temporal	Recommended	0.5	Findability
Contact point	dcat:contactPoint	Recommended	0.5	Reusability
Publisher	dct:publisher	Recommended	0.5	Reusability
Access rights	dct:accessRights	Optional	0.25	Reusability
Access rights vocabulary	dct:accessRights	Optional	0.25	Reusability
Date of issue	dct:issued	Optional	0.25	Findability
Modification date	dct:modified	Optional	0.25	Findability
Identifier (dct:identifier)	dct:identifier	Optional	0.25	Accessibility
Identifier (adms:identifier)	adms:identifier	Optional	0.25	Accessibility
Related resource	dct:relation	Optional	0.25	Findability
Provenance	dct:provenance	Optional	0.25	Reusability
Applicable legislation	dcatap:applicableLegislation	Optional	0.25	Findability
Landing page	dcat:landingPage	Optional	0.25	Accessibility

18 checks, weights summing to 7.5.

Distribution (maximum 7.5)

Metric	Property	Importance	Weight	Dimension
Access URL	dcat:accessURL	Mandatory	1.0	Accessibility
Format	dct:format	Recommended	0.5	Accessibility
License information	dct:license	Recommended	0.5	Reusability
License vocabulary	dct:license	Recommended	0.5	Reusability
Format from vocabulary	dct:format	Recommended	0.5	Interoperability
Non-proprietary format	dct:format	Recommended	0.5	Interoperability
Machine-readable format	dct:format	Recommended	0.5	Interoperability
Description	dct:description	Recommended	0.5	Findability
Download URL	dcat:downloadURL	Optional	0.25	Accessibility
Media type	dcat:mediaType	Optional	0.25	Accessibility
File size	dcat:byteSize	Optional	0.25	Accessibility
Media type from vocabulary	dcat:mediaType	Optional	0.25	Interoperability
Rights	dct:rights	Optional	0.25	Reusability
Date of issue	dct:issued	Optional	0.25	Findability
Modification date	dct:modified	Optional	0.25	Findability
Access URL accessibility test	dcat:accessURL	Optional	0.25	Accessibility
Download URL accessibility test	dcat:downloadURL	Optional	0.25	Accessibility
Documentation	foaf:page	Optional	0.25	Reusability
Status	adms:status	Optional	0.25	Reusability
Title	dct:title	Optional	0.25	Findability

20 checks, weights summing to 7.5.

Data Service (maximum 7.5)

Metric	Property	Importance	Weight	Dimension
Title	dct:title	Mandatory	1.0	Findability
Endpoint URL	dcat:endpointURL	Mandatory	1.0	Accessibility
Keyword usage	dcat:keyword	Recommended	0.5	Findability
Categories	dcat:theme	Recommended	0.5	Findability
Conforms To	dct:conformsTo	Recommended	0.5	Interoperability
Endpoint description	dcat:endpointDescription	Recommended	0.5	Interoperability
Publisher	dct:publisher	Recommended	0.5	Reusability
Serves dataset	dcat:servesDataset	Recommended	0.5	Interoperability
Endpoint URL accessibility test	dcat:endpointURL	Optional	0.25	Accessibility
Format	dct:format	Optional	0.25	Accessibility
Format from vocabulary	dct:format	Optional	0.25	Interoperability
License information	dct:license	Optional	0.25	Reusability
License vocabulary	dct:license	Optional	0.25	Reusability
Access rights	dct:accessRights	Optional	0.25	Reusability
Description	dct:description	Optional	0.25	Reusability
Documentation	foaf:page	Optional	0.25	Reusability
Landing page	dcat:landingPage	Optional	0.25	Accessibility
Landing page accessibility test	dcat:landingPage	Optional	0.25	Accessibility

18 checks, weights summing to 7.5.

This gives 56 checks in total (18 + 20 + 18) in the default configuration — the same table that drives the annotator, accessibility, and score services (see Configuring and customizing the metrics for how to change it).

Check kinds

Check kind	Result is 1 when…	Used for
Set	The property is present with at least one value	Most metrics
Set & valid URL	The property is present and its value is a syntactically valid URI/URL	dcat:accessURL, dcat:downloadURL, dcat:endpointURL
Vocabulary membership	The value is found in the relevant controlled vocabulary	Format-from-vocabulary (EU file-type authority table), media-type-from-vocabulary (IANA media types), license vocabulary, access-rights vocabulary
Non-proprietary / machine-readable	The dct:format value is flagged as such in the file-type vocabulary	The two Distribution/Data-Service format-quality metrics
Accessibility test	The URL is set and an HTTP HEAD request returns a 2xx status; everything else — including redirects (3xx) — is 0	The four "…accessibility test" metrics

The accessibility test is deliberately binary: unlike v1 (which accepted a broader 200–399 range), v2 only counts an actual 2xx success as reachable.

Entity scoring

For each Dataset, Distribution, and Data Service:

entity_score = Σ (weight of checked property × its binary result)
entity_max   = Σ (weights) = 7.5

Scores are stored and reported as raw numbers between 0 and 7.5 — there is no normalization to a percentage, and the metrics services do not derive rating bands (bands, if a portal wants them, are a presentation-layer decision).

Dataset roll-up

A dataset's own score does not stand alone — it is combined with its distributions and, if any, their embedded data services:

dataset_final = ( Dataset_score
                 + avg(distribution_scores)
                 + avg(dataService_scores) ) / N

• N is the number of groups actually present — 2 or 3. A Data Service can only exist embedded in a Distribution (via dcat:accessService) or standing alone at the catalogue root — never attached directly to a dataset — so a dataset can only have N = 2 (Dataset + distributions) or N = 3 (Dataset + distributions + embedded data services).
• A dataset with no distributions at all (a metadata-only record) is scored on its own: dataset_final = Dataset_score (N = 1).
• Each of the up-to-three groups carries equal weight — roughly one third each, when all three are present.

Worked examples:

Example	Inputs	Calculation	dataset_final
1	Dataset 4.25; one distribution 3.25	(4.25 + 3.25) / 2	3.75
2	Dataset 5.5; distributions [4.25, 4.25, 4.75, 4, 3.25] (avg 4.1); embedded data services [4.5, 4.2] (avg 4.35)	(5.5 + 4.1 + 4.35) / 3	4.65

The second worked example above, as a roll-up diagram:

graph TD
    DS["Dataset entity score: 5.5"]
    DIST["5 distributions: 4.25, 4.25, 4.75, 4.0, 3.25 - avg 4.1"]
    DSV["2 data services: 4.5, 4.2 - avg 4.35"]
    FIN["Dataset final: (5.5 + 4.1 + 4.35) / 3 = 4.65"]
    DS --> FIN
    DIST --> FIN
    DSV --> FIN

Standalone data services

A dcat:DataService published directly at the catalogue root — i.e. one that is not reachable through any distribution's dcat:accessService — is scored directly as its own entity, using the same entity-scoring formula as any Data Service. It does not go through the dataset roll-up.

Worked example: a standalone data service scoring 5.25 stays 5.25 — no averaging.

Catalogue score

A catalogue's overall score is the average of every dataset's final score and every standalone data service's score:

catalogue_score = average( all dataset_final scores
                          + all standalone data-service scores )

Each dataset and each standalone data service counts as one unit. Data services embedded in a distribution are only counted within their dataset's roll-up above — they are not added again at the catalogue level, to avoid double counting.

Worked example, combining the dataset roll-up examples above with the standalone data service: (3.75 + 4.65 + 5.25) / 3 = 4.55.

The catalogue-level aggregation of that worked example:

graph TD
    F1["Dataset final: 3.75"]
    F2["Dataset final: 4.65"]
    SD["Standalone data service: 5.25"]
    CAT["Catalogue score: (3.75 + 4.65 + 5.25) / 3 = 4.55"]
    F1 --> CAT
    F2 --> CAT
    SD --> CAT

One consequence of this method is worth knowing: an embedded data service influences the catalogue score only indirectly (through its dataset's averaged roll-up), while a standalone data service counts as a full unit on its own — so the same service can carry different weight in the catalogue score depending on whether it is embedded or standalone.

Per-metric coverage

In addition to scores, the catalogue-level output reports per-metric coverage — the one place in v2 where a percentage is used:

coverage% = (number of evaluated resources where the check passed)
          / (number of evaluated resources of that type) × 100

Coverage is reported separately for the Dataset, Distribution, and Data Service resource blocks (embedded and standalone data services are combined into one Data Service block). Everywhere else — entity scores, dataset-final scores, and catalogue scores — stays a raw 0–7.5 number; only coverage is a percentage.

Deprecation

• Signal: a distribution carrying adms:status set to http://purl.org/adms/status/Deprecated.
• Behaviour: a deprecated distribution is ignored entirely. It is not evaluated, no measurements are written for it, and it is excluded from scoring, from the distribution average in the dataset roll-up, and from catalogue coverage.
• All-deprecated dataset: if every distribution of a dataset is deprecated, the whole dataset is excluded from catalogue aggregation. This is different from a metadata-only dataset with no distributions at all, which is still scored on its own (N = 1, see above).
• Deprecation only applies to distributions — Data Services have no equivalent status property in DCAT-AP 3.

FAIR dimensions

Each metric in the tables above carries an internal FAIR label — Findability, Accessibility, Interoperability, or Reusability. v1's fifth dimension, Contextuality, is dropped in v2; its former properties have been redistributed across the remaining four. Dimensions are informational labels on individual metrics only — v2 never sums or reports a score per dimension, and there is no dimension-based filtering of results.

DCAT-AP compliance (SHACL)

DCAT-AP compliance, as validated by the SHACL validator, is kept but no longer scored in v2. The validating-SHACL service is unchanged: it still validates metadata in the pipeline and via its standalone API, and the validation report is still produced and can still be displayed — it simply no longer contributes a weighted property to the quality score.