Skip to content

Data Storage - piveau-hub#

Introduction#

Hub is the central (meta) data management component of piveau. It employs a triplestore as primary database and offers first-class support for DCAT-AP.

Features:

  • DCAT-AP RESTful API
  • URI management
  • Access control with Keycloack
  • Controlled vocabulary integration
  • Single-Page-Application Frontend

Technology:

  • Vert.x Java Framework
  • Apache Jena
  • Elasticsearch (ES)
  • Virtuoso Triplestore
  • Vue.js

Demos:

Troubleshoot#

strict_dynamic_mapping_exception while indexing#

If you use a custom ES mapping and get a [es/index] failed: [strict_dynamic_mapping_exception] error while indexing a resource, double-check your ES mapping to ensure all fields you want to index are explicitly defined. With "dynamic": "strict" mapping, ES rejects documents containing unknown fields.

Assuming you run ES locally with port 9200, you can inspect the fields by checking: 

http://localhost:9200/your-index-name/_mapping

Error example: 

[es/index] failed: [strict_dynamic_mapping_exception] [1:4582] mapping set to strict, dynamic introduction of [sample_field_name] within [_doc] is not allowed

F.A.Q.#

Questions and Answers:

  1. Is there no validation during data submission via the API?
    Currently, there is no validation of mandatory fields during data submission via the API.

  2. Is there a way to patch records via the API instead of submitting the entire dataset for updates?
    Handling updates in the RDF world is complex. It is often unclear whether a property should be replaced or added, especially for properties that can accept multiple values. This ambiguity makes straightforward patching challenging.

  3. How can we trigger re-indexing in the search engine?
    To trigger re-indexing of the entire repository or a specific dataset, you can use a command in the Hub-Repo CLI. For example, you can execute the command index -v my-catalogue to re-index one or more catalogs.