Store

The store represents all files that make up a deployment of ldproxy besides the application itself. That includes all configuration files but also other resources like file databases or caches. In the most simple case all of these files will exist in the local data directory, but that for example would not be a scalable approach for cloud environments.

The flexibility to meet such different demands is provided by store sources, which in theory allow to integrate any imaginable local or remote file source. The store can be composed of any number of sources, how that works is described in the following paragraphs.

Options

These are the configuration options for key store in the global configuration.

NameDefaultDescriptionTypeSince
mode
RW
RW or RO. Set to RO if ldproxy should not be allowed to write anything. Otherwise, see mode for Store sources.
string
v2.0
sources
[{type:FS,src:.}]
List of Store sources. The default is the data directory. The list is appendable, which means entries from configuration files will be appended to the default.
array
v3.5

Store Sources

Options

These are general options common to all source types. Specific options might be described in Source Types.

NameDefaultDescriptionTypeSince
type
string
v3.5
content
ALL
string
v3.5
mode
RO
Set to RW to make the source writable.
string
v3.5
src
The source path, see Source Types for details.
string
v3.5
prefix
null
Prefix for file paths from a source, may be needed to align directory structures.
string
v3.5
include
[]
Glob expressions for file paths from a source that should be included, others will be ignored. May be needed to align directory structures.
array
v3.6
exclude
[]
Glob expressions for file paths from a source that should be ignored. May be needed to align directory structures.
array
v3.6
archiveRoot
/
Can be set to use a subdirectory from a ZIP file.
string
v3.5
parts
[]
List of partial sources for content type MULTI.
array
v3.5

Source Types

Store sources may have different source types, which allows to integrate any local or remote files.

FS

Access files from the local file system. src must be a path to a directory or a ZIP file. The path can either be absolute or relative to the data directory. FS directory sources are writable by default.

S3

Access files from any S3 compatible object store. src must be a relative path composed of a host and a bucket, e.g. my-minio/demo or s3.eu-central-1.amazonaws.com/demo.

Additionally an accessKey and a secretKey have to be specified for the source.

HTTP

Access files from a web server. src must be a valid URL pointing to a ZIP file. Can be neither writable nor watchable.

GITHUB

Access files from a GitHub repository. Convenience wrapper for HTTP. src must be a relative path composed of the organization, the repository and an optional branch (default is main), e.g. ldproxy/demo:main.

GITLAB

Access files from a GitLab repository. Convenience wrapper for HTTP. src must be a relative path composed of an optional host (default is gitlab.com), the organization, the repository and an optional branch (default is main), e.g. my-gitlab/ldproxy/demo:main.

GITEA

Access files from a Gitea repository. Convenience wrapper for HTTP. src must be a relative path composed of a host, the organization, the repository and an optional branch (default is main), e.g. my-gitea/ldproxy/demo:main.

Content Types

Store sources may have different content types, which allows a fine-granular composition of the store. The ALL type would be sufficient since it contains all other types, but then all sources would have to follow the required directory structure. The other types basically allow to include files from an arbitrary directory structure, which may just be convenient or even necessary when the files are also used in another context.

ALL

Store sources with content type ALL are a container for these other content types:

  • CFG in path cfg.yml
  • ENTITIES in path entities/
  • VALUES in path values/
  • RESOURCES in path resources/

CFG

Store sources with content type CFG contain a single YAML file with global configuration settings.

ENTITIES

Entities make up the main user-defined part of the application, for example APIs and Data Providers. The configuration of these entities is defined in YAML files.

Store sources with content type ENTITIES are a container for these other content types:

  • INSTANCES in path instances/
  • DEFAULTS in path defaults/
  • OVERRIDES in path overrides/

INSTANCES

Store sources with content type INSTANCES contain the main definitions of entities. The paths are made up of an entity type and an entity id, for example services/foo.yml. An instance has to be unique, so it can only be defined once across all store sources.

DEFAULTS

Store sources with content type DEFAULTS may contain default configurations for the different entity types that are applied before the instance configuration. The paths are made up of an entity type and an optional entity subtype, for example services.yml would contain common defaults for all subtypes and services/ogc_api.yml would contain defaults for the subtype ogc_api. Some entity types may also allow another level of files with partial defaults, for example services/ogc_api/metadata.yml.

OVERRIDES

Store sources with content type OVERRIDES may contain override configurations for entities that are applied after the instance configuration. The paths have to match the instance paths, for example services/foo.yml.

VALUES

Store sources with content type VALUES may contain auxiliary user-defined configurations, e.g. codelists. The configurations are defined in YAML or JSON files. The paths are made up of a value type and an arbitrary file path, for example codelists/foo/bar.yml. The actual path patterns are defined by the components that need them, their documentation will state something like "are values with type codelists and path foo/bar". Some features may also need a writable source to work properly.

RESOURCES

Store sources with content type RESOURCES may contain any other files that are needed by the application. The paths are defined by the components that need them, their documentation will state something like "are resources with path foo/bar". Some features may also need a writable source to work properly.

MULTI

Store sources with content type MULTI are a container with a common root that contains a list of other content types. This type can be used for convenience for example for an ALL-like source with a different directory structure.

Order

CFG

Global configuration files are read and merged in the order of the sources on startup.

In theory every global configuration file could define additional store sources, which in turn could contain additional global configuration files. Such chains are not allowed, only global configuration files on the first level are considered.

ENTITIES

Entity configuration files are read in the order of the sources on startup. First all DEFAULTS are read and merged in the order of the sources. Then the INSTANCES are created in order of the sources. If an instance exists in multiple sources, the first one will win since an instance can only exist once. Duplicates will be logged as error. Last all OVERRIDES are applied to the instances in order of the sources.

VALUES

Value configuration files are read in the order of the sources on startup. If the same path exists in multiple sources, the last one will win. When the application wants to write a value with a specific path, the sources are checked in reverse order for the first that is writable for values with the given prefix. So if more than one source could take the given value, the one defined later will win.

RESOURCES

Resources are only accessed on-demand. When the application wants to read a resource with a specific path, the sources are checked in reverse order for the existence of that path. So if a path exists in more than one source, the one defined later will win. When the application wants to write a resource with a specific path, the sources are checked in reverse order for the first that is writable for resources with the given prefix. So if more than one source could take the given resource, the one defined later will win.

Examples

Entity instances from local ZIP file


store:
  sources:
  - type: FS
    content: INSTANCES
    src: /path/to.zip

ZIP


services/
  foo.yml
  bar.yml

GeoPackages from remote ZIP file


store:
  sources:
  - type: HTTP
    content: RESOURCES
    prefix: features
    src: https://example.org/path/to.zip

ZIP


foo.gpkg
bar.gpkg

Subfolder from a GitHub Repository


store:
  sources:
  - type: GITHUB
    src: org/repo
    archiveRoot: /path/to/store

Data directory with old store layout

You could use this if you want to keep using the old layout with v4.


store:
  sources:
  - type: FS
    content: MULTI
    src: /old/data
    parts:
    - content: CFG
      src: cfg.yml
    - content: ENTITIES
      src: store
    - content: RESOURCES
      src: api-resources
    - content: RESOURCES
      src: api-resources/resources
      prefix: api-resources
    - content: RESOURCES
      src: store/resources
      mode: RO
    - content: RESOURCES
      src: cache/tiles
      prefix: tiles
    - content: RESOURCES
      src: cache/tiles3d
      prefix: tiles3d
    - content: RESOURCES
      src: proj
      prefix: proj
    - content: RESOURCES
      src: templates/html
      prefix: html/templates