feat: add extended metadata fields for fields and metrics

[jklahr]

Summary

This PR proposes a new extended metadata layer for the Apache Ossie core spec — a standardized, optional set of non-executional, interpretability metadata fields for fields and metrics.

A companion announcement and discussion will be sent to dev@ossie.apache.org.

Motivation

OSI/Ossie defines what data is. There is currently limited standardization for how data should be interpreted and presented. As a result, consumers (BI tools, AI agents, developers) must repeatedly infer or redefine units, display conventions, aggregation behavior, KPI polarity, and field visibility.

This proposal addresses those gaps through optional metadata that does not affect query execution.

Extended proposal document (design rationale, field-by-field detail, open questions): https://docs.google.com/document/d/1DLuteF0WwlVbpD5kmHhcqa6vItqNFL4MoAQrImy6dKo

Changes

Three files in core-spec/ are updated:

• spec.yaml — adds extended metadata fields to fields and metrics
• osi-schema.json — adds new $defs: SemanticType, Measurement, DefaultSort, SemanticMapping, DesiredDirection, DefaultAggregation, DefaultTimeGranularity, UnitSystem
• spec.md — documents all new fields with tables, type descriptions, and examples

New fields

Field	Applies to	Purpose
display_label	fields, metrics	Human-readable display name
semantic_type	fields, metrics	Token or URI classification (e.g., monetary, or https://www.iso20022.org/glossary/LEI)
measurement	fields, metrics	Unit/quantity metadata (quantity_kind, unit ISO 4217, unit_system)
display_format	fields, metrics	Excel-compatible format string (e.g., $#,##0.00)
default_sort	fields, metrics	Default sort direction, null handling, and optional sort-by-field
semantic_mappings	fields, metrics	Links to external ontologies via SKOS-based open predicate vocabulary (target, predicate, provenance)
hidden	fields, metrics	Hide from consumer UIs while remaining available in expressions
group_label	fields, metrics	Organizational grouping for UI presentation
desired_direction	metrics only	KPI polarity: higher_is_better, lower_is_better, neutral
default_aggregation	fields only	Default aggregation when field is used as a measure
default_time_granularity	fields only	Default time bucket for temporal fields: day, week, month, quarter, year

Design principles

• Non-executional: no field affects query execution
• Optional and backward compatible: all existing models remain valid
• Consumer-focused: designed for BI tools, AI agents, and developers

Community input incorporated

Dragos Crintea (maintainer of semantido, a SQLAlchemy-based semantic layer used in capital-markets/regulatory-reporting contexts) provided substantive feedback that has been incorporated into this PR:

1. semantic_mappings uses an open predicate vocabulary (SKOS baseline: exactMatch, closeMatch, broadMatch, narrowMatch, relatedMatch) rather than a closed enum, allowing non-positive assertions like DISTINCT_FROM via extensions. This addresses the regulatory homonym problem — e.g., "Counterparty" under EMIR and "Counterparty" under MiFIR are distinct concepts that should not be conflated.
2. semantic_type accepts either a well-known token or a URI, keeping simple cases simple while supporting governed external type systems (ISO 20022, FIBO, SKOS vocabularies, LEI, ISIN, etc.).

Dragos has also offered to contribute a reference implementation in the semantido-OSI converter and benchmark data showing accuracy improvements from this class of metadata. Those contributions will follow in subsequent PRs.

Impact on existing implementations

None — all additions are optional. The JSON schema uses additionalProperties: false per existing convention; all new fields are added explicitly to the Field and Metric definitions.

Related discussions

• Discussion Native support for units #42 (Semantic typing): Native support for units #42
• Discussion Native support for currencies #43 (Aggregation semantics): Native support for currencies #43
• Discussion Semantic Field Types: dimension_type, data_type, and pii_classification #55 (Metadata extensibility): Semantic Field Types: dimension_type, data_type, and pii_classification #55
• Issue Extended Metatada Proposal for OSI #100: Extended Metatada Proposal for OSI #100

AI disclosure

This contribution was developed with AI assistance (Snowflake Cortex Code). The author (Josh Klahr) has reviewed all changes and is personally responsible for the content per the ASF Generative Tooling Guidance.

.... Generated with Cortex Code

Co-Authored-By: Cortex Code noreply@snowflake.com