Schema library v2 - #75
Conversation
Deploying schema-library with
|
| Latest commit: |
893be13
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://2cf805ce.schema-library.pages.dev |
| Branch Preview URL: | https://bgi-schema-library-v2.schema-library.pages.dev |
petercrocker
left a comment
petercrocker
left a comment
There was a problem hiding this comment.
Reviewed this from the angle of importing the NetBox devicetype-library into it (that's what #76 is for), but items 3 and 5 stand independently of that work. Evidence below is from the branch as it stands plus testing against Infrahub 1.10.6.
1. Blocking — DcimModuleBay.position cannot hold NetBox bay positions
extensions/device_module/device_module.yml:
- name: position
kind: Number
parameters: {min_value: 1}
optional: false
uniqueness_constraints:
- ["device", "position__value"]NetBox device-type definitions are the main external source of bay data, and their position is a free-form string that is not unique within a device. Checked against three real published device types, all three fail, each on a different constraint:
| Device type | Bays | Fails on |
|---|---|---|
| Arista DCS-7508N | 24 | 14 non-numeric positions — F1–F6, PSU-1–PSU-8 |
| Cisco Catalyst 9300-48P | 6 | A and B non-numeric, and duplicate position 1 (Network Module + FAN 1) violating [device, position__value] |
| Juniper EX4300-48T | 3 | position 0 violates min_value: 1 |
Three separate constraints, three devices, 100% failure — so this isn't an edge case in the source data, it's the normal shape of it.
Proposed: make position kind: Text and drop the ["device", "position__value"] uniqueness constraint. If a numeric sort key is wanted, an additional optional Number attribute alongside the verbatim string would give that without rejecting real data.
2. Blocking — NetBox's bay label has nowhere to go
DcimModuleBay has name, position and description, but no equivalent of NetBox's bay label — the short role marker (Supervisor, Line Card). On a DCS-7508N, 10 of 24 bays carry one, and it's the only thing distinguishing supervisor slots from line-card slots.
Could we add an optional bay_label? One caution worth baking into the name: do not call it label. Infrahub auto-populates an attribute literally named label from name when it is unset, and title-cases it — verified on 1.10.6, where an unlabelled bay came back labelled with its own name, and a probe named LabelTest Bay became Labeltest Bay. That makes "no label supplied" indistinguishable from "label equals name".
3. Blocking — three extensions inherit generics this PR deletes
The PR body says:
extensions/modules_linecardsandextensions/modules_routing_enginenow depend onextensions/device_moduleinstead of the removedextensions/modules.
Only .metadata.yml was updated. The schema files still inherit the old Device-namespaced generics, which nothing on this branch defines any more (device_module defines DcimGenericModule / DcimGenericModuleType):
experimental/modules_linecards/linecard.yml:19 - DeviceGenericModuleType
experimental/modules_linecards/linecard.yml:34 - DeviceGenericModule
experimental/modules_routing_engine/routing_engine.yml:16 - DeviceGenericModuleType
experimental/modules_routing_engine/routing_engine.yml:30 - DeviceGenericModule
experimental/optical_transport/optical_transport.yml:648 - DeviceGenericModuleType
experimental/optical_transport/optical_transport.yml:702 - DeviceGenericModule
experimental/optical_transport/optical_transport.yml:818 - DeviceGenericModule
experimental/optical_transport/optical_transport.yml:1122 - DeviceGenericModule
experimental/optical_transport also references them and isn't in the breaking-changes list at all — its header comment still points at extensions/modules/modules.yml. As shipped, all three should fail to load.
4. Question — can DcimModule be templated, and can a spare exist?
DcimGenericModule.module_bay is optional: false, and DcimModule.computed_name is {{ module_bay__computed_name__value }} ({{ module_type__name__value }}), so a module's identity derives from the bay it sits in.
Two consequences worth confirming were intended:
- Object templates. Importing NetBox module types as reusable blueprints means creating a
TemplateDcimModule, which by definition isn't installed anywhere. Infrahub relaxes mandatory relationships and strips unique attributes on generated templates — I've verified both on 1.10.6 — but a computed identity dereferencing an absent relationship is a different case, and I haven't been able to test it. If it doesn't resolve, module types can't be templated at all under this model. - Spares. The comment says spares awaiting installation aren't modelled. That's a legitimate scoping call, but it means a PSU on a shelf can't be tracked, which is a common inventory ask — worth stating in the extension README rather than only in a YAML comment.
5. Smaller items
DcimGenericModuleType has no weight attribute. NetBox module types publish one (a DCS-7500-SUP2 is 5 lb). Since Infrahub has no float kind and integer kilograms round a supervisor or transceiver to 0, an optional weight_grams Number would keep them distinct and sortable.
DcimDeviceType.weight remains an integer Number. Same root cause, pre-existing rather than introduced here — but a large share of the published library is sub-500 g (transceivers, access points) and rounds to 0, and a zero reads as data rather than as a missing value. A weight_grams companion would fix it; happy to raise separately if it's out of scope for v2.
The comment on DcimModuleBay.device contradicts the code:
- name: device
peer: DcimPhysicalDevice
# DcimGenericDevice (not DcimPhysicalDevice) so `name` is resolvable in computed_name above -
# PhysicalDevice itself carries no identifying attribute.The peer is DcimPhysicalDevice, and it works because this PR adds name to that generic — so the comment describes an approach that was abandoned. Worth updating before it misleads someone.
name is now declared unique: true on both DcimGenericDevice and DcimPhysicalDevice, and DcimDevice inherits both. DcimPhysicalDevice.name carries allow_override: any and the note "Here for computed attribute". Is the second unique declaration needed, or would anchoring the computed attribute at DcimGenericDevice avoid duplicating identity across two co-inherited generics?
Items 1 and 2 are what block #76 specifically: without them a modular chassis can't be imported at all, and bay labels have nowhere to land. Happy to open a PR against this branch for any of these if that's easier than patching them yourself.
petercrocker
left a comment
petercrocker
left a comment
There was a problem hiding this comment.
Second pass, this time over the PR as a whole rather than the module extensions. I ran a cross-reference check over all 57 schema files (189 kinds defined, 569 references) and diffed the results against main, so everything below is new in this branch unless marked otherwise.
All static analysis — I have not been able to load v2 against a running Infrahub, so finding 4 in particular is inferred rather than observed.
1. This branch reverts a fix that is already on main
The branch is 2 commits behind main, and the single file it overlaps on is the one those commits fixed:
merge-base: a7ccf78
main since then: d5ae01b fix(security): update stale Infra kind references to current Dcim/Ipam namespaces
fa16de3 Merge pull request #74
overlapping file: experimental/security/security.yml
main now has DcimGenericDevice / DcimInterface / DcimEndpoint / IpamIPAddress / IpamPrefix in that file. This branch still has InfraGenericDevice / InfraInterface / InfraEndpoint / InfraIPAddress / InfraPrefix:
experimental/security/security.yml:137 peer: InfraIPAddress
experimental/security/security.yml:158 peer: InfraPrefix
experimental/security/security.yml:486 - InfraGenericDevice
experimental/security/security.yml:493 menu_placement: "InfraGenericDevice"
experimental/security/security.yml:616 - InfraInterface
experimental/security/security.yml:617 - InfraEndpoint
Merging as-is either silently undoes #74 or produces a conflict that is easy to resolve the wrong way. A rebase on main fixes it.
2. experimental/tenancy was promoted but not deleted
The body says extensions/tenancy is "promoted from experimental/tenancy, and rebuilt", and the .metadata.yml entry for experimental/tenancy is indeed gone — but the directory is still present (2 files) and still defines TenancyTenant. So the kind is declared twice, and the old copy is orphaned: no metadata entry, invisible to the dependency graph, still loadable by anyone pointing at the path.
Assuming the promotion was intended to replace it, experimental/tenancy/ should be removed.
3. extensions/meet_me_room is removed with no mention anywhere
Extensions removed on this branch: extensions/modules, extensions/topology, extensions/meet_me_room, plus extensions/firewall_policer moved to experimental/. The first two and the move are covered in the body; meet_me_room gets zero mentions, including in the breaking-changes list. Either it should be listed as a removal or it was deleted by accident.
4. IpamVLANGroupScope is declared twice, and base is always loaded
# base/ipam.yml:14 — bare stub, no description, no relationships
- name: VLANGroupScope
namespace: Ipam
include_in_menu: false
# extensions/vlan/vlan.yml:11 — the real definition
- name: VLANGroupScope
namespace: Ipam
description: Mixin for objects that can scope a VLAN group (e.g. Region, Site, Rack).
relationships:
- name: vlan_groups
peer: IpamVLANGroup
identifier: vlan_group__scopeThe stub looks deliberate — it lets extensions/location_minimal and extensions/location_site inherit IpamVLANGroupScope without taking a dependency on extensions/vlan. The problem is that base is always loaded, so base + extensions/vlan puts two declarations of one kind into a single load. Per opsmill/infrahub#10139, two declarations of the same kind merge by payload order and one is silently discarded; if the stub wins, vlan_groups disappears with no error.
Worth either guaranteeing the load order explicitly, or forward-declaring in a way that cannot lose the relationship.
5. LocationSite (and LocationRegion) declared in two extensions with no mutual-exclusion signal
LocationSite—extensions/location_siteandextensions/location_minimalLocationRegion—experimental/location_extendedandextensions/location_minimal
The location_extended vs location_minimal pair reads as obviously choose-one, and LocationCountry / LocationRack already overlap that way on main. But location_site and location_minimal are both under extensions/, so nothing signals they can't be loaded together. A note in .metadata.yml or in the two READMEs would prevent it.
6. Pre-existing dangling references, not introduced here
Flagging only because v2 is billed as a full audit pass, so these may be in scope:
| Unresolved kind | Referenced from |
|---|---|
CircuitEndpoint |
experimental/circuit_service |
InfraIPAddress, InfraPrefix, InfraVLAN |
experimental/topology |
Note that extensions/topology is removed on this branch, so the surviving topology extension is the broken experimental one.
Clean: yamllint passes across base/, extensions/, experimental/ and .metadata.yml; find_cycles.py reports no cycles in either the dependency or reverse-dependency graph; every .metadata.yml entry maps to a real directory; and there are no duplicate kinds or unresolved references beyond those listed above.
2 participants
Schema Library v2.0
Version 2.0 is a full pass over the base DCIM and IPAM schemas and most extensions, rather than an incremental update. It combines two kinds of work: closing data-modeling gaps found in an audit of the previous schema — VRF route targets that could only hold one target, interfaces with no valid default MTU, two competing topology and location models, inconsistent use of Dropdown versus plain-text enums, typos in published attribute names, among others — and adding capabilities the library didn't have before, such as tracking swappable device modules and power supplies, assigning IP aggregates to a registry, and scoping tenancy to devices and prefixes. Because so much of the base and extension schemas changed shape as part of that cleanup, v2.0 is not compatible with schemas or data built on v1.x: loading it as-is over an existing v1.x deployment will fail or silently drop data on the attributes and relationships listed below, so plan a migration rather than an in-place upgrade.
Warning
Breaking changes in this release
This release changes attribute and relationship names, types, and cardinalities across
base/dcim.yml,base/ipam.yml,base/location.yml, and most extensions. Loading v2.0 over data created with v1.x schemas requires migrating that data first.Highlights
extensions/modulesshipped no ready-to-use node types, only a generic to build on.extensions/device_modulenow ships a workingModule/ModuleTypepair tied to aDcimModuleBayslot, andextensions/device_psu_moduleadds a PSU flavor with wattage and hot-swap attributes. See Track modules and power supplies as inventory on a device.extensions/aggregateextension adds anAggregatenode tied to anRIRnode. See Assign IP aggregates to a registry.location_minimalhierarchy.extensions/rackandextensions/location_sitesplit racks and sites into their own extensions, so either can be loaded independently. See Model racks and standalone sites independently of a full location hierarchy.extensions/tenancy(promoted out of experimental) wires aTenantdirectly to devices, prefixes, IP addresses, and hosted locations. See Scope tenancy to devices, prefixes, and hosted locations.Track modules and power supplies as inventory on a device
Devices with swappable hardware — fan trays, line cards, power supplies — can now be modeled as installed modules rather than static device attributes.
What changed:
extensions/device_module(replacingextensions/modules) adds aDcimModuleBaynode representing a physical slot on a device, and a ready-to-useModule/ModuleTypepair that installs into a bay. A module can only be tracked once it is installed in a bay; spares awaiting installation are not modeled.extensions/device_psu_module(new, depends onextensions/device_module) adds a PSU-specific module type withwattage(Number) andhotswappable(Boolean) attributes.extensions/modules_linecardsandextensions/modules_routing_enginenow depend onextensions/device_moduleinstead of the removedextensions/modules.extensions/patch_panelno longer defines its ownDcimPatchPanelModulenode; a patch panel's modules are now tracked through the sameDcimModuleBaymechanism as any other device.Assign IP aggregates to a registry
Top-level address space can now be tracked back to the registry or private authority that assigned it, rather than only appearing as a role on a generic prefix.
What changed:
extensions/aggregate(new) adds anAggregatenode for top-level IPv4/IPv6 blocks, and anRIRnode (with anis_privateflag) that an aggregate points to.Model racks and standalone sites independently of a full location hierarchy
A rack or a site can now be loaded on its own, without pulling in the rest of a country/metro/site hierarchy.
What changed:
extensions/rack(new) splitsLocationRackout ofextensions/location_minimalinto its own extension, and relates it to a site through an explicitsite/racksrelationship rather than hierarchical nesting. It addsstatus,serial_number, andasset_tagattributes that the rack did not have before.extensions/location_site(new) adds a standaloneLocationSitenode (facility, physical address, timezone, status) with no region or country tier above it. It defines the sameLocation.Sitenode asextensions/location_minimal, so load one or the other, not both.extensions/location_minimal's hierarchy changes fromCountry → Metro → Site → RacktoRegion → Country → Site(theMetrotier is removed, aRegiontier is added aboveCountry, andRackmoves to its own extension as described above).Scope tenancy to devices, prefixes, and hosted locations
A tenant can now own the devices, address space, and locations it's responsible for directly, instead of only circuits and buildings.
What changed:
extensions/tenancy(promoted fromexperimental/tenancy, and rebuilt) wires aTenantnode toDcimGenericDevice(devices),IpamPrefix(prefixes),IpamIPAddress(addresses), andLocationHosting(locations). It no longer depends onextensions/circuit; extending tenancy onto circuits or other optional nodes is documented as a pattern you apply yourself intenancy.yml.Align VRFs, interfaces, and VLANs with industry standards
Several base and extension schemas are retyped or restructured to match industry standards, and to fix values that couldn't previously be expressed at all.
What changed:
IpamVRF.import_rtandexport_rtchange fromcardinality: onetocardinality: many, so a VRF can import or export more than one route target — the correspondingIpamRouteTargetrelationship splits intoimport_vrf/export_vrf.DcimInterface's defaultmtuof1514(the full Ethernet frame size) is removed from the shared generic;InterfacePhysicalnow sets a default of1500(the IP payload size) directly.InterfaceVirtualno longer carries anmtuattribute.DcimInterface.roledrops thelagchoice (LAG is already modeled byInterfaceLag) and renamescusttocustomer.DcimInterface.statusdropsdeletedandoutageand is now mandatory.qinq_roleDropdown patched onto the genericIpamVLAN(extensions/qinq/qinq.yaml) to dedicatedIpamSVLANandIpamCVLANnode types (extensions/qinq/qinq.yml— note the corrected.ymlextension) built on a newGenericVLANbase. ACVLAN's name is computed from its parentSVLANand VLAN ID.IpamL2Domainis replaced byIpamVLANGroup, scoped to a location the same wayextensions/aggregatescopes to an RIR.extensions/topology's schema is removed;experimental/topologyremains the only topology model in the library.SnmpCommunityV2.access,SnmpCommunityV3.auth_protocol/privacy_protocol) andBGPSession.session_typechange fromkind: Textwith anenum:list tokind: Dropdown, and their stored values change case (e.g.EXTERNAL→external,Read-Only→read_only).Bug fixes
RoutingBGPSession.import_routing_policies/export_routing_policiespointed to the genericRoutingPolicypeer and reused the same relationship identifiers (bgp__import_policies/bgp__export_policies) as the equivalent relationships onRoutingBGPPeerGroup. Both now point toRoutingPolicyBGPand use distinct identifiers, so a session's routing policies no longer risk being conflated with a peer group's.IpamIPAddress.interfaceandInterfaceLayer3.ip_addressesare now linked with a matchingidentifier, so the two sides resolve as one relationship instead of being treated as unrelated.Minor changes
Documentation
docs/docs/reference/*.mdx) for every base and extension schema to reflect the v2.0 model, and added dedicated reference pages foraggregate,device_module,device_psu_module,location_site, andrack.docs.infrahub.appor OpsMill for architectural review.Developer Experience
load_demo_ipam_dciminvoke task that loads the base schema plus a curated set of extensions (aggregate, cable, circuit, compute, cluster, hosting_cluster, lag, location_minimal, vlan, qinq, rack, vrf) for a demo environment.Reliability
extensions/firewall_policermoves toexperimental/firewall_policer.Full changelog
Added
extensions/aggregate—AggregateandRIRnodes for top-level IP address space. (#75)extensions/device_module—DcimModuleBay,Module, andModuleTypenodes, replacingextensions/modules. (#75)extensions/device_psu_module— PSU-specific module type withwattageandhotswappable. (#75)extensions/rack— standaloneLocationRack, split out ofextensions/location_minimal. (#75)extensions/location_site— standaloneLocationSitewith no region/country hierarchy. (#75)extensions/tenancy— promoted fromexperimental/tenancy, wired to devices, prefixes, addresses, and hosted locations. (#75)DcimCircuit.commit_rate,DcimCircuitEndpoint.side,IpamVRF.enforce_unique,IpamIPAddress.status/role,SnmpCommunity.devices,SnmpClient.ip_address. (#75)tasks/schemas.py:load_demo_ipam_dciminvoke task. (#75)Changed
IpamVRF.import_rt/export_rt—cardinality: onetomany. (#75)DcimInterface.mtu— default1514to1500, moved from the shared generic toInterfacePhysicalonly. (#75)DcimInterface.role/statuschoices reduced and renamed (cust→customer,lagremoved;deleted/outageremoved). (#75)DcimDevice.statuschoices —drainedremoved,reserved/deprecatedadded. (#75)extensions/qinq— rebuilt aroundIpamSVLAN/IpamCVLAN; file renamedqinq.yaml→qinq.yml. (#75)IpamL2Domain→IpamVLANGroup, scoped via the newIpamVLANGroupScopemixin. (#75)extensions/location_minimalhierarchy —Country → Metro → Site → RacktoRegion → Country → Site;Site.facility_id→facility. (#75)DcimCircuit.circuit_typechoices —upstreamreplaced byinternet_access,point_to_pointadded. (#75)DcimCircuitEndpoint.name— now a computed attribute (circuit ID + side), replacing free text. (#75)VRRPGroup.group→vrid(TexttoNumber);VRRPGroup.ip_address→ip_addresses. (#75)hosting_cluster'scluster_type→technology; cloud-specific choices (aws,gcp) replaced bypublic_cloud. (#75)BGPSession.session_type, SNMP community/client enums —Text+enumtoDropdown, values changed to lowercase/snake_case. (#75)extensions/firewall_policermoved toexperimental/firewall_policer. (#75)extensions/topologyschema removed (superseded byexperimental/topology). (#75)LocationGeneric/LocationHosting—shortnameremoved;human_friendly_idswitches toname. (#75)IpamPrefix—organization,location,gatewayrelationships removed in favor of the newscoperelationship (IpamPrefixScope);rolechoices fully replaced ({loopback, management, public, server, supernet, technical, loopback-vtep}→{management, link, customer, supernet, backbone}). (#75)Fixed
DcimCircuit.enpoints→endpointstypo. (#75)DcimInterface.rolecust→customertypo/abbreviation. (#75)RoutingBGPSessionrouting-policy relationships retyped toRoutingPolicyBGPwith distinct identifiers, no longer colliding withRoutingBGPPeerGroup's. (#75)IpamIPAddress.interface/InterfaceLayer3.ip_addressesgiven a matchingidentifierso both sides resolve as one relationship. (#75)