Skip to content

Implement SCORE Tool Management workflow - #831

Draft
AlexanderLanin wants to merge 1 commit into
eclipse-score:mainfrom
etas-contrib:tool-verification-report
Draft

AlexanderLanin wants to merge 1 commit into
eclipse-score:mainfrom
etas-contrib:tool-verification-report

Conversation

@AlexanderLanin

@AlexanderLanin AlexanderLanin commented Sep 15, 2026

Copy link
Copy Markdown
Member

Why

The previous PR described tool use cases and malfunctions as a documentation proof of concept, but it did not execute the SCORE Tool Management workflow. In particular, report ownership, conditional malfunction classification, derived safety/TCL results, lifecycle status, qualification completeness, and generated evidence views were not validated consistently. This made it possible for the written report, the graph, and the qualification result to diverge.

The PR now makes the Sphinx-Needs representation an executable model of the workflow while preserving the distinction between evaluation/classification and qualification.

What changed

Metamodel

  • tool_usecase is the explicit usage-context element.
  • Every tool_usecase has explicit ownership by a doc_tool through belongs_to and remains linked to its tool_req needs through realized_by.
  • potential_tool_malfunction requires safety_affected.
  • detection_sufficient is conditional and safety_measures remains free text.
  • No separate qualification-test or safety-measure Need type was introduced.

Executable validation

  • Safety-relevant malfunctions require detection_sufficient.
  • Positive detection claims require non-empty safety_measures.
  • Non-safety malfunctions omit detection_sufficient; meaningless values are rejected.
  • LOW-confidence malfunctions must violate at least one tool_req.
  • TVR safety relevance and TCL are derived from the owned use-case/malfunction graph and checked against the stored SCORE fields.
  • draft remains permissive, while evaluated, qualified, and released enforce the relevant workflow completeness.
  • LOW qualification requires passed testcase evidence with fully_verifies for every relevant tool_req. Partial verification alone is not sufficient.
  • Qualification does not change detection_sufficient or raise TCL from LOW to HIGH.

Generated report

The qualification report template now follows:

doc_tool
  -> belongs_to backlink -> tool_usecase
  -> parent_needs backlink -> potential_tool_malfunction
  -> violates -> tool_req
  -> fully_verifies / partially_verifies -> testcase

It generates the evaluation table, derived safety/confidence summary, qualification matrix, and testcase evidence view from the structured graph.

Documentation and migration

  • perform_tool_verification.md is aligned with the implemented workflow.
  • classification.rst documents the evaluation model, conditional malfunction data, and derived result.
  • qualification.rst documents qualification evidence, testcase interpretation, and lifecycle semantics.
  • The requirements index now explains the relationship between classification and qualification.
  • Existing tool use cases were migrated to explicit belongs_to ownership.
  • Classification and qualification remain separate pages because the HowTo defines them as two distinct analysis questions and views of one TVR workflow.

Regression coverage

The PR adds unit and RST/Sphinx integration coverage for conditional malfunction rules, ownership, graph-derived safety/TCL, LOW requirement targets, qualification evidence, status progression, mixed use cases, and the rule that qualification does not reclassify the tool.

Design decisions and migration

  • detection_sufficient on a non-safety malfunction is rejected as meaningless data.
  • A passed testcase must fully verify a tool requirement for qualification; partial evidence remains visible but does not complete qualification.
  • Existing reports that use the structured model are migrated. Reports in evaluated state can remain LOW until their qualification evidence is complete; no filename-specific migration exception was added.
  • Classification and qualification are intentionally not merged: they are separate views of the same Tool Verification Report workflow.

Validation

  • bazel test //...
  • bazel run //:docs_check
  • bazel run //:docs
  • Commit hooks including Ruff, BasedPyright, and copyright checks

@AlexanderLanin AlexanderLanin changed the title feat: add tool qualification verification report model POC: add tool qualification verification report model Sep 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Documentation preview for this pull request is available at:
pr-831: https://eclipse-score.github.io/docs-as-code/pr-831/

This report evaluates the Docs-as-Code toolchain as a qualified tool for
producing and verifying safety-relevant documentation.

.. tool-usecase:: Build and CI behavior

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the author of the doc_as_code.rst this is derived from, I wonder why there is only two use cases here?

For me, each malfunction is a separate use case and their distinct description is important. Also, reference to the process requirements are important. This is all lost here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

POC scope has ended there :p

:post_template: tool_qualification_report

This report evaluates the Docs-as-Code toolchain as a qualified tool for
producing and verifying safety-relevant documentation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security evaluation is missing but required by S-CORE process.

Comment thread docs/internals/requirements/tool_qualification_example.rst Outdated
:detection_sufficient: NO
:additional_safety_measure_required: YES
:safety_measure: Pull-request review checks changes to safety-critical links before merge.
:violates: tool_req__docs_common_attr_safety_link_check

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No way that this single requirement is sufficient. I expect a list of ten or more requirements for this one. Similar for the other malfunctions.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: dont review the content. this is a POC

@attifunel
attifunel self-requested a review September 16, 2026 13:38
@AlexanderLanin
AlexanderLanin force-pushed the tool-verification-report branch from 2ca13d2 to 47270ce Compare September 17, 2026 06:37
@AlexanderLanin AlexanderLanin changed the title POC: add tool qualification verification report model Implement SCORE Tool Management workflow Sep 22, 2026
@AlexanderLanin
AlexanderLanin force-pushed the tool-verification-report branch from 431a148 to 272653e Compare September 22, 2026 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants