docs: Add AI SDLC / SpecKit tooling evaluation comparison matrix (Iss…

[aryansingh0012]

Add AI SDLC tooling comparison matrix for Issue #3115

[github-actions]

The created documentation from the pull request is available at: docu-html

[masc2023]

Check existing DRs, and template, this DR does not follow the required rules, please update
https://eclipse-score.github.io/score/main/design_decisions/index.html

https://eclipse-score.github.io/process_description//main/folder_templates/platform/docs/change/decision_record.html

[FScholPer]

Naming Convention: https://github.com/eclipse-score/score/blob/main/docs/design_decisions/index.rst

[aryansingh0012]

Added two new decision records under design_decisions and verified the docs build in WSL with bazel run //:docs.

[dirkmacke]

Do I understand DR-10 correctly, that the decision (or recommendation) is based solely on an agent's summary of the tools' repos and no real-world experience with these tools on the S-CORE codebase, or on how they scale with a rather large community and high-frequency changes like S-CORE? I might be misinterpreting the "evidence" section, though, hence the question.

[praveen-ltts]

Do I understand DR-10 correctly, that the decision (or recommendation) is based solely on an agent's summary of the tools' repos and no real-world experience with these tools on the S-CORE codebase, or on how they scale with a rather large community and high-frequency changes like S-CORE? I might be misinterpreting the "evidence" section, though, hence the question.

@dirkmacke , a good study has been done on these tools to be part of SCORE ecosystem and evaluated which of them is more suitable. Also, we have created an EPIC to add these tools in SCORE ecosystem.

@FScholPer , please add more information if I missed something.

[dirkmacke]

@dirkmacke , a good study has been done on these tools to be part of SCORE ecosystem and evaluated which of them is more suitable. Also, we have created an EPIC to add these tools in SCORE ecosystem.

No doubts about that, but did you use/test-drive them, too?

I did spent quite some time with test-driving Openspec (BTW, I was surprised not to see it on the list) and did some evaluation on BMAD as well. Goal was a PoC for series SW development - no safety requirements/pure PoC, but we wanted to see, how SDD scales for concurrent development (multiple devs/agents working in parallel on a design base). Our findings in a nutshell: It does not plus getting Openspec to adhere even to its own very simple SDLC process requires constant supervision and/or supporting tooling. Both in plain scripting and agent/skill plumbing.

Key reasons for this: All the SDD tools do not forsee loopbacks within an ongoing "feature" (or "change", as Openspec calls it), so after you have your tasks, the agent implements. And iterates, And iterates, And iterates. If you still have implemented what was originally spec'd, you'll need to check, since the process/tool will not loop back to check if all the changes done/encountered during implementation and testing are still in line with what was initially spec'd. You'll also not be looped back to spec should missing requirements, e.g. for non-functional testing, being discovered in a later phase of the process. You'll have to notice that and force them. At one point I had an agent so unhappy with doing that, that his recommendation was to re-write Openspec in order not to get the spec fixed and - of course - to not start the whole process again (check that plan & tasks are in line, check that implementation is still in line with the updated tasks etc.) which his own skills should have forced him to do - but they didn't get used, because at least in Openspec there's no plumbing in the agents for that.

I have still very limited understanding of the sphinx/sphinx-needs toolchain, but just using Openspec or Speckit most likely won't work as intended, since you'll basically need to fork the framework to replace the default process and artifacts with the S-CORE ones. Otherwise you'll end up with a lot of additional i.e. Speckit artifacts just to create the S-CORE artifacts, like DRs or test specs etc. to ultimately at some point create some code (and you'll have some unaligned artifacts for this, too).

Maybe this is what is intended (=writing a "S-CORE Speckit") and I simply didn't read it out of the DRs and the associated PR.

Thanks for reading until here anyways 😉

[qor-lb]

I am missing the context/motivation this DR is based on. What problem do we try to solve here?

[aryansingh0012]

Thanks for the feedback. The motivation behind this DR is:
Evaluate existing AI SDLC frameworks for their suitability within the S-CORE project.
Identify a workflow that supports end-to-end traceability (Requirements → Specifications → ADRs → Code → Tests → Documentation).
Ensure the workflow is compatible with S-CORE engineering goals, including ASPICE alignment, governance, reproducibility, and long-term maintainability.
Compare the strengths and limitations of candidate frameworks (SpecKit, Syspilot, BMAD, and Pharaoh).

[qor-lb]

@aryansingh0012 this is a task description but not a motivation. Why do we need this evaluation at all and what problems do we try to address with this DR? E.g. Identify a workflow that supports end-to-end traceability sounds like the current process description/development workflow is not sufficient. Is this actually the case? And if so, why not?

In plain what I am looking for: what does this DR intend to decide and why do we need it?

After that we can start discussing the impact of the intended change. @dirkmacke concerns are also very valid and should be picked up IMO.

[FScholPer]

I think the concerns are valid. My proposal would be to generate the artifacts / rules for the agents out of our process meta model. Lets discuss that in detail in the next meeting

[dirkmacke]

@FScholPer I think this is the right approach if the intent of this exercise is to create something like "reference agents" to run the S-CORE dev process. Emphasized "if", because this would IMO not match/fit to what's stated in the DR this PR is based upon.

In what meeting is this being discussed?

[FScholPer]

@dirkmacke Thanks for the feedback. We went now away from doubling the meta model and artifacts towards a cached sidecar graph: https://github.com/eclipse-score/mcp-servers/blob/e116ad1ec15184e3d413d1a5818a55427cbba343/packages/metamodel-flow/model/process_graph.json. I think would be helpful to get some feedback on this

[aryansingh0012]

Hi @qor-lb @dirkmacke

Thanks, this is a fair point. Let me clarify the motivation behind this DR.

The actual motivation is not that S-CORE lacks a development process today. Rather, we currently do not have a defined approach for integrating AI-assisted SDLC workflows into the existing engineering process while maintaining traceability, governance, reviewability, and long-term project ownership.

The question this DR is trying to answer is:

Should S-CORE adopt an existing AI SDLC framework, depend on multiple external frameworks, or build and own a solution that integrates only the capabilities required by S-CORE?

This evaluation was initiated because the available frameworks each solve only part of the problem:

• SpecKit provides a strong specification-first workflow, but lacks native Sphinx-Needs traceability, change-impact analysis, and ASPICE-oriented artifact relationships.
• Syspilot demonstrates requirement-driven traceability and change-impact analysis, but is currently positioned as a research-stage project.
• Pharaoh contains many concepts relevant for ASPICE-style traceability, lifecycle governance, and impact analysis, but the project has been archived.
• BMAD focuses on agent collaboration and productivity workflows rather than governed, traceable engineering artifacts.

As we discussed in the AI recurring meeting the practical outcome of this evaluation is already influencing ongoing implementation work. Rather than adopting any framework wholesale, we are currently prototyping an S-CORE-owned SDLC Harness in eclipse-score/mcp-servers.

The objective is to combine:

• SpecKit for specification-first workflows.
• Sphinx-Needs as the traceability source of truth.
• Concepts from Syspilot for requirement-driven traceability and change-impact analysis.
• Concepts from Pharaoh for ASPICE-oriented lifecycle governance and traceability.
• MCP-based automation implemented and maintained by S-CORE.

We are currently implementing MCP tools that provide:

• Requirement validation and normalization.
• SpecKit → Sphinx-Needs conversion.
• Traceability queries across lifecycle artifacts.
• Implementation evidence collection.
• Progress tracking and reporting.
• Lifecycle loopback tracking.
• Change-impact analysis.
• Requirement → Architecture → Task → Code → Test → Evidence traceability.

The target workflow is:

flowchart TD
    A[SpecKit Requirements]
    B[Sphinx-Needs Requirements]
    C[Architecture / ADRs]
    D[Tasks]
    E[Code]
    F[Tests]
    G[Implementation Evidence]
    H[Documentation]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H

    B -. Traceability .-> G
    B -. Impact Analysis .-> C
    B -. Impact Analysis .-> D
    B -. Impact Analysis .-> E
    B -. Impact Analysis .-> F

Loading

[FScholPer]

I think the concerns are valid. My proposal would be to generate the artifacts / rules for the agents out of our process meta model. Lets discuss that in detail in the next meeting

[dirkmacke]

@FScholPer , @aryansingh0012 We have the dev/design process including associated artifacts as an integral part of S-CORE. That's basically what Speckit or Openspec would provide, if you don't have any process. The additional plumbing provided by these frameworks is what's to my understanding is already in place via sphinx-needs. The remaining bullet points are IMO covered by the git rules amd Github actions in place.

So if the intent is to provide "reference" agents and/or skills to work within the existing process and tool framework, I'd propose to do just that. An initial discussion could be on the needed granularity of the agents in light of the rather large artifacts they'll have to deal with. Do we want an "S-CORE Requirements Engineer" agent with full knowledge of the process, potentially eating through tokens and its context windows even for simple requests, or would we want to split that into an "S-CORE Process Engineer" being able to defer to (to be indentified) skills depending on process step and/or artifact and "S-CORE Requirements Engineer" agent, so that context is kept manageable? And so on.

This can be done, although keeping the agents on the process will be a challenge. However, I think the key question is: Would we still be able to qualify for ASIL-B, if this gets part of the S-CORE toolchain?? Because if there is not a clear "yes", I think one should be very cautious to make this canon!

[masc2023]

@FScholPer , @aryansingh0012 We have the dev/design process including associated artifacts as an integral part of S-CORE. That's basically what Speckit or Openspec would provide, if you don't have any process. The additional plumbing provided by these frameworks is what's to my understanding is already in place via sphinx-needs. The remaining bullet points are IMO covered by the git rules amd Github actions in place.

So if the intent is to provide "reference" agents and/or skills to work within the existing process and tool framework, I'd propose to do just that. An initial discussion could be on the needed granularity of the agents in light of the rather large artifacts they'll have to deal with. Do we want an "S-CORE Requirements Engineer" agent with full knowledge of the process, potentially eating through tokens and its context windows even for simple requests, or would we want to split that into an "S-CORE Process Engineer" being able to defer to (to be indentified) skills depending on process step and/or artifact and "S-CORE Requirements Engineer" agent, so that context is kept manageable? And so on.

This can be done, although keeping the agents on the process will be a challenge. However, I think the key question is: Would we still be able to qualify for ASIL-B, if this gets part of the S-CORE toolchain?? Because if there is not a clear "yes", I think one should be very cautious to make this canon!

Idea from safety perspective, keep agents only on a workflow (one process step), and let humans always sign-off.

[dirkmacke]

@masc2023 Yes, having a human always sign everything off is always the plan... 😉 While I can comprehend formally signing off something your trusted team has come up with, would you sign off something a plethora of agents has created? Would you review the agents' definition first? Probably you would want to review the artifacts, before signing off... How many people are there, which will create S-CORE artifacts with agents (especially if these are "provided") and how many people are there capable (competence, capacity) reviewing this?

Agentic coding is here to stay and IMO the only way to deal with it is adopting SDD and handling the requirements "as code". Code becomes a black box and with a TDD approach, only code would be generated for which a requirement and an associated test case does exist. So we'd have (theoretically) no code without requirement and if all of our TCs pass, we'd (theoretically) have prove, that all requirements are implemented correctly.

This obviously means, that requirements must be complete, including all the non-functional requirements SW devs have usually in their heads and usually don't write down. Hence LLMs don't know much about these...! TARAs and HARAs would need to generate/result in explicit requirements, too.

But if this is done, I think code could be treated in many (most?) cases as black box, solving the issue in the 1st paragraph. It's a big "if", though.

[MaximilianSoerenPollak]

But if this is done, I think code could be treated in many (most?) cases as black box, solving the issue in the 1st paragraph. It's a big "if", though.

Treating code as a black box in my opinion unless it's a one time use script or something that doesn't matter if it fails or breaks is a very very dangerous precedent to be set.

If I can no longer understand the code that it produces then what exactly am I reviewing, the text that it says with it? It can say anything. Talk is cheap, let me see the code (though with generation that is now also cheap).
If one day I have to go in and extend this code or fix it / debug it, and the only way I can understand it is via another clanker/AI, then I have lost and I could just as well auto approve everything without caring.

But that is just my opinion.

[masc2023]

But if this is done, I think code could be treated in many (most?) cases as black box, solving the issue in the 1st paragraph. It's a big "if", though.

Treating code as a black box in my opinion unless it's a one time use script or something that doesn't matter if it fails or breaks is a very very dangerous precedent to be set.

If I can no longer understand the code that it produces then what exactly am I reviewing, the text that it says with it? It can say anything. Talk is cheap, let me see the code (though with generation that is now also cheap). If one day I have to go in and extend this code or fix it / debug it, and the only way I can understand it is via another clanker/AI, then I have lost and I could just as well auto approve everything without caring.

But that is just my opinion.

Agree, as we threat everything as code, also requirements, it must be understood by the people, who sign-it off, same for source code, no black-box

[dirkmacke]

I don't think, that "not understanding code" and "not having read every line of code" is the same thing.

It's also not my professional experience, that people signing off a product have read (or even understood) every line of code or even every requirement. They usually trust their teams and if they say "it's OK", usually backed-up by testing evidence, they sign off.

I don't see much of a difference here, especially if there is systematic proof, that implementation is according to specs. I'd definetly not say these days, that just because a human wrote the code it's per default better or more reliable.

Back to FuSa: What is it, what is to be demonstrated at the end of the day: That a feature works according to specification, usually proven by evidence from test cases and a breadcrumb trail from requirement to test, or that the code is written in any specific way (and yes, "specific" in a way that memory protection, FFI etc. is secured of course - but that can/must be formulated as requirements anyways; and mybe that part of the implementation is then to be reviewied by a human (if not anyway this is handled by using some already qualified software component)).