"Add Test Coverage": reports "0 test files found (0%)" on a repo with 89 RSpec files and 1406 examples

[justi]

Summary

The Add Test Coverage recommendation (severity MEDIUM) fires on a Ruby gem with this text:

Only 0 test files found (0%). Consider adding more tests.
Action: Focus on testing critical paths and high-complexity files
Prevents regressions and improves confidence

The repo has 89 spec files and 1406 examples.

Repro: https://codeflow-five.vercel.app/?repo=justi%2Fruby_llm-contract
Source: https://github.com/justi/ruby_llm-contract (MIT)

Measurements

spec/**/*_spec.rb	89 files
All .rb under spec/	90 (89 specs + spec_helper.rb)
Examples, per RSpec itself (rspec --dry-run)	1406 examples, 0 failures
Lines of spec code	21,948
Lines under lib/	7,269
Test-to-code ratio	~3:1

Coverage is also already instrumented. .simplecov at the repo root:

SimpleCov.start do
  enable_coverage :branch
  primary_coverage :branch

  add_filter "/spec/"
  add_filter "/examples/"
  add_filter "/internal/"
  add_filter "/tmp/"

  track_files "lib/**/*.rb"

  if ENV["CI"] == "true" || ENV["SIMPLECOV_STRICT"] == "1"
    minimum_coverage line: 89
    minimum_coverage branch: 75
  end

  command_name "RSpec"
end

So the report advises adding tests to a repo that has 1406 of them, plus branch-coverage instrumentation and coverage floors that engage under CI=true.

Likely cause: the RSpec layout

The repo has no test/ directory and no *_test.rb files. It uses RSpec's convention exclusively:

spec/
  spec_helper.rb              (1 file, not a spec)
  ruby_llm_contract_spec.rb   (1 spec)
  ruby_llm/                   (84 specs)
  integration/                (4 specs)

That is 89 _spec.rb files and 90 .rb files in total.

Discovery is wired the normal way - Rakefile:6 is RSpec::Core::RakeTask.new(:spec) with task default: :spec on line 8, and .rspec:3 sets --require spec_helper.

spec/**/*_spec.rb is RSpec's standard layout - it is what rspec --init generates, and RSpec is one of the two mainstream test frameworks in Ruby alongside Minitest (whose layout is test/**/*_test.rb). I do not know which patterns you match, so I cannot say whether the Minitest layout works and only _spec.rb is missing, or whether something broader is going on.

One alternative I could not rule out: if the default Excludes patterns strip spec/ before analysis, the count would be a consequence of that exclusion rather than a pattern gap. That would be worth catching, because the recommendation would then be computed after removing the exact files it claims are absent - and it would read to the user as "you have no tests" rather than "we didn't look."

This also touches #76

I filed #76 about false positives in the unused functions category on this same repo. Four of the 35 entries there have callers only in spec/:

Reported unused	Callers in lib/	Call sites in spec/
reset_configuration!	none - definition only, contract.rb:19	55 files
stub_step	none - definition only, concerns/stub_helpers.rb:22	18
stub_steps	none - definition only, concerns/stub_helpers.rb:40	7
stub_all_steps	none - definition only, concerns/stub_helpers.rb:63	7

(The three stub helpers total 32 call sites across 3 spec files. In lib/ they appear only in their own definitions and in comments - no invocations.)

If spec/ is not indexed, that single cause produces both this "0 test files" report and those four false positives - one fix, two categories.

To be clear about the limits of that: it does not explain all of #76. run_all_evals is called from rake_task.rb:60, inside lib/, and was still reported unused. So at least one other cause is at work there, independent of this one.

Suggestions

1. Add spec/**/*_spec.rb to the test-file patterns, alongside whatever test/ patterns already exist. Ruby has two standard layouts - RSpec's spec/**/*_spec.rb and Minitest's test/**/*_test.rb - and matching both should cover most Ruby repos.
2. Verify spec/ is being indexed at all, not just for the test count. If it is excluded by default, the "unused functions" category will systematically miss test-only callers - which is exactly what "unused functions": 34 of 35 entries on a Ruby gem are false positives, and the 1 real hit is reported as self #76 shows for helper methods a gem exports for its users' test suites.
3. When the count is zero, say which patterns were tried. "0 test files found" is indistinguishable from "we looked for *.test.js and this is a Ruby repo." Listing the attempted globs would make a miss self-diagnosing.
4. Consider corroborating a zero count before firing a MEDIUM. A zero is ambiguous between "no tests" and "we did not recognise them", and cheap signals can disambiguate. In this repo, Gemfile:9 declares rspec and Gemfile:12 declares simplecov, and .simplecov exists at the root - two independent hints that a suite is expected, both available without parsing a single spec file. (I would not lean on "has CI" as a signal, incidentally: this repo has no CI config at all, and still has 1406 examples.)

Note on scope

I checked this one recommendation on one Ruby repo. I have not looked at codeflow's source, so the cause above is inferred from the output - the numbers on my side are measured, the explanation is not.

Related: #76 ("unused functions"), #77 ("Circular Dependencies"), #78 ("High Complexity Files"), all on the same repo.

[braedonsaunders]

Fixed in #80 (merged). Two causes stacked here, and your issue correctly identified the first:

1. The pattern gap you diagnosed: test detection only matched JS conventions (.test., .spec., __tests__/). Your suggestion 1 is implemented as a shared isTestFile helper covering spec/**/*_spec.rb (RSpec), test/**/*_test.rb (Minitest), test_*.py/*_test.py/conftest.py (pytest), *_test.go, *Test.java/*Spec.scala-style JVM names, and Elixir *_test.exs — used by the coverage suggestion, the dead-function filter, and the architecture-layer classifier. Re-run on your repo: 90 spec files recognized, and the "Add Test Coverage" recommendation no longer fires.

2. The alternative you couldn't rule out — spec/ stripped before analysis — was also (nearly) right, just with a different mechanism than excludes: unauthenticated GitHub access allows 60 requests/hour, each file costs 2, and failed fetches were silently dropped. Your run fetched ~50 of 226 files and never reached spec/ at all. That single cause is also what produced the four spec-only-caller false positives you cross-referenced from "unused functions": 34 of 35 entries on a Ruby gem are false positives, and the 1 real hit is reported as self #76. Partial fetches are now surfaced as a critical "Files Not Fetched — Partial Analysis" issue instead of being invisible (your suggestion 2/3 in spirit — a zero is no longer indistinguishable from "we didn't look").

Your suggestion 4 (corroborate a zero via Gemfile/simplecov signals before firing a MEDIUM) is a nice belt-and-braces idea; with both causes fixed the recommendation is correct on your repo, so we've left that for future hardening.

Thanks for the cross-issue analysis connecting this to #76 — "one fix, two categories" was exactly right for the fetch half.