TASK: SOTA Research Initiative — Priority Matrix & Repository Impact Analysis

[carochacs]

Overview

This task provides the research prioritization framework and detailed impact analysis for the FeedBack SOTA tool review across all 19 repositories.

Related: feedback#X - Complete Methodology

---

High-Priority Research Areas (Significant SOTA Gap)

1. 🔴 Pitch Extraction: pYIN → CREPE

Repository: feedback-plugin-lyrics-karaoke
Issue: feedback-plugin-lyrics-karaoke#2
Current Tool: librosa.pyin (Probabilistic YIN, 2013)
SOTA Alternatives: CREPE (neural), SPICE (Google), Ensemble voting
Impact If Adopted:

• Eliminates octave-error heuristics (implicit in CREPE)
• Better handling of vibrato, extended range, poor audio quality
• Downstream benefit: cleaner karaoke pitch tracking, more accurate lyrics sync

Dependencies:

• Stem quality (demucs v3/v4) — see Research: Evaluate demucs v4 and pyguitarpro GP8 support #4
• Lyrics alignment accuracy — see Preserve Guitar Pro strum direction #2 (lyrics-sync)

Effort Estimate:

• Phase 1–2: 8–12 hours (PoC)
• Phase 3: 16–24 hours (benchmark MIR-1K dataset, listening tests)
• Phase 4: 4–6 hours (decision + implementation planning)
• Total: 28–42 hours

Risk: Medium (already have torchcrepe as dependency; GPU latency trade-off)

---

2. 🔴 Stem Separation: demucs v3 → v4

Repository: feedback (main)
Issue: feedback#4
Current Tool: demucs v3 (Facebook Meta)
SOTA Alternatives: demucs v4 (improved models), Roformer (already in use, competitive)
Impact If Adopted:

• Better vocal isolation → cleaner for pitch extraction (karaoke)
• Better vocal isolation → cleaner for lyrics transcription (WhisperX)
• Better drum/bass isolation → benefit chord detection, editor tools
• Foundational for entire karaoke + editor pipeline

Downstream Benefits:

• Improves quality of #2 - lyrics-karaoke
• Improves quality of #2 - lyrics-sync
• Improves quality of #3 - chord-detector

Effort Estimate:

• Phase 1–2: 6–10 hours (PoC, model availability check)
• Phase 3: 20–28 hours (benchmark MUSDB18, SDR calculations)
• Phase 4: 4–6 hours (decision + integration planning)
• Total: 30–44 hours

Risk: Medium (model API changes; potential latency regression)

---

3. 🔴 Lyrics Alignment: WhisperX → Faster-Whisper

Repository: feedback-plugin-lyrics-sync
Issue: feedback-plugin-lyrics-sync#2
Current Tool: WhisperX v3.8.0+ (Whisper + forced alignment)
SOTA Alternatives: Faster-Whisper (OpenAI optimized), Distil-Whisper (smaller models)
Impact If Adopted:

• 2–3× latency reduction (CPU inference)
• Drop-in replacement (no architecture change)
• Better accessibility for edge/embedded deployments

Dependencies:

• Stem quality (demucs v3/v4) — see Research: Evaluate demucs v4 and pyguitarpro GP8 support #4
• Pitch extraction quality — see Preserve Guitar Pro strum direction #2 (lyrics-karaoke) for pipeline validation

Effort Estimate:

• Phase 1–2: 6–8 hours (PoC, trivial integration)
• Phase 3: 12–16 hours (benchmark Common Voice singing subset, WER measurement)
• Phase 4: 3–4 hours (decision, minimal implementation effort if adopted)
• Total: 21–28 hours

Risk: Low (drop-in replacement; high confidence in adoption)

---

4. 🔴 Chord Detection: Design Decision (Not Yet Implemented)

Repository: feedback-plugin-chord-detector (template)
Issue: feedback-plugin-chord-detector#3
Current State: Template only; no implementation yet
SOTA Approaches:

• Chroma-based (fast, interpretable)
• Deep Learning (CNN/RNN, high accuracy)
• Essentia (pre-trained models)
• Hybrid ensemble

Impact If Implemented:

• New plugin for editor + music theory analysis
• Foundational for future harmonic analysis features
• Depends on high-quality stem separation (see Research: Evaluate demucs v4 and pyguitarpro GP8 support #4)

Effort Estimate:

• Phase 1–2: 8–12 hours (literature review, PoC both approaches)
• Phase 3: 20–28 hours (benchmark Isophonics dataset, accuracy measurements)
• Phase 4: 4–6 hours (design decision, architecture planning)
• Total: 32–46 hours (+ implementation phase if approved)

Risk: Medium (design choice affects long-term architecture)

---

5. 🔴 Adaptive Difficulty: EMA → Kalman/IRT

Repository: feedback-plugin-difficulty-ladder
Issue: feedback-plugin-difficulty-ladder#31
Current Tool: EMA + fixed thresholds (simple, effective but limited)
SOTA Alternatives:

• Kalman filtering (state-space model for skill tracking)
• IRT (Item Response Theory, psychometric)
• Flow-state targeting (Csikszentmihalyi)
• Reinforcement Learning (long-term optimization)

Impact If Adopted:

• Better player engagement + retention
• Per-player skill models (enables leaderboards, adaptive curriculum)
• Flow-state targeting (evidence-based difficulty selection)
• Downstream benefit: player satisfaction, practice effectiveness

Effort Estimate:

• Phase 1–2: 8–12 hours (literature review, psychological research, PoC Kalman)
• Phase 3: 20–30 hours (A/B testing with real players, retention measurement)
• Phase 4: 4–6 hours (decision, implementation planning)
• Total: 32–48 hours (+ longitudinal player study if pursuing)

Risk: Medium-High (requires player data for validation; behavioral change)

---

Medium-Priority Reviews (Working Well, Incremental Improvements)

🟡 Stem Separation Pipeline (Roformer, demucs-server)

Repository: feedback-demucs-server
Issue: feedback#4 (see also demucs-server-specific review if needed)
Current State: Roformer is competitive; no urgent change
Action: Monitor demucs v4 improvements; evaluate if SDR gain > 1 dB

---

🟡 Lyrics Transcription (WhisperX Version Tracking)

Repository: feedback-plugin-lyrics-sync
Action: Monitor OpenAI Whisper v3 release; compare to WhisperX on singing voice

---

Low-Priority (Already Optimal)

🟢 Web Audio API (Metronome)

Repository: feedback-plugin-metronome
Status: Web Audio API is solved problem; no SOTA review needed

🟢 Frontend Stack (Tailwind, Canvas, Three.js)

Repositories: All plugins
Status: Industry-standard, well-maintained; no SOTA review needed

---

Effort & Timeline Summary

Research Area	Hours (Low–High)	Timeline	Owner	Status
Pitch extraction (CREPE)	28–42h	2–3 weeks	[Assign]	Not started
Stem separation (demucs v4)	30–44h	2–3 weeks	[Assign]	Not started
Lyrics alignment (Faster-Whisper)	21–28h	1.5–2 weeks	[Assign]	Not started
Chord detection (design)	32–46h	2–3 weeks	[Assign]	Not started
Difficulty (Kalman/IRT)	32–48h	2–3 weeks	[Assign]	Not started
TOTAL	143–208 hours	7–12 weeks	—	—

Resource allocation recommendation: Assign 2–3 researchers in parallel; stagger start dates to avoid bottlenecks on shared benchmarks (MUSDB18 for stems).

---

Repository Impact Map

Stem Separation (demucs v4) — FOUNDATIONAL
├─→ Pitch extraction quality (CREPE)
├─→ Lyrics alignment quality (Faster-Whisper)
└─→ Chord detection quality (any approach)

Lyrics Alignment (Faster-Whisper) — PERFORMANCE
└─→ User experience (latency reduction)

Pitch Extraction (CREPE) — QUALITY
└─→ Karaoke accuracy, lyrics sync

Chord Detection (design) — NEW FEATURE
└─→ Editor, music theory analysis

Adaptive Difficulty (Kalman) — ENGAGEMENT
└─→ Player retention, practice effectiveness

Recommendation: Start with #4 (stem separation) as a dependency for #2, #3, and #5. Parallel track #1 (pitch) and #2 (lyrics) once stems are benchmarked.

---

How to Use This Guide

1. Assign researchers to each high-priority area (5 researchers ideal)
2. Use effort estimates to plan sprints and deadlines
3. Reference repository impact map when deciding execution order
4. Track Phase completion via linked research issues
5. Share findings in GitHub issue comments as they emerge

---

Questions? Dependencies?

• Clarify dependencies between research areas? Comment below.
• Need access to specific benchmark datasets? We can coordinate.
• Researcher unavailable? Comment and we'll replan.