Skip to content

Add LoRA analysis: SVD spectrum, effective rank, Frobenius norms#1

Closed
harelc wants to merge 4 commits into
matanby:mainfrom
harelc:lora_analysis
Closed

Add LoRA analysis: SVD spectrum, effective rank, Frobenius norms#1
harelc wants to merge 4 commits into
matanby:mainfrom
harelc:lora_analysis

Conversation

@harelc
Copy link
Copy Markdown

@harelc harelc commented Mar 4, 2026

  • Add LoRA pair detection (lora.py) with support for common naming conventions (lora_A/B, lora_down/up, down/up)
  • Add tensor data loading and SVD computation (data.py) with QR-based optimization to avoid full-matrix SVD
  • Add LoraScreen with DataTable listing all LoRA pairs, columns for rank, ||A||, ||B|| Frobenius norms, and effective (stable) rank
  • Add SvdScreen with full-width vertical bar histogram (sqrt scale) showing singular value spectrum with effective rank statistics
  • Add LoraHelpScreen (?) explaining stable rank, Frobenius norm, and SVD spectrum metrics
  • Context-aware L key: filters pairs to current view, jumps directly to SVD when only one pair matches
  • Async stats computation with 4-thread pool: norms first (cheap), then stable rank (SVD-based), cells update progressively
  • Export (on user request) and load (automatically) the LoRA statistics analysis as it's computationally heavy.
Screenshot 2026-03-04 at 10 52 44 Screenshot 2026-03-04 at 10 52 53 Screenshot 2026-03-04 at 11 08 02

- LoRA pair detection with support for common naming conventions
- Tensor data loading and SVD via QR-based optimization on small rank×rank matrix
- LoraScreen: full-screen DataTable with rank, ||A||/||B|| norms, effective rank
- SvdScreen: vertical bar histogram (√ scale) with rank statistics
- LoraHelpScreen (?) explains stable rank, Frobenius norm, SVD spectrum
- Context-aware L key: filters to visible pairs, jumps to SVD for single match
- Async 4-thread pool for progressive stats computation, cached across screen opens
- Load/export analysis as JSON (e key); auto-loads on startup if present
- Clean thread shutdown on quit and screen dismiss
- numpy is a required dependency

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@harelc harelc force-pushed the lora_analysis branch 6 times, most recently from 9409d05 to 94579b9 Compare March 4, 2026 11:16
- Add CompactifyScreen to TUI: LoRA screen -> 'c' -> enter target rank
  to truncate all pairs via SVD and save a smaller safetensors file
- Add truncate_lora_pair(), load_all_tensors(), write_safetensors() to data.py
- Fix critical SVD bug in compute_lora_svd(): QR of b.T had wrong dimensions,
  silently fell back to full (4096x4096) matrix SVD instead of (64x64).
  Now uses qr(b) correctly — ~1000x faster effective rank computation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
harelc and others added 2 commits March 6, 2026 15:07
- Replace read-only MetadataScreen with interactive editor:
  field-by-field view, edit (e), add (a), delete (d), save (Ctrl+S)
- Add ScaleScreen for scaling LoRA pairs by alpha coefficient
  (accessible via 'a' key from LoRA analysis view)
- Add scale_lora_a() and rewrite_with_metadata() to data.py
- Load sigma0 from cached JSON on startup

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add interactive metadata editor and LoRA scaling
@matanby
Copy link
Copy Markdown
Owner

matanby commented May 18, 2026

Hi @harelc — thanks a lot for putting this together, the QR-SVD trick and the per-pair stats screen were both great ideas. I ended up reimplementing the core pieces against the new multi-command structure rather than merging the branch directly, but most of the work is now in v0.2.0:

What landed (inspired by this PR):

  • QR-accelerated SVDsrc/sft/ops/lora/svd.py, the same qa, ra = qr(a.T); qb, rb = qr(b); svd(rb @ ra) trick. Used by sft lora svd, sft lora resize, and sft lora stack --target-rank.
  • LoRA Mode in the TUI (L key) — LoraModeScreen with per-pair rank / ||A|| / ||B|| / effective rank columns; Enter on a row opens the singular-value spectrum modal (SvdSpectrumScreen).
  • Async stats worker — norms fill in first, stable rank is computed in the background, cells update progressively.
  • PEFT + Kohya pair detection, plus a sft lora convert command to move between them.

Not (yet) backported, in case you want to follow up:

  • Bare down/up naming (SDXL-style without lora_ prefix) — detect.py currently recognizes PEFT (lora_A/lora_B) and Kohya (lora_down/lora_up); the third convention from your PR isn't covered.
  • Dedicated ? help modal explaining stable rank / Frobenius norm / SVD — we currently surface those definitions in command --help text and the Compress prompt, not as a modal inside LoRA Mode.
  • Persisted analysis cache (your export/auto-load idea) — with the QR-SVD path the recompute is fast enough that we skipped this, but it would still be nice for very large adapters.
  • Context-aware L filtering to the current viewL always opens the full LoRA list right now.

Closing this PR since the core features are in v0.2.0; happy to take a follow-up PR for any of the gaps above. Thanks again!

@matanby matanby closed this May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants