From 8d9f965f0349689a4a62504958e80af1da84e284 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Sat, 15 Aug 2026 09:57:51 -0700 Subject: [PATCH] docs: add feature reference and fix three bugs Audit the documentation by running every example. Three claims turned out to be false, and each traced to a real bug rather than a wrong sentence. Fix Marshal support in Bayes. marshal_dump left out min_word_length, so a restored classifier raised "comparison of Integer with nil failed" on its first classify. Bayes was the only classifier affected. A dump from an older version still loads and takes the configured default. Fix LSI#highest_relative_content, which ended in a bare .map and so returned an Enumerator instead of the documented array. Fix LSI#highest_ranked_stems, which looked up each weight with arr.index(value). Tied weights all resolved to the same index, so one stem repeated. It now ranks indices directly. Correct the README where it contradicted the code: - The logistic regression example never called fit, so it raised NotFittedError. - The keywords examples scored text before any fit, but keywords ships no pre-trained model, so the first command a reader ran failed. - The incremental LSI example used the default auto_rebuild, under which incremental mode never starts. Each add rebuilds immediately, the index builds from the first two documents, and the next add trips the vocabulary growth threshold and disables the mode for good. Add docs/ covering both commands, each classifier, persistence, streaming, and configuration, and ship it in the gem. Add test/docs/documentation_test.rb, which asserts every published value. rake test now fails when the docs and the code disagree. Add a .claude/skills/audit-docs skill and a static checker for dashes, relative links, link fragments, and documented executables. Track project skills by narrowing the gitignore to .claude/* with an exception. --- .claude/skills/audit-docs/SKILL.md | 87 +++++ .claude/skills/audit-docs/check_docs.rb | 77 +++++ .gitignore | 5 +- CHANGELOG.md | 12 + README.md | 84 +++-- classifier.gemspec | 2 +- docs/README.md | 49 +++ docs/bayes.md | 108 ++++++ docs/cli.md | 144 ++++++++ docs/configuration.md | 100 ++++++ docs/keywords.md | 171 ++++++++++ docs/knn.md | 92 +++++ docs/logistic-regression.md | 107 ++++++ docs/lsi.md | 219 ++++++++++++ docs/persistence.md | 124 +++++++ docs/streaming.md | 104 ++++++ docs/tfidf.md | 137 ++++++++ lib/classifier/bayes.rb | 7 +- lib/classifier/lsi.rb | 6 +- test/bayes/bayesian_test.rb | 31 ++ test/docs/documentation_test.rb | 437 ++++++++++++++++++++++++ test/lsi/lsi_test.rb | 27 ++ 22 files changed, 2091 insertions(+), 39 deletions(-) create mode 100644 .claude/skills/audit-docs/SKILL.md create mode 100755 .claude/skills/audit-docs/check_docs.rb create mode 100644 docs/README.md create mode 100644 docs/bayes.md create mode 100644 docs/cli.md create mode 100644 docs/configuration.md create mode 100644 docs/keywords.md create mode 100644 docs/knn.md create mode 100644 docs/logistic-regression.md create mode 100644 docs/lsi.md create mode 100644 docs/persistence.md create mode 100644 docs/streaming.md create mode 100644 docs/tfidf.md create mode 100644 test/docs/documentation_test.rb diff --git a/.claude/skills/audit-docs/SKILL.md b/.claude/skills/audit-docs/SKILL.md new file mode 100644 index 00000000..4f06677d --- /dev/null +++ b/.claude/skills/audit-docs/SKILL.md @@ -0,0 +1,87 @@ +--- +name: audit-docs +description: Audit README.md and docs/ for accuracy. Runs every documented code example, checks links, anchors, and prose rules, and verifies the CLI examples against the built gem. Use when documentation changes, before a release, or when the user asks to check the docs, verify code samples, or confirm the examples still work. +--- + +# Audit the documentation + +The docs make claims about how this gem behaves. This audit proves each claim or +finds the ones that are false. + +A documented example that no longer runs is a bug. Fix the code or fix the docs. +Never weaken an assertion to make a check pass. + +## 1. Run the executable examples + +`test/docs/documentation_test.rb` holds one assertion per value printed in +README.md and docs/*.md. + +```bash +bundle exec ruby -Ilib -Itest test/docs/documentation_test.rb +``` + +Every failure means the docs and the code disagree. Read the failure, decide +which side is wrong, and correct that side. + +## 2. Run the static checks + +```bash +ruby .claude/skills/audit-docs/check_docs.rb +``` + +This covers what a unit test cannot: + +- em dashes and en dashes, which this repo's prose rules forbid +- relative markdown links that point at a missing file +- link fragments that point at a missing heading +- documented executables that do not exist + +## 3. Run the full suite + +```bash +bundle exec rake test +bundle exec rubocop +``` + +The documentation test is part of `rake test`, so a red suite blocks a release. + +## 4. Verify the CLI examples against a real install + +The shell examples are not covered by the unit test. Build the gem, install it +into a throwaway `GEM_HOME`, and run the commands as a new user would. + +```bash +gem build classifier.gemspec +export GEM_HOME=$(mktemp -d) GEM_PATH=$GEM_HOME PATH=$GEM_HOME/bin:$PATH +gem install classifier-*.gem --no-document +``` + +Then walk the examples in `docs/cli.md` and `docs/keywords.md` in order, from an +empty directory. Order matters. A reader runs the commands top to bottom, so a +command that needs a model must come after the command that builds one. + +Check that: + +- the first example a new user meets actually succeeds +- printed output matches what the page shows +- exit codes match the documented table +- an error path prints the documented message + +Delete the temporary `GEM_HOME` and the built `.gem` when finished. + +## 5. Cross-check new public API + +List what the code exposes and confirm the docs cover it: + +```bash +grep -rn "^\s*def \(self\.\)\?[a-z_]" lib/classifier/*.rb | grep -v "def _" +``` + +Anything public and undocumented is a gap. Add it to the right page in `docs/` +and add an assertion to `test/docs/documentation_test.rb`. + +## Reporting + +Report every finding with the file, the claim, and the observed behavior. Say +plainly which side you changed. If the audit finds nothing, say the docs are +accurate and name what you verified, so the result is checkable. diff --git a/.claude/skills/audit-docs/check_docs.rb b/.claude/skills/audit-docs/check_docs.rb new file mode 100755 index 00000000..b974c010 --- /dev/null +++ b/.claude/skills/audit-docs/check_docs.rb @@ -0,0 +1,77 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Static checks for the documentation set. +# +# Run from the repository root: +# ruby .claude/skills/audit-docs/check_docs.rb +# +# Checks that need no Ruby runtime behavior live here. The executable examples +# live in test/docs/documentation_test.rb, which `rake test` runs. + +DASHES = { '—' => 'em dash', '–' => 'en dash', '‒' => 'figure dash', '―' => 'horizontal bar' }.freeze + +def markdown_files + (Dir['*.md'] + Dir['docs/**/*.md']).reject { |f| f == 'CLAUDE.md' }.sort +end + +def report(findings) + findings.each { |f| puts " #{f}" } + puts ' none' if findings.empty? + findings.size +end + +total = 0 + +puts 'Dash characters (docs must use plain prose, not em or en dashes):' +total += report(markdown_files.flat_map do |file| + File.readlines(file).each_with_index.filter_map do |line, index| + hit = DASHES.keys.find { |d| line.include?(d) } + "#{file}:#{index + 1}: #{DASHES[hit]} in #{line.strip[0, 70]}" if hit + end +end) + +puts +puts 'Internal links (every relative markdown link must resolve):' +total += report(markdown_files.flat_map do |file| + dir = File.dirname(file) + File.read(file).scan(/\[[^\]]*\]\(([^)#][^)]*)\)/).flatten.filter_map do |target| + next if target.start_with?('http://', 'https://', 'mailto:') + + path = target.split('#').first + next if path.nil? || path.empty? + + resolved = File.expand_path(path, dir) + "#{file}: broken link to #{target}" unless File.exist?(resolved) + end +end) + +puts +puts 'Anchors (every relative link with a fragment must hit a real heading):' +total += report(markdown_files.flat_map do |file| + dir = File.dirname(file) + File.read(file).scan(/\[[^\]]*\]\(([^)]*#[^)]+)\)/).flatten.filter_map do |target| + next if target.start_with?('http://', 'https://') + + path, fragment = target.split('#', 2) + resolved = path.empty? ? file : File.expand_path(path, dir) + next unless File.exist?(resolved) + + slugs = File.read(resolved).scan(/^#+\s+(.+)$/).flatten.map do |heading| + heading.downcase.gsub(/[^\w\s-]/, '').strip.gsub(/\s+/, '-') + end + "#{file}: no heading ##{fragment} in #{path.empty? ? File.basename(file) : path}" unless slugs.include?(fragment) + end +end) + +puts +puts 'Documented commands exist:' +total += report( + %w[classifier keywords].filter_map do |exe| + "exe/#{exe} is documented but missing" unless File.exist?("exe/#{exe}") + end +) + +puts +puts total.zero? ? 'PASS: no static documentation problems.' : "FAIL: #{total} problem(s)." +exit(total.zero? ? 0 : 1) diff --git a/.gitignore b/.gitignore index ea84b02f..48f2dd4f 100644 --- a/.gitignore +++ b/.gitignore @@ -25,6 +25,7 @@ mkmf.log # OS files .DS_Store -# Claude Code local settings -.claude/ +# Claude Code local settings, except tracked project skills +.claude/* +!.claude/skills/ sig/generated/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 1b5f572f..2e718133 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,18 @@ frequent original word in the text. - Add the `min_df` and `max_df` readers to `Classifier::TFIDF`. - Accept a `MultiIO` in `TFIDF#fit_from_stream` and `Streaming::LineReader`. +- Fix `Marshal` support in `Classifier::Bayes`. The dump left out + `min_word_length`, so the restored classifier raised + `ArgumentError: comparison of Integer with nil failed` on its first + `classify`. A dump written by an older version still loads, and takes the + configured default. +- Fix `LSI#highest_relative_content`, which returned an Enumerator rather than + the documented array of documents. +- Fix `LSI#highest_ranked_stems`, which repeated one stem when the document + vector held equal weights. It looked up each weight by value, so tied weights + all resolved to the same index. +- Add a `docs/` reference for the command line tools, each classifier, + persistence, streaming, and configuration. ## 2.6.0 - 2026-06-25 diff --git a/README.md b/README.md index 0089a091..85f3b75d 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ Text classification in Ruby. Five algorithms, native performance, streaming support. -**[Documentation](https://rubyclassifier.com/docs)** · **[Tutorials](https://rubyclassifier.com/docs/tutorials)** · **[API Reference](https://rubydoc.info/gems/classifier)** +**[Reference](docs/)** · **[Documentation](https://rubyclassifier.com/docs)** · **[Tutorials](https://rubyclassifier.com/docs/tutorials)** · **[API Reference](https://rubydoc.info/gems/classifier)** ## Why This Library? @@ -32,7 +32,7 @@ brew install cardmagic/tap/classifier ## Command Line -Classify text instantly with pre-trained models—no coding required: +Classify text instantly with pre-trained models. No code required: ```bash # Detect spam @@ -63,42 +63,46 @@ classifier "Great product, highly recommend" # => positive ``` -Extract keywords and analyze term importance using TF-IDF instantly: +The `keywords` command scores term importance with TF-IDF. It has no +pre-trained models, so build a vocabulary first. Every later command reads +that model: ```bash -# Extract from a raw string +# Fit from multiple files. Each line becomes a separate document. +keywords fit corpus/*.txt +# => Saved to "/path/to/keywords.json" + +# Fit from stdin +cat documents.txt | keywords fit + +# Tune the vocabulary filters during the fit +keywords fit --min-df 2 --max-df 0.85 --ngram 1,2 corpus/*.txt +``` + +Then score any text against that vocabulary: + +```bash +# Score a raw string keywords "Ruby is a programming language" -# => ruby:0.52 programming:0.41 language:0.38 +# => language:0.58 programming:0.58 ruby:0.58 -# Extract from a file +# Score a file keywords extract article.txt -# => machine:0.61 learning:0.58 neural:0.45 network:0.42 +# => machine:0.58 network:0.47 neural:0.47 learning:0.47 # Pipeline with stdin and web data curl -s https://example.com/article | keywords extract -# Get top 5 terms only +# Get the top 5 terms only keywords -n 5 "long document with many terms..." -# Use a custom model file +# Use a different model file keywords -m custom_model.json "Ruby is a programming language" ``` -Build your own vocabulary (fit data): -```bash -# Fit from multiple files -keywords fit corpus/*.txt - -# Fit from stdin (each line is treated as a separate document) -cat documents.txt | keywords fit +Inspect the model: -# Tune vocabulary filters during fitting -keywords fit --min-df 2 --max-df 0.85 --ngram 1,2 corpus/*.txt -``` - -Inspect your model: ```bash -# Check model statistics and parameters keywords info # => Documents: 1,234 # => Vocabulary: 5,678 @@ -106,7 +110,14 @@ keywords info # => Max DF: 1.0 ``` -[CLI Guide →](https://rubyclassifier.com/docs/guides/cli/basics) +The output maps stems back to whole words, so a model built from `programming` +prints `programming`, not `program`. An n-gram label joins its parts with a +space, as in `machine learning:0.35`. + +Run `keywords --help` for the full option list. A usage error exits 2 and any +other error exits 1, so scripts can tell the two apart. + +[keywords reference →](docs/keywords.md) · [CLI Guide →](https://rubyclassifier.com/docs/guides/cli/basics) ### Claude Code Plugin @@ -143,6 +154,7 @@ classifier.classify("Cheap pills!") # => "Spam" classifier = Classifier::LogisticRegression.new(:positive, :negative) classifier.train(positive: "love amazing great wonderful") classifier.train(negative: "hate terrible awful bad") +classifier.fit # required before the first classify classifier.classify("I love it!") # => "Positive" ``` [Logistic Regression Guide →](https://rubyclassifier.com/docs/guides/logisticregression/basics) @@ -171,7 +183,7 @@ knn.classify("programming code") # => "tech" ```ruby tfidf = Classifier::TFIDF.new tfidf.fit(["Ruby is great", "Python is great", "Ruby on Rails"]) -tfidf.transform("Ruby programming") # => {:rubi => 1.0} +tfidf.transform("Ruby programming") # => {rubi: 1.0} ``` [TF-IDF Guide →](https://rubyclassifier.com/docs/guides/tfidf/basics) @@ -179,19 +191,29 @@ tfidf.transform("Ruby programming") # => {:rubi => 1.0} ### Incremental LSI -Add documents without rebuilding the entire index—400x faster for streaming data: +Add documents without a rebuild of the whole index. Turn `auto_rebuild` off, add +the starting corpus, then build once: ```ruby -lsi = Classifier::LSI.new(incremental: true) -lsi.add(tech: ["Ruby is elegant", "Python is popular"]) +lsi = Classifier::LSI.new(incremental: true, auto_rebuild: false) +lsi.add(tech: [ + "Ruby is an elegant programming language for web development", + "Python is a popular programming language for data science", + "JavaScript runs in browsers and powers modern web applications", + "Java is a compiled language used for enterprise backend systems", + "Rust provides memory safety without a garbage collector runtime" +]) lsi.build_index -# These use Brand's algorithm—no full rebuild -lsi.add(tech: "Go is fast") -lsi.add(tech: "Rust is safe") +# This uses Brand's algorithm. No full rebuild. +lsi.add(tech: "Go is a fast compiled language for backend systems") +lsi.incremental_enabled? # => true ``` -[Learn more →](https://rubyclassifier.com/docs/guides/lsi/basics) +Incremental mode needs the starting corpus in place before the first build, and +it falls back to a full rebuild when one document grows the vocabulary too far. + +[Incremental LSI →](docs/lsi.md#incremental-mode) · [Learn more →](https://rubyclassifier.com/docs/guides/lsi/basics) ### Persistence diff --git a/classifier.gemspec b/classifier.gemspec index a8b6ee0d..ee3883b9 100644 --- a/classifier.gemspec +++ b/classifier.gemspec @@ -19,7 +19,7 @@ Gem::Specification.new do |s| } s.required_ruby_version = '>= 3.1' s.files = Dir['{lib,sig,exe}/**/*.{rb,rbs}', 'ext/**/*.{c,h,rb}', 'exe/*', 'bin/*', 'LICENSE', - 'README.md', 'CHANGELOG.md', 'test/*'] + 'README.md', 'CHANGELOG.md', 'docs/**/*.md', 'test/*'] s.bindir = 'exe' s.executables = %w[classifier keywords] s.extensions = ['ext/classifier/extconf.rb'] diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..59909e56 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,49 @@ +# Classifier reference + +Feature reference for the `classifier` gem. Every example here runs against the +version in this repository. + +The [README](../README.md) gives the short tour. These pages give the detail. + +## Command line + +| Page | Contents | +|:--|:--| +| [classifier](cli.md) | Train, classify, and manage models from the shell | +| [keywords](keywords.md) | TF-IDF keyword extraction and term scores | + +## Classifiers + +| Page | Use it for | +|:--|:--| +| [Bayes](bayes.md) | Fast probabilistic classification. The default choice | +| [Logistic Regression](logistic-regression.md) | Linear classification with calibrated probabilities | +| [LSI](lsi.md) | Semantic similarity, search, related documents, and summaries | +| [k-Nearest Neighbors](knn.md) | Classification with the nearest examples and their votes | + +## Vectorization + +| Page | Contents | +|:--|:--| +| [TF-IDF](tfidf.md) | Term weights, n-grams, document frequency filters | + +## Shared behavior + +| Page | Contents | +|:--|:--| +| [Persistence](persistence.md) | Save, load, storage backends, and custom backends | +| [Streaming](streaming.md) | Training on data larger than memory | +| [Configuration](configuration.md) | Global settings and the native extension | + +## Which classifier + +Start with Bayes. It trains in one pass, needs no fit step, and handles most +text classification tasks. + +- Choose **Logistic Regression** when you need a probability per category, and + you accept a `fit` step after training. +- Choose **LSI** when you need similarity, search, or related documents, and not + only a label. +- Choose **k-Nearest Neighbors** when you want to see which examples drove the + answer. +- Choose **TF-IDF** when you want term weights rather than a category. diff --git a/docs/bayes.md b/docs/bayes.md new file mode 100644 index 00000000..42f7fd44 --- /dev/null +++ b/docs/bayes.md @@ -0,0 +1,108 @@ +# Bayes + +`Classifier::Bayes` is a Naive Bayesian classifier. It trains in one pass, needs +no fit step, and suits most text classification tasks. + +It uses log probabilities for numerical stability, and add-one (Laplace) +smoothing, where `P(word|category) = (count + 1) / (total + vocabulary_size)`. + +## Train and classify + +```ruby +require "classifier" + +classifier = Classifier::Bayes.new(:spam, :ham) +classifier.train(spam: "Buy viagra cheap pills now") +classifier.train(spam: "You won million dollars prize") +classifier.train(ham: ["Meeting tomorrow at 3pm", "Quarterly report attached"]) + +classifier.classify("Cheap pills!") +# => "Spam" +``` + +A category name comes back capitalized. Pass an array to train several documents +against one category in a single call. + +## Scores per category + +```ruby +classifier.classifications("Cheap pills!") +# => {"Spam" => -8.579980179515003, "Ham" => -9.680344001221918} +``` + +These are log probabilities, so they are negative, and the highest value wins. + +## Dynamic training methods + +A `train_` method exists for every category: + +```ruby +classifier = Classifier::Bayes.new(:spam, :ham) +classifier.train_spam("cheap pills") +classifier.train_ham("meeting tomorrow") +classifier.classify("pills") +# => "Spam" +``` + +`untrain_` removes a document the same way. + +## Manage categories + +```ruby +classifier.categories +# => ["Spam", "Ham"] + +classifier.add_category(:other) +classifier.categories +# => ["Spam", "Ham", "Other"] + +classifier.remove_category(:other) +classifier.categories +# => ["Spam", "Ham"] +``` + +`remove_category` also removes that category's word counts. `append_category` is +an alias of `add_category`. + +## Untrain + +```ruby +classifier.untrain(spam: "Buy viagra cheap pills now") +``` + +Untrain the same text you trained. A document you never trained corrupts the +counts. + +## Short words + +The tokenizer drops words shorter than `min_word_length`, which defaults to 3. +Raise or lower it per classifier: + +```ruby +classifier = Classifier::Bayes.new(:spam, :ham, min_word_length: 2) +``` + +See [Configuration](configuration.md) to change the default for every +classifier. + +## Constructor + +```ruby +Classifier::Bayes.new(*categories, min_word_length: 3) +``` + +An array of categories also works: + +```ruby +Classifier::Bayes.new([:spam, :ham]) +``` + +## Save and load + +```ruby +classifier.save_to_file("model.json") +loaded = Classifier::Bayes.load_from_file("model.json") +``` + +See [Persistence](persistence.md) for storage backends, and +[Streaming](streaming.md) for corpora larger than memory. diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 00000000..159bfff9 --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,144 @@ +# classifier + +`classifier` trains and runs text classifiers from the shell. For term scores +rather than categories, use [`keywords`](keywords.md). + +## Pre-trained models + +The `-r` flag pulls a model from the registry, so classification works with no +training: + +```console +$ classifier -r sms-spam-filter "You won a free iPhone" +spam + +$ classifier -r imdb-sentiment "This movie was absolutely amazing" +positive + +$ classifier models +``` + +## Train your own + +```console +$ classifier train positive reviews/good/*.txt +$ classifier train negative reviews/bad/*.txt + +$ classifier "Great product, highly recommend" +positive +``` + +Training reads standard input when you name no file: + +```console +$ echo "amazing fantastic" | classifier train positive +``` + +The default model file is `./classifier.json`. Use `-f` for any other path. Each +`train` call updates that file in place. + +## Probabilities + +```console +$ classifier -p "Great product, highly recommend" +positive:0.89 negative:0.11 +``` + +## Model information + +```console +$ classifier info +{ + "file": "./classifier.json", + "type": "bayes", + "categories": [ + "Positive", + "Negative" + ], + "category_stats": { + "Positive": { + "unique_words": 7, + "total_words": 7 + }, + "Negative": { + "unique_words": 7, + "total_words": 7 + } + } +} +``` + +## Other classifiers + +`-m` selects the algorithm. The default is `bayes`. + +```console +$ classifier -m knn -k 3 train tech docs/tech/*.txt +$ classifier -m lr train positive reviews/good/*.txt +$ classifier -m lsi train dogs corpus/dogs/*.txt +``` + +Logistic regression needs a fit step after training: + +```console +$ classifier -m lr train positive reviews/good/*.txt +$ classifier -m lr train negative reviews/bad/*.txt +$ classifier -m lr fit +Model fitted successfully +$ classifier -m lr "I love it" +positive +``` + +## Search and related documents + +These two commands need an LSI model. + +```console +$ classifier -m lsi search "machine learning" +$ classifier -m lsi related article.txt +``` + +`-n` sets how many results come back. The default is 10. + +## Commands + +| Command | Action | +|:--|:--| +| `train [files...]` | Train a category from files or standard input | +| `info` | Print model information | +| `fit` | Fit the model. Logistic regression only | +| `search ` | Semantic search. LSI only | +| `related ` | Find related documents. LSI only | +| `models [registry]` | List the models in a registry | +| `pull ` | Download a model from the registry | +| `push ` | Contribute a model to the registry | +| `` | Classify the text. The default action | + +## Options + +| Option | Meaning | +|:--|:--| +| `-f`, `--file FILE` | Model file. Default `./classifier.json` | +| `-m`, `--model TYPE` | Algorithm: `bayes`, `lsi`, `knn`, or `lr`. Default `bayes` | +| `-r`, `--remote MODEL` | Use a remote model, by name or `@user/repo:name` | +| `--search TEXT` | Search remote models by name and description, and local models by name | +| `-o`, `--output FILE` | Output path for `pull` | +| `-p` | Print probabilities | +| `-n`, `--count N` | Result count for `search` and `related`. Default 10 | +| `-k`, `--neighbors N` | Neighbor count for kNN. Default 5 | +| `--weighted` | Use distance-weighted voting for kNN | +| `--learning-rate N` | Learning rate for logistic regression. Default 0.1 | +| `--regularization N` | L2 regularization for logistic regression. Default 0.01 | +| `--max-iterations N` | Maximum iterations for logistic regression. Default 100 | +| `-q` | Quiet mode | +| `--local` | List locally cached models, with the `models` command | +| `-v`, `--version` | Print the gem version | +| `-h`, `--help` | Print the full usage | + +## Install without Ruby + +Homebrew installs the command line tools on their own: + +```bash +brew install cardmagic/tap/classifier +``` diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 00000000..77e819c4 --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,100 @@ +# Configuration + +## Global settings + +`Classifier.configure` sets the defaults for every classifier: + +```ruby +require "classifier" + +Classifier.configure do |config| + config.min_word_length = 2 +end + +Classifier.config.min_word_length +# => 2 +``` + +| Setting | Default | Effect | +|:--|:--|:--| +| `min_word_length` | 3 | The tokenizer drops any word shorter than this | + +Set the configuration once at startup. The lazy setup is not thread-safe, so do +not first touch it from several threads at once. + +Every classifier also takes `min_word_length` on its own, which overrides the +global value: + +```ruby +Classifier::Bayes.new(:spam, :ham, min_word_length: 2) +``` + +## Tokenization + +The tokenizer downcases the text, strips punctuation, drops the stop words in +`CORPUS_SKIP_WORDS`, drops words shorter than `min_word_length`, and reduces +each remaining word to its Porter stem. + +```ruby +"Ruby programming is elegant".word_hash +# => {rubi: 1, program: 1, eleg: 1} +``` + +`clean_word_hash` skips the punctuation strip when the text is already clean. +`stem_to_word_hash` maps each stem back to the most frequent original word: + +```ruby +"Ruby programming is elegant and programming rocks".stem_to_word_hash +# => {rubi: "ruby", program: "programming", eleg: "elegant", rock: "rocks"} +``` + +## Native extension + +LSI uses a C extension for its linear algebra. It has no external dependency +and builds during `gem install`. Pure Ruby runs when the extension is absent, +with the same results and less speed. + +```ruby +Classifier::LSI.backend +# => :native +``` + +The value is `:native` or `:ruby`. + +Force pure Ruby with an environment variable, which is useful to compare the +two: + +```bash +NATIVE_VECTOR=true bundle exec rake test +``` + +Build the extension from a checkout: + +```bash +bundle exec rake compile +``` + +Silence the startup notice about the missing extension: + +```bash +SUPPRESS_LSI_WARNING=true +``` + +## Errors + +Every error inherits from `Classifier::Error`. + +| Error | Raised when | +|:--|:--| +| `Classifier::NotFittedError` | A model is used before its fit. Logistic regression and TF-IDF | +| `Classifier::UnsavedChangesError` | `reload!` would discard unsaved changes | +| `Classifier::StorageError` | A storage backend operation fails | + +```ruby +begin + classifier.classify("text") +rescue Classifier::NotFittedError + classifier.fit + retry +end +``` diff --git a/docs/keywords.md b/docs/keywords.md new file mode 100644 index 00000000..83976ee8 --- /dev/null +++ b/docs/keywords.md @@ -0,0 +1,171 @@ +# keywords + +`keywords` scores the terms of a text with TF-IDF. It prints `term:score` pairs +in descending order of score. + +The gem installs this command next to `classifier`. + +## A model comes first + +`keywords` ships no pre-trained models. Build a vocabulary before you score any +text: + +```console +$ keywords fit corpus/*.txt +Saved to "/path/to/keywords.json" +``` + +A command that needs a model and finds none exits 2: + +```console +$ keywords "Ruby is elegant" +Error: No model found; run 'keywords fit' first or pass correct model using the '-m' option. +``` + +The default model path is `./keywords.json`. Use `-m` for any other path. + +## Commands + +| Command | Action | +|:--|:--| +| `keywords fit ` | Build a vocabulary from files or standard input | +| `keywords extract ` | Score the contents of one file | +| `keywords info` | Print the model statistics | +| `keywords ` | Score the text given as arguments | + +With no arguments and no piped input, `keywords` prints a short guide. + +## fit + +Each **line** becomes a separate document. The document count drives the inverse +document frequency, so a file of 200 lines contributes 200 documents. + +```console +$ keywords fit corpus/*.txt +Saved to "/path/to/keywords.json" + +$ cat documents.txt | keywords fit +Saved to "/path/to/keywords.json" + +$ keywords fit --min-df 2 --max-df 0.85 --ngram 1,2 corpus/*.txt +Saved to "/path/to/keywords.json" +``` + +`fit` skips a directory and keeps the files beside it, so a shell glob works +even when the directory holds a subdirectory: + +```console +$ ls corpus +a.txt b.txt archive/ +$ keywords fit corpus/* +Saved to "/path/to/keywords.json" +``` + +A path that matches nothing stops the run, so a typo never produces a smaller +model in silence: + +```console +$ keywords fit corpus/a.txt corpus/NOPE.txt +Error: No files matched "corpus/NOPE.txt" +``` + +An argument set with no readable file at all reports `No files to fit`. Empty +input reports `No documents found to save the model`. Neither writes a model. + +`fit` reads one file at a time, so a corpus larger than the file descriptor +limit still works. + +## extract + +```console +$ keywords extract article.txt +machine:0.58 network:0.47 neural:0.47 learning:0.47 + +$ curl -s https://example.com/article | keywords extract +``` + +`extract` requires a real file. A path that does not exist, or a directory, +exits 2. To score literal text, use the bare form instead. + +## info + +```console +$ keywords info +Documents: 1,234 +Vocabulary: 5,678 +Min DF: 1 +Max DF: 1.0 +``` + +## Options + +| Option | Meaning | +|:--|:--| +| `-m`, `--model FILE` | Model file. Default `./keywords.json` | +| `-n`, `--top N` | Print the top N terms only. N must be positive | +| `-q` | Quiet. Suppress the `Saved to` line from `fit` | +| `--min-df N` | Minimum document frequency, as a count. Default 1 | +| `--max-df N` | Maximum document frequency, as a ratio from 0.0 to 1.0. Default 1.0 | +| `--ngram MIN,MAX` | N-gram range. Default `1,1` | +| `-v`, `--version` | Print the gem version | +| `-h`, `--help` | Print the full usage | + +`--min-df` and `--max-df` apply during `fit`. The model stores them, and `info` +reports them back. + +`-q` suppresses progress text but never the term scores. A scripted `fit` stays +silent, and a scripted score still produces its data. + +## Output format + +Terms print as `term:score`, separated by spaces, sorted by descending score. + +The command maps stems back to whole words. A model built from `programming` +prints `programming`, not the `program` stem: + +```console +$ keywords "Ruby is a programming language" +language:0.58 programming:0.58 ruby:0.58 +``` + +An n-gram label joins its parts with a space: + +```console +$ keywords fit --ngram 1,2 -m ng.json corpus/*.txt +$ keywords -m ng.json "machine learning neural networks" +machine learning:0.46 machine:0.46 neural networks:0.38 networks:0.38 neural:0.38 learning:0.38 +``` + +Each score depends on the corpus you fitted, so your numbers will differ. + +That space sits inside a label in an otherwise space-separated stream. Parse +n-gram output on the `:` separator, not on whitespace. + +## Exit codes + +| Code | Meaning | +|:--|:--| +| 0 | Success | +| 1 | An unexpected error | +| 2 | A usage error, such as a bad option, a missing model, or a path that matches nothing | + +Scripts can rely on 2 for every input mistake: + +```bash +keywords fit corpus/*.txt || echo "fit failed with $?" +``` + +## Equivalent Ruby + +The command wraps [`Classifier::TFIDF`](tfidf.md). This code does what +`keywords fit` does: + +```ruby +require "classifier" + +tfidf = Classifier::TFIDF.new(min_df: 2, max_df: 0.85, ngram_range: [1, 2]) +tfidf.fit_from_stream( + Classifier::Streaming::MultiIO.new(Dir["corpus/*.txt"]) +) +tfidf.save_to_file("keywords.json") +``` diff --git a/docs/knn.md b/docs/knn.md new file mode 100644 index 00000000..48b4ebc9 --- /dev/null +++ b/docs/knn.md @@ -0,0 +1,92 @@ +# k-Nearest Neighbors + +`Classifier::KNN` classifies text by the categories of its nearest examples. It +shows which examples drove the answer, which makes it useful when you must +explain a result. + +It builds on [LSI](lsi.md) for the similarity measure. + +## Add examples and classify + +```ruby +require "classifier" + +knn = Classifier::KNN.new(k: 3) +%w[laptop coding software developer programming].each { |w| knn.add(tech: w) } +%w[football basketball soccer goal team].each { |w| knn.add(sports: w) } + +knn.classify("programming code") +# => "tech" +``` + +`train` is an alias of `add`, so the call reads the same as Bayes: + +```ruby +knn.train(tech: "compiler", sports: "referee") +``` + +## See the neighbors + +```ruby +knn.classify_with_neighbors("programming code") +``` + +The result holds the winning category, the neighbors that voted, the vote +tally, and a confidence value: + +```ruby +{ + category: "tech", + neighbors: [ + { item: "programming", category: "tech", similarity: 0.9999999999999993 }, + { item: "coding", category: "tech", similarity: 0.9999999999999992 }, + { item: "team", category: "sports", similarity: 2.6e-16 } + ], + votes: { "tech" => 2.0, "sports" => 1.0 }, + confidence: 0.6666666666666666 +} +``` + +`confidence` is the winning share of the votes. + +## Choose k + +`k` sets how many neighbors vote. It defaults to 5. + +```ruby +knn = Classifier::KNN.new(k: 3) +knn.k # => 3 +knn.k = 5 +``` + +A small `k` follows the data closely and reacts to noise. A large `k` smooths +the boundary. Keep `k` at or below the number of examples you added. + +## Weighted voting + +By default every neighbor casts an equal vote. Weighted voting scales each vote +by similarity, so a close neighbor counts for more: + +```ruby +knn = Classifier::KNN.new(k: 5, weighted: true) +``` + +Set it later with `knn.weighted = true`. + +## Inspect the model + +```ruby +knn.categories # => ["tech", "sports"] +knn.items # every added example +knn.categories_for("programming") +knn.remove_item("programming") +``` + +## Save and load + +```ruby +knn.save_to_file("model.json") +loaded = Classifier::KNN.load_from_file("model.json") +``` + +See [Persistence](persistence.md). diff --git a/docs/logistic-regression.md b/docs/logistic-regression.md new file mode 100644 index 00000000..f91c6707 --- /dev/null +++ b/docs/logistic-regression.md @@ -0,0 +1,107 @@ +# Logistic Regression + +`Classifier::LogisticRegression` is a linear classifier. It gives calibrated +probabilities that sum to 1.0, which Bayes does not. + +## Train, fit, then classify + +Unlike Bayes, this classifier needs a `fit` call after training. `classify` +raises `Classifier::NotFittedError` before that call. + +```ruby +require "classifier" + +classifier = Classifier::LogisticRegression.new(:positive, :negative) +classifier.train(positive: "love amazing great wonderful") +classifier.train(negative: "hate terrible awful bad") +classifier.fit + +classifier.classify("I love it!") +# => "Positive" +``` + +Train more documents at any time. Call `fit` again before the next classify. + +```ruby +classifier.fitted? +# => true +``` + +## Probabilities + +```ruby +classifier.probabilities("I love it!") +# => {"Positive" => 0.7398506195705559, "Negative" => 0.26014938042944413} +``` + +The values sum to 1.0. Use `classifications` for the raw scores before the +sigmoid: + +```ruby +classifier.classifications("I love it!") +# => {"Positive" => 0.5225961471158276, "Negative" => -0.5225961471158275} +``` + +## Inspect the weights + +`weights` shows which terms drive a category: + +```ruby +classifier.weights("positive") +# => {hate: -0.5225, terribl: -0.5225, aw: -0.5225, bad: -0.5225, +# love: 0.5225, amaz: 0.5225, great: 0.5225, wonder: 0.5225} +``` + +The keys are Porter stems. The order runs by **absolute** value, so the terms +that matter most come first whichever way they point. A positive weight argues +for the category and a negative weight argues against it. + +`limit` caps the count: + +```ruby +classifier.weights("positive", limit: 3) +``` + +Terms of equal absolute weight tie, and a tie has no defined order. A toy +corpus like the one above gives every term the same magnitude, so `limit` there +returns an arbitrary three. Real training data separates the weights. + +## Tuning + +```ruby +Classifier::LogisticRegression.new( + :positive, :negative, + learning_rate: 0.1, + regularization: 0.01, + max_iterations: 100 +) +``` + +| Parameter | Default | Effect | +|:--|:--|:--| +| `learning_rate` | 0.1 | Step size per iteration. Raise it to train faster, lower it for stability | +| `regularization` | 0.01 | L2 penalty. Raise it to reduce overfit | +| `max_iterations` | 100 | Gradient descent iterations during `fit` | + +## Categories + +```ruby +classifier.categories +# => ["Positive", "Negative"] + +classifier.add_category(:neutral) +``` + +Call `fit` again after you add a category and train it. + +## Save and load + +```ruby +classifier.save_to_file("model.json") +loaded = Classifier::LogisticRegression.load_from_file("model.json") +``` + +A saved model keeps its fitted weights, so a loaded model classifies with no +further `fit`. + +See [Persistence](persistence.md) and [Streaming](streaming.md). diff --git a/docs/lsi.md b/docs/lsi.md new file mode 100644 index 00000000..d019b191 --- /dev/null +++ b/docs/lsi.md @@ -0,0 +1,219 @@ +# LSI + +`Classifier::LSI` implements Latent Semantic Indexing. It finds documents that +share meaning, not only shared words, so it answers similarity, search, and +related-document questions that a word-count classifier cannot. + +It uses Singular Value Decomposition. A [native C extension](configuration.md) +makes that 5 to 50 times faster, and pure Ruby runs when the extension is +absent. + +## Classify + +```ruby +require "classifier" + +lsi = Classifier::LSI.new +lsi.add(dog: "dog puppy canine bark fetch", cat: "cat kitten feline meow purr") + +lsi.classify("My puppy barks") +# => "dog" +``` + +## Confidence + +```ruby +lsi.classify_with_confidence("My puppy barks") +# => ["dog", 1.0] +``` + +The second value runs from 0.0 to 1.0. + +## Search + +```ruby +lsi.search("puppy", 2) +# => ["dog puppy canine bark fetch", "cat kitten feline meow purr"] +``` + +The second argument caps the result count. Results come back in descending +order of similarity. + +## Related documents + +```ruby +lsi.find_related("dog puppy canine bark fetch", 1) +``` + +## Add documents + +`add` takes categories as keywords: + +```ruby +lsi.add(dog: "dog puppy canine bark fetch") +lsi.add(tech: ["Ruby is elegant", "Python is popular"]) +``` + +`add_item` takes the item first, then its categories, and accepts a block that +converts the item to text: + +```ruby +lsi.add_item("dog puppy canine", :dog) +lsi.add_item(article, :tech) { |a| a.body } +``` + +## The index + +LSI builds an index before it answers a query. By default it rebuilds whenever +it needs to. Turn that off to add many documents and rebuild once: + +```ruby +lsi = Classifier::LSI.new(auto_rebuild: false) +lsi.add(dog: "dog puppy canine bark fetch") +lsi.add(cat: "cat kitten feline meow purr") +lsi.build_index +``` + +```ruby +lsi.needs_rebuild? +# => false +``` + +## Incremental mode + +Incremental mode adds documents through Brand's algorithm, with no full +rebuild. + +Turn `auto_rebuild` off. Incremental mode needs the whole starting corpus in +place before the first index build: + +```ruby +lsi = Classifier::LSI.new(incremental: true, auto_rebuild: false, max_rank: 100) +lsi.add(tech: [ + "Ruby is an elegant programming language for web development", + "Python is a popular programming language for data science", + "JavaScript runs in browsers and powers modern web applications", + "Java is a compiled language used for enterprise backend systems", + "Rust provides memory safety without a garbage collector runtime" +]) +lsi.build_index + +lsi.incremental_enabled? +# => true + +lsi.add(tech: "Go is a fast compiled language for backend systems") +lsi.incremental_enabled? +# => true +``` + +`build_index` stores the U matrix that later updates need. It stores that +matrix only while incremental mode is on. + +**Leave `auto_rebuild` at its default and incremental mode never starts.** Each +`add` rebuilds at once, so the index builds from the first two documents, and +the next `add` measures its vocabulary growth against that tiny start. The +growth trips the threshold below, incremental mode switches off, and a later +`build_index` cannot turn it back on. + +### The fallback + +An added document that grows the vocabulary by more than 20 percent of its +size at the first build is too large a shift for an incremental update. LSI +then turns incremental mode off and rebuilds in full. The results stay correct. +The speed advantage stops. + +The fallback is permanent. Call `enable_incremental_mode!` to resume: + +```ruby +lsi.enable_incremental_mode!(max_rank: 100) +lsi.build_index(force: true) +``` + +`current_rank` reports the count of positive singular values. +`disable_incremental_mode!` turns the mode off by hand. + +A corpus of a few documents grows its vocabulary quickly, so incremental mode +suits a large starting corpus and small later additions. + +## Inspect the model + +```ruby +lsi.items # every indexed document +lsi.categories_for("dog puppy canine bark fetch") +lsi.remove_item("dog puppy canine bark fetch") +``` + +`singular_values` returns the raw values after `build_index`, and +`singular_value_spectrum` returns the variance each dimension explains. + +`highest_ranked_stems` names the stems that carry a document: + +```ruby +lsi.highest_ranked_stems("dog puppy canine bark fetch loyal", 3) +# => [:dog, :puppi, :canin] +``` + +The document must already be indexed, or the call raises. + +`highest_relative_content` returns the documents nearest the center of the +whole set, which describes what a corpus is mostly about: + +```ruby +lsi.highest_relative_content(2) +``` + +It returns an empty array while the index still needs a rebuild. + +## Add without categories + +`<<` indexes a document with no category, for search and similarity only: + +```ruby +lsi << "bird sparrow robin fly nest feather" +``` + +## Add in batches + +`add_batch` turns `auto_rebuild` off for the run, adds everything, then builds +once. It reports progress like the streaming API: + +```ruby +lsi.add_batch( + tech: ["Ruby is elegant", "Python is popular"], + sports: ["soccer goal", "basketball hoop"] +) { |progress| puts progress.completed } +``` + +See [Streaming](streaming.md). + +## Summaries + +The gem adds `summary` to `String`: + +```ruby +text = "The dog barks loudly. The cat sleeps quietly. " \ + "Birds sing sweetly in the morning light." + +text.summary(1) +# => "The cat sleeps quietly." +``` + +The argument sets how many sentences come back. + +## Constructor options + +| Option | Default | Meaning | +|:--|:--|:--| +| `auto_rebuild` | `true` | Rebuild the index automatically after a change | +| `incremental` | `false` | Use Brand's algorithm to add documents | +| `max_rank` | 100 | Rank cap in incremental mode | +| `min_word_length` | 3 | Drop words shorter than this | + +## Save and load + +```ruby +lsi.save_to_file("model.json") +loaded = Classifier::LSI.load_from_file("model.json") +``` + +See [Persistence](persistence.md). diff --git a/docs/persistence.md b/docs/persistence.md new file mode 100644 index 00000000..15e7acfa --- /dev/null +++ b/docs/persistence.md @@ -0,0 +1,124 @@ +# Persistence + +Every classifier saves and loads the same way. `Classifier::Bayes`, +`Classifier::LogisticRegression`, `Classifier::LSI`, `Classifier::KNN`, and +`Classifier::TFIDF` all share this API. + +## Files + +```ruby +require "classifier" + +classifier = Classifier::Bayes.new(:spam, :ham) +classifier.train(spam: "cheap pills", ham: "meeting tomorrow") + +classifier.save_to_file("model.json") + +loaded = Classifier::Bayes.load_from_file("model.json") +loaded.classify("pills") +# => "Spam" +``` + +The format is JSON, so a saved model is readable and portable between Ruby +versions. + +## Storage backends + +A backend separates the model from where it lives. Assign one, then call `save` +and `load` with no path: + +```ruby +classifier.storage = Classifier::Storage::File.new(path: "model.json") +classifier.save + +loaded = Classifier::Bayes.load(storage: classifier.storage) +``` + +The gem ships two backends: + +| Backend | Use it for | +|:--|:--| +| `Classifier::Storage::File` | A model on disk | +| `Classifier::Storage::Memory` | Tests, and a model that lives for one process | + +```ruby +storage = Classifier::Storage::Memory.new + +classifier = Classifier::Bayes.new(:a, :b) +classifier.train(a: "alpha", b: "beta") +classifier.storage = storage +classifier.save + +Classifier::Bayes.load(storage: storage).categories +# => ["A", "B"] +``` + +## Write your own backend + +Subclass `Classifier::Storage::Base` and implement four methods: + +```ruby +Classifier::Storage::Base.instance_methods(false).sort +# => [:delete, :exists?, :read, :write] +``` + +| Method | Contract | +|:--|:--| +| `write(key, data)` | Store the serialized model | +| `read(key)` | Return what `write` stored, or nil | +| `exists?(key)` | Report whether a model is stored under the key | +| `delete(key)` | Remove the stored model | + +A Redis backend looks like this: + +```ruby +class RedisStorage < Classifier::Storage::Base + def initialize(redis:, namespace: "classifier") + @redis = redis + @namespace = namespace + end + + def write(key, data) = @redis.set(namespaced(key), data) + def read(key) = @redis.get(namespaced(key)) + def exists?(key) = @redis.exists?(namespaced(key)) + def delete(key) = @redis.del(namespaced(key)) + + private + + def namespaced(key) = "#{@namespace}:#{key}" +end +``` + +The same shape covers S3, a SQL table, or any other store. + +## Track unsaved changes + +`dirty?` reports whether the model changed since the last save: + +```ruby +classifier.dirty? +``` + +`reload` discards unsaved changes and reads the stored model again. `reload!` +does the same and raises when no stored model exists. + +## Marshal + +Every classifier also supports `Marshal`: + +```ruby +data = Marshal.dump(classifier) +restored = Marshal.load(data) +``` + +Prefer JSON. Only load a marshalled model from a source you trust. + +## Checkpoints + +Streaming training writes checkpoints, so a long run resumes after a failure: + +```ruby +Classifier::Bayes.load_checkpoint(storage: storage, checkpoint_id: "run-1") +``` + +See [Streaming](streaming.md). diff --git a/docs/streaming.md b/docs/streaming.md new file mode 100644 index 00000000..e9dac3f6 --- /dev/null +++ b/docs/streaming.md @@ -0,0 +1,104 @@ +# Streaming + +Streaming trains a classifier on data larger than memory. The reader pulls one +batch of lines at a time, so peak memory stays flat whatever the corpus size. + +Each **line** is one document. + +## Train from a stream + +```ruby +require "classifier" + +classifier = Classifier::Bayes.new(:spam, :ham) +classifier.train_from_stream(:spam, File.open("spam_corpus.txt")) +``` + +`Classifier::LogisticRegression` and `Classifier::KNN` accept the same call. +`Classifier::TFIDF` uses `fit_from_stream`, because it fits a vocabulary rather +than a category. + +## Progress + +Pass a block to watch the run: + +```ruby +classifier.train_from_stream(:spam, File.open("spam_corpus.txt")) do |progress| + puts "completed=#{progress.completed}" +end +``` + +`Classifier::Streaming::Progress` reports `completed`, and `total` when the +reader can estimate the line count from the file size. + +## Batch size + +The reader groups lines into batches. The default is +`Classifier::Streaming::DEFAULT_BATCH_SIZE`. + +```ruby +classifier.train_from_stream(:spam, File.open("corpus.txt"), batch_size: 500) +``` + +A larger batch does less bookkeeping and uses more memory. + +## Train from an array + +`train_batch` takes documents already in memory and uses the same batching: + +```ruby +classifier.train_batch(:spam, ["cheap pills", "you won a prize"]) +``` + +`Classifier::LSI` and `Classifier::KNN` name the same method `add_batch`, which +matches their `add` API: + +```ruby +lsi.add_batch(tech: ["Ruby is elegant", "Python is popular"]) +``` + +## Read several files as one stream + +`Classifier::Streaming::MultiIO` presents many sources as one sequential +stream. It accepts file paths, IO objects, or both: + +```ruby +multi = Classifier::Streaming::MultiIO.new(["a.txt", "b.txt"]) +multi.each_line { |line| puts line } +``` + +```ruby +Classifier::Streaming::MultiIO.new(Dir["corpus/*.txt"]).each_line.to_a +``` + +Given a path, `MultiIO` opens the file, reads it, and closes it before it moves +to the next one. Only one file is ever open, so a corpus larger than the file +descriptor limit still works. Given an IO object, it reads that object and +leaves the closing to you. + +`each_line` returns an Enumerator when you pass no block. + +Combine it with a vectorizer to fit a whole corpus: + +```ruby +tfidf = Classifier::TFIDF.new +tfidf.fit_from_stream( + Classifier::Streaming::MultiIO.new(Dir["corpus/*.txt"]) +) +tfidf.num_documents +``` + +## Checkpoints + +A long run writes checkpoints through the assigned storage backend, so a +failure does not cost the whole pass: + +```ruby +classifier.storage = Classifier::Storage::File.new(path: "model.json") +resumed = Classifier::Bayes.load_checkpoint( + storage: classifier.storage, + checkpoint_id: "run-1" +) +``` + +See [Persistence](persistence.md) for the backends. diff --git a/docs/tfidf.md b/docs/tfidf.md new file mode 100644 index 00000000..380dc607 --- /dev/null +++ b/docs/tfidf.md @@ -0,0 +1,137 @@ +# TF-IDF + +`Classifier::TFIDF` turns text into term weights. It answers "which terms matter +in this document", not "which category is this". For the same thing from a +shell, use [`keywords`](keywords.md). + +TF-IDF raises the weight of a term that is frequent in one document, and lowers +it for a term that is common across every document. + +## Fit and transform + +```ruby +require "classifier" + +tfidf = Classifier::TFIDF.new +tfidf.fit(["Ruby is great", "Python is great", "Ruby on Rails"]) + +tfidf.transform("Ruby programming") +# => {rubi: 1.0} +``` + +The keys are Porter stems. `fit_transform` does both steps at once: + +```ruby +tfidf.fit_transform(["Ruby is great", "Python is great"]) +``` + +`transform` raises `Classifier::NotFittedError` before a fit. + +## The vocabulary + +```ruby +tfidf.feature_names # every term in the vocabulary +tfidf.vocabulary # term => column index +tfidf.idf # term => inverse document frequency +tfidf.num_documents # documents seen during the fit +tfidf.fitted? # => true +``` + +## Document frequency filters + +`min_df` and `max_df` drop terms that are too rare or too common. + +```ruby +tfidf = Classifier::TFIDF.new(min_df: 2, max_df: 0.85) +``` + +| Parameter | Type | Meaning | +|:--|:--|:--| +| `min_df` | Integer | Keep a term only when at least this many documents hold it | +| `min_df` | Float | The same, as a ratio of the document count | +| `max_df` | Float | Drop a term that appears in more than this ratio of documents | +| `max_df` | Integer | The same, as an absolute document count | + +An Integer means a count and a Float means a ratio. A Float must fall between +0.0 and 1.0, and an Integer must not be negative. + +Both values are readable after construction, and a saved model keeps them: + +```ruby +tfidf.min_df # => 2 +tfidf.max_df # => 0.85 +``` + +## N-grams + +`ngram_range` sets the shortest and longest phrase to index. It defaults to +`[1, 1]`, which indexes single words only. + +```ruby +tfidf = Classifier::TFIDF.new(ngram_range: [1, 2]) +tfidf.fit(["machine learning rocks", "machine learning is fun"]) + +tfidf.feature_names.sort.first(6) +# => [:fun, :learn, :learn_fun, :learn_rock, :machin, :machin_learn] +``` + +An n-gram key joins its stems with an underscore. Both bounds must be 1 or +more, and the first must not exceed the second. + +## Sublinear term frequency + +`sublinear_tf: true` replaces the raw count with `1 + log(count)`, which damps +the effect of a term repeated many times in one document: + +```ruby +tfidf = Classifier::TFIDF.new(sublinear_tf: true) +``` + +## Constructor + +```ruby +Classifier::TFIDF.new( + min_df: 1, + max_df: 1.0, + ngram_range: [1, 1], + sublinear_tf: false, + min_word_length: 3 +) +``` + +## Fit from a stream + +`fit_from_stream` reads line by line, so a corpus larger than memory still +fits. Each line is one document. + +```ruby +tfidf = Classifier::TFIDF.new +tfidf.fit_from_stream(File.open("corpus.txt")) +``` + +Pass a [`MultiIO`](streaming.md) to read several files as one stream: + +```ruby +tfidf.fit_from_stream( + Classifier::Streaming::MultiIO.new(Dir["corpus/*.txt"]) +) +``` + +## Map stems back to words + +`transform` returns stems. `String#stem_to_word_hash` maps each stem back to the +most frequent original word, which is how `keywords` prints whole words: + +```ruby +"Ruby programming is elegant and programming rocks".stem_to_word_hash +# => {rubi: "ruby", program: "programming", eleg: "elegant", rock: "rocks"} +``` + +## Save and load + +```ruby +tfidf.save_to_file("vectorizer.json") +loaded = Classifier::TFIDF.load_from_file("vectorizer.json") +``` + +See [Persistence](persistence.md). diff --git a/lib/classifier/bayes.rb b/lib/classifier/bayes.rb index 622360d4..ff9df0b4 100644 --- a/lib/classifier/bayes.rb +++ b/lib/classifier/bayes.rb @@ -279,14 +279,17 @@ def add_category(category) # Custom marshal serialization to exclude mutex state # @rbs () -> Array[untyped] def marshal_dump - [@categories, @total_words, @category_counts, @category_word_count, @dirty] + [@categories, @total_words, @category_counts, @category_word_count, @dirty, + @min_word_length] end # Custom marshal deserialization to recreate mutex # @rbs (Array[untyped]) -> void def marshal_load(data) mu_initialize - @categories, @total_words, @category_counts, @category_word_count, @dirty = data + @categories, @total_words, @category_counts, @category_word_count, @dirty, + @min_word_length = data + @min_word_length ||= Classifier.config.min_word_length @cached_training_count = nil @cached_vocab_size = nil @storage = nil diff --git a/lib/classifier/lsi.rb b/lib/classifier/lsi.rb index 3476cd90..8dcd1f26 100644 --- a/lib/classifier/lsi.rb +++ b/lib/classifier/lsi.rb @@ -348,7 +348,7 @@ def highest_relative_content(max_chunks = 10) avg_density = {} @items.each_key { |x| avg_density[x] = proximity_array_for_content_unlocked(x).sum { |pair| pair[1] } } - avg_density.keys.sort_by { |x| avg_density[x] }.reverse[0..(max_chunks - 1)].map + avg_density.keys.sort_by { |x| avg_density[x] }.reverse[0..(max_chunks - 1)] end end @@ -480,8 +480,8 @@ def highest_ranked_stems(doc, count = 3) raise 'Requested stem ranking on non-indexed content!' unless @items[doc] arr = node_for_content_unlocked(doc).lsi_vector.to_a - top_n = arr.sort.reverse[0..(count - 1)] - top_n.collect { |x| @word_list.word_for_index(arr.index(x)) } + top_indices = arr.each_index.sort_by { |index| -arr[index] }.first(count) + top_indices.collect { |index| @word_list.word_for_index(index) } end end diff --git a/test/bayes/bayesian_test.rb b/test/bayes/bayesian_test.rb index f59d1f68..8ea83f2c 100644 --- a/test/bayes/bayesian_test.rb +++ b/test/bayes/bayesian_test.rb @@ -634,4 +634,35 @@ def test_loaded_classifier_can_continue_training assert_equal 'Uninteresting', loaded.classify('boring content') end end + + def test_marshal_round_trip_can_classify + @classifier.train_interesting 'here are some good words' + @classifier.train_uninteresting 'here are some bad words' + + loaded = Marshal.load(Marshal.dump(@classifier)) + + assert_equal 'Interesting', loaded.classify('good') + end + + def test_marshal_round_trip_keeps_min_word_length + classifier = Classifier::Bayes.new('Interesting', 'Uninteresting', min_word_length: 2) + classifier.train_interesting 'go db ok' + classifier.train_uninteresting 'be at up' + + loaded = Marshal.load(Marshal.dump(classifier)) + + assert_equal 2, loaded.instance_variable_get(:@min_word_length) + assert_equal 'Interesting', loaded.classify('go db') + end + + def test_marshal_load_tolerates_a_payload_without_min_word_length + @classifier.train_interesting 'here are some good words' + legacy = @classifier.marshal_dump[0, 5] + + loaded = Classifier::Bayes.allocate + loaded.marshal_load(legacy) + + assert_equal Classifier.config.min_word_length, + loaded.instance_variable_get(:@min_word_length) + end end diff --git a/test/docs/documentation_test.rb b/test/docs/documentation_test.rb new file mode 100644 index 00000000..2fd13cc9 --- /dev/null +++ b/test/docs/documentation_test.rb @@ -0,0 +1,437 @@ +require_relative '../test_helper' + +# Executes the code examples published in README.md and docs/*.md. +# +# Every assertion mirrors a value printed in the documentation. A failure here +# means the docs claim something the code no longer does. Fix the docs or the +# code, never the assertion alone. +module Docs + class DocumentationTest < Minitest::Test + def setup + @tmpdir = Dir.mktmpdir + end + + def teardown + FileUtils.remove_entry(@tmpdir) if @tmpdir && File.exist?(@tmpdir) + end + + # --- README.md and docs/bayes.md --------------------------------------- + + def test_bayes_quick_start + classifier = Classifier::Bayes.new(:spam, :ham) + classifier.train(spam: 'Buy viagra cheap pills now') + classifier.train(spam: 'You won million dollars prize') + classifier.train(ham: ['Meeting tomorrow at 3pm', 'Quarterly report attached']) + + assert_equal 'Spam', classifier.classify('Cheap pills!') + assert_equal %w[Spam Ham], classifier.categories + end + + def test_bayes_classifications_are_negative_log_probabilities + classifier = trained_bayes + scores = classifier.classifications('Cheap pills!') + + assert_equal %w[Spam Ham], scores.keys + assert_operator scores['Spam'], :>, scores['Ham'] + assert_operator scores['Spam'], :<, 0 + end + + def test_bayes_dynamic_training_methods + classifier = Classifier::Bayes.new(:spam, :ham) + classifier.train_spam('cheap pills') + classifier.train_ham('meeting tomorrow') + + assert_equal 'Spam', classifier.classify('pills') + end + + def test_bayes_category_management + classifier = trained_bayes + + classifier.add_category(:other) + + assert_equal %w[Spam Ham Other], classifier.categories + + classifier.remove_category(:other) + + assert_equal %w[Spam Ham], classifier.categories + end + + def test_bayes_accepts_an_array_of_categories + assert_equal %w[Spam Ham], Classifier::Bayes.new(%i[spam ham]).categories + end + + # --- README.md and docs/logistic-regression.md ------------------------- + + def test_logistic_regression_requires_fit_before_classify + classifier = untrained_logistic_regression + + assert_raises(Classifier::NotFittedError) { classifier.classify('I love it!') } + end + + def test_logistic_regression_quick_start + classifier = untrained_logistic_regression + classifier.fit + + assert_predicate classifier, :fitted? + assert_equal 'Positive', classifier.classify('I love it!') + end + + def test_logistic_regression_probabilities_sum_to_one + classifier = untrained_logistic_regression + classifier.fit + probabilities = classifier.probabilities('I love it!') + + assert_in_delta 1.0, probabilities.values.sum + assert_operator probabilities['Positive'], :>, probabilities['Negative'] + end + + def test_logistic_regression_weights_rank_by_absolute_value + classifier = untrained_logistic_regression + classifier.fit + weights = classifier.weights('positive') + + assert_includes weights.keys, :amaz + assert_operator weights[:love], :>, 0 + assert_operator weights[:hate], :<, 0 + + magnitudes = weights.values.map(&:abs) + + assert_equal magnitudes.sort.reverse, magnitudes + end + + def test_logistic_regression_weights_limit_caps_the_count + classifier = untrained_logistic_regression + classifier.fit + + assert_equal 3, classifier.weights('positive', limit: 3).size + end + + def test_logistic_regression_survives_a_round_trip_without_refitting + classifier = untrained_logistic_regression + classifier.fit + path = File.join(@tmpdir, 'lr.json') + classifier.save_to_file(path) + + loaded = Classifier::LogisticRegression.load_from_file(path) + + assert_equal 'Positive', loaded.classify('I love it!') + end + + # --- README.md and docs/lsi.md ----------------------------------------- + + def test_lsi_quick_start + lsi = Classifier::LSI.new + lsi.add(dog: 'dog puppy canine bark fetch', cat: 'cat kitten feline meow purr') + + assert_equal 'dog', lsi.classify('My puppy barks') + assert_equal ['dog', 1.0], lsi.classify_with_confidence('My puppy barks') + end + + def test_lsi_search_returns_documents_ranked_by_similarity + lsi = Classifier::LSI.new + lsi.add(dog: 'dog puppy canine bark fetch', cat: 'cat kitten feline meow purr') + + assert_equal 'dog puppy canine bark fetch', lsi.search('puppy', 2).first + assert_equal 2, lsi.items.size + end + + def test_lsi_manual_index_build + lsi = Classifier::LSI.new(auto_rebuild: false) + lsi.add(dog: 'dog puppy canine bark fetch') + lsi.add(cat: 'cat kitten feline meow purr') + lsi.build_index + + refute_predicate lsi, :needs_rebuild? + end + + def test_lsi_incremental_mode_needs_manual_index_control + lsi = Classifier::LSI.new(incremental: true, auto_rebuild: false, max_rank: 100) + lsi.add(tech: incremental_corpus) + lsi.build_index + + assert_predicate lsi, :incremental_enabled? + + lsi.add(tech: 'Go is a fast compiled language for backend systems') + + assert_predicate lsi, :incremental_enabled? + assert_equal 5, lsi.current_rank + end + + def test_lsi_incremental_mode_never_starts_under_auto_rebuild + lsi = Classifier::LSI.new(incremental: true) + lsi.add(tech: incremental_corpus) + lsi.build_index + + refute_predicate lsi, :incremental_enabled? + end + + def test_lsi_highest_ranked_stems_are_distinct + lsi = animal_lsi + + assert_equal %i[dog puppi canin], + lsi.highest_ranked_stems('dog puppy canine bark fetch loyal', 3) + end + + def test_lsi_highest_relative_content_returns_documents + result = animal_lsi.highest_relative_content(2) + + assert_kind_of Array, result + assert_equal 2, result.size + end + + def test_lsi_shovel_adds_without_a_category + lsi = animal_lsi + + assert_equal 3, lsi.items.size + end + + def test_lsi_add_batch + lsi = Classifier::LSI.new + lsi.add_batch(tech: ['Ruby is elegant', 'Python is popular'], + sports: ['soccer goal', 'basketball hoop']) + + assert_equal 4, lsi.items.size + end + + def test_bayes_append_category_is_an_alias + classifier = Classifier::Bayes.new(:spam) + classifier.append_category(:ham) + + assert_equal %w[Spam Ham], classifier.categories + end + + def test_lsi_backend_is_native_or_ruby + assert_includes %i[native ruby], Classifier::LSI.backend + end + + def test_string_summary + text = 'The dog barks loudly. The cat sleeps quietly. ' \ + 'Birds sing sweetly in the morning light.' + + assert_equal 'The cat sleeps quietly.', text.summary(1) + end + + # --- README.md and docs/knn.md ----------------------------------------- + + def test_knn_quick_start + knn = trained_knn + + assert_equal 'tech', knn.classify('programming code') + assert_equal %w[tech sports], knn.categories + end + + def test_knn_classify_with_neighbors + result = trained_knn.classify_with_neighbors('programming code') + + assert_equal 'tech', result[:category] + assert_equal 3, result[:neighbors].size + assert_equal({ 'tech' => 2.0, 'sports' => 1.0 }, result[:votes]) + assert_in_delta 2.0 / 3.0, result[:confidence] + end + + def test_knn_train_is_an_alias_of_add + knn = Classifier::KNN.new(k: 1) + knn.train(tech: 'compiler') + + assert_equal ['tech'], knn.categories + end + + def test_knn_exposes_and_updates_k + knn = Classifier::KNN.new(k: 3) + + assert_equal 3, knn.k + + knn.k = 5 + + assert_equal 5, knn.k + end + + # --- README.md and docs/tfidf.md --------------------------------------- + + def test_tfidf_quick_start + tfidf = Classifier::TFIDF.new + tfidf.fit(['Ruby is great', 'Python is great', 'Ruby on Rails']) + + assert_equal({ rubi: 1.0 }, tfidf.transform('Ruby programming')) + assert_predicate tfidf, :fitted? + end + + def test_tfidf_raises_before_fit + assert_raises(Classifier::NotFittedError) { Classifier::TFIDF.new.transform('Ruby') } + end + + def test_tfidf_ngram_feature_names + tfidf = Classifier::TFIDF.new(ngram_range: [1, 2]) + tfidf.fit(['machine learning rocks', 'machine learning is fun']) + + assert_equal %i[fun learn learn_fun learn_rock machin machin_learn], + tfidf.feature_names.sort.first(6) + end + + def test_tfidf_exposes_document_frequency_bounds + tfidf = Classifier::TFIDF.new(min_df: 2, max_df: 0.85) + + assert_equal 2, tfidf.min_df + assert_in_delta 0.85, tfidf.max_df + end + + def test_tfidf_fit_from_stream_counts_each_line_as_a_document + tfidf = Classifier::TFIDF.new + tfidf.fit_from_stream(Classifier::Streaming::MultiIO.new(corpus_paths)) + + assert_equal 4, tfidf.num_documents + end + + # --- docs/configuration.md --------------------------------------------- + + def test_word_hash_returns_stemmed_counts + assert_equal({ rubi: 1, program: 1, eleg: 1 }, 'Ruby programming is elegant'.word_hash) + end + + def test_stem_to_word_hash_maps_stems_to_whole_words + mapping = 'Ruby programming is elegant and programming rocks'.stem_to_word_hash + + assert_equal({ rubi: 'ruby', program: 'programming', eleg: 'elegant', rock: 'rocks' }, mapping) + end + + def test_default_min_word_length + assert_equal 3, Classifier.config.min_word_length + end + + def test_error_hierarchy + assert_operator Classifier::NotFittedError, :<, Classifier::Error + assert_operator Classifier::UnsavedChangesError, :<, Classifier::Error + assert_operator Classifier::StorageError, :<, Classifier::Error + end + + # --- docs/persistence.md ----------------------------------------------- + + def test_save_and_load_a_file + path = File.join(@tmpdir, 'model.json') + trained_bayes.save_to_file(path) + + assert_equal 'Spam', Classifier::Bayes.load_from_file(path).classify('pills') + end + + def test_file_storage_backend + classifier = trained_bayes + classifier.storage = Classifier::Storage::File.new(path: File.join(@tmpdir, 'model.json')) + classifier.save + + loaded = Classifier::Bayes.load(storage: classifier.storage) + + assert_equal 'Spam', loaded.classify('pills') + end + + def test_memory_storage_backend + storage = Classifier::Storage::Memory.new + classifier = Classifier::Bayes.new(:a, :b) + classifier.train(a: 'alpha', b: 'beta') + classifier.storage = storage + classifier.save + + assert_equal %w[A B], Classifier::Bayes.load(storage: storage).categories + end + + def test_storage_base_interface + assert_equal %i[delete exists? read write], + Classifier::Storage::Base.instance_methods(false).sort + end + + def test_marshal_round_trip + restored = Marshal.load(Marshal.dump(trained_bayes)) + + assert_equal 'Spam', restored.classify('pills') + end + + # --- docs/streaming.md ------------------------------------------------- + + def test_train_from_stream_with_progress + classifier = Classifier::Bayes.new(:spam, :ham) + seen = [] + classifier.train_from_stream(:spam, File.open(corpus_paths.first)) do |progress| + seen << progress.completed + end + + assert_equal %w[Spam Ham], classifier.categories + refute_empty seen + end + + def test_multi_io_reads_paths_in_order + lines = Classifier::Streaming::MultiIO.new(corpus_paths).each_line.to_a + + assert_equal ["cheap pills now\n", "buy viagra cheap\n", + "meeting tomorrow\n", "quarterly report\n"], lines + end + + def test_multi_io_accepts_io_objects + lines = [] + Classifier::Streaming::MultiIO.new([File.open(corpus_paths.first)]).each_line { |l| lines << l } + + assert_equal ["cheap pills now\n", "buy viagra cheap\n"], lines + end + + def test_multi_io_returns_an_enumerator_without_a_block + assert_instance_of Enumerator, Classifier::Streaming::MultiIO.new(corpus_paths).each_line + end + + def test_train_batch_accepts_an_array + classifier = Classifier::Bayes.new(:spam, :ham) + classifier.train_batch(:spam, ['cheap pills', 'you won a prize']) + + assert_equal 'Spam', classifier.classify('pills') + end + + private + + def trained_bayes + classifier = Classifier::Bayes.new(:spam, :ham) + classifier.train(spam: 'Buy viagra cheap pills now') + classifier.train(spam: 'You won million dollars prize') + classifier.train(ham: ['Meeting tomorrow at 3pm', 'Quarterly report attached']) + classifier + end + + def untrained_logistic_regression + classifier = Classifier::LogisticRegression.new(:positive, :negative) + classifier.train(positive: 'love amazing great wonderful') + classifier.train(negative: 'hate terrible awful bad') + classifier + end + + def trained_knn + knn = Classifier::KNN.new(k: 3) + %w[laptop coding software developer programming].each { |w| knn.add(tech: w) } + %w[football basketball soccer goal team].each { |w| knn.add(sports: w) } + knn + end + + def animal_lsi + lsi = Classifier::LSI.new + lsi.add(dog: 'dog puppy canine bark fetch loyal', + cat: 'cat kitten feline meow purr independent') + lsi << 'bird sparrow robin fly nest feather' + lsi + end + + def incremental_corpus + [ + 'Ruby is an elegant programming language for web development', + 'Python is a popular programming language for data science', + 'JavaScript runs in browsers and powers modern web applications', + 'Java is a compiled language used for enterprise backend systems', + 'Rust provides memory safety without a garbage collector runtime' + ] + end + + def corpus_paths + @corpus_paths ||= begin + a = File.join(@tmpdir, 'a.txt') + b = File.join(@tmpdir, 'b.txt') + File.write(a, "cheap pills now\nbuy viagra cheap\n") + File.write(b, "meeting tomorrow\nquarterly report\n") + [a, b] + end + end + end +end diff --git a/test/lsi/lsi_test.rb b/test/lsi/lsi_test.rb index 08f2977f..5c37e1ac 100644 --- a/test/lsi/lsi_test.rb +++ b/test/lsi/lsi_test.rb @@ -270,6 +270,33 @@ def test_keyword_search assert_equal %i[dog text deal], lsi.highest_ranked_stems(@str1) end + def test_highest_ranked_stems_returns_distinct_stems + lsi = Classifier::LSI.new + lsi.add(dog: 'dog puppy canine bark fetch loyal', cat: 'cat kitten feline meow purr independent') + lsi.add_item 'bird sparrow robin fly nest feather', 'Bird' + + stems = lsi.highest_ranked_stems('dog puppy canine bark fetch loyal', 3) + + assert_equal 3, stems.size + assert_equal stems.uniq, stems + end + + def test_highest_relative_content_returns_an_array + lsi = Classifier::LSI.new + lsi.add_item @str1, 'Dog' + lsi.add_item @str2, 'Dog' + lsi.add_item @str3, 'Cat' + + result = lsi.highest_relative_content(2) + + assert_kind_of Array, result + assert_equal 2, result.size + end + + def test_highest_relative_content_is_empty_before_a_build + assert_empty Classifier::LSI.new.highest_relative_content(2) + end + def test_summary summary = [@str1, @str2, @str3, @str4, @str5].join.summary(2) # Summary should contain 2 sentences separated by [...]