feat: add support for GBNF grammar - #20
Open
oglego wants to merge 2 commits into
Open
Conversation
fix: update comment on GBNF grammar
fix: update comments in model.h for GBNF grammar
test: update test suite for GBNF grammar support
refactor: clean up grammar test formatting
feat: update GBNF grammar support with example and reset fix
Update GBNF grammar support to ModelConfig via a `grammar` string and
`grammar_root` rule name, applied to the sampler chain in
initialize_context(). load_grammar_file() reads a .gbnf file into
ModelConfig::grammar.
The grammar sampler retains its parse position across calls to
generate_from_tokens(), so a completed grammar from one turn forces
EOS immediately on the next. Reset only the grammar sampler (not the
whole chain) at the start of each turn, via a non-owning pointer
kept on Model, so the dist sampler's RNG and other stateful samplers
are left untouched when an explicit seed is configured.
Add examples/grammar, a minimal sentiment classifier constrained to
{"sentiment": "positive"|"negative"|"neutral"} via sentiment.gbnf,
demonstrating the feature end-to-end. Document grammar-constrained
output in the README and register the example in CMakeLists.txt.
fix: allow custom grammar roots in grammar example
Add a -r flag to the grammar demo, update the README usage examples,
and polish the grammar documentation wording.
fix: comment formatting in model.cpp
fix: update formatting for comments in model.h
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR partially addresses #18 - adding support for GBNF grammar. I believe that I have everything for this wired in correctly, I also tried to follow the current coding style as best as I could, but if anything needs to be changed please let me know.
Changes
The main items that I have added on this PR are:
ModelConfig gains grammar (GBNF string) and grammar_root (rule name, defaults to "root").
load_grammar_file() reads a .gbnf file into ModelConfig::grammar.
initialize_context() applies the grammar to the sampler chain when set.
examples/grammar/ — minimal sentiment classifier demonstrating grammar-constrained output.
tests/test_grammar.cpp — config defaults, file loading, missing-file error handling.
For testing this locally and interacting with it I just used the recommended model (granite-4.0-micro-Q8_0.gguf).
Thanks in advance for the review!