Fix size_hint underflow on empty-string sentence iterators - #181
Merged
Manishearth merged 1 commit intoSep 1, 2026
Merged
Conversation
USentenceBounds::size_hint subtracted 1 from the inner iterator's lower bound, which is 0 for an empty string, so size_hint() on any of the three public sentence iterators panicked with 'attempt to subtract with overflow' in debug builds and returned a lower bound of usize::MAX in release builds. The cmp::max(0, ...) wrappers around the expression are no-ops on usize and hid the underflow. Both bounds now use saturating_sub(1). The regression test drives size_hint on every public iterator of the crate over a corpus led by the empty string and asserts lower <= yielded <= upper in both profiles.
Manishearth
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
size_hint()on any of the three public sentence iterators panics on the empty string in debug builds:In release builds the same calls return
(18446744073709551615, Some(0))- a lower bound above the upper bound, which theIterator::size_hintcontract forbids. A consumer that passes the lower bound toVec::with_capacitygets a capacity-overflow panic; that appears to be exactly what the reporter of #146 ran into in a comment on that thread ("something in that logic managed to summon a value that madeVec::with_capacity()panic with a Capacity overflow by just passing the lower bound unaltered") - the issue was closed on theminmisreading, but this underflow was live underneath it.Cause:
USentenceBounds::size_hintsubtracts 1 from the inner iterator's lower bound, and that bound is 0 for an empty string. The subtraction is onusize, and thecmp::max(0, ...)wrappers around both bounds are no-ops on an unsigned type that read as saturation while the underflow sat inside them.Fix: both bounds use
saturating_sub(1); the wrappers and the now-unused top-leveluse core::cmpare removed. For every non-empty input the arithmetic is unchanged, sincecmp::max(0, x - 1)andx.saturating_sub(1)agree whereverx >= 1; only the empty-string case moves, from a panic (debug) or a wrapped bound (release) to(0, Some(0)).Test: the new
test_size_hint_is_a_valid_bounddrivessize_hinton all ten public iterators over a corpus led by the empty string plus the crate's own test tables, assertinglower <= yielded <= upper. On master it fails with the debug panic above; with the fix,cargo test,cargo fmt --check,cargo clippy --all-targets --allandcargo +1.85.0 test(the MSRV job's command) all pass.One related observation, kept out of this PR to keep it minimal: the inner
SentenceBreaks::size_hintderives its bounds from the whole string's length rather than the unconsumed remainder, so the outer hint never shrinks as items are yielded (a lower bound of 1 remains after the last item). Happy to follow up separately if that is of interest.Provenance, for transparency: this defect was found by an automated audit loop I run against open-source projects; the patch and its verification were reviewed by me before filing, and I am happy to answer any questions during review.