Return Long from count() instead of silently narrowing to Integer - #16323
Return Long from count() instead of silently narrowing to Integer#16323codeconsole wants to merge 5 commits into
Conversation
9f921fd to
cdeffcf
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 8.0.x #16323 +/- ##
==================================================
- Coverage 55.0529% 55.0410% -0.0118%
Complexity 20773 20773
==================================================
Files 2110 2110
Lines 101368 101368
Branches 18005 18005
==================================================
- Hits 55806 55794 -12
- Misses 37517 37532 +15
+ Partials 8045 8042 -3
🚀 New features to boost your workflow:
|
|
Under compilestatic code wouldn't this be considered a breaking change since there wouldn't be type coercion? Would it then be a better change to return a type that extends Number, ie: |
|
You're right that it's a breaking change for statically compiled callers — that's intentional, and
The generic signature does remove the compile error, but it doesn't add coercion — it erases to
So there's no non-breaking option here: any type wide enough to hold what the datastores actually return is unassignable to Worth noting where this actually bites: MongoDB aggregates counts with |
|
We discussed this in the weekly and there was agreement the breaking change is acceptable for 8.0. Reviews can proceed now. |
jdaugherty
left a comment
There was a problem hiding this comment.
I think we should keep this review focused on count() if we're only going to partially implement the Long in the pagination code. If we fully implement Long in the pagination, then I'm fine with updating the paging logic
| not an error in Groovy, so the probe changed nothing it could observe. An application that relied on the | ||
| import being *omitted* when the package was absent sees no difference in compiled output. | ||
|
|
||
| ==== 54. `count()` Returns `Long` |
There was a problem hiding this comment.
This file conflicts with 8.0.x on merge, and 8.0.x now numbers sections through 57, so after a rebase this becomes §58.
There was a problem hiding this comment.
Rebased onto current 8.0.x — it's §58 now.
Separately: 8.0.x already has ==== 54. twice, Request Processing Behaviour Changes (3256) and Non-Public Bean Classes Are Marshalled, and Reported Once (3454). Left both alone since neither is mine, but you may want to renumber the second.
There was a problem hiding this comment.
Can you go ahead and fix the other numbering problem please? @codeconsole
There was a problem hiding this comment.
Done in af45fc2 — should have just fixed it rather than pointing at it, sorry.
"Non-Public Bean Classes Are Marshalled, and Reported Once" is §58 and count() moves to §59. Top-level sections now run 1–59 with no duplicates and no gaps.
| n == 0L | ||
| } | ||
|
|
||
| void "count() returns a Long so a large table is not truncated"() { |
There was a problem hiding this comment.
The name promises the truncation case, but the body only checks declared types. Nothing pushes a value past Integer.MAX_VALUE through the longValue() branch this PR changed, which is the behaviour that was broken. Stubbing the session gets at it directly, and this passes on the branch (needs org.grails.datastore.mapping.query.Query imported):
void "count() preserves a datastore count above Integer.MAX_VALUE"() {
given:
def ds = Stub(Datastore)
def session = Stub(Session)
def query = Mock(Query)
ds.getMappingContext() >> datastore.mappingContext
ds.connect() >> session
session.getDatastore() >> ds
session.createQuery(GormStaticApiThing) >> query
query.projections() >> Mock(Query.ProjectionList)
query.singleResult() >> 3_000_000_000L
def api = new GormStaticApi(GormStaticApiThing, ds, [])
expect:
api.count() == 3_000_000_000L
}Worth also asserting the trait path, since that is what domain classes actually expose: GormStaticApiThing.count() instanceof Long and GormStaticApiThing.count instanceof Long both hold against the SimpleMapDatastore this spec already has. The two getMethod(...).returnType == Long lines can go once the behaviour is covered; they restate the signature.
There was a problem hiding this comment.
Taken as written, plus GormStaticApiThing.count() / .count for the trait path, and the two getMethod(...).returnType lines are gone.
Checked it isn't vacuous the way the old one was: reverting longValue() to intValue() makes it fail, restoring it makes it pass.
|
|
||
| final String countFieldName = namingConvention.getCount(entity) | ||
| final GraphQLOutputType countOutputType = (GraphQLOutputType) typeManager.getType(Integer) | ||
| final GraphQLOutputType countOutputType = (GraphQLOutputType) typeManager.getType(Long) |
There was a problem hiding this comment.
Nothing asserts the type the count field is built with; ReadOnlyOpSpec only checks the field exists. A type == ExtendedScalars.GraphQLLong assertion next to that check (or in SchemaSpec) would pin this.
Also worth a sentence in the upgrade note: paginated list responses already type totalCount as Long through the same type manager (DefaultGraphQLPaginationResponseHandler), so the count field now matches an existing scalar in the schema rather than introducing a new one.
There was a problem hiding this comment.
Added — in SchemaSpec rather than ReadOnlyOpSpec, because the only test in the latter is @Ignored, so an assertion there would never execute. It asserts every *Count query field is ExtendedScalars.GraphQLLong, with a non-empty guard so it can't pass by matching nothing.
Added the totalCount sentence to the upgrade note too — confirmed at DefaultGraphQLPaginationResponseHandler:52, which already builds that field with typeManager.getType(Long).
| * @param total The total number of objects to be paginated | ||
| */ | ||
| void paginate(Object object, Integer total) | ||
| void paginate(Object object, Number total) |
There was a problem hiding this comment.
The paginate options seem additional to this review, Number allows BigDecimal, why aren't we making this a Long too?
There was a problem hiding this comment.
Agreed — Number admits BigDecimal and says less than the code knows. All five paginate overloads take Long now, and so does links(Map, Object, Long total), which was pre-existing but truncating for the same reason.
| */ | ||
| //TODO: Once GROOVY-9662 is fixed, remove explicit delegate call and typecast to StreamingJsonDelegate | ||
| void paginate(Object object, Integer total, Integer offset = null, Integer max = null, String sort = null, String order = null) { | ||
| void paginate(Object object, Number total, Integer offset = null, Integer max = null, String sort = null, String order = null) { |
There was a problem hiding this comment.
Why aren't we changing this to Long?
There was a problem hiding this comment.
Done — Long total.
| ((StreamingJsonBuilder.StreamingJsonDelegate) delegate).call(TYPE_ATTRIBUTE, contentTypeMimeType ?: contentType) | ||
| } | ||
| List<Link> links = getPaginationLinks(object, total, max, offset, sort, order) as List<Link> | ||
| List<Link> links = getPaginationLinks(object, total?.intValue(), max, offset, sort, order) as List<Link> |
There was a problem hiding this comment.
Why are we forcing this to intValue() and not converting it to long too?
There was a problem hiding this comment.
Gone. getPaginationLinks takes Long, and so do the offsets derived from a total — getNextOffset, getLastOffset, getPrevOffset, paramsWithOffset, buildPaginateParams.
The offsets mattered as much as the total: getLastOffset computes (ceil(total / max) - 1) * max, so the offset overflows the moment the total stops fitting in an Int. Parameters already exposed long(name, default) for the request path.
4d95dc6 to
38f736e
Compare
GormStaticOperations declared `Integer count()`, but the datastore has always produced a Long: COUNT(*) is a 64-bit value in SQL. Every backend narrowed the result to satisfy the declared type, and each did it differently — GormStaticApi called intValue(), Hibernate 5 cast the Number, Hibernate 7 coerced with `as Integer`. intValue() truncates silently, so a table with more than Integer.MAX_VALUE rows reported a wrong and possibly negative count without failing. Declare what the query actually returns. RestfulServiceController loses its Math.toIntExact call, because the service it delegates to already returned a Long and was narrowing only to fit the controller signature. Dynamic Groovy is unaffected: `Book.count() == 5` and `int n = Book.count()` still work, because Groovy converts and compares numeric types by value. Statically compiled callers and anything overriding these methods need updating, so this is documented in the 8.0 upgrade notes. Typed views bind their model by reflective Field.set with no coercion, in GSP and in JSON views alike, so a view naming a concrete numeric type fails when the other scaffolding path supplies the other type. The scaffolded index.gsp and the two product index.gson views now declare Number, and HalViewHelper.paginate accepts a Number total so a count can reach it without being narrowed again on the way in. The scaffolded index.gsp change also appears in the fix for the Integer declaration it previously carried; the two resolve to the same line.
CountEntityDataFetcher declared Integer and returned staticApi.count()
directly, so it narrowed the same way the datastore APIs did. The class is
@CompileStatic, so once count() returns Long this no longer compiles:
[Static type checking] - Possible loss of precision
from java.lang.Long to java.lang.Integer
The schema field was typed with typeManager.getType(Integer), and GraphQL's
Int is 32 bits by specification, so counts above Integer.MAX_VALUE were
truncated here as well. Type the field with Long, which the type manager
maps to ExtendedScalars.GraphQLLong.
This changes the published schema from `personCount: Int` to
`personCount: Long`. Queries selecting the field and the JSON they return
are unchanged, but generated clients and schema snapshots need
regenerating, so it is called out in the upgrade notes.
The note claimed the datastore always produced a Long. It does not.
Hibernate counts with SQL COUNT(*) and Neo4j with Cypher count(*), both
64-bit, but MongoDB aggregates with {$sum: 1}, which returns Int32 and
promotes to Int64 once the total exceeds it, and the in-memory datastore
returns a list size. GormStaticApi normalises all of them with
longValue(), so the code was already correct; only the explanation was
wrong. Long is the right declaration because it is the only type that
holds every value a datastore can produce, not because every datastore
already produced one.
Renumber the upgrade note to 58, since 8.0.x has since used 54 through 57, and fix the Java example: the trait's static methods land on the domain class, so the caller writes Book.count(), and `long total = Book.count()` shows the auto-unboxing directly. Replace the count spec with one that exercises the behaviour instead of restating the signature. It stubs a datastore count of 3_000_000_000 and asserts the value survives, which fails if longValue() goes back to intValue(), and covers the trait members a domain class actually exposes. Assert the GraphQL count scalar, in SchemaSpec rather than ReadOnlyOpSpec because the only test in the latter is @ignore'd and would never run. Note in the upgrade note that paged results already type totalCount as Long through the same type manager. Carry Long through the JSON view pagination rather than converting on the way in: paginate, links, getPaginationLinks, and the offsets derived from a total. An offset computed from a total past 2^31 would otherwise overflow the moment the total stopped doing so. Parameters already exposed long(name, default) for the request path, and the two product index.gson views now declare the Long their controller supplies.
38f736e to
04f871c
Compare
Two sections were numbered 54: "Request Processing Behaviour Changes" and "Non-Public Bean Classes Are Marshalled, and Reported Once", the latter having been appended after 55 through 57 already existed. Number it 58, and move the count() section that followed it to 59. The top-level sections now run 1 to 59 with no duplicates and no gaps.
| int offset = params.int(PAGINATION_OFFSET, 0) | ||
| protected List<Link> getPaginationLinks(Object object, Long total, Parameters params) { | ||
| long offset = params.long(PAGINATION_OFFSET, 0L) | ||
| int max = params.int(PAGINATION_MAX, 10) |
There was a problem hiding this comment.
Shouldn't max be a long too?
There was a problem hiding this comment.
No — max is a page size, and GORM takes it as an int at the source (Query.max(int), Query.maxResults(int)). total and offset are positions in the result set, which is what can exceed 2^31.
Every expression mixing them already widens through the Long operand — offset + max, Math.max(offset - max, 0L), laststep * max — so nothing truncates.
🚨 TestLens detected 1 failed test 🚨Here is what you can do:
Test SummaryCI / Functional Tests (Java 21, indy=true) > :grails-test-examples-scaffolding:integrationTest
🏷️ Commit: af45fc2 Test FailuresUserControllerSpec > User list (:grails-test-examples-scaffolding:integrationTest in CI / Functional Tests (Java 21, indy=true))UserControllerSpec > User list (:grails-test-examples-scaffolding:integrationTest in CI / Functional Tests (Java 21, indy=true) | Attempt 1/3)Rerun ControlsSelect tests to mute in this pull request:
Reuse successful test results:
Click the checkbox to trigger a rerun:
Learn more about TestLens at testlens.app/docs. |
AI ReviewThanks, the direction is right and the core change is complete: every A few things to address before merge. 1. The JSON API pagination path still truncates
Integer total = (Integer) paginationArgs.get(PAGINATION_TOTAL)
...
List<Link> links = getPaginationLinks(resource, total, params)Under Long total = ((Number) paginationArgs.get(PAGINATION_TOTAL)).longValue()and a row in 2.
|
|
@codeconsole looks like Mattias found some more ints for you to change =) I'm good to merge once these are changed. |
Description
count()returnsLonginstead ofInteger, matching what the datastore actually produces.The problem
GormStaticOperationsdeclares:How wide the value underneath actually is depends on the datastore:
COUNT(*)— 64-bitcount(*)— 64-bit{$sum: 1}— Int32, promoting to Int64 once the total exceeds itentityList.size()—intGormStaticApinormalises all of them withlongValue(), and the declared return type then narrowed the result again. Every site did that narrowing a different way:GormStaticApi((Number)res).intValue()CriteriaQuery<Long>, then casts(Integer)select count(*) ... as IntegerCountEntityDataFetcherInteger, so the returnedLongis narrowed on returnInt, which is 32-bit by GraphQL specificationMongoDB and Neo4j need no change of their own: neither overrides
count(), so both inheritGormStaticApiand are fixed by the core change. MongoDB is the sharpest case for declaringLong, sincethe width it returns depends on how many documents are in the collection — a collection that grows past
2³¹ starts returning Int64, which is the correct answer, and
intValue()then wrapped it.intValue()truncates silently, so a table with more thanInteger.MAX_VALUErows reported a wrong and possibly negative count with no error at all.Longis the only type that holds every value a datastore can produce here, so declaring it removes the narrowing rather than moving it around.RestfulServiceControllergets simpler as a direct consequence — it was callingMath.toIntExact(getService().count(params))on a service that already returned aLong, narrowing purely to fit the controller signature. That call is gone.Scope
GormStaticOperations.count()/getCount(), thecount()/countmembers generated on every domain class viaGormEntity,TenantDelegatingGormOperations, both Hibernate backends, andRestfulController.countResources().Plus the GraphQL count fetcher and its schema field, which changes the published schema from
personCount: InttopersonCount: Long(graphql-javamapsLongtoExtendedScalars.GraphQLLong). Queries selecting the field and the JSON they return are unchanged, but generated clients and schema snapshots need regenerating.grails-datamapping-rxneeded no change:RxGormStaticOperations.count()already returnsObservable<Number>and never committed toInteger.Compatibility
Dynamic Groovy is unaffected.
Book.count() == 5,int n = Book.count(), and arithmetic on the result all continue to work, because Groovy converts between numeric types on assignment and compares them by value. What breaks is statically compiled code assigning the result to anInteger, Java callers relying on unboxing, and subclasses overridingcountResources(). Documented as section 54 of the 8.0 upgrade notes with before/after examples.Typed views
Worth calling out, because it is the same defect twice. Typed view models are bound by reflective
Field.setwith no coercion — in GSP (GroovyPage.applyModelFieldsFromBinding) and in JSON views (WritableScriptTemplate.FieldSetter) alike. A view that names a concrete numeric type therefore fails outright when a controller supplies the other one, and the two scaffolding paths supply different types:generate-viewspairs with a generated service declaringLong count(), whilestatic scaffold = Xis backed byRestfulController.countResources().So the scaffolded
index.gspand bothproduct/index.gsonviews now declareNumber, which accepts whatever a controller supplies.HalViewHelper.paginateaccordingly takes aNumber total(converted once at the internalgetPaginationLinksboundary) so a count can reach it without being narrowed again on the way in.The
paginatesignature change is the one piece here that is not strictly aboutcount(). It is included because without it the in-tree JSON views cannot compile against aLongcount. Happy to split it out if reviewers prefer.Relationship to #16322
#16322 changes the scaffolded
index.gspcount field toNumberfor the same underlying reason. This branch is cut from8.0.x, where that file still declaresInteger, and thegrails-test-examples-gormscaffolding functional tests fail without it — so the identical one-line change is included here to keep this PR green on its own. The two resolve to the same line and should merge cleanly in either order.