From bdc6395d2ea58b21f2415dfa65b5c28a1d2de551 Mon Sep 17 00:00:00 2001 From: Alex Soffronow-Pagonidis Date: Fri, 14 Aug 2026 15:53:19 +0200 Subject: [PATCH 1/4] Translate standard DateTime members and methods (#55) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The provider registered no date/time member translator, so only a direct comparison worked: every member and method threw "The LINQ expression ... could not be translated". Add one shared translator, which serves DateTime and DateOnly because the ClickHouse function is the same for each. Components map to the to* extraction functions. Those return UInt8/UInt16, which the provider's integer mappings already widen on read. .DayOfWeek maps to toDayOfWeek(x, 2). Week mode 2 agrees with System.DayOfWeek exactly, so no arithmetic correction is applied. The result carries a number-backed enum mapping, because this provider maps a C# enum to a ClickHouse string and that mapping would otherwise render x.DayOfWeek == DayOfWeek.Sunday as a comparison against 'Sunday'. .TimeOfDay maps to toTime64(x, 7); precision 7 is one .NET tick, so the fraction survives, which toTime would drop. AddYears and AddMonths take an int, so they map straight onto addYears and addMonths, which clamp the day of month the way .NET does. The other Add* methods take a double, which .NET scales to whole ticks. The matching ClickHouse function takes a whole number of its own unit and discards the rest, so addDays(x, 1.5) would add only one day. A constant is therefore folded to ticks and expressed in the coarsest unit that holds it exactly: the natural function when possible, otherwise addMilliseconds. Preferring the natural function keeps the source's store type and keeps Date/Date32 columns working, which addMilliseconds rejects. A sub-millisecond offset, a non-constant offset, and a value outside the DateTime range are left untranslated rather than rounded to fit. addNanoseconds would express a tick exactly, but promotes the result to DateTime64(9), whose Int64 nanosecond count cannot span the DateTime64 range — that would trade a rounding error for a silently wrong date. Server-side rounding was rejected too: ClickHouse round() is banker's rounding, so it disagrees with .NET. An untranslated call still gives the correct value through client evaluation in a projection, and reports a reason in a predicate. Also report a clear reason for arithmetic on two date/time values. ClickHouse has no operator that matches the .NET result: one date minus another gives a TimeSpan while dateDiff counts whole units, Time64 subtraction gives a Decimal of seconds, and a date plus a TimeSpan is rejected outright. These used to fail with an internal cast or coercion error naming CLR types the user never wrote. Reporting the reason also restores client evaluation in a projection, where the .NET result is correct. DateTimeOffset follows in the next commit, once the store mapping this branch is stacked on is in place. The Northwind GroupJoin_aggregate_anonymous_key_selectors2 query now translates, so its "not translatable" override is removed. Co-Authored-By: Claude --- CHANGELOG.md | 14 + README.md | 58 ++ .../ClickHouseDateTimeMemberTranslator.cs | 243 +++++++ .../ClickHouseDateTimeMethodTranslator.cs | 166 ++++- .../ClickHouseMemberTranslatorProvider.cs | 1 + ...ickHouseSqlTranslatingExpressionVisitor.cs | 55 ++ .../Query/NorthwindJoinQueryClickHouseTest.cs | 5 - .../DateTimeMemberTranslationTests.cs | 602 ++++++++++++++++++ 8 files changed, 1137 insertions(+), 7 deletions(-) create mode 100644 src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs create mode 100644 test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs diff --git a/CHANGELOG.md b/CHANGELOG.md index dc29d64..917906e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,10 +3,24 @@ v0.3.1 (Unreleased) ### Query translation * **`toStartOf*` date-time functions** via `EF.Functions`: `ToStartOfYear`, `ToStartOfQuarter`, `ToStartOfMonth`, `ToStartOfWeek` (with optional week `mode`), `ToStartOfDay`, `ToStartOfHour`, `ToStartOfMinute`, `ToStartOfSecond`, the fixed buckets `ToStartOfFiveMinutes` / `ToStartOfTenMinutes` / `ToStartOfFifteenMinutes`, and the general `ToStartOfInterval(source, value, unit)`. Each maps to the matching ClickHouse function and works in `GROUP BY`. Return types follow ClickHouse: the calendar buckets (`Year`/`Quarter`/`Month`/`Week`) return `Date`, the day/hour/minute buckets return `DateTime`, and `ToStartOfSecond` returns `DateTime64`. All accept `DateTime`/`DateTime64` columns, and the plain truncation functions also accept `DateOnly`; `ToStartOfInterval` requires a `DateTime`/`DateTime64` column on older ClickHouse, which rejects a `DateOnly` (Date/Date32) source with `Illegal type Date32 of 1st argument` for every unit; recent versions accept it. `ToStartOfInterval` takes a `ClickHouseInterval` unit (`Second`…`Year`) and emits `toStartOfInterval(source, toInterval(value))`; the unit must be a constant so it can be translated. The default `Date`/`DateTime` result types only span 1970–2149/2106, so ClickHouse narrows out-of-range values — enable `enable_extended_results_for_datetime_functions` (e.g. `set_enable_extended_results_for_datetime_functions=1` in the connection string) for range-preserving `Date32`/`DateTime64` results. +* **Standard `DateTime` members and methods now translate to SQL.** Previously the provider registered no date/time member translator, so only a direct comparison worked and every member threw `The LINQ expression ... could not be translated`. One shared translator serves `DateTime` and `DateOnly`, because the ClickHouse function is the same for each. ([#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55)) + * **Components** — `.Year` → `toYear`, `.Month` → `toMonth`, `.Day` → `toDayOfMonth`, `.Hour` → `toHour`, `.Minute` → `toMinute`, `.Second` → `toSecond`, `.Millisecond` → `toMillisecond`, `.DayOfYear` → `toDayOfYear`. These ClickHouse functions return `UInt8`/`UInt16`, which the provider's integer mappings widen to `int` on read. `DateOnly` gets the date components only, matching the members it declares. + * **`.DayOfWeek`** → `toDayOfWeek(x, 2)`. Week mode 2 agrees with `System.DayOfWeek` exactly (Sunday 0 … Saturday 6), so no arithmetic correction is applied — the default mode 0 starts the week on Monday, which is why the mode argument is always sent. The result carries a number-backed enum mapping, because this provider maps a C# `enum` to a ClickHouse string and that mapping would otherwise render `x.DayOfWeek == DayOfWeek.Sunday` as a comparison against `'Sunday'`. + * **`.Date`** → `toStartOfDay`, which keeps the timezone of the source. Note that `toStartOfDay` returns a `DateTime`, whose range is 1970–2106, and ClickHouse **wraps** a value outside that window rather than reporting it — so `.Date` on a `DateTime64` column holding a pre-1970 date reads back wrong. Enable `enable_extended_results_for_datetime_functions` (for example `set_enable_extended_results_for_datetime_functions=1` in the connection string) to get a range-preserving `DateTime64` result. This is the same caveat that already applies to `EF.Functions.ToStartOfDay`. + * **`.TimeOfDay`** → `toTime64(x, 7)`; precision 7 is one .NET tick, so no part of the value is lost (`toTime` would drop the fraction). + * **`DateTime.UtcNow`** → `now64(7, 'UTC')`, **`DateTime.Now`** → `now64(7)` and **`DateTime.Today`** → `toStartOfDay(now())`. `today()` is not used for `.Today` because it returns a `Date`, whereas the member's type is `DateTime`. + * **`.AddYears(n)`** → `addYears` and **`.AddMonths(n)`** → `addMonths`. Both take an `int` in .NET, and ClickHouse clamps the day of month the same way .NET does, so `2026-01-31` plus one month gives `2026-02-28` in both. + * **`.AddDays`/`.AddHours`/`.AddMinutes`/`.AddSeconds`/`.AddMilliseconds`** take a `double` in .NET, which .NET scales to whole **ticks** (100 ns), rounding half away from zero — so `AddSeconds(0.1234567)` adds exactly 1 234 567 ticks. The matching ClickHouse function takes a whole number of its own unit and discards the rest, so `addDays(x, 1.5)` would add only one day. A constant argument is therefore folded to ticks during translation and then expressed in the coarsest unit that holds it exactly: a whole number of the unit emits the natural function (`addDays(x, 1)`), and otherwise `addMilliseconds` carries the exact count (`AddDays(1.5)` → `addMilliseconds(x, 129600000)`). Preferring the natural function keeps the store type of the source and keeps `Date`/`Date32` columns working, since `addMilliseconds` rejects those outright. + * A **sub-millisecond** offset, a **non-constant** offset, and a value **outside the `DateTime` range** are deliberately left untranslated rather than rounded to fit. Milliseconds are as fine as the translation goes, because `addNanoseconds` would express a tick exactly but promotes the result to `DateTime64(9)`, whose Int64 nanosecond count cannot span the `DateTime64` range — that would trade a rounding error for a silently wrong date. An untranslated call still gives the correct .NET value through client evaluation in a projection, and reports a clear reason in a predicate. `DateOnly.AddDays` takes an `int`, so it always emits `addDays`. + * A previously-unsupported Northwind query, `GroupJoin_aggregate_anonymous_key_selectors2`, now passes as a result of these translations; its provider-specific "not translatable" override is removed. + * Not yet translated: `.Ticks`, `.AddTicks`, and the `.Microsecond`/`.Nanosecond` members. `DateTimeOffset` is not covered yet either — it has no store mapping until [#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53), so translating its members would silently drop the offset. The translator is shaped to take the CLR type as a parameter, so it gains `DateTimeOffset` with that mapping. The ClickHouse-specific functions such as `dateDiff` and `dateTrunc` are tracked in [#58](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/58). + * **Behaviour change:** `DateTime.Now` and `DateTime.Today` in a *projection* used to be evaluated on the client; they now read the **server** clock. The value therefore follows the server's timezone rather than the client's, and comes back with `DateTimeKind.Unspecified` instead of `Local`. Use `DateTime.UtcNow` for an instant that does not depend on server configuration. In a predicate all three were untranslatable before, so nothing changes there. + ### Types * **`DateTimeOffset` is now a supported property type**, and maps to `DateTime64(7, 'UTC')`. The provider previously had no mapping for it, so EF Core fell back to `DateTimeOffsetToStringConverter` and made a `String` column with no warning — which broke queries against a real `DateTime64` column with `TYPE_MISMATCH`, and stopped `SaveChanges` from writing the value at all. The store type pins the timezone to `'UTC'` so the instant does not depend on the server's `session_timezone`, and precision 7 is one .NET tick (100 ns), so the round trip is exact. `HasPrecision(n)` is honoured for a property with no `HasColumnType`, and keeps the UTC pin. Note that ClickHouse has no type that stores a UTC offset: `DateTime64` holds an instant, and a declared timezone only decides how that instant is rendered. The instant is kept and the offset is not, so a value read back carries the offset of the column's declared timezone. A non-UTC column such as `DateTime64(6, 'Asia/Tokyo')` reads correctly, as does a fixed-offset column such as `DateTime64(7, 'Fixed/UTC+05:30:00')`, and `DateTimeOffset` composes into `Array`, `Map` and `Tuple` columns. Where an instant cannot be reproduced exactly the read throws rather than returning a value that is quietly wrong: the repeated hour when clocks go back in a zone whose candidate offsets are both non-zero (`Europe/Paris`), and a value before 1900 in a named zone, where the offset is Local Mean Time to the second. Writing a value the store type cannot hold throws for the same reason — ClickHouse wraps it rather than reporting it, which matters for precision 8 and 9, whose ranges are narrower than `DateTimeOffset`. **Behaviour change:** a `DateTimeOffset` property that relied on the old `String` column now resolves to `DateTime64(7, 'UTC')` — add `HasConversion()` to keep the previous shape. See the [DateTimeOffset section of the README](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore#datetimeoffset) for the daylight-saving and `MinValue`/`MaxValue` limits. ([#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53)) ### Bug fixes +* **Subtracting one date/time value from another no longer fails with an internal error.** `dt1 - dt2` gives a `TimeSpan`, which ClickHouse has no operator for — `dateDiff` returns a count of whole units instead. The expression used to reach type-mapping inference and fail with an `InvalidCastException` or a bare `No coercion operator is defined between types ...`, both of which name CLR types the user never wrote. The subtraction is now reported as not translatable, with the reason attached. In a projection EF Core can therefore fall back to the client and return the correct `TimeSpan`; in a predicate, where no fallback exists, the message explains why and what to do instead. ([#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55)) * `ToStartOfWeek` now rejects row-dependent week modes during query translation, and `ToStartOfInterval` likewise rejects row-dependent interval sizes. ClickHouse requires these operands to be constant for the query; literals and captured query parameters remain supported. * **Composite columns now convert their components on read.** `Array(T)`, `Map(K, V)` and `Tuple(...)` read the whole column through `GetValue`, so a component mapping's own read pipeline never ran, and any component whose CLR type differs from the driver's type threw `InvalidCastException`. This is not new with `DateTimeOffset` — `DateOnly[]`, `Dictionary` and `Tuple` were already affected, because `DateOnly` also arrives from the driver as a `DateTime`. The composite is now rebuilt component by component, with the same two steps EF Core applies to a scalar column: the mapping's data-reader conversion, then its `ValueConverter`. An `enum` component, a `List` component and a nested composite therefore all read correctly, and a component that needs no conversion keeps the direct cast. **Known limit:** *writing* a component that needs a `ValueConverter` still does not work, because the bulk insert path passes model values to the driver without applying converters ([#54](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/54)) — an `enum` inside a composite is written as its raw ordinal. * **`Array(Nullable(T))` DDL is no longer double-wrapped.** For a value-type element the store type came out as `Array(Nullable(Nullable(T)))`, which ClickHouse rejects with `Nested type Nullable(T) cannot be inside Nullable type`, so `EnsureCreated` and migrations both failed. The component mapping is resolved from a store type that already carries the wrapper, and `HasColumnType(...)` text is kept verbatim, so the nullable-element wrapper added a second one. It now adds the wrapper only when the inner store type does not already have one, including through `LowCardinality(Nullable(T))`. Reference-type elements were never affected. diff --git a/README.md b/README.md index 5371e84..a7bb99a 100644 --- a/README.md +++ b/README.md @@ -221,6 +221,64 @@ ClickHouse returns `NULL` from a scalar subquery that matches no rows, where sta ### Date/Time Functions +#### Standard members and methods + +The standard .NET date/time members translate to ClickHouse functions, for both `DateTime` and `DateOnly`: + +| .NET | ClickHouse | +| --- | --- | +| `.Year` `.Month` `.Day` | `toYear` `toMonth` `toDayOfMonth` | +| `.Hour` `.Minute` `.Second` `.Millisecond` | `toHour` `toMinute` `toSecond` `toMillisecond` | +| `.DayOfYear` | `toDayOfYear` | +| `.DayOfWeek` | `toDayOfWeek(x, 2)` | +| `.Date` | `toStartOfDay` | +| `.TimeOfDay` | `toTime64(x, 7)` | +| `.AddYears(n)` `.AddMonths(n)` | `addYears` `addMonths` | +| `.AddDays(n)` `.AddHours(n)` `.AddMinutes(n)` `.AddSeconds(n)` `.AddMilliseconds(n)` | `addDays` `addHours` … (see below) | +| `DateTime.UtcNow` | `now64(7, 'UTC')` | +| `DateTime.Now` | `now64(7)` | +| `DateTime.Today` | `toStartOfDay(now())` | + +```csharp +// Runs entirely on the server +var busyHours = await ctx.Events + .Where(e => e.Timestamp.Year == 2026 && e.Timestamp.DayOfWeek == DayOfWeek.Sunday) + .GroupBy(e => e.Timestamp.Hour) + .Select(g => new { Hour = g.Key, Count = g.Count() }) + .ToListAsync(); + +var recent = await ctx.Events + .Where(e => e.Timestamp > DateTime.UtcNow.AddDays(-7)) + .ToListAsync(); +``` + +`DateOnly` gets the date components only, which are the members it declares. `DateTimeOffset` is not covered yet, because it has no store mapping — see [#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53). + +Five points are worth knowing: + +**`.DayOfWeek` needs no correction.** ClickHouse week mode 2 agrees with `System.DayOfWeek` exactly — Sunday is 0 through to Saturday 6 — so the value is used as it comes back. The mode argument is always sent, because the default mode starts the week on Monday. + +**`.Now` and `.Today` read the server clock**, so they follow the *server's* timezone, not the client's, and they come back with `DateTimeKind.Unspecified`. Use `DateTime.UtcNow` when you need an instant that does not depend on server configuration. + +**`.Date` narrows outside 1970–2106.** `toStartOfDay` returns a `DateTime`, and ClickHouse *wraps* a value outside that window instead of reporting it — so `.Date` on a `DateTime64` column holding a pre-1970 date reads back wrong. Enable [`enable_extended_results_for_datetime_functions`](https://clickhouse.com/docs/operations/settings/settings#enable_extended_results_for_datetime_functions) — for example `set_enable_extended_results_for_datetime_functions=1` in the connection string — to get a range-preserving `DateTime64` result. + +**A fractional `Add*` argument is exact or is not translated.** `AddDays` and the other time-based methods take a `double`, which .NET scales to whole *ticks* (100 ns), so `AddSeconds(0.1234567)` adds exactly 1 234 567 ticks. The ClickHouse `addDays` function takes a whole number of days and discards the rest, so it cannot be used directly. A constant argument is folded to ticks and then expressed in the coarsest unit that holds it exactly: + +```csharp +e.Timestamp.AddDays(1) // addDays(ts, 1) +e.Timestamp.AddDays(1.5) // addMilliseconds(ts, 129600000) +e.Timestamp.AddMilliseconds(0.5) // not translated — 5 000 ticks is below millisecond resolution +e.Timestamp.AddDays(offsetVariable) // not translated — cannot be checked for exactness +``` + +The natural function keeps the column's store type, and it is the only form that works on a `Date`/`Date32` column — ClickHouse rejects `addMilliseconds` on those. Anything the provider cannot express exactly is left untranslated rather than rounded to fit, so a projection still gives the correct .NET value through client evaluation, while a predicate reports why. `DateOnly.AddDays` takes an `int`, so it always emits `addDays`. + +**Arithmetic on two date/time values is not translated.** `dt1 - dt2` and `time1 - time2` give a `TimeSpan`, and `date + timeSpan` mixes types ClickHouse rejects; `dateDiff` returns a count of whole units, and `Time64` subtraction returns a decimal number of seconds. In a projection EF Core reads the columns and does the arithmetic on the client, which gives the correct result. In a predicate there is no client fallback, so the query fails with an explanation. + +Not yet translated: `.Ticks`, `.AddTicks`, and the `.Microsecond`/`.Nanosecond` members. + +#### `toStartOf*` bucketing + The ClickHouse `toStartOf*` family is exposed through `EF.Functions`, so you can bucket and truncate timestamps directly in queries, including in `GROUP BY`: `ToStartOfYear`, `ToStartOfQuarter`, `ToStartOfMonth`, `ToStartOfWeek` (with an optional ClickHouse week `mode`), `ToStartOfDay`, `ToStartOfHour`, `ToStartOfMinute`, `ToStartOfSecond`, `ToStartOfFiveMinutes`, `ToStartOfTenMinutes`, `ToStartOfFifteenMinutes`, and the general `ToStartOfInterval(source, value, unit)`. diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs new file mode 100644 index 0000000..4c5f01b --- /dev/null +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs @@ -0,0 +1,243 @@ +using System.Reflection; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Diagnostics; +using Microsoft.EntityFrameworkCore.Query; +using Microsoft.EntityFrameworkCore.Query.SqlExpressions; +using Microsoft.EntityFrameworkCore.Storage; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; + +namespace ClickHouse.EntityFrameworkCore.Query.ExpressionTranslators.Internal; + +/// +/// Translates the standard date/time members of , +/// and to ClickHouse functions. +/// +/// +/// One class serves all three CLR types, because the ClickHouse function is the same for each: the +/// to* extraction functions accept Date, Date32, DateTime and +/// DateTime64 alike. registers the members that the +/// given type declares, so gets the date components only. +/// +public class ClickHouseDateTimeMemberTranslator : IMemberTranslator +{ + /// + /// Week mode 2 makes toDayOfWeek agree with exactly: Sunday is 0 + /// through to Saturday is 6. The default mode 0 starts the week on Monday, so the argument is + /// required and no arithmetic correction is needed. + /// + private const byte SundayFirstWeekMode = 2; + + /// + /// One tick is 100 ns, which is Time64 precision 7, and one .NET + /// tick is also the resolution of now64(7). Asking for that precision keeps + /// exact, whereas toTime drops the fraction. + /// + private const int TickPrecision = 7; + + /// Members that map to a ClickHouse function taking the source value alone. + private static readonly Dictionary ComponentFunctions = []; + + private static readonly HashSet DayOfWeekMembers = []; + private static readonly HashSet DateMembers = []; + private static readonly HashSet TimeOfDayMembers = []; + + /// Static members that read the server clock. + private static readonly Dictionary ServerClockMembers = []; + + private readonly ISqlExpressionFactory _sqlExpressionFactory; + private readonly IRelationalTypeMappingSource _typeMappingSource; + + /// How a server-clock member is built: a function name, and whether it pins UTC. + private enum ServerClock + { + /// now64(7, 'UTC') — an exact instant, independent of server settings. + UtcNow, + + /// now64(7) — the server's local clock, in its configured timezone. + LocalNow, + + /// toStartOfDay(now()) — midnight today on the server's local clock. + LocalToday + } + + /// + /// The mapping given to a result. Built once, because it composes a + /// converter over the Int32 mapping and nothing about it varies per translation. + /// + private readonly RelationalTypeMapping? _dayOfWeekMapping; + + static ClickHouseDateTimeMemberTranslator() + { + RegisterInstanceMembers(typeof(DateTime), hasTimeComponents: true); + RegisterInstanceMembers(typeof(DateOnly), hasTimeComponents: false); + + ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.UtcNow)), ServerClock.UtcNow); + ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.Now)), ServerClock.LocalNow); + ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.Today)), ServerClock.LocalToday); + + // DateTimeOffset is deliberately absent, even though every function here would serve it. The + // provider has no DateTimeOffset store mapping yet, so such a property resolves to String: the + // extraction functions then fail on the server, and addDays silently drops the offset and the + // sub-second part. Register it here together with the mapping (issue #53). + } + + public ClickHouseDateTimeMemberTranslator( + ISqlExpressionFactory sqlExpressionFactory, + IRelationalTypeMappingSource typeMappingSource) + { + _sqlExpressionFactory = sqlExpressionFactory; + _typeMappingSource = typeMappingSource; + + // DayOfWeek is an enum, and this provider maps a C# enum to a ClickHouse string. That mapping + // would render the other side of a comparison as 'Sunday' against a number, so the result + // carries a number-backed enum mapping instead. + _dayOfWeekMapping = typeMappingSource.FindMapping(typeof(int)) is { } intMapping + ? (RelationalTypeMapping)intMapping.WithComposedConverter(new EnumToNumberConverter()) + : null; + } + + public SqlExpression? Translate( + SqlExpression? instance, + MemberInfo member, + Type returnType, + IDiagnosticsLogger logger) + { + if (instance is null) + { + return TranslateServerClock(member, returnType); + } + + // toYear and friends return UInt8/UInt16, which the provider's integer mappings widen on read. + if (ComponentFunctions.TryGetValue(member, out var function)) + { + return _sqlExpressionFactory.Function( + name: function, + arguments: [instance], + nullable: true, + argumentsPropagateNullability: [true], + returnType: returnType, + typeMapping: _typeMappingSource.FindMapping(returnType)); + } + + if (DayOfWeekMembers.Contains(member)) + { + return _sqlExpressionFactory.Function( + name: "toDayOfWeek", + arguments: [instance, _sqlExpressionFactory.Constant(SundayFirstWeekMode)], + nullable: true, + // Only the source propagates nullability; the week mode is a constant. + argumentsPropagateNullability: [true, false], + returnType: returnType, + typeMapping: _dayOfWeekMapping); + } + + // toStartOfDay keeps the timezone of the source, which is what DateTime.Date means for a + // column: midnight on the same calendar day that the column renders. + // + // Note that toStartOfDay returns a DateTime, whose range is 1970-2106. ClickHouse wraps a value + // outside that window rather than reporting it, so a DateTime64 column holding a date before + // 1970 reads back wrong unless the session enables + // enable_extended_results_for_datetime_functions, which widens the result to DateTime64. This + // matches EF.Functions.ToStartOfDay and is documented alongside it. + if (DateMembers.Contains(member)) + { + return _sqlExpressionFactory.Function( + name: "toStartOfDay", + arguments: [instance], + nullable: true, + argumentsPropagateNullability: [true], + returnType: returnType, + // Reuse the source's mapping only when it describes the member's own CLR type. A mapping + // for a different type cannot be coerced during materialization. + typeMapping: instance.TypeMapping?.ClrType == returnType + ? instance.TypeMapping + : _typeMappingSource.FindMapping(returnType)); + } + + if (TimeOfDayMembers.Contains(member)) + { + return _sqlExpressionFactory.Function( + name: "toTime64", + arguments: [instance, _sqlExpressionFactory.Constant(TickPrecision)], + nullable: true, + argumentsPropagateNullability: [true, false], + returnType: returnType, + typeMapping: _typeMappingSource.FindMapping($"Time64({TickPrecision})")); + } + + return null; + } + + private SqlExpression? TranslateServerClock(MemberInfo member, Type returnType) + { + if (!ServerClockMembers.TryGetValue(member, out var clock)) + { + return null; + } + + var mapping = _typeMappingSource.FindMapping(returnType); + + // DateTime.Today is midnight today. today() returns a Date, whereas the member's type is + // DateTime, so truncate the clock value instead and keep a DateTime store type. + if (clock == ServerClock.LocalToday) + { + return _sqlExpressionFactory.Function( + name: "toStartOfDay", + arguments: [Niladic("now", returnType, mapping)], + nullable: false, + argumentsPropagateNullability: [false], + returnType: returnType, + typeMapping: mapping); + } + + List arguments = [_sqlExpressionFactory.Constant(TickPrecision)]; + if (clock == ServerClock.UtcNow) + { + arguments.Add(_sqlExpressionFactory.Constant("UTC")); + } + + return _sqlExpressionFactory.Function( + name: "now64", + arguments: arguments, + nullable: false, + argumentsPropagateNullability: arguments.Select(_ => false), + returnType: returnType, + typeMapping: mapping); + } + + private SqlExpression Niladic(string name, Type returnType, RelationalTypeMapping? typeMapping) + => _sqlExpressionFactory.Function( + name: name, + arguments: [], + nullable: false, + argumentsPropagateNullability: [], + returnType: returnType, + typeMapping: typeMapping); + + private static void RegisterInstanceMembers(Type type, bool hasTimeComponents) + { + ComponentFunctions.Add(Property(type, nameof(DateTime.Year)), "toYear"); + ComponentFunctions.Add(Property(type, nameof(DateTime.Month)), "toMonth"); + ComponentFunctions.Add(Property(type, nameof(DateTime.Day)), "toDayOfMonth"); + ComponentFunctions.Add(Property(type, nameof(DateTime.DayOfYear)), "toDayOfYear"); + + DayOfWeekMembers.Add(Property(type, nameof(DateTime.DayOfWeek))); + + if (!hasTimeComponents) + { + return; + } + + ComponentFunctions.Add(Property(type, nameof(DateTime.Hour)), "toHour"); + ComponentFunctions.Add(Property(type, nameof(DateTime.Minute)), "toMinute"); + ComponentFunctions.Add(Property(type, nameof(DateTime.Second)), "toSecond"); + ComponentFunctions.Add(Property(type, nameof(DateTime.Millisecond)), "toMillisecond"); + + DateMembers.Add(Property(type, nameof(DateTime.Date))); + TimeOfDayMembers.Add(Property(type, nameof(DateTime.TimeOfDay))); + } + + private static MemberInfo Property(Type type, string name) + => type.GetProperty(name) + ?? throw new InvalidOperationException($"Property {type.Name}.{name} was not found."); +} diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs index fa31725..fb88ea0 100644 --- a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs @@ -9,13 +9,28 @@ namespace ClickHouse.EntityFrameworkCore.Query.ExpressionTranslators.Internal; /// -/// Translates the EF.Functions.ToStartOf* extension methods -/// () to their ClickHouse SQL functions. +/// Translates date/time method calls to ClickHouse SQL functions: the +/// EF.Functions.ToStartOf* extension methods +/// (), and the standard Add* methods of +/// , and . /// public class ClickHouseDateTimeMethodTranslator : IMethodCallTranslator { private readonly ISqlExpressionFactory _sqlExpressionFactory; + /// + /// Add* methods that take an , keyed to their ClickHouse function. An + /// integer count needs no rounding, so these always translate. + /// + private static readonly Dictionary IntegralAddMethods = []; + + /// + /// Add* methods that take a , keyed to their ClickHouse function and + /// the tick length of the method's own unit. Keeping the two families in separate dictionaries + /// makes a unit with no fixed tick length (a month, a year) unrepresentable here. + /// + private static readonly Dictionary FractionalAddMethods = []; + /// /// Maps the generic method definitions that take only the source value (and, for /// ToStartOfWeek(source, mode), an extra scalar argument) directly to a ClickHouse function name. @@ -105,8 +120,49 @@ void RegisterSourceOnly(string methodName, string sqlFunction) && parameters[2].ParameterType == typeof(int) && parameters[3].ParameterType == typeof(ClickHouseInterval); }) ?? throw new InvalidOperationException("Method ToStartOfInterval with strict signature not found."); + + RegisterAddMethods(typeof(DateTime), hasTimeComponents: true); + + // DateOnly declares no time-based Add* method, and its AddDays takes an int. + RegisterAddMethods(typeof(DateOnly), hasTimeComponents: false); + + // DateTimeOffset is deliberately absent: the provider has no DateTimeOffset store mapping yet, + // so such a property resolves to String and these functions would either fail on the server or + // silently drop the offset. Add it here together with the mapping (issue #53). } + /// + /// Registers the Add* methods that declares. + /// + /// + /// AddYears and AddMonths take an on every supported type, so they + /// map straight onto addYears/addMonths. The time-based methods take a + /// on , which needs the exactness check that + /// applies. On , AddDays takes an + /// instead, so it is registered as integral. + /// + private static void RegisterAddMethods(Type type, bool hasTimeComponents) + { + IntegralAddMethods.Add(Method(type, nameof(DateTime.AddYears), typeof(int)), "addYears"); + IntegralAddMethods.Add(Method(type, nameof(DateTime.AddMonths), typeof(int)), "addMonths"); + + if (!hasTimeComponents) + { + IntegralAddMethods.Add(Method(type, nameof(DateOnly.AddDays), typeof(int)), "addDays"); + return; + } + + FractionalAddMethods.Add(Method(type, nameof(DateTime.AddDays), typeof(double)), ("addDays", TimeSpan.TicksPerDay)); + FractionalAddMethods.Add(Method(type, nameof(DateTime.AddHours), typeof(double)), ("addHours", TimeSpan.TicksPerHour)); + FractionalAddMethods.Add(Method(type, nameof(DateTime.AddMinutes), typeof(double)), ("addMinutes", TimeSpan.TicksPerMinute)); + FractionalAddMethods.Add(Method(type, nameof(DateTime.AddSeconds), typeof(double)), ("addSeconds", TimeSpan.TicksPerSecond)); + FractionalAddMethods.Add(Method(type, nameof(DateTime.AddMilliseconds), typeof(double)), ("addMilliseconds", TimeSpan.TicksPerMillisecond)); + } + + private static MethodInfo Method(Type type, string name, Type argumentType) + => type.GetRuntimeMethod(name, [argumentType]) + ?? throw new InvalidOperationException($"Method {type.Name}.{name}({argumentType.Name}) was not found."); + public ClickHouseDateTimeMethodTranslator(ISqlExpressionFactory sqlExpressionFactory) { _sqlExpressionFactory = sqlExpressionFactory; @@ -118,6 +174,20 @@ public ClickHouseDateTimeMethodTranslator(ISqlExpressionFactory sqlExpressionFac IReadOnlyList arguments, IDiagnosticsLogger logger) { + if (instance is not null) + { + if (IntegralAddMethods.TryGetValue(method, out var integralFunction)) + { + return AddFunction(integralFunction, instance, arguments[0], method.ReturnType); + } + + if (FractionalAddMethods.TryGetValue(method, out var fractional)) + { + return TranslateFractionalAdd( + instance, fractional.Function, fractional.TicksPerUnit, arguments[0], method.ReturnType); + } + } + var genericMethod = method.IsGenericMethod ? method.GetGenericMethodDefinition() : method; if (SupportedMethods.TryGetValue(genericMethod, out var function)) @@ -180,6 +250,98 @@ public ClickHouseDateTimeMethodTranslator(ISqlExpressionFactory sqlExpressionFac return null; } + /// + /// Translates one Add* call whose .NET argument is a , or returns + /// when no exact translation exists. + /// + /// + /// + /// .NET scales the argument to whole ticks and rounds half away from zero, so + /// AddSeconds(0.1234567) adds exactly 1 234 567 ticks. The matching ClickHouse function takes + /// a whole number of its own unit and discards the rest, so addDays(x, 1.5) would add one day. + /// The two agree only when the tick count divides exactly into the function's unit. + /// + /// + /// A constant is therefore folded to ticks here and then expressed in the coarsest unit that holds + /// it exactly. The natural unit is preferred, and not only for readability: it keeps the store type + /// of the source, and addMilliseconds rejects a Date or Date32 source outright + /// (ILLEGAL_TYPE_OF_ARGUMENT). + /// + /// + /// Everything else is left untranslated on purpose, rather than rounded to fit. This covers a + /// sub-millisecond offset and any value that is not a constant. Milliseconds are as fine as this + /// goes: addNanoseconds would express a tick exactly but promotes the result to + /// DateTime64(9), whose Int64 nanosecond count cannot span the DateTime64 range, which + /// would trade a rounding error for a silently wrong date. An untranslated call still gives the + /// correct .NET value through client evaluation in a projection, and reports a clear reason in a + /// predicate. + /// + /// + private SqlExpression? TranslateFractionalAdd( + SqlExpression instance, + string function, + long ticksPerUnit, + SqlExpression value, + Type returnType) + { + if (value is not SqlConstantExpression { Value: double constantValue }) + { + return null; + } + + if (TicksFor(constantValue, ticksPerUnit) is not { } ticks) + { + return null; + } + + if (ticks % ticksPerUnit == 0) + { + return AddFunction(function, instance, _sqlExpressionFactory.Constant(ticks / ticksPerUnit), returnType); + } + + if (ticks % TimeSpan.TicksPerMillisecond != 0) + { + return null; + } + + return AddFunction( + "addMilliseconds", + instance, + _sqlExpressionFactory.Constant(ticks / TimeSpan.TicksPerMillisecond), + returnType); + } + + /// + /// The tick count that .NET would add, or when .NET would not produce one. + /// + /// + /// Mirrors 's scaling: multiply by the unit's tick length, then + /// round half away from zero. A value that cannot represent is rejected, so + /// the comes from .NET during client evaluation instead of + /// from a wrapped Int64 on the server, which ClickHouse reports as a decimal overflow — or, for the + /// larger magnitudes, does not report at all. + /// + private static long? TicksFor(double value, long ticksPerUnit) + { + if (double.IsNaN(value) || double.IsInfinity(value)) + { + return null; + } + + var scaled = value * ticksPerUnit + (value >= 0 ? 0.5 : -0.5); + + return double.Abs(scaled) > DateTime.MaxValue.Ticks ? null : (long)scaled; + } + + private SqlExpression AddFunction(string function, SqlExpression instance, SqlExpression value, Type returnType) + => _sqlExpressionFactory.Function( + name: function, + arguments: [instance, value], + nullable: true, + argumentsPropagateNullability: [true, true], + returnType: returnType, + typeMapping: instance.TypeMapping); + private static bool IsQueryConstant(SqlExpression expression) => expression is SqlConstantExpression or SqlParameterExpression; diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseMemberTranslatorProvider.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseMemberTranslatorProvider.cs index c97a393..aab320b 100644 --- a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseMemberTranslatorProvider.cs +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseMemberTranslatorProvider.cs @@ -16,6 +16,7 @@ public ClickHouseMemberTranslatorProvider( [ new ClickHouseArrayMethodTranslator(sqlExpressionFactory, typeMappingSource), new ClickHouseStringMethodTranslator(sqlExpressionFactory), + new ClickHouseDateTimeMemberTranslator(sqlExpressionFactory, typeMappingSource), ]); } } diff --git a/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs b/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs index dc66ad3..d1d04a9 100644 --- a/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs +++ b/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs @@ -44,4 +44,59 @@ protected override Expression VisitMethodCall(MethodCallExpression methodCallExp => _arrayLinqTranslator.TryTranslate(methodCallExpression, out var translated) ? translated : base.VisitMethodCall(methodCallExpression); + + /// + /// Reports a clear reason when two date/time values are added or subtracted. + /// + /// + /// + /// ClickHouse has no operator for any of these shapes, and each one fails differently: + /// + /// + /// + /// One date minus another gives a in .NET, whereas dateDiff + /// returns a count of whole units. + /// + /// + /// One time of day minus another gives a in .NET, whereas ClickHouse + /// Time64 subtraction gives a Decimal number of seconds. + /// + /// + /// A date plus or minus a keeps the date type in .NET, whereas ClickHouse + /// rejects the mixed operands outright (Illegal types ... of arguments of function plus). + /// + /// + /// + /// Left alone, each of these reaches type-mapping inference or the server and fails with an internal + /// cast error or raw SQL error that names types the user never wrote. Reporting the reason here turns + /// that into EF Core's normal "could not be translated" message with an explanation attached — which + /// also restores client evaluation in a projection, where the .NET result is correct. + /// + /// + protected override Expression VisitBinary(BinaryExpression binaryExpression) + { + if (binaryExpression.NodeType is ExpressionType.Add or ExpressionType.Subtract + && IsDateOrTimeType(binaryExpression.Left.Type) + && IsDateOrTimeType(binaryExpression.Right.Type)) + { + AddTranslationErrorDetails( + "Arithmetic on two date or time values is not supported, because ClickHouse has no " + + "operator that matches the .NET result. Compare the two values directly, or project " + + "them and do the arithmetic on the client."); + + return QueryCompilationContext.NotTranslatedExpression; + } + + return base.VisitBinary(binaryExpression); + } + + private static bool IsDateOrTimeType(Type type) + { + var unwrapped = Nullable.GetUnderlyingType(type) ?? type; + return unwrapped == typeof(DateTime) + || unwrapped == typeof(DateTimeOffset) + || unwrapped == typeof(DateOnly) + || unwrapped == typeof(TimeSpan) + || unwrapped == typeof(TimeOnly); + } } diff --git a/test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindJoinQueryClickHouseTest.cs b/test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindJoinQueryClickHouseTest.cs index 6aaffeb..7c26242 100644 --- a/test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindJoinQueryClickHouseTest.cs +++ b/test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindJoinQueryClickHouseTest.cs @@ -36,11 +36,6 @@ public override Task Take_in_collection_projection_with_FirstOrDefault_on_top_le public override Task SelectMany_with_client_eval_with_constructor(bool async) => AssertUnsupported(() => base.SelectMany_with_client_eval_with_constructor(async)); - // Complex LINQ pattern not translatable - public override Task GroupJoin_aggregate_anonymous_key_selectors2(bool async) - => Assert.ThrowsAsync( - () => base.GroupJoin_aggregate_anonymous_key_selectors2(async)); - private static async Task AssertUnsupported(Func test) => await Assert.ThrowsAsync(test); } diff --git a/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs b/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs new file mode 100644 index 0000000..8b5be82 --- /dev/null +++ b/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs @@ -0,0 +1,602 @@ +using Microsoft.EntityFrameworkCore; +using Xunit; + +namespace EFCore.ClickHouse.Tests; + +public class DateTimeMemberEntity +{ + public long Id { get; set; } + + /// Mapped to ClickHouse DateTime, which holds whole seconds only. + public DateTime Timestamp { get; set; } + + /// Mapped to ClickHouse DateTime64(7). Precision 7 is one .NET tick. + public DateTime Timestamp64 { get; set; } + + /// Mapped to ClickHouse Date32. + public DateOnly Date { get; set; } +} + +public class DateTimeMemberDbContext : DbContext +{ + public DbSet Events => Set(); + + private readonly string _connectionString; + + public DateTimeMemberDbContext(string connectionString) + { + _connectionString = connectionString; + } + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + { + optionsBuilder.UseClickHouse(_connectionString); + } + + protected override void OnModelCreating(ModelBuilder modelBuilder) + { + modelBuilder.Entity(entity => + { + entity.ToTable("datetime_member_test"); + entity.HasKey(e => e.Id); + entity.Property(e => e.Id).HasColumnName("id"); + entity.Property(e => e.Timestamp).HasColumnName("ts"); + entity.Property(e => e.Timestamp64).HasColumnName("ts64").HasColumnType("DateTime64(7)"); + entity.Property(e => e.Date).HasColumnName("d"); + }); + } +} + +public class DateTimeMemberFixture : IAsyncLifetime +{ + public string ConnectionString { get; private set; } = string.Empty; + + /// + /// Row 1's instant: Sunday 2026-08-16 13:47:32.1234567. A Sunday on purpose — ClickHouse + /// toDayOfWeek gives Sunday 7 in its default mode and 0 in mode 2, so a Sunday is the day + /// that proves the mode argument reaches the server. + /// + public static readonly DateTime Instant = new DateTime(2026, 8, 16, 13, 47, 32).AddTicks(1_234_567); + + /// Row 1's time of day, to one tick. + public static readonly TimeSpan InstantTimeOfDay = TimeSpan.FromTicks(496_521_234_567); + + public async Task InitializeAsync() + { + ConnectionString = await SharedContainer.GetConnectionStringAsync(); + + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(ConnectionString); + await connection.OpenAsync(); + + using var createCmd = connection.CreateCommand(); + createCmd.CommandText = """ + CREATE TABLE datetime_member_test ( + id Int64, + ts DateTime, + ts64 DateTime64(7), + d Date32 + ) ENGINE = MergeTree() + ORDER BY id + """; + await createCmd.ExecuteNonQueryAsync(); + + using var insertCmd = connection.CreateCommand(); + // Row 2 is the last day of a month, so AddMonths and AddYears have a day to clamp. + insertCmd.CommandText = """ + INSERT INTO datetime_member_test (id, ts, ts64, d) VALUES + (1, '2026-08-16 13:47:32', '2026-08-16 13:47:32.1234567', '2026-08-16'), + (2, '2026-01-31 00:00:00', '2026-01-31 00:00:00.0000000', '2026-01-31') + """; + await insertCmd.ExecuteNonQueryAsync(); + } + + public Task DisposeAsync() => Task.CompletedTask; +} + +public class DateTimeMemberTranslationTest : IClassFixture +{ + private readonly DateTimeMemberFixture _fixture; + + public DateTimeMemberTranslationTest(DateTimeMemberFixture fixture) + { + _fixture = fixture; + } + + private async Task SelectSingleAsync( + Func, IQueryable> selector, + long id = 1) + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + return await selector(context.Events.AsNoTracking().Where(e => e.Id == id)).SingleAsync(); + } + + private async Task> WhereIdsAsync( + System.Linq.Expressions.Expression> predicate) + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + return await context.Events.AsNoTracking().Where(predicate) + .OrderBy(e => e.Id).Select(e => e.Id).ToListAsync(); + } + + // ---------------------------------------------------------------- components + + [Fact] + public async Task Year_translates_to_toYear() + => Assert.Equal(2026, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Year))); + + [Fact] + public async Task Month_translates_to_toMonth() + => Assert.Equal(8, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Month))); + + [Fact] + public async Task Day_translates_to_toDayOfMonth() + => Assert.Equal(16, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Day))); + + [Fact] + public async Task Hour_translates_to_toHour() + => Assert.Equal(13, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Hour))); + + [Fact] + public async Task Minute_translates_to_toMinute() + => Assert.Equal(47, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Minute))); + + [Fact] + public async Task Second_translates_to_toSecond() + => Assert.Equal(32, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Second))); + + [Fact] + public async Task Millisecond_translates_to_toMillisecond() + => Assert.Equal(123, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Millisecond))); + + [Fact] + public async Task DayOfYear_translates_to_toDayOfYear() + => Assert.Equal(228, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.DayOfYear))); + + [Fact] + public async Task Components_agree_with_dotnet_on_a_second_precision_column() + { + var result = await SelectSingleAsync(q => q.Select(e => new { e.Timestamp.Year, e.Timestamp.Hour, e.Timestamp.Second })); + + Assert.Equal(DateTimeMemberFixture.Instant.Year, result.Year); + Assert.Equal(DateTimeMemberFixture.Instant.Hour, result.Hour); + Assert.Equal(DateTimeMemberFixture.Instant.Second, result.Second); + } + + // ---------------------------------------------------------------- DayOfWeek + + [Fact] + public async Task DayOfWeek_projects_the_dotnet_value() + { + var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.DayOfWeek)); + + // .NET DayOfWeek.Sunday is 0. ClickHouse mode 2 agrees; the default mode would give 7. + Assert.Equal(DayOfWeek.Sunday, result); + } + + [Fact] + public async Task DayOfWeek_compares_against_a_dotnet_constant() + { + // The provider maps a C# enum to a ClickHouse string, so this is the test that proves the + // constant renders as a number rather than as 'Sunday'. + Assert.Equal([1L], await WhereIdsAsync(e => e.Timestamp64.DayOfWeek == DayOfWeek.Sunday)); + } + + [Fact] + public async Task DayOfWeek_of_a_Saturday_is_six() + { + // Row 2 is 2026-01-31, a Saturday. + Assert.Equal(DayOfWeek.Saturday, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.DayOfWeek), id: 2)); + } + + // ---------------------------------------------------------------- Date / TimeOfDay + + [Fact] + public async Task Date_translates_to_toStartOfDay() + => Assert.Equal(new DateTime(2026, 8, 16), await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Date))); + + [Fact] + public async Task TimeOfDay_keeps_tick_precision() + { + var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.TimeOfDay)); + + // toTime would drop the fraction; toTime64(x, 7) keeps every tick. + Assert.Equal(DateTimeMemberFixture.InstantTimeOfDay, result); + } + + // ---------------------------------------------------------------- DateOnly + + [Fact] + public async Task DateOnly_components_translate() + { + var result = await SelectSingleAsync(q => q.Select(e => new + { + e.Date.Year, + e.Date.Month, + e.Date.Day, + e.Date.DayOfYear, + e.Date.DayOfWeek + })); + + Assert.Equal(2026, result.Year); + Assert.Equal(8, result.Month); + Assert.Equal(16, result.Day); + Assert.Equal(228, result.DayOfYear); + Assert.Equal(DayOfWeek.Sunday, result.DayOfWeek); + } + + // ---------------------------------------------------------------- Add* + + [Fact] + public async Task AddYears_translates_to_addYears() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddYears(1), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddYears(1)))); + + [Fact] + public async Task AddMonths_translates_to_addMonths() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddMonths(1), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMonths(1)))); + + [Fact] + public async Task AddMonths_clamps_the_day_like_dotnet() + { + // Row 2 is 2026-01-31, so one month lands on 2026-02-28 in both systems. + var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMonths(1)), id: 2); + + Assert.Equal(new DateTime(2026, 2, 28), result); + Assert.Equal(new DateTime(2026, 1, 31).AddMonths(1), result); + } + + [Fact] + public async Task AddDays_with_a_whole_number_translates_to_addDays() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddDays(1), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddDays(1)))); + + [Fact] + public async Task AddDays_with_a_negative_whole_number_translates_to_addDays() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddDays(-1), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddDays(-1)))); + + [Fact] + public async Task AddDays_with_a_fraction_keeps_dotnet_semantics() + { + // addDays(x, 1.5) would discard the fraction and add one day. The translation must not. + var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddDays(1.5))); + + Assert.Equal(DateTimeMemberFixture.Instant.AddDays(1.5), result); + Assert.Equal(new DateTime(2026, 8, 18, 1, 47, 32).AddTicks(1_234_567), result); + } + + [Fact] + public async Task AddHours_translates() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddHours(2), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddHours(2)))); + + [Fact] + public async Task AddMinutes_with_a_fraction_keeps_dotnet_semantics() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddMinutes(0.5), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMinutes(0.5)))); + + [Fact] + public async Task AddSeconds_with_a_fraction_keeps_dotnet_semantics() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddSeconds(1.5), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddSeconds(1.5)))); + + [Fact] + public async Task AddMilliseconds_translates() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddMilliseconds(1), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMilliseconds(1)))); + + [Fact] + public async Task AddDays_on_a_Date32_column_keeps_the_date_store_type() + { + // DateOnly.AddDays takes an int, so this must use addDays — addMilliseconds rejects a Date32. + var result = await SelectSingleAsync(q => q.Select(e => e.Date.AddDays(1))); + + Assert.Equal(new DateOnly(2026, 8, 17), result); + } + + [Fact] + public async Task AddMonths_on_a_Date32_column_clamps_like_dotnet() + => Assert.Equal( + new DateOnly(2026, 2, 28), + await SelectSingleAsync(q => q.Select(e => e.Date.AddMonths(1)), id: 2)); + + [Fact] + public async Task AddSeconds_below_millisecond_resolution_keeps_dotnet_semantics() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddSeconds(0.1234567)); + + // .NET adds exactly 1 234 567 ticks. No ClickHouse unit holds that without promoting the result + // to DateTime64(9), so the call is left untranslated and the client supplies the exact value. + Assert.DoesNotContain("addSeconds", query.ToQueryString()); + Assert.DoesNotContain("addMilliseconds", query.ToQueryString()); + + var result = await query.SingleAsync(); + Assert.Equal(DateTimeMemberFixture.Instant.AddSeconds(0.1234567), result); + Assert.Equal(DateTimeMemberFixture.Instant.AddTicks(1_234_567), result); + } + + [Fact] + public async Task AddMilliseconds_below_millisecond_resolution_keeps_dotnet_semantics() + { + // .NET rounds to the nearest tick, so this adds 5 000 ticks — not 0 ms and not 1 ms. + var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMilliseconds(0.5))); + + Assert.Equal(DateTimeMemberFixture.Instant.AddMilliseconds(0.5), result); + Assert.Equal(DateTimeMemberFixture.Instant.AddTicks(5_000), result); + } + + [Fact] + public async Task AddDays_with_a_parameter_keeps_dotnet_semantics() + { + var days = 1.5; + + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddDays(days)); + + // A parameter cannot be checked for exactness, so it is not translated. Rounding it on the server + // would disagree with .NET, because ClickHouse round() is banker's rounding. + Assert.DoesNotContain("addDays", query.ToQueryString()); + Assert.Equal(DateTimeMemberFixture.Instant.AddDays(1.5), await query.SingleAsync()); + } + + [Fact] + public async Task AddDays_with_a_parameter_in_a_predicate_reports_a_reason() + { + var days = 1.5; + + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Timestamp64.AddDays(days) > e.Timestamp); + + await Assert.ThrowsAsync(() => query.ToListAsync()); + } + + [Fact] + public async Task AddDays_beyond_the_dotnet_range_throws_the_dotnet_exception() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddDays(1e30)); + + // Folding this would wrap to Int64.MaxValue and give a server decimal-overflow error, or worse a + // silently wrong date. Left untranslated, .NET raises its own exception on the client. + await Assert.ThrowsAsync(() => query.SingleAsync()); + } + + [Fact] + public async Task Add_composes_with_a_component_member() + => Assert.Equal(2027, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddYears(1).Year))); + + // ---------------------------------------------------------------- server clock + + [Fact] + public async Task UtcNow_runs_on_the_server() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // Both bounds are offset by a century so the outcome does not depend on the day the suite runs. + var query = context.Events.AsNoTracking().Where(e => e.Timestamp64 < DateTime.UtcNow.AddYears(100)) + .Select(e => e.Id); + + // If EF Core evaluated DateTime.UtcNow on the client, the SQL would carry a literal instead. + Assert.Contains("now64", query.ToQueryString()); + Assert.Equal([1L, 2L], await query.OrderBy(id => id).ToListAsync()); + + var none = await context.Events.AsNoTracking() + .Where(e => e.Timestamp64 < DateTime.UtcNow.AddYears(-100)) + .Select(e => e.Id).ToListAsync(); + + Assert.Empty(none); + } + + [Fact] + public async Task Today_runs_on_the_server() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Timestamp64 < DateTime.Today) + .Select(e => e.Id); + + Assert.Contains("toStartOfDay", query.ToQueryString()); + Assert.Contains("now()", query.ToQueryString()); + + // Execute it too: a SQL-shape assertion alone would not catch the server rejecting the call. + var beforeToday = await query.ToListAsync(); + + // Row 2 is 2026-01-31, which is before today on any day this suite can run. + Assert.Contains(2L, beforeToday); + } + + // ---------------------------------------------------------------- subtraction + + [Fact] + public async Task Subtracting_two_date_times_in_a_projection_now_works() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // Previously this failed with an opaque cast/coercion error from type-mapping inference. The + // subtraction is now reported as not translatable, so EF Core reads both columns and subtracts + // on the client, which is the correct .NET result. + var result = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64 - e.Timestamp).SingleAsync(); + + Assert.Equal(TimeSpan.FromTicks(1_234_567), result); + } + + [Fact] + public async Task Subtracting_two_date_times_in_a_predicate_reports_a_clear_reason() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // A predicate cannot fall back to the client, so this is where the reason must surface. + var query = context.Events.AsNoTracking() + .Where(e => e.Timestamp64 - e.Timestamp > TimeSpan.Zero); + + var exception = await Assert.ThrowsAsync(() => query.ToListAsync()); + + Assert.Contains("Arithmetic on two date or time values", exception.Message); + } + + [Fact] + public async Task Subtracting_two_times_of_day_in_a_projection_works() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // ClickHouse Time64 subtraction gives a Decimal of seconds, not a TimeSpan, so this must stay + // on the client rather than emit SQL that materializes into the wrong type. + var result = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.TimeOfDay - e.Timestamp.TimeOfDay).SingleAsync(); + + Assert.Equal(TimeSpan.FromTicks(1_234_567), result); + } + + [Fact] + public async Task Adding_a_time_of_day_to_a_date_in_a_projection_works() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // ClickHouse rejects DateTime + Time64 outright, so this must stay on the client too. + var result = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.Date + e.Timestamp64.TimeOfDay).SingleAsync(); + + Assert.Equal(DateTimeMemberFixture.Instant, result); + } + + [Fact] + public async Task Adding_a_TimeSpan_to_a_date_in_a_projection_works() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + var result = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64 + TimeSpan.FromHours(1)).SingleAsync(); + + Assert.Equal(DateTimeMemberFixture.Instant.AddHours(1), result); + } +} + +public class DateTimeMemberTranslationOfflineTest +{ + private sealed class OfflineContext : DbContext + { + public DbSet Events => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + { + optionsBuilder.UseClickHouse("Host=localhost;Protocol=http;Port=8123;Database=test"); + } + + protected override void OnModelCreating(ModelBuilder modelBuilder) + { + modelBuilder.Entity(entity => + { + entity.ToTable("datetime_member_test"); + entity.HasKey(e => e.Id); + entity.Property(e => e.Timestamp64).HasColumnType("DateTime64(7)"); + }); + } + } + + private static string Sql(Func, IQueryable> selector) + { + using var context = new OfflineContext(); + return selector(context.Events).ToQueryString(); + } + + [Fact] + public void Year_emits_toYear() + => Assert.Contains("toYear(", Sql(q => q.Select(e => e.Timestamp64.Year))); + + [Fact] + public void Day_emits_toDayOfMonth() + => Assert.Contains("toDayOfMonth(", Sql(q => q.Select(e => e.Timestamp64.Day))); + + [Fact] + public void DayOfWeek_emits_week_mode_two() + { + var sql = Sql(q => q.Select(e => e.Timestamp64.DayOfWeek)); + + // Assert the mode argument itself — a bare "2" would also match toDayOfWeek(x, 12). + Assert.Contains(", 2)", sql); + Assert.Contains("toDayOfWeek(", sql); + } + + [Fact] + public void DayOfWeek_comparison_emits_a_number_not_a_string() + { + var sql = Sql(q => q.Where(e => e.Timestamp64.DayOfWeek == DayOfWeek.Sunday).Select(e => e.Id)); + + Assert.DoesNotContain("'Sunday'", sql); + Assert.Contains("= 0", sql); + } + + [Fact] + public void TimeOfDay_emits_toTime64_with_tick_precision() + => Assert.Contains("toTime64(", Sql(q => q.Select(e => e.Timestamp64.TimeOfDay))); + + [Fact] + public void AddDays_with_a_whole_number_emits_addDays() + { + var sql = Sql(q => q.Select(e => e.Timestamp64.AddDays(1))); + + Assert.Contains("addDays(", sql); + Assert.DoesNotContain("addMilliseconds", sql); + } + + [Fact] + public void AddDays_with_a_fraction_emits_addMilliseconds() + { + var sql = Sql(q => q.Select(e => e.Timestamp64.AddDays(1.5))); + + // 1.5 days is exactly 129 600 000 ms, folded at translation time. + Assert.Contains("addMilliseconds(", sql); + Assert.Contains("129600000", sql); + } + + [Fact] + public void AddMilliseconds_below_millisecond_resolution_emits_no_add_function() + { + // 0.5 ms is 5 000 ticks, which no ClickHouse unit holds exactly without promoting to + // DateTime64(9), so the call must not be translated. + var sql = Sql(q => q.Select(e => e.Timestamp64.AddMilliseconds(0.5))); + + Assert.DoesNotContain("addMilliseconds", sql); + Assert.DoesNotContain("addNanoseconds", sql); + } + + [Fact] + public void AddSeconds_with_a_whole_number_of_milliseconds_emits_addMilliseconds() + { + // 1.5 s is 1 500 ms exactly, so it is still translatable — just not in seconds. + var sql = Sql(q => q.Select(e => e.Timestamp64.AddSeconds(1.5))); + + Assert.Contains("addMilliseconds(", sql); + Assert.Contains("1500", sql); + } + + [Fact] + public void AddHours_with_a_whole_number_emits_addHours() + => Assert.Contains("addHours(", Sql(q => q.Select(e => e.Timestamp64.AddHours(3)))); + + [Fact] + public void UtcNow_emits_a_utc_pinned_now64() + => Assert.Contains("now64(7, 'UTC')", Sql(q => q.Where(e => e.Timestamp64 < DateTime.UtcNow).Select(e => e.Id))); + + [Fact] + public void Now_emits_a_timezone_less_now64() + { + var sql = Sql(q => q.Where(e => e.Timestamp64 < DateTime.Now).Select(e => e.Id)); + + Assert.Contains("now64(7)", sql); + Assert.DoesNotContain("'UTC'", sql); + } +} From 0c10af9cdeac16acb1cdc10bb6124825f2be5f4a Mon Sep 17 00:00:00 2001 From: Alex Soffronow-Pagonidis Date: Fri, 14 Aug 2026 15:59:01 +0200 Subject: [PATCH 2/4] Extend the date/time translators to DateTimeOffset (#55) The shared translator already takes the CLR type as a parameter, so serving DateTimeOffset is a registration. It was held back only because the type had no store mapping: such a property resolved to String, where the extraction functions fail on the server and addDays silently drops both the offset and the sub-second part. The mapping from #53, which this branch is stacked on, removes that obstacle. The result is in the timezone the column declares, which the store type pins to UTC. That agrees with .NET, because a value read back from such a column carries the +00:00 offset, so .Hour and .Date describe the same instant on both sides. DateTimeOffset.Now translates to the same UTC-pinned now64 as UtcNow, since a DateTimeOffset is an instant. Co-Authored-By: Claude --- CHANGELOG.md | 5 +- README.md | 9 +- .../ClickHouseDateTimeMemberTranslator.cs | 16 ++- .../ClickHouseDateTimeMethodTranslator.cs | 5 +- .../DateTimeMemberTranslationTests.cs | 98 ++++++++++++++++++- 5 files changed, 115 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 917906e..2ed0393 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,7 @@ v0.3.1 (Unreleased) ### Query translation * **`toStartOf*` date-time functions** via `EF.Functions`: `ToStartOfYear`, `ToStartOfQuarter`, `ToStartOfMonth`, `ToStartOfWeek` (with optional week `mode`), `ToStartOfDay`, `ToStartOfHour`, `ToStartOfMinute`, `ToStartOfSecond`, the fixed buckets `ToStartOfFiveMinutes` / `ToStartOfTenMinutes` / `ToStartOfFifteenMinutes`, and the general `ToStartOfInterval(source, value, unit)`. Each maps to the matching ClickHouse function and works in `GROUP BY`. Return types follow ClickHouse: the calendar buckets (`Year`/`Quarter`/`Month`/`Week`) return `Date`, the day/hour/minute buckets return `DateTime`, and `ToStartOfSecond` returns `DateTime64`. All accept `DateTime`/`DateTime64` columns, and the plain truncation functions also accept `DateOnly`; `ToStartOfInterval` requires a `DateTime`/`DateTime64` column on older ClickHouse, which rejects a `DateOnly` (Date/Date32) source with `Illegal type Date32 of 1st argument` for every unit; recent versions accept it. `ToStartOfInterval` takes a `ClickHouseInterval` unit (`Second`…`Year`) and emits `toStartOfInterval(source, toInterval(value))`; the unit must be a constant so it can be translated. The default `Date`/`DateTime` result types only span 1970–2149/2106, so ClickHouse narrows out-of-range values — enable `enable_extended_results_for_datetime_functions` (e.g. `set_enable_extended_results_for_datetime_functions=1` in the connection string) for range-preserving `Date32`/`DateTime64` results. -* **Standard `DateTime` members and methods now translate to SQL.** Previously the provider registered no date/time member translator, so only a direct comparison worked and every member threw `The LINQ expression ... could not be translated`. One shared translator serves `DateTime` and `DateOnly`, because the ClickHouse function is the same for each. ([#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55)) +* **Standard `DateTime` members and methods now translate to SQL.** Previously the provider registered no date/time member translator, so only a direct comparison worked and every member threw `The LINQ expression ... could not be translated`. One shared translator serves `DateTime`, `DateTimeOffset` and `DateOnly`, because the ClickHouse function is the same for each. ([#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55)) * **Components** — `.Year` → `toYear`, `.Month` → `toMonth`, `.Day` → `toDayOfMonth`, `.Hour` → `toHour`, `.Minute` → `toMinute`, `.Second` → `toSecond`, `.Millisecond` → `toMillisecond`, `.DayOfYear` → `toDayOfYear`. These ClickHouse functions return `UInt8`/`UInt16`, which the provider's integer mappings widen to `int` on read. `DateOnly` gets the date components only, matching the members it declares. * **`.DayOfWeek`** → `toDayOfWeek(x, 2)`. Week mode 2 agrees with `System.DayOfWeek` exactly (Sunday 0 … Saturday 6), so no arithmetic correction is applied — the default mode 0 starts the week on Monday, which is why the mode argument is always sent. The result carries a number-backed enum mapping, because this provider maps a C# `enum` to a ClickHouse string and that mapping would otherwise render `x.DayOfWeek == DayOfWeek.Sunday` as a comparison against `'Sunday'`. * **`.Date`** → `toStartOfDay`, which keeps the timezone of the source. Note that `toStartOfDay` returns a `DateTime`, whose range is 1970–2106, and ClickHouse **wraps** a value outside that window rather than reporting it — so `.Date` on a `DateTime64` column holding a pre-1970 date reads back wrong. Enable `enable_extended_results_for_datetime_functions` (for example `set_enable_extended_results_for_datetime_functions=1` in the connection string) to get a range-preserving `DateTime64` result. This is the same caveat that already applies to `EF.Functions.ToStartOfDay`. @@ -13,7 +13,8 @@ v0.3.1 (Unreleased) * **`.AddDays`/`.AddHours`/`.AddMinutes`/`.AddSeconds`/`.AddMilliseconds`** take a `double` in .NET, which .NET scales to whole **ticks** (100 ns), rounding half away from zero — so `AddSeconds(0.1234567)` adds exactly 1 234 567 ticks. The matching ClickHouse function takes a whole number of its own unit and discards the rest, so `addDays(x, 1.5)` would add only one day. A constant argument is therefore folded to ticks during translation and then expressed in the coarsest unit that holds it exactly: a whole number of the unit emits the natural function (`addDays(x, 1)`), and otherwise `addMilliseconds` carries the exact count (`AddDays(1.5)` → `addMilliseconds(x, 129600000)`). Preferring the natural function keeps the store type of the source and keeps `Date`/`Date32` columns working, since `addMilliseconds` rejects those outright. * A **sub-millisecond** offset, a **non-constant** offset, and a value **outside the `DateTime` range** are deliberately left untranslated rather than rounded to fit. Milliseconds are as fine as the translation goes, because `addNanoseconds` would express a tick exactly but promotes the result to `DateTime64(9)`, whose Int64 nanosecond count cannot span the `DateTime64` range — that would trade a rounding error for a silently wrong date. An untranslated call still gives the correct .NET value through client evaluation in a projection, and reports a clear reason in a predicate. `DateOnly.AddDays` takes an `int`, so it always emits `addDays`. * A previously-unsupported Northwind query, `GroupJoin_aggregate_anonymous_key_selectors2`, now passes as a result of these translations; its provider-specific "not translatable" override is removed. - * Not yet translated: `.Ticks`, `.AddTicks`, and the `.Microsecond`/`.Nanosecond` members. `DateTimeOffset` is not covered yet either — it has no store mapping until [#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53), so translating its members would silently drop the offset. The translator is shaped to take the CLR type as a parameter, so it gains `DateTimeOffset` with that mapping. The ClickHouse-specific functions such as `dateDiff` and `dateTrunc` are tracked in [#58](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/58). + * For a `DateTimeOffset` property the result is in the timezone the column declares, which the store type pins to UTC. That agrees with .NET, because a value read back from such a column carries the `+00:00` offset, so `.Hour` and `.Date` describe the same instant on both sides. + * Not yet translated: `.Ticks`, `.AddTicks`, and the `.Microsecond`/`.Nanosecond` members. The ClickHouse-specific functions such as `dateDiff` and `dateTrunc` are tracked in [#58](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/58). * **Behaviour change:** `DateTime.Now` and `DateTime.Today` in a *projection* used to be evaluated on the client; they now read the **server** clock. The value therefore follows the server's timezone rather than the client's, and comes back with `DateTimeKind.Unspecified` instead of `Local`. Use `DateTime.UtcNow` for an instant that does not depend on server configuration. In a predicate all three were untranslatable before, so nothing changes there. ### Types diff --git a/README.md b/README.md index a7bb99a..05d74e2 100644 --- a/README.md +++ b/README.md @@ -176,8 +176,9 @@ ambiguous, and it does not change before 1900. Two points of its own do: `List`, `Dictionary` and `Tuple` all round trip. -`DateTimeOffset` members such as `.Year` and `.UtcDateTime` do not translate to SQL yet. This -applies to `DateTime` as well — see [#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55). +The standard members and methods — `.Year`, `.DayOfWeek`, `.AddDays(n)` and the rest — translate to +SQL; see [Date/Time Functions](#datetime-functions). `.UtcDateTime`, `.LocalDateTime` and `.Offset` +do not. ## Current Status @@ -223,7 +224,7 @@ ClickHouse returns `NULL` from a scalar subquery that matches no rows, where sta #### Standard members and methods -The standard .NET date/time members translate to ClickHouse functions, for both `DateTime` and `DateOnly`: +The standard .NET date/time members translate to ClickHouse functions, for `DateTime`, `DateTimeOffset` and `DateOnly` alike: | .NET | ClickHouse | | --- | --- | @@ -252,7 +253,7 @@ var recent = await ctx.Events .ToListAsync(); ``` -`DateOnly` gets the date components only, which are the members it declares. `DateTimeOffset` is not covered yet, because it has no store mapping — see [#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53). +`DateOnly` gets the date components only, which are the members it declares. For a `DateTimeOffset` property the result is in the timezone the column declares, which the store type pins to UTC — that agrees with .NET, because a value read back carries the `+00:00` offset. Five points are worth knowing: diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs index 4c5f01b..2ae1ba7 100644 --- a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs @@ -13,10 +13,17 @@ namespace ClickHouse.EntityFrameworkCore.Query.ExpressionTranslators.Internal; /// and to ClickHouse functions. /// /// +/// /// One class serves all three CLR types, because the ClickHouse function is the same for each: the /// to* extraction functions accept Date, Date32, DateTime and /// DateTime64 alike. registers the members that the /// given type declares, so gets the date components only. +/// +/// +/// For the result is in the timezone the column declares, which the +/// provider pins to UTC. That agrees with .NET, because a value read back from such a column carries +/// the +00:00 offset, so .Hour and .Date describe the same instant on both sides. +/// /// public class ClickHouseDateTimeMemberTranslator : IMemberTranslator { @@ -69,16 +76,17 @@ private enum ServerClock static ClickHouseDateTimeMemberTranslator() { RegisterInstanceMembers(typeof(DateTime), hasTimeComponents: true); + RegisterInstanceMembers(typeof(DateTimeOffset), hasTimeComponents: true); RegisterInstanceMembers(typeof(DateOnly), hasTimeComponents: false); ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.UtcNow)), ServerClock.UtcNow); ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.Now)), ServerClock.LocalNow); ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.Today)), ServerClock.LocalToday); - // DateTimeOffset is deliberately absent, even though every function here would serve it. The - // provider has no DateTimeOffset store mapping yet, so such a property resolves to String: the - // extraction functions then fail on the server, and addDays silently drops the offset and the - // sub-second part. Register it here together with the mapping (issue #53). + // A DateTimeOffset is an instant, and its store type is UTC-pinned, so both of its clock + // members read the same UTC value. + ServerClockMembers.Add(Property(typeof(DateTimeOffset), nameof(DateTimeOffset.UtcNow)), ServerClock.UtcNow); + ServerClockMembers.Add(Property(typeof(DateTimeOffset), nameof(DateTimeOffset.Now)), ServerClock.UtcNow); } public ClickHouseDateTimeMemberTranslator( diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs index fb88ea0..aa3f29e 100644 --- a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs @@ -122,13 +122,10 @@ void RegisterSourceOnly(string methodName, string sqlFunction) }) ?? throw new InvalidOperationException("Method ToStartOfInterval with strict signature not found."); RegisterAddMethods(typeof(DateTime), hasTimeComponents: true); + RegisterAddMethods(typeof(DateTimeOffset), hasTimeComponents: true); // DateOnly declares no time-based Add* method, and its AddDays takes an int. RegisterAddMethods(typeof(DateOnly), hasTimeComponents: false); - - // DateTimeOffset is deliberately absent: the provider has no DateTimeOffset store mapping yet, - // so such a property resolves to String and these functions would either fail on the server or - // silently drop the offset. Add it here together with the mapping (issue #53). } /// diff --git a/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs b/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs index 8b5be82..c62c13c 100644 --- a/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs +++ b/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs @@ -15,6 +15,9 @@ public class DateTimeMemberEntity /// Mapped to ClickHouse Date32. public DateOnly Date { get; set; } + + /// Mapped to ClickHouse DateTime64(7, 'UTC') by the DateTimeOffset mapping. + public DateTimeOffset Offset { get; set; } } public class DateTimeMemberDbContext : DbContext @@ -43,6 +46,7 @@ protected override void OnModelCreating(ModelBuilder modelBuilder) entity.Property(e => e.Timestamp).HasColumnName("ts"); entity.Property(e => e.Timestamp64).HasColumnName("ts64").HasColumnType("DateTime64(7)"); entity.Property(e => e.Date).HasColumnName("d"); + entity.Property(e => e.Offset).HasColumnName("off"); }); } } @@ -74,7 +78,8 @@ CREATE TABLE datetime_member_test ( id Int64, ts DateTime, ts64 DateTime64(7), - d Date32 + d Date32, + off DateTime64(7, 'UTC') ) ENGINE = MergeTree() ORDER BY id """; @@ -83,9 +88,9 @@ ORDER BY id using var insertCmd = connection.CreateCommand(); // Row 2 is the last day of a month, so AddMonths and AddYears have a day to clamp. insertCmd.CommandText = """ - INSERT INTO datetime_member_test (id, ts, ts64, d) VALUES - (1, '2026-08-16 13:47:32', '2026-08-16 13:47:32.1234567', '2026-08-16'), - (2, '2026-01-31 00:00:00', '2026-01-31 00:00:00.0000000', '2026-01-31') + INSERT INTO datetime_member_test (id, ts, ts64, d, off) VALUES + (1, '2026-08-16 13:47:32', '2026-08-16 13:47:32.1234567', '2026-08-16', '2026-08-16 13:47:32.1234567'), + (2, '2026-01-31 00:00:00', '2026-01-31 00:00:00.0000000', '2026-01-31', '2026-01-31 00:00:00.0000000') """; await insertCmd.ExecuteNonQueryAsync(); } @@ -224,6 +229,91 @@ public async Task DateOnly_components_translate() Assert.Equal(DayOfWeek.Sunday, result.DayOfWeek); } + // ---------------------------------------------------------------- DateTimeOffset + + [Fact] + public async Task DateTimeOffset_components_translate() + { + var result = await SelectSingleAsync(q => q.Select(e => new + { + e.Offset.Year, + e.Offset.Month, + e.Offset.Day, + e.Offset.Hour, + e.Offset.Minute, + e.Offset.Second, + e.Offset.DayOfYear, + e.Offset.DayOfWeek + })); + + // The store type is UTC-pinned, and a value read back carries +00:00, so every component + // describes the same instant on both sides. + Assert.Equal(2026, result.Year); + Assert.Equal(8, result.Month); + Assert.Equal(16, result.Day); + Assert.Equal(13, result.Hour); + Assert.Equal(47, result.Minute); + Assert.Equal(32, result.Second); + Assert.Equal(228, result.DayOfYear); + Assert.Equal(DayOfWeek.Sunday, result.DayOfWeek); + } + + [Fact] + public async Task DateTimeOffset_components_agree_with_dotnet() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + var expected = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Offset).SingleAsync(); + var actual = await SelectSingleAsync(q => q.Select(e => new { e.Offset.Year, e.Offset.Hour, e.Offset.Second })); + + Assert.Equal(expected.Year, actual.Year); + Assert.Equal(expected.Hour, actual.Hour); + Assert.Equal(expected.Second, actual.Second); + } + + [Fact] + public async Task DateTimeOffset_Date_translates() + => Assert.Equal( + new DateTime(2026, 8, 16), + await SelectSingleAsync(q => q.Select(e => e.Offset.Date))); + + [Fact] + public async Task DateTimeOffset_TimeOfDay_keeps_tick_precision() + => Assert.Equal( + DateTimeMemberFixture.InstantTimeOfDay, + await SelectSingleAsync(q => q.Select(e => e.Offset.TimeOfDay))); + + [Fact] + public async Task DateTimeOffset_AddDays_translates() + { + var result = await SelectSingleAsync(q => q.Select(e => e.Offset.AddDays(1))); + + Assert.Equal(new DateTimeOffset(2026, 8, 17, 13, 47, 32, TimeSpan.Zero).AddTicks(1_234_567), result); + } + + [Fact] + public async Task DateTimeOffset_AddMonths_clamps_like_dotnet() + => Assert.Equal( + new DateTimeOffset(2026, 2, 28, 0, 0, 0, TimeSpan.Zero), + await SelectSingleAsync(q => q.Select(e => e.Offset.AddMonths(1)), id: 2)); + + [Fact] + public async Task DateTimeOffset_DayOfWeek_compares_against_a_dotnet_constant() + => Assert.Equal([1L], await WhereIdsAsync(e => e.Offset.DayOfWeek == DayOfWeek.Sunday)); + + [Fact] + public async Task DateTimeOffset_UtcNow_runs_on_the_server() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking() + .Where(e => e.Offset < DateTimeOffset.UtcNow.AddYears(100)) + .Select(e => e.Id); + + Assert.Contains("now64", query.ToQueryString()); + Assert.Equal([1L, 2L], await query.OrderBy(id => id).ToListAsync()); + } + // ---------------------------------------------------------------- Add* [Fact] From b5037c4fc25e80be6b826d77b06f2e558ecf2139 Mon Sep 17 00:00:00 2001 From: Alex Soffronow-Pagonidis Date: Thu, 20 Aug 2026 11:07:12 +0200 Subject: [PATCH 3/4] Address PR 64 review feedback --- CHANGELOG.md | 8 +- README.md | 13 +- .../ClickHouseDateTimeMemberTranslator.cs | 13 +- .../ClickHouseDateTimeMethodTranslator.cs | 149 ++++++++++-- ...ickHouseSqlTranslatingExpressionVisitor.cs | 26 +- .../ClickHouseDateTimeOffsetTypeMapping.cs | 9 + .../DateTimeMemberTranslationTests.cs | 230 +++++++++++++++++- 7 files changed, 400 insertions(+), 48 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2ed0393..c4cd714 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,12 +10,12 @@ v0.3.1 (Unreleased) * **`.TimeOfDay`** → `toTime64(x, 7)`; precision 7 is one .NET tick, so no part of the value is lost (`toTime` would drop the fraction). * **`DateTime.UtcNow`** → `now64(7, 'UTC')`, **`DateTime.Now`** → `now64(7)` and **`DateTime.Today`** → `toStartOfDay(now())`. `today()` is not used for `.Today` because it returns a `Date`, whereas the member's type is `DateTime`. * **`.AddYears(n)`** → `addYears` and **`.AddMonths(n)`** → `addMonths`. Both take an `int` in .NET, and ClickHouse clamps the day of month the same way .NET does, so `2026-01-31` plus one month gives `2026-02-28` in both. - * **`.AddDays`/`.AddHours`/`.AddMinutes`/`.AddSeconds`/`.AddMilliseconds`** take a `double` in .NET, which .NET scales to whole **ticks** (100 ns), rounding half away from zero — so `AddSeconds(0.1234567)` adds exactly 1 234 567 ticks. The matching ClickHouse function takes a whole number of its own unit and discards the rest, so `addDays(x, 1.5)` would add only one day. A constant argument is therefore folded to ticks during translation and then expressed in the coarsest unit that holds it exactly: a whole number of the unit emits the natural function (`addDays(x, 1)`), and otherwise `addMilliseconds` carries the exact count (`AddDays(1.5)` → `addMilliseconds(x, 129600000)`). Preferring the natural function keeps the store type of the source and keeps `Date`/`Date32` columns working, since `addMilliseconds` rejects those outright. + * **`.AddDays`/`.AddHours`/`.AddMinutes`/`.AddSeconds`/`.AddMilliseconds`** take a `double` in .NET. .NET splits the integral and fractional parts, scales each to whole **ticks** (100 ns), and truncates any fractional tick toward zero — so `AddSeconds(0.1234567)` adds exactly 1 234 567 ticks, while `AddMilliseconds(0.99995)` adds 9 999 ticks rather than one millisecond. The matching ClickHouse function takes a whole number of its own unit and discards the rest, so `addDays(x, 1.5)` would add only one day. A constant argument is therefore folded with the .NET algorithm during translation and then expressed in the coarsest unit that holds it exactly: a whole number of the unit emits the natural function (`addDays(x, 1)`), and otherwise `addMilliseconds` carries the exact count (`AddDays(1.5)` → `addMilliseconds(x, 129600000)`). Preferring the natural function keeps the store type of the source and keeps `Date`/`Date32` columns working, since `addMilliseconds` rejects those outright. * A **sub-millisecond** offset, a **non-constant** offset, and a value **outside the `DateTime` range** are deliberately left untranslated rather than rounded to fit. Milliseconds are as fine as the translation goes, because `addNanoseconds` would express a tick exactly but promotes the result to `DateTime64(9)`, whose Int64 nanosecond count cannot span the `DateTime64` range — that would trade a rounding error for a silently wrong date. An untranslated call still gives the correct .NET value through client evaluation in a projection, and reports a clear reason in a predicate. `DateOnly.AddDays` takes an `int`, so it always emits `addDays`. * A previously-unsupported Northwind query, `GroupJoin_aggregate_anonymous_key_selectors2`, now passes as a result of these translations; its provider-specific "not translatable" override is removed. - * For a `DateTimeOffset` property the result is in the timezone the column declares, which the store type pins to UTC. That agrees with .NET, because a value read back from such a column carries the `+00:00` offset, so `.Hour` and `.Date` describe the same instant on both sides. - * Not yet translated: `.Ticks`, `.AddTicks`, and the `.Microsecond`/`.Nanosecond` members. The ClickHouse-specific functions such as `dateDiff` and `dateTrunc` are tracked in [#58](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/58). - * **Behaviour change:** `DateTime.Now` and `DateTime.Today` in a *projection* used to be evaluated on the client; they now read the **server** clock. The value therefore follows the server's timezone rather than the client's, and comes back with `DateTimeKind.Unspecified` instead of `Local`. Use `DateTime.UtcNow` for an instant that does not depend on server configuration. In a predicate all three were untranslatable before, so nothing changes there. + * For a `DateTimeOffset` property, component results use the timezone the column declares. The default store type pins that timezone to UTC; explicitly configured named and fixed-offset zones are preserved. `DateTimeOffset.Add*` translates only for UTC and `Fixed/UTC±HH:MM:SS` mappings: .NET preserves the instance offset, whereas ClickHouse applies a named timezone's calendar rules and can change both the offset and instant across a daylight-saving transition. Named-zone and timezone-less additions remain client-evaluated in projections and report the limitation in predicates. + * Not yet translated: `.Ticks`, `.AddTicks`, the `.Microsecond`/`.Nanosecond` members, and `DateTimeOffset.Now`. The ClickHouse-specific functions such as `dateDiff` and `dateTrunc` are tracked in [#58](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/58). + * **Behaviour change:** `DateTime.Now` and `DateTime.Today` in a *projection* used to be evaluated on the client; they now read the **server** clock. The value therefore follows the server's timezone rather than the client's, and comes back with `DateTimeKind.Unspecified` instead of `Local`. Use `DateTime.UtcNow` for an instant that does not depend on server configuration. `DateTimeOffset.UtcNow` also reads a UTC-pinned server clock; `DateTimeOffset.Now` stays on the client so its local offset is preserved. In a predicate the `DateTime` clock members were untranslatable before, so nothing changes there. ### Types * **`DateTimeOffset` is now a supported property type**, and maps to `DateTime64(7, 'UTC')`. The provider previously had no mapping for it, so EF Core fell back to `DateTimeOffsetToStringConverter` and made a `String` column with no warning — which broke queries against a real `DateTime64` column with `TYPE_MISMATCH`, and stopped `SaveChanges` from writing the value at all. The store type pins the timezone to `'UTC'` so the instant does not depend on the server's `session_timezone`, and precision 7 is one .NET tick (100 ns), so the round trip is exact. `HasPrecision(n)` is honoured for a property with no `HasColumnType`, and keeps the UTC pin. Note that ClickHouse has no type that stores a UTC offset: `DateTime64` holds an instant, and a declared timezone only decides how that instant is rendered. The instant is kept and the offset is not, so a value read back carries the offset of the column's declared timezone. A non-UTC column such as `DateTime64(6, 'Asia/Tokyo')` reads correctly, as does a fixed-offset column such as `DateTime64(7, 'Fixed/UTC+05:30:00')`, and `DateTimeOffset` composes into `Array`, `Map` and `Tuple` columns. Where an instant cannot be reproduced exactly the read throws rather than returning a value that is quietly wrong: the repeated hour when clocks go back in a zone whose candidate offsets are both non-zero (`Europe/Paris`), and a value before 1900 in a named zone, where the offset is Local Mean Time to the second. Writing a value the store type cannot hold throws for the same reason — ClickHouse wraps it rather than reporting it, which matters for precision 8 and 9, whose ranges are narrower than `DateTimeOffset`. **Behaviour change:** a `DateTimeOffset` property that relied on the old `String` column now resolves to `DateTime64(7, 'UTC')` — add `HasConversion()` to keep the previous shape. See the [DateTimeOffset section of the README](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore#datetimeoffset) for the daylight-saving and `MinValue`/`MaxValue` limits. ([#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53)) diff --git a/README.md b/README.md index 05d74e2..cfeecb0 100644 --- a/README.md +++ b/README.md @@ -239,6 +239,7 @@ The standard .NET date/time members translate to ClickHouse functions, for `Date | `DateTime.UtcNow` | `now64(7, 'UTC')` | | `DateTime.Now` | `now64(7)` | | `DateTime.Today` | `toStartOfDay(now())` | +| `DateTimeOffset.UtcNow` | `now64(7, 'UTC')` | ```csharp // Runs entirely on the server @@ -253,17 +254,17 @@ var recent = await ctx.Events .ToListAsync(); ``` -`DateOnly` gets the date components only, which are the members it declares. For a `DateTimeOffset` property the result is in the timezone the column declares, which the store type pins to UTC — that agrees with .NET, because a value read back carries the `+00:00` offset. +`DateOnly` gets the date components only, which are the members it declares. For a `DateTimeOffset` property the result is in the timezone the column declares. The default mapping pins that timezone to UTC, while an explicit store type can select a named or fixed-offset timezone; the value read by .NET carries that same declared-zone offset. -Five points are worth knowing: +Points worth knowing: **`.DayOfWeek` needs no correction.** ClickHouse week mode 2 agrees with `System.DayOfWeek` exactly — Sunday is 0 through to Saturday 6 — so the value is used as it comes back. The mode argument is always sent, because the default mode starts the week on Monday. -**`.Now` and `.Today` read the server clock**, so they follow the *server's* timezone, not the client's, and they come back with `DateTimeKind.Unspecified`. Use `DateTime.UtcNow` when you need an instant that does not depend on server configuration. +**`DateTime.Now` and `DateTime.Today` read the server clock**, so they follow the *server's* timezone, not the client's, and they come back with `DateTimeKind.Unspecified`. Use `DateTime.UtcNow` when you need an instant that does not depend on server configuration. `DateTimeOffset.UtcNow` also reads a UTC-pinned server clock. `DateTimeOffset.Now` remains client-evaluated in a projection, because its observable local offset cannot be reconstructed from a UTC-pinned server value. **`.Date` narrows outside 1970–2106.** `toStartOfDay` returns a `DateTime`, and ClickHouse *wraps* a value outside that window instead of reporting it — so `.Date` on a `DateTime64` column holding a pre-1970 date reads back wrong. Enable [`enable_extended_results_for_datetime_functions`](https://clickhouse.com/docs/operations/settings/settings#enable_extended_results_for_datetime_functions) — for example `set_enable_extended_results_for_datetime_functions=1` in the connection string — to get a range-preserving `DateTime64` result. -**A fractional `Add*` argument is exact or is not translated.** `AddDays` and the other time-based methods take a `double`, which .NET scales to whole *ticks* (100 ns), so `AddSeconds(0.1234567)` adds exactly 1 234 567 ticks. The ClickHouse `addDays` function takes a whole number of days and discards the rest, so it cannot be used directly. A constant argument is folded to ticks and then expressed in the coarsest unit that holds it exactly: +**A fractional `Add*` argument is exact or is not translated.** `AddDays` and the other time-based methods take a `double`. .NET splits the integral and fractional parts, scales each to *ticks* (100 ns), and truncates any fractional tick toward zero, so `AddSeconds(0.1234567)` adds exactly 1 234 567 ticks. The ClickHouse `addDays` function takes a whole number of days and discards the rest, so it cannot be used directly. A constant argument is folded with the .NET algorithm and then expressed in the coarsest unit that holds it exactly: ```csharp e.Timestamp.AddDays(1) // addDays(ts, 1) @@ -274,9 +275,11 @@ e.Timestamp.AddDays(offsetVariable) // not translated — cannot be checked f The natural function keeps the column's store type, and it is the only form that works on a `Date`/`Date32` column — ClickHouse rejects `addMilliseconds` on those. Anything the provider cannot express exactly is left untranslated rather than rounded to fit, so a projection still gives the correct .NET value through client evaluation, while a predicate reports why. `DateOnly.AddDays` takes an `int`, so it always emits `addDays`. +**`DateTimeOffset.Add*` requires a UTC or fixed-offset column.** .NET preserves the instance's offset during addition. On a named timezone with daylight saving, ClickHouse applies calendar rules instead, so `addDays` across a clock change can advance the instant by 23 or 25 hours and return a different offset. The provider therefore translates these methods for the default `'UTC'` mapping and `Fixed/UTC±HH:MM:SS` mappings only. A named-zone or timezone-less source stays on the client in a projection and reports this limitation in a predicate. + **Arithmetic on two date/time values is not translated.** `dt1 - dt2` and `time1 - time2` give a `TimeSpan`, and `date + timeSpan` mixes types ClickHouse rejects; `dateDiff` returns a count of whole units, and `Time64` subtraction returns a decimal number of seconds. In a projection EF Core reads the columns and does the arithmetic on the client, which gives the correct result. In a predicate there is no client fallback, so the query fails with an explanation. -Not yet translated: `.Ticks`, `.AddTicks`, and the `.Microsecond`/`.Nanosecond` members. +Not yet translated: `.Ticks`, `.AddTicks`, the `.Microsecond`/`.Nanosecond` members, and `DateTimeOffset.Now`. #### `toStartOf*` bucketing diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs index 2ae1ba7..221afbd 100644 --- a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs @@ -20,9 +20,10 @@ namespace ClickHouse.EntityFrameworkCore.Query.ExpressionTranslators.Internal; /// given type declares, so gets the date components only. /// /// -/// For the result is in the timezone the column declares, which the -/// provider pins to UTC. That agrees with .NET, because a value read back from such a column carries -/// the +00:00 offset, so .Hour and .Date describe the same instant on both sides. +/// For the result is in the timezone the column declares. The default +/// mapping pins that timezone to UTC; an explicitly configured named or fixed-offset timezone is +/// preserved instead. In either case, a materialized value carries the same declared-zone offset, so +/// its components agree with the server result. /// /// public class ClickHouseDateTimeMemberTranslator : IMemberTranslator @@ -83,10 +84,10 @@ static ClickHouseDateTimeMemberTranslator() ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.Now)), ServerClock.LocalNow); ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.Today)), ServerClock.LocalToday); - // A DateTimeOffset is an instant, and its store type is UTC-pinned, so both of its clock - // members read the same UTC value. + // UtcNow is an instant and the default DateTimeOffset store type is UTC-pinned. + // DateTimeOffset.Now deliberately stays untranslated: unlike UtcNow, it exposes the client's + // current local offset, which a UTC-pinned server value cannot preserve. ServerClockMembers.Add(Property(typeof(DateTimeOffset), nameof(DateTimeOffset.UtcNow)), ServerClock.UtcNow); - ServerClockMembers.Add(Property(typeof(DateTimeOffset), nameof(DateTimeOffset.Now)), ServerClock.UtcNow); } public ClickHouseDateTimeMemberTranslator( diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs index aa3f29e..7f672df 100644 --- a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs @@ -1,5 +1,6 @@ using System.Reflection; using ClickHouse.EntityFrameworkCore.Metadata; +using ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping; using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore.Diagnostics; using Microsoft.EntityFrameworkCore.Query; @@ -12,7 +13,9 @@ namespace ClickHouse.EntityFrameworkCore.Query.ExpressionTranslators.Internal; /// Translates date/time method calls to ClickHouse SQL functions: the /// EF.Functions.ToStartOf* extension methods /// (), and the standard Add* methods of -/// , and . +/// , and . DateTimeOffset +/// addition is limited to UTC and fixed-offset mappings so daylight-saving calendar rules cannot +/// change .NET's offset-preserving semantics. /// public class ClickHouseDateTimeMethodTranslator : IMethodCallTranslator { @@ -20,7 +23,8 @@ public class ClickHouseDateTimeMethodTranslator : IMethodCallTranslator /// /// Add* methods that take an , keyed to their ClickHouse function. An - /// integer count needs no rounding, so these always translate. + /// integer count needs no precision check. calls still require a + /// fixed-offset source mapping so daylight-saving rules cannot change their semantics. /// private static readonly Dictionary IntegralAddMethods = []; @@ -29,7 +33,14 @@ public class ClickHouseDateTimeMethodTranslator : IMethodCallTranslator /// the tick length of the method's own unit. Keeping the two families in separate dictionaries /// makes a unit with no fixed tick length (a month, a year) unrepresentable here. /// - private static readonly Dictionary FractionalAddMethods = []; + private static readonly Dictionary FractionalAddMethods = []; + + private enum TickConversionResult + { + Success, + NonConstant, + OutOfRange + } /// /// Maps the generic method definitions that take only the source value (and, for @@ -149,13 +160,18 @@ private static void RegisterAddMethods(Type type, bool hasTimeComponents) return; } - FractionalAddMethods.Add(Method(type, nameof(DateTime.AddDays), typeof(double)), ("addDays", TimeSpan.TicksPerDay)); - FractionalAddMethods.Add(Method(type, nameof(DateTime.AddHours), typeof(double)), ("addHours", TimeSpan.TicksPerHour)); - FractionalAddMethods.Add(Method(type, nameof(DateTime.AddMinutes), typeof(double)), ("addMinutes", TimeSpan.TicksPerMinute)); - FractionalAddMethods.Add(Method(type, nameof(DateTime.AddSeconds), typeof(double)), ("addSeconds", TimeSpan.TicksPerSecond)); - FractionalAddMethods.Add(Method(type, nameof(DateTime.AddMilliseconds), typeof(double)), ("addMilliseconds", TimeSpan.TicksPerMillisecond)); + RegisterFractionalAdd(type, nameof(DateTime.AddDays), "addDays", TimeSpan.TicksPerDay); + RegisterFractionalAdd(type, nameof(DateTime.AddHours), "addHours", TimeSpan.TicksPerHour); + RegisterFractionalAdd(type, nameof(DateTime.AddMinutes), "addMinutes", TimeSpan.TicksPerMinute); + RegisterFractionalAdd(type, nameof(DateTime.AddSeconds), "addSeconds", TimeSpan.TicksPerSecond); + RegisterFractionalAdd(type, nameof(DateTime.AddMilliseconds), "addMilliseconds", TimeSpan.TicksPerMillisecond); } + private static void RegisterFractionalAdd(Type type, string methodName, string function, long ticksPerUnit) + => FractionalAddMethods.Add( + Method(type, methodName, typeof(double)), + (function, ticksPerUnit, DateTime.MaxValue.Ticks / ticksPerUnit)); + private static MethodInfo Method(Type type, string name, Type argumentType) => type.GetRuntimeMethod(name, [argumentType]) ?? throw new InvalidOperationException($"Method {type.Name}.{name}({argumentType.Name}) was not found."); @@ -173,6 +189,11 @@ public ClickHouseDateTimeMethodTranslator(ISqlExpressionFactory sqlExpressionFac { if (instance is not null) { + if (IsAddMethod(method) && !CanTranslateDateTimeOffsetAdd(method, instance)) + { + return null; + } + if (IntegralAddMethods.TryGetValue(method, out var integralFunction)) { return AddFunction(integralFunction, instance, arguments[0], method.ReturnType); @@ -181,7 +202,12 @@ public ClickHouseDateTimeMethodTranslator(ISqlExpressionFactory sqlExpressionFac if (FractionalAddMethods.TryGetValue(method, out var fractional)) { return TranslateFractionalAdd( - instance, fractional.Function, fractional.TicksPerUnit, arguments[0], method.ReturnType); + instance, + fractional.Function, + fractional.TicksPerUnit, + fractional.MaxUnitCount, + arguments[0], + method.ReturnType); } } @@ -253,7 +279,8 @@ public ClickHouseDateTimeMethodTranslator(ISqlExpressionFactory sqlExpressionFac /// /// /// - /// .NET scales the argument to whole ticks and rounds half away from zero, so + /// .NET separates the integral and fractional parts, scales each to ticks, and truncates any + /// fractional tick toward zero. Consequently, /// AddSeconds(0.1234567) adds exactly 1 234 567 ticks. The matching ClickHouse function takes /// a whole number of its own unit and discards the rest, so addDays(x, 1.5) would add one day. /// The two agree only when the tick count divides exactly into the function's unit. @@ -278,15 +305,11 @@ public ClickHouseDateTimeMethodTranslator(ISqlExpressionFactory sqlExpressionFac SqlExpression instance, string function, long ticksPerUnit, + long maxUnitCount, SqlExpression value, Type returnType) { - if (value is not SqlConstantExpression { Value: double constantValue }) - { - return null; - } - - if (TicksFor(constantValue, ticksPerUnit) is not { } ticks) + if (TryGetTicks(value, ticksPerUnit, maxUnitCount, out var ticks) != TickConversionResult.Success) { return null; } @@ -309,27 +332,107 @@ public ClickHouseDateTimeMethodTranslator(ISqlExpressionFactory sqlExpressionFac } /// - /// The tick count that .NET would add, or when .NET would not produce one. + /// Computes the tick count that .NET would add and reports why it cannot be folded when necessary. /// /// - /// Mirrors 's scaling: multiply by the unit's tick length, then - /// round half away from zero. A value that cannot represent is rejected, so + /// Mirrors .NET 10's DateTime.AddUnits implementation: reject values outside that method's + /// per-unit bound, split the integral and fractional parts, and truncate fractional ticks toward + /// zero. A value that cannot represent is rejected, so /// the comes from .NET during client evaluation instead of /// from a wrapped Int64 on the server, which ClickHouse reports as a decimal overflow — or, for the /// larger magnitudes, does not report at all. /// - private static long? TicksFor(double value, long ticksPerUnit) + private static TickConversionResult TryGetTicks( + SqlExpression value, + long ticksPerUnit, + long maxUnitCount, + out long ticks) { - if (double.IsNaN(value) || double.IsInfinity(value)) + ticks = default; + + if (value is not SqlConstantExpression { Value: double constantValue }) + { + return TickConversionResult.NonConstant; + } + + if (!double.IsFinite(constantValue) || Math.Abs(constantValue) > maxUnitCount) + { + return TickConversionResult.OutOfRange; + } + + var integralPart = Math.Truncate(constantValue); + var fractionalPart = constantValue - integralPart; + ticks = (long)integralPart * ticksPerUnit; + ticks += (long)(fractionalPart * ticksPerUnit); + + return TickConversionResult.Success; + } + + internal static bool IsAddMethod(MethodInfo method) + => IntegralAddMethods.ContainsKey(method) || FractionalAddMethods.ContainsKey(method); + + /// + /// Returns a provider-specific explanation when a recognized Add* method was deliberately + /// left untranslated. + /// + internal static string? GetUnsupportedAddTranslationErrorDetails( + MethodInfo method, + SqlExpression instance, + SqlExpression value) + { + if (!IsAddMethod(method)) { return null; } - var scaled = value * ticksPerUnit + (value >= 0 ? 0.5 : -0.5); + var displayName = $"{method.DeclaringType?.Name}.{method.Name}"; - return double.Abs(scaled) > DateTime.MaxValue.Ticks ? null : (long)scaled; + if (!CanTranslateDateTimeOffsetAdd(method, instance)) + { + var timezone = (instance.TypeMapping as ClickHouseDateTimeOffsetTypeMapping)?.Timezone; + var timezoneDescription = timezone is null ? "no declared timezone" : $"timezone '{timezone}'"; + + return $"The '{displayName}' method cannot be translated for a DateTimeOffset column with " + + $"{timezoneDescription}. .NET preserves the instance offset, while ClickHouse applies " + + "the column timezone's calendar rules and may change the offset across a daylight-saving " + + "transition. Use a UTC or Fixed/UTC offset store type, or perform the addition on the client."; + } + + if (!FractionalAddMethods.TryGetValue(method, out var fractional)) + { + return null; + } + + var tickResult = TryGetTicks(value, fractional.TicksPerUnit, fractional.MaxUnitCount, out var ticks); + if (tickResult == TickConversionResult.NonConstant) + { + return $"The '{displayName}' argument must be a constant so its exact .NET tick count can be " + + "checked before translating it to ClickHouse. Perform the addition on the client when " + + "the offset is row-dependent or parameterized."; + } + + if (tickResult == TickConversionResult.OutOfRange) + { + return $"The '{displayName}' argument is outside the range that .NET accepts for that unit, so " + + "it cannot be translated safely. Let .NET evaluate the call to preserve its " + + "ArgumentOutOfRangeException."; + } + + if (ticks % fractional.TicksPerUnit != 0 + && ticks % TimeSpan.TicksPerMillisecond != 0) + { + return $"The '{displayName}' argument produces a sub-millisecond tick offset that ClickHouse " + + "cannot represent exactly without narrowing the supported DateTime64 range. Only exact " + + "whole-unit or whole-millisecond offsets are translated."; + } + + return null; } + private static bool CanTranslateDateTimeOffsetAdd(MethodInfo method, SqlExpression instance) + => method.DeclaringType != typeof(DateTimeOffset) + || instance.TypeMapping is ClickHouseDateTimeOffsetTypeMapping { HasFixedOffset: true }; + private SqlExpression AddFunction(string function, SqlExpression instance, SqlExpression value, Type returnType) => _sqlExpressionFactory.Function( name: function, diff --git a/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs b/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs index d1d04a9..899abfd 100644 --- a/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs +++ b/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs @@ -1,4 +1,5 @@ using System.Linq.Expressions; +using ClickHouse.EntityFrameworkCore.Query.ExpressionTranslators.Internal; using Microsoft.EntityFrameworkCore.Query; using Microsoft.EntityFrameworkCore.Query.SqlExpressions; @@ -41,9 +42,28 @@ public override SqlExpression GenerateGreatest(IReadOnlyList expr /// Select(...).Contains(...) lambda patterns. /// protected override Expression VisitMethodCall(MethodCallExpression methodCallExpression) - => _arrayLinqTranslator.TryTranslate(methodCallExpression, out var translated) - ? translated - : base.VisitMethodCall(methodCallExpression); + { + if (_arrayLinqTranslator.TryTranslate(methodCallExpression, out var arrayTranslation)) + { + return arrayTranslation; + } + + var translated = base.VisitMethodCall(methodCallExpression); + + if (translated == QueryCompilationContext.NotTranslatedExpression + && ClickHouseDateTimeMethodTranslator.IsAddMethod(methodCallExpression.Method) + && methodCallExpression.Object is { } methodInstance + && methodCallExpression.Arguments is [var methodArgument] + && Visit(methodInstance) is SqlExpression sqlInstance + && Visit(methodArgument) is SqlExpression sqlArgument + && ClickHouseDateTimeMethodTranslator.GetUnsupportedAddTranslationErrorDetails( + methodCallExpression.Method, sqlInstance, sqlArgument) is { } errorDetails) + { + AddTranslationErrorDetails(errorDetails); + } + + return translated; + } /// /// Reports a clear reason when two date/time values are added or subtracted. diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeOffsetTypeMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeOffsetTypeMapping.cs index 945ef02..0d8d922 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeOffsetTypeMapping.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeOffsetTypeMapping.cs @@ -96,6 +96,15 @@ public class ClickHouseDateTimeOffsetTypeMapping : RelationalTypeMapping, IClick /// public string? Timezone { get; } + /// + /// Whether the declared timezone has one offset for every instant. Date/time addition can only + /// preserve semantics for these mappings: named zones with daylight + /// saving may change offset while .NET deliberately keeps the instance offset. + /// + internal bool HasFixedOffset + => Timezone == DefaultTimezone + || Timezone is not null && FixedOffsetRegex.IsMatch(Timezone); + public ClickHouseDateTimeOffsetTypeMapping() : this(DefaultPrecision, DefaultTimezone) { diff --git a/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs b/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs index c62c13c..f27d9dc 100644 --- a/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs +++ b/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs @@ -1,3 +1,4 @@ +using System.Linq.Expressions; using Microsoft.EntityFrameworkCore; using Xunit; @@ -18,6 +19,15 @@ public class DateTimeMemberEntity /// Mapped to ClickHouse DateTime64(7, 'UTC') by the DateTimeOffset mapping. public DateTimeOffset Offset { get; set; } + + /// Mapped to a named timezone with daylight-saving transitions. + public DateTimeOffset OffsetLondon { get; set; } + + /// Mapped to a fixed-offset ClickHouse timezone. + public DateTimeOffset OffsetFixed { get; set; } + + /// Mapped to a timezone-less ClickHouse date/time type. + public DateTimeOffset OffsetNaive { get; set; } } public class DateTimeMemberDbContext : DbContext @@ -47,6 +57,12 @@ protected override void OnModelCreating(ModelBuilder modelBuilder) entity.Property(e => e.Timestamp64).HasColumnName("ts64").HasColumnType("DateTime64(7)"); entity.Property(e => e.Date).HasColumnName("d"); entity.Property(e => e.Offset).HasColumnName("off"); + entity.Property(e => e.OffsetLondon).HasColumnName("off_london") + .HasColumnType("DateTime64(7, 'Europe/London')"); + entity.Property(e => e.OffsetFixed).HasColumnName("off_fixed") + .HasColumnType("DateTime64(7, 'Fixed/UTC+05:30:00')"); + entity.Property(e => e.OffsetNaive).HasColumnName("off_naive") + .HasColumnType("DateTime64(7)"); }); } } @@ -79,7 +95,10 @@ CREATE TABLE datetime_member_test ( ts DateTime, ts64 DateTime64(7), d Date32, - off DateTime64(7, 'UTC') + off DateTime64(7, 'UTC'), + off_london DateTime64(7, 'Europe/London'), + off_fixed DateTime64(7, 'Fixed/UTC+05:30:00'), + off_naive DateTime64(7) ) ENGINE = MergeTree() ORDER BY id """; @@ -88,9 +107,14 @@ ORDER BY id using var insertCmd = connection.CreateCommand(); // Row 2 is the last day of a month, so AddMonths and AddYears have a day to clamp. insertCmd.CommandText = """ - INSERT INTO datetime_member_test (id, ts, ts64, d, off) VALUES - (1, '2026-08-16 13:47:32', '2026-08-16 13:47:32.1234567', '2026-08-16', '2026-08-16 13:47:32.1234567'), - (2, '2026-01-31 00:00:00', '2026-01-31 00:00:00.0000000', '2026-01-31', '2026-01-31 00:00:00.0000000') + INSERT INTO datetime_member_test + (id, ts, ts64, d, off, off_london, off_fixed, off_naive) VALUES + (1, '2026-08-16 13:47:32', '2026-08-16 13:47:32.1234567', '2026-08-16', + '2026-08-16 13:47:32.1234567', '2026-08-16 13:47:32.1234567+00:00', + '2026-08-16 13:47:32.1234567+00:00', '2026-08-16 13:47:32.1234567'), + (2, '2026-01-31 00:00:00', '2026-01-31 00:00:00.0000000', '2026-01-31', + '2026-01-31 00:00:00.0000000', '2026-03-28 12:00:00.0000000+00:00', + '2026-01-31 00:00:00.0000000+00:00', '2026-01-31 00:00:00.0000000') """; await insertCmd.ExecuteNonQueryAsync(); } @@ -287,11 +311,83 @@ public async Task DateTimeOffset_TimeOfDay_keeps_tick_precision() [Fact] public async Task DateTimeOffset_AddDays_translates() { - var result = await SelectSingleAsync(q => q.Select(e => e.Offset.AddDays(1))); + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Offset.AddDays(1)); + Assert.Contains("addDays", query.ToQueryString()); + var result = await query.SingleAsync(); Assert.Equal(new DateTimeOffset(2026, 8, 17, 13, 47, 32, TimeSpan.Zero).AddTicks(1_234_567), result); } + [Fact] + public async Task DateTimeOffset_AddDays_on_a_fixed_offset_mapping_translates() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var source = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.OffsetFixed).SingleAsync(); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.OffsetFixed.AddDays(1)); + + Assert.Contains("addDays", query.ToQueryString()); + Assert.Equal(source.AddDays(1), await query.SingleAsync()); + } + + [Fact] + public async Task DateTimeOffset_AddDays_on_a_dst_mapping_uses_client_semantics_across_the_transition() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var source = await context.Events.AsNoTracking().Where(e => e.Id == 2) + .Select(e => e.OffsetLondon).SingleAsync(); + var query = context.Events.AsNoTracking().Where(e => e.Id == 2) + .Select(e => e.OffsetLondon.AddDays(1)); + + // The UK moves from +00:00 to +01:00 on 2026-03-29. DateTimeOffset.AddDays preserves + // the source's +00:00 offset; ClickHouse addDays would instead apply London's calendar + // rules and return a value one hour earlier as an instant. + Assert.Equal(new DateTimeOffset(2026, 3, 28, 12, 0, 0, TimeSpan.Zero), source); + Assert.DoesNotContain("addDays", query.ToQueryString()); + Assert.Equal(source.AddDays(1), await query.SingleAsync()); + } + + [Fact] + public async Task DateTimeOffset_AddDays_on_a_timezone_less_mapping_uses_client_evaluation() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var source = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.OffsetNaive).SingleAsync(); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.OffsetNaive.AddDays(1)); + + Assert.DoesNotContain("addDays", query.ToQueryString()); + Assert.Equal(source.AddDays(1), await query.SingleAsync()); + } + + [Fact] + public async Task DateTimeOffset_AddMonths_on_a_dst_mapping_in_a_predicate_reports_the_reason() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking() + .Where(e => e.OffsetLondon.AddMonths(1) > e.OffsetLondon); + + var exception = await Assert.ThrowsAsync(() => query.ToListAsync()); + + Assert.Contains("timezone 'Europe/London'", exception.Message); + Assert.Contains("daylight-saving transition", exception.Message); + } + + [Fact] + public async Task DateTimeOffset_AddDays_on_a_timezone_less_mapping_in_a_predicate_reports_the_reason() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking() + .Where(e => e.OffsetNaive.AddDays(1) > e.OffsetNaive); + + var exception = await Assert.ThrowsAsync(() => query.ToListAsync()); + + Assert.Contains("no declared timezone", exception.Message); + } + [Fact] public async Task DateTimeOffset_AddMonths_clamps_like_dotnet() => Assert.Equal( @@ -384,6 +480,30 @@ public async Task AddMilliseconds_translates() DateTimeMemberFixture.Instant.AddMilliseconds(1), await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMilliseconds(1)))); + [Fact] + public async Task AddMilliseconds_truncates_positive_fractional_ticks_like_dotnet() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddMilliseconds(0.99995)); + + // .NET 10 truncates 9 999.5 fractional ticks toward zero. Rounding would incorrectly + // turn this into one whole millisecond and make it look translatable. + Assert.DoesNotContain("addMilliseconds", query.ToQueryString()); + Assert.Equal(DateTimeMemberFixture.Instant.AddTicks(9_999), await query.SingleAsync()); + } + + [Fact] + public async Task AddMilliseconds_truncates_negative_fractional_ticks_like_dotnet() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddMilliseconds(-0.99995)); + + Assert.DoesNotContain("addMilliseconds", query.ToQueryString()); + Assert.Equal(DateTimeMemberFixture.Instant.AddTicks(-9_999), await query.SingleAsync()); + } + [Fact] public async Task AddDays_on_a_Date32_column_keeps_the_date_store_type() { @@ -419,7 +539,7 @@ public async Task AddSeconds_below_millisecond_resolution_keeps_dotnet_semantics [Fact] public async Task AddMilliseconds_below_millisecond_resolution_keeps_dotnet_semantics() { - // .NET rounds to the nearest tick, so this adds 5 000 ticks — not 0 ms and not 1 ms. + // This is exactly 5 000 ticks — not 0 ms and not 1 ms. var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMilliseconds(0.5))); Assert.Equal(DateTimeMemberFixture.Instant.AddMilliseconds(0.5), result); @@ -449,7 +569,57 @@ public async Task AddDays_with_a_parameter_in_a_predicate_reports_a_reason() await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); var query = context.Events.AsNoTracking().Where(e => e.Timestamp64.AddDays(days) > e.Timestamp); - await Assert.ThrowsAsync(() => query.ToListAsync()); + var exception = await Assert.ThrowsAsync(() => query.ToListAsync()); + + Assert.Contains("argument must be a constant", exception.Message); + } + + [Fact] + public async Task AddMilliseconds_below_millisecond_resolution_in_a_predicate_reports_the_reason() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking() + .Where(e => e.Timestamp64.AddMilliseconds(0.99995) > e.Timestamp64); + + var exception = await Assert.ThrowsAsync(() => query.ToListAsync()); + + Assert.Contains("sub-millisecond tick offset", exception.Message); + } + + [Fact] + public async Task AddSeconds_above_the_positive_dotnet_unit_bound_is_not_translated() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddSeconds(315_537_897_599.5)); + + Assert.DoesNotContain("addSeconds", query.ToQueryString()); + Assert.DoesNotContain("addMilliseconds", query.ToQueryString()); + await Assert.ThrowsAsync(() => query.SingleAsync()); + } + + [Fact] + public async Task AddSeconds_below_the_negative_dotnet_unit_bound_is_not_translated() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddSeconds(-315_537_897_599.5)); + + Assert.DoesNotContain("addSeconds", query.ToQueryString()); + Assert.DoesNotContain("addMilliseconds", query.ToQueryString()); + await Assert.ThrowsAsync(() => query.SingleAsync()); + } + + [Fact] + public async Task AddSeconds_outside_the_dotnet_unit_bound_in_a_predicate_reports_the_reason() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking() + .Where(e => e.Timestamp64.AddSeconds(315_537_897_599.5) > e.Timestamp64); + + var exception = await Assert.ThrowsAsync(() => query.ToListAsync()); + + Assert.Contains("outside the range that .NET accepts", exception.Message); } [Fact] @@ -592,6 +762,9 @@ protected override void OnModelCreating(ModelBuilder modelBuilder) entity.ToTable("datetime_member_test"); entity.HasKey(e => e.Id); entity.Property(e => e.Timestamp64).HasColumnType("DateTime64(7)"); + entity.Property(e => e.OffsetLondon).HasColumnType("DateTime64(7, 'Europe/London')"); + entity.Property(e => e.OffsetFixed).HasColumnType("DateTime64(7, 'Fixed/UTC+05:30:00')"); + entity.Property(e => e.OffsetNaive).HasColumnType("DateTime64(7)"); }); } } @@ -602,6 +775,17 @@ private static string Sql(Func, IQueryable> predicate) + { + using var context = new OfflineContext(); + var query = context.Events.Where(predicate); + + var exception = Assert.Throws(() => query.ToQueryString()); + + Assert.Contains("outside the range that .NET accepts", exception.Message); + } + [Fact] public void Year_emits_toYear() => Assert.Contains("toYear(", Sql(q => q.Select(e => e.Timestamp64.Year))); @@ -677,6 +861,28 @@ public void AddSeconds_with_a_whole_number_of_milliseconds_emits_addMilliseconds public void AddHours_with_a_whole_number_emits_addHours() => Assert.Contains("addHours(", Sql(q => q.Select(e => e.Timestamp64.AddHours(3)))); + [Fact] + public void DateTimeOffset_AddDays_on_a_fixed_offset_mapping_emits_addDays() + => Assert.Contains("addDays(", Sql(q => q.Select(e => e.OffsetFixed.AddDays(1)))); + + [Fact] + public void DateTimeOffset_AddDays_on_a_dst_mapping_emits_no_add_function() + => Assert.DoesNotContain("addDays(", Sql(q => q.Select(e => e.OffsetLondon.AddDays(1)))); + + [Fact] + public void AddSeconds_with_nan_reports_out_of_range() + => AssertNonFiniteAddReportsOutOfRange(e => e.Timestamp64.AddSeconds(double.NaN) > e.Timestamp64); + + [Fact] + public void AddSeconds_with_positive_infinity_reports_out_of_range() + => AssertNonFiniteAddReportsOutOfRange( + e => e.Timestamp64.AddSeconds(double.PositiveInfinity) > e.Timestamp64); + + [Fact] + public void AddSeconds_with_negative_infinity_reports_out_of_range() + => AssertNonFiniteAddReportsOutOfRange( + e => e.Timestamp64.AddSeconds(double.NegativeInfinity) > e.Timestamp64); + [Fact] public void UtcNow_emits_a_utc_pinned_now64() => Assert.Contains("now64(7, 'UTC')", Sql(q => q.Where(e => e.Timestamp64 < DateTime.UtcNow).Select(e => e.Id))); @@ -689,4 +895,14 @@ public void Now_emits_a_timezone_less_now64() Assert.Contains("now64(7)", sql); Assert.DoesNotContain("'UTC'", sql); } + + [Fact] + public void DateTimeOffset_UtcNow_emits_a_utc_pinned_now64() + => Assert.Contains( + "now64(7, 'UTC')", + Sql(q => q.Where(e => e.Offset < DateTimeOffset.UtcNow).Select(e => e.Id))); + + [Fact] + public void DateTimeOffset_Now_is_left_for_client_evaluation() + => Assert.DoesNotContain("now64", Sql(q => q.Select(_ => DateTimeOffset.Now))); } From a4a0b065e6bc65a38b3df9390f679d24e5eee16c Mon Sep 17 00:00:00 2001 From: Alex Soffronow-Pagonidis Date: Thu, 20 Aug 2026 13:23:52 +0200 Subject: [PATCH 4/4] Gate Add* on the source timezone, not on the CLR type The daylight-saving gate only asked whether the method belonged to DateTimeOffset, so a DateTime column on a named timezone kept translating and returned values .NET does not produce. Measured on ClickHouse 26.7 against a DateTime64(3, 'Europe/London') column holding 2024-03-30 12:00: AddHours(24) answered 13:00 where .NET answers 12:00, and AddDays(1.5) answered 01:00 where .NET answers 00:00. Both halves of the ClickHouse add* family disagree with .NET on such a column, in different ways. The calendar functions (addDays, addMonths, addYears) keep the wall clock, which normally matches, but cannot produce the hour the clocks skip: addDays on 2026-03-28 01:30 London answers 00:30 rather than 01:30. The absolute functions (addHours through addMilliseconds, including the addMilliseconds fallback a fractional AddDays uses) move the instant, so any interval crossing a transition shifts the wall clock by an hour. That also made AddDays(1) and AddDays(1.5) resolve on different clocks, purely because of which function the coarsest-exact-unit rule picked. The gate now asks the source mapping what timezone it declares. UTC and Fixed/UTC offsets translate, because they have one offset for every instant. For DateTime and DateOnly a store type with no declared timezone also translates, since the driver reads such a column as a UTC wall clock. Everything else falls back to the client, which gives the correct .NET value in a projection and a reason in a predicate. DateTimeOffset keeps its stricter rule, because .NET also preserves the instance offset there. The question is now asked through IClickHouseTimezoneTypeMapping, which the DateTime, DateTime64 and DateTimeOffset mappings implement, with the fixed-offset pattern held once in ClickHouseTimezones instead of privately in the DateTimeOffset mapping. One residual limit is documented rather than translated around: a timezone-less DateTime column still has its calendar arithmetic evaluated in the server's session_timezone, which cannot be seen while translating. Refusing that would give up the default mapping. Also from the same review: - AddYears and AddMonths now honour the bound .NET applies to the argument itself, 10000 years and 120000 months. ClickHouse saturates at the year 9999 where .NET raises ArgumentOutOfRangeException, so AddYears(20000) used to answer 9999-03-30 instead of raising. Only the argument can be checked; whether the result fits depends on the column value. - .TimeOfDay is documented as needing ClickHouse 25.6 or later. It maps to toTime64, which arrived with the Time64 type in 25.6; 25.5 and 24.8 LTS answer "Function with name 'toTime64' does not exist". Every other function this translator emits works on 24.8. - Shifting a date by a TimeSpan gets its own message. The advice for subtracting two dates did not fit it, because it does have a translatable equivalent: x.AddDays(-7) works where x - TimeSpan.FromDays(7) does not. - A reason is attached at most once per translation. The same call can be reached twice, and building the reason for an Add* call re-visits its operands, so the message used to repeat. Co-Authored-By: Claude --- CHANGELOG.md | 9 +- README.md | 14 +- .../ClickHouseDateTimeMethodTranslator.cs | 101 ++++++-- ...ickHouseSqlTranslatingExpressionVisitor.cs | 36 ++- .../ClickHouseDateTime64TypeMapping.cs | 2 +- .../ClickHouseDateTimeOffsetTypeMapping.cs | 16 +- .../Mapping/ClickHouseDateTimeTypeMapping.cs | 2 +- .../Mapping/IClickHouseTimezoneTypeMapping.cs | 68 +++++ .../DateTimeMemberTranslationTests.cs | 243 +++++++++++++++++- 9 files changed, 442 insertions(+), 49 deletions(-) create mode 100644 src/EFCore.ClickHouse/Storage/Internal/Mapping/IClickHouseTimezoneTypeMapping.cs diff --git a/CHANGELOG.md b/CHANGELOG.md index c4cd714..f4362ee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,13 +7,14 @@ v0.3.1 (Unreleased) * **Components** — `.Year` → `toYear`, `.Month` → `toMonth`, `.Day` → `toDayOfMonth`, `.Hour` → `toHour`, `.Minute` → `toMinute`, `.Second` → `toSecond`, `.Millisecond` → `toMillisecond`, `.DayOfYear` → `toDayOfYear`. These ClickHouse functions return `UInt8`/`UInt16`, which the provider's integer mappings widen to `int` on read. `DateOnly` gets the date components only, matching the members it declares. * **`.DayOfWeek`** → `toDayOfWeek(x, 2)`. Week mode 2 agrees with `System.DayOfWeek` exactly (Sunday 0 … Saturday 6), so no arithmetic correction is applied — the default mode 0 starts the week on Monday, which is why the mode argument is always sent. The result carries a number-backed enum mapping, because this provider maps a C# `enum` to a ClickHouse string and that mapping would otherwise render `x.DayOfWeek == DayOfWeek.Sunday` as a comparison against `'Sunday'`. * **`.Date`** → `toStartOfDay`, which keeps the timezone of the source. Note that `toStartOfDay` returns a `DateTime`, whose range is 1970–2106, and ClickHouse **wraps** a value outside that window rather than reporting it — so `.Date` on a `DateTime64` column holding a pre-1970 date reads back wrong. Enable `enable_extended_results_for_datetime_functions` (for example `set_enable_extended_results_for_datetime_functions=1` in the connection string) to get a range-preserving `DateTime64` result. This is the same caveat that already applies to `EF.Functions.ToStartOfDay`. - * **`.TimeOfDay`** → `toTime64(x, 7)`; precision 7 is one .NET tick, so no part of the value is lost (`toTime` would drop the fraction). + * **`.TimeOfDay`** → `toTime64(x, 7)`; precision 7 is one .NET tick, so no part of the value is lost (`toTime` would drop the fraction). This is the one member here that needs a recent server: `toTime64` arrived with the `Time64` type in **ClickHouse 25.6**, and an earlier server answers `Function with name 'toTime64' does not exist`. Every other function in this entry works on 24.8 LTS. * **`DateTime.UtcNow`** → `now64(7, 'UTC')`, **`DateTime.Now`** → `now64(7)` and **`DateTime.Today`** → `toStartOfDay(now())`. `today()` is not used for `.Today` because it returns a `Date`, whereas the member's type is `DateTime`. - * **`.AddYears(n)`** → `addYears` and **`.AddMonths(n)`** → `addMonths`. Both take an `int` in .NET, and ClickHouse clamps the day of month the same way .NET does, so `2026-01-31` plus one month gives `2026-02-28` in both. + * **`.AddYears(n)`** → `addYears` and **`.AddMonths(n)`** → `addMonths`. Both take an `int` in .NET, and ClickHouse clamps the day of month the same way .NET does, so `2026-01-31` plus one month gives `2026-02-28` in both. A constant beyond the bound .NET applies to the argument itself — 10 000 years or 120 000 months — is not translated, because ClickHouse saturates at the year 9999 where .NET raises `ArgumentOutOfRangeException`. Only the argument can be checked while translating; whether the *result* fits depends on the column value, and a result past 9999 still saturates on the server. * **`.AddDays`/`.AddHours`/`.AddMinutes`/`.AddSeconds`/`.AddMilliseconds`** take a `double` in .NET. .NET splits the integral and fractional parts, scales each to whole **ticks** (100 ns), and truncates any fractional tick toward zero — so `AddSeconds(0.1234567)` adds exactly 1 234 567 ticks, while `AddMilliseconds(0.99995)` adds 9 999 ticks rather than one millisecond. The matching ClickHouse function takes a whole number of its own unit and discards the rest, so `addDays(x, 1.5)` would add only one day. A constant argument is therefore folded with the .NET algorithm during translation and then expressed in the coarsest unit that holds it exactly: a whole number of the unit emits the natural function (`addDays(x, 1)`), and otherwise `addMilliseconds` carries the exact count (`AddDays(1.5)` → `addMilliseconds(x, 129600000)`). Preferring the natural function keeps the store type of the source and keeps `Date`/`Date32` columns working, since `addMilliseconds` rejects those outright. * A **sub-millisecond** offset, a **non-constant** offset, and a value **outside the `DateTime` range** are deliberately left untranslated rather than rounded to fit. Milliseconds are as fine as the translation goes, because `addNanoseconds` would express a tick exactly but promotes the result to `DateTime64(9)`, whose Int64 nanosecond count cannot span the `DateTime64` range — that would trade a rounding error for a silently wrong date. An untranslated call still gives the correct .NET value through client evaluation in a projection, and reports a clear reason in a predicate. `DateOnly.AddDays` takes an `int`, so it always emits `addDays`. * A previously-unsupported Northwind query, `GroupJoin_aggregate_anonymous_key_selectors2`, now passes as a result of these translations; its provider-specific "not translatable" override is removed. - * For a `DateTimeOffset` property, component results use the timezone the column declares. The default store type pins that timezone to UTC; explicitly configured named and fixed-offset zones are preserved. `DateTimeOffset.Add*` translates only for UTC and `Fixed/UTC±HH:MM:SS` mappings: .NET preserves the instance offset, whereas ClickHouse applies a named timezone's calendar rules and can change both the offset and instant across a daylight-saving transition. Named-zone and timezone-less additions remain client-evaluated in projections and report the limitation in predicates. + * For a `DateTimeOffset` property, component results use the timezone the column declares. The default store type pins that timezone to UTC; explicitly configured named and fixed-offset zones are preserved. + * **`Add*` translates only for a column whose timezone has one offset for every instant.** ClickHouse arithmetic follows the declared timezone, and a named zone with daylight saving disagrees with .NET in two ways. The calendar functions (`addDays`, `addMonths`, `addYears`) keep the wall clock but cannot produce the hour the clocks skip — on a `DateTime64(7, 'Europe/London')` column holding `2026-03-28 01:30`, `addDays(x, 1)` answers `00:30` where .NET answers `01:30`. The absolute functions (`addHours` … `addMilliseconds`, including the `addMilliseconds` fallback that a fractional `AddDays` uses) move the instant, so an interval crossing a transition shifts the wall clock by an hour. For `DateTimeOffset` there is a third difference: .NET preserves the instance offset, which the column's zone can change. So `Add*` is translated for `'UTC'` and `Fixed/UTC±HH:MM:SS` store types, and — for `DateTime`/`DateOnly` only — for a store type that declares no timezone, which the driver reads as a UTC wall clock. Everything else is client-evaluated in a projection and reports the limitation in a predicate. **Known limit:** a timezone-less `DateTime` column still has its calendar arithmetic evaluated in the server's `session_timezone`, which is invisible during translation; declare the timezone in the store type when that setting observes daylight saving. * Not yet translated: `.Ticks`, `.AddTicks`, the `.Microsecond`/`.Nanosecond` members, and `DateTimeOffset.Now`. The ClickHouse-specific functions such as `dateDiff` and `dateTrunc` are tracked in [#58](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/58). * **Behaviour change:** `DateTime.Now` and `DateTime.Today` in a *projection* used to be evaluated on the client; they now read the **server** clock. The value therefore follows the server's timezone rather than the client's, and comes back with `DateTimeKind.Unspecified` instead of `Local`. Use `DateTime.UtcNow` for an instant that does not depend on server configuration. `DateTimeOffset.UtcNow` also reads a UTC-pinned server clock; `DateTimeOffset.Now` stays on the client so its local offset is preserved. In a predicate the `DateTime` clock members were untranslatable before, so nothing changes there. @@ -21,7 +22,7 @@ v0.3.1 (Unreleased) * **`DateTimeOffset` is now a supported property type**, and maps to `DateTime64(7, 'UTC')`. The provider previously had no mapping for it, so EF Core fell back to `DateTimeOffsetToStringConverter` and made a `String` column with no warning — which broke queries against a real `DateTime64` column with `TYPE_MISMATCH`, and stopped `SaveChanges` from writing the value at all. The store type pins the timezone to `'UTC'` so the instant does not depend on the server's `session_timezone`, and precision 7 is one .NET tick (100 ns), so the round trip is exact. `HasPrecision(n)` is honoured for a property with no `HasColumnType`, and keeps the UTC pin. Note that ClickHouse has no type that stores a UTC offset: `DateTime64` holds an instant, and a declared timezone only decides how that instant is rendered. The instant is kept and the offset is not, so a value read back carries the offset of the column's declared timezone. A non-UTC column such as `DateTime64(6, 'Asia/Tokyo')` reads correctly, as does a fixed-offset column such as `DateTime64(7, 'Fixed/UTC+05:30:00')`, and `DateTimeOffset` composes into `Array`, `Map` and `Tuple` columns. Where an instant cannot be reproduced exactly the read throws rather than returning a value that is quietly wrong: the repeated hour when clocks go back in a zone whose candidate offsets are both non-zero (`Europe/Paris`), and a value before 1900 in a named zone, where the offset is Local Mean Time to the second. Writing a value the store type cannot hold throws for the same reason — ClickHouse wraps it rather than reporting it, which matters for precision 8 and 9, whose ranges are narrower than `DateTimeOffset`. **Behaviour change:** a `DateTimeOffset` property that relied on the old `String` column now resolves to `DateTime64(7, 'UTC')` — add `HasConversion()` to keep the previous shape. See the [DateTimeOffset section of the README](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore#datetimeoffset) for the daylight-saving and `MinValue`/`MaxValue` limits. ([#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53)) ### Bug fixes -* **Subtracting one date/time value from another no longer fails with an internal error.** `dt1 - dt2` gives a `TimeSpan`, which ClickHouse has no operator for — `dateDiff` returns a count of whole units instead. The expression used to reach type-mapping inference and fail with an `InvalidCastException` or a bare `No coercion operator is defined between types ...`, both of which name CLR types the user never wrote. The subtraction is now reported as not translatable, with the reason attached. In a projection EF Core can therefore fall back to the client and return the correct `TimeSpan`; in a predicate, where no fallback exists, the message explains why and what to do instead. ([#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55)) +* **Subtracting one date/time value from another no longer fails with an internal error.** `dt1 - dt2` gives a `TimeSpan`, which ClickHouse has no operator for — `dateDiff` returns a count of whole units instead. The expression used to reach type-mapping inference and fail with an `InvalidCastException` or a bare `No coercion operator is defined between types ...`, both of which name CLR types the user never wrote. The subtraction is now reported as not translatable, with the reason attached. In a projection EF Core can therefore fall back to the client and return the correct `TimeSpan`; in a predicate, where no fallback exists, the message explains why and what to do instead. Shifting a date by a `TimeSpan` gets its own message, because it does have a translatable equivalent: `x.AddDays(-7)` works where `x - TimeSpan.FromDays(7)` does not. ([#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55)) * `ToStartOfWeek` now rejects row-dependent week modes during query translation, and `ToStartOfInterval` likewise rejects row-dependent interval sizes. ClickHouse requires these operands to be constant for the query; literals and captured query parameters remain supported. * **Composite columns now convert their components on read.** `Array(T)`, `Map(K, V)` and `Tuple(...)` read the whole column through `GetValue`, so a component mapping's own read pipeline never ran, and any component whose CLR type differs from the driver's type threw `InvalidCastException`. This is not new with `DateTimeOffset` — `DateOnly[]`, `Dictionary` and `Tuple` were already affected, because `DateOnly` also arrives from the driver as a `DateTime`. The composite is now rebuilt component by component, with the same two steps EF Core applies to a scalar column: the mapping's data-reader conversion, then its `ValueConverter`. An `enum` component, a `List` component and a nested composite therefore all read correctly, and a component that needs no conversion keeps the direct cast. **Known limit:** *writing* a component that needs a `ValueConverter` still does not work, because the bulk insert path passes model values to the driver without applying converters ([#54](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/54)) — an `enum` inside a composite is written as its raw ordinal. * **`Array(Nullable(T))` DDL is no longer double-wrapped.** For a value-type element the store type came out as `Array(Nullable(Nullable(T)))`, which ClickHouse rejects with `Nested type Nullable(T) cannot be inside Nullable type`, so `EnsureCreated` and migrations both failed. The component mapping is resolved from a store type that already carries the wrapper, and `HasColumnType(...)` text is kept verbatim, so the nullable-element wrapper added a second one. It now adds the wrapper only when the inner store type does not already have one, including through `LowCardinality(Nullable(T))`. Reference-type elements were never affected. diff --git a/README.md b/README.md index cfeecb0..fa6ca90 100644 --- a/README.md +++ b/README.md @@ -233,7 +233,7 @@ The standard .NET date/time members translate to ClickHouse functions, for `Date | `.DayOfYear` | `toDayOfYear` | | `.DayOfWeek` | `toDayOfWeek(x, 2)` | | `.Date` | `toStartOfDay` | -| `.TimeOfDay` | `toTime64(x, 7)` | +| `.TimeOfDay` | `toTime64(x, 7)` (needs ClickHouse 25.6 or later) | | `.AddYears(n)` `.AddMonths(n)` | `addYears` `addMonths` | | `.AddDays(n)` `.AddHours(n)` `.AddMinutes(n)` `.AddSeconds(n)` `.AddMilliseconds(n)` | `addDays` `addHours` … (see below) | | `DateTime.UtcNow` | `now64(7, 'UTC')` | @@ -275,10 +275,20 @@ e.Timestamp.AddDays(offsetVariable) // not translated — cannot be checked f The natural function keeps the column's store type, and it is the only form that works on a `Date`/`Date32` column — ClickHouse rejects `addMilliseconds` on those. Anything the provider cannot express exactly is left untranslated rather than rounded to fit, so a projection still gives the correct .NET value through client evaluation, while a predicate reports why. `DateOnly.AddDays` takes an `int`, so it always emits `addDays`. -**`DateTimeOffset.Add*` requires a UTC or fixed-offset column.** .NET preserves the instance's offset during addition. On a named timezone with daylight saving, ClickHouse applies calendar rules instead, so `addDays` across a clock change can advance the instant by 23 or 25 hours and return a different offset. The provider therefore translates these methods for the default `'UTC'` mapping and `Fixed/UTC±HH:MM:SS` mappings only. A named-zone or timezone-less source stays on the client in a projection and reports this limitation in a predicate. +**`.TimeOfDay` needs ClickHouse 25.6 or later.** It maps to `toTime64`, which arrived with the `Time64` type in 25.6. On an earlier server the query fails with `Function with name 'toTime64' does not exist`. Every other function on this page works on 24.8 LTS. + +**An integral `Add*` argument outside the .NET bound is not translated.** .NET rejects more than 10 000 years or 120 000 months whatever the instance holds, while ClickHouse saturates at the end of its own range — `addYears(x, 20000)` answers the year 9999. Such a call is therefore left on the client, so the `ArgumentOutOfRangeException` still happens. Note that only the argument can be checked during translation: whether the *result* also fits depends on the column value, and a result past the year 9999 still saturates on the server. + +**`Add*` requires a column whose timezone has one offset.** ClickHouse arithmetic follows the timezone the column declares, and a named zone with daylight saving disagrees with .NET in two ways. The calendar functions (`addDays`, `addMonths`, `addYears`) keep the wall clock, but cannot produce the hour the clocks skip: on a `DateTime64(7, 'Europe/London')` column holding `2026-03-28 01:30`, `addDays(x, 1)` answers `00:30` where .NET answers `01:30`. The absolute functions (`addHours` … `addMilliseconds`) move the instant, so any interval that crosses a transition shifts the wall clock by an hour. For a `DateTimeOffset` there is a third difference: .NET preserves the instance's offset, which the column's zone can change. + +The provider therefore translates `Add*` only when the store type declares `'UTC'` or a `Fixed/UTC±HH:MM:SS` offset — plus, for `DateTime` and `DateOnly`, when it declares no timezone at all, because the driver reads such a column as a UTC wall clock. Anything else stays on the client in a projection and reports the limitation in a predicate. + +> **One residual limit.** A `DateTime` column that declares no timezone still has its *calendar* arithmetic (`AddDays`, `AddMonths`, `AddYears`) evaluated in the server's `session_timezone`, which the provider cannot see while translating. If that setting names a zone with daylight saving, those results can differ from .NET by an hour. Declare the timezone in the store type — for example `HasColumnType("DateTime64(3, 'UTC')")` — to remove the ambiguity. `DateTimeOffset` is not affected, because its default store type already pins `'UTC'`. **Arithmetic on two date/time values is not translated.** `dt1 - dt2` and `time1 - time2` give a `TimeSpan`, and `date + timeSpan` mixes types ClickHouse rejects; `dateDiff` returns a count of whole units, and `Time64` subtraction returns a decimal number of seconds. In a projection EF Core reads the columns and does the arithmetic on the client, which gives the correct result. In a predicate there is no client fallback, so the query fails with an explanation. +For a rolling window, use the `Add*` methods rather than a `TimeSpan`. `x > DateTime.UtcNow.AddDays(-7)` translates; `x > DateTime.UtcNow - TimeSpan.FromDays(7)` does not, and says so. + Not yet translated: `.Ticks`, `.AddTicks`, the `.Microsecond`/`.Nanosecond` members, and `DateTimeOffset.Now`. #### `toStartOf*` bucketing diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs index 7f672df..743a230 100644 --- a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs @@ -13,20 +13,29 @@ namespace ClickHouse.EntityFrameworkCore.Query.ExpressionTranslators.Internal; /// Translates date/time method calls to ClickHouse SQL functions: the /// EF.Functions.ToStartOf* extension methods /// (), and the standard Add* methods of -/// , and . DateTimeOffset -/// addition is limited to UTC and fixed-offset mappings so daylight-saving calendar rules cannot -/// change .NET's offset-preserving semantics. +/// , and . Addition is +/// limited to source columns whose declared timezone has one offset for every instant, so +/// daylight-saving rules cannot make the server disagree with .NET — see +/// . /// public class ClickHouseDateTimeMethodTranslator : IMethodCallTranslator { private readonly ISqlExpressionFactory _sqlExpressionFactory; + /// The bound .NET puts on the argument of AddYears, independently of the instance. + private const int MaxAddYears = 10_000; + + /// The bound .NET puts on the argument of AddMonths, independently of the instance. + private const int MaxAddMonths = 120_000; + /// - /// Add* methods that take an , keyed to their ClickHouse function. An - /// integer count needs no precision check. calls still require a - /// fixed-offset source mapping so daylight-saving rules cannot change their semantics. + /// Add* methods that take an , keyed to their ClickHouse function and the + /// largest count .NET accepts. An integer count needs no precision check, but it still needs the + /// range check: ClickHouse saturates at the end of its own range where .NET throws, so + /// AddYears(20000) would return the year 9999 instead of raising + /// . /// - private static readonly Dictionary IntegralAddMethods = []; + private static readonly Dictionary IntegralAddMethods = []; /// /// Add* methods that take a , keyed to their ClickHouse function and @@ -147,16 +156,19 @@ void RegisterSourceOnly(string methodName, string sqlFunction) /// map straight onto addYears/addMonths. The time-based methods take a /// on , which needs the exactness check that /// applies. On , AddDays takes an - /// instead, so it is registered as integral. + /// instead, so it is registered as integral; its only bound is the end of the + /// range. /// private static void RegisterAddMethods(Type type, bool hasTimeComponents) { - IntegralAddMethods.Add(Method(type, nameof(DateTime.AddYears), typeof(int)), "addYears"); - IntegralAddMethods.Add(Method(type, nameof(DateTime.AddMonths), typeof(int)), "addMonths"); + IntegralAddMethods.Add(Method(type, nameof(DateTime.AddYears), typeof(int)), ("addYears", MaxAddYears)); + IntegralAddMethods.Add(Method(type, nameof(DateTime.AddMonths), typeof(int)), ("addMonths", MaxAddMonths)); if (!hasTimeComponents) { - IntegralAddMethods.Add(Method(type, nameof(DateOnly.AddDays), typeof(int)), "addDays"); + IntegralAddMethods.Add( + Method(type, nameof(DateOnly.AddDays), typeof(int)), + ("addDays", DateOnly.MaxValue.DayNumber)); return; } @@ -189,14 +201,16 @@ public ClickHouseDateTimeMethodTranslator(ISqlExpressionFactory sqlExpressionFac { if (instance is not null) { - if (IsAddMethod(method) && !CanTranslateDateTimeOffsetAdd(method, instance)) + if (IsAddMethod(method) && !CanTranslateAdd(method, instance)) { return null; } - if (IntegralAddMethods.TryGetValue(method, out var integralFunction)) + if (IntegralAddMethods.TryGetValue(method, out var integral)) { - return AddFunction(integralFunction, instance, arguments[0], method.ReturnType); + return IsWithinIntegralBound(arguments[0], integral.MaxUnitCount) + ? AddFunction(integral.Function, instance, arguments[0], method.ReturnType) + : null; } if (FractionalAddMethods.TryGetValue(method, out var fractional)) @@ -371,6 +385,15 @@ private static TickConversionResult TryGetTicks( internal static bool IsAddMethod(MethodInfo method) => IntegralAddMethods.ContainsKey(method) || FractionalAddMethods.ContainsKey(method); + /// + /// Whether an integral Add* argument is inside the bound .NET applies to it regardless of the + /// instance. A non-constant argument cannot be checked, and is translated because an + /// needs no exactness check. + /// + private static bool IsWithinIntegralBound(SqlExpression value, long maxUnitCount) + => value is not SqlConstantExpression { Value: int constantValue } + || Math.Abs((long)constantValue) <= maxUnitCount; + /// /// Returns a provider-specific explanation when a recognized Add* method was deliberately /// left untranslated. @@ -387,15 +410,29 @@ internal static bool IsAddMethod(MethodInfo method) var displayName = $"{method.DeclaringType?.Name}.{method.Name}"; - if (!CanTranslateDateTimeOffsetAdd(method, instance)) + if (!CanTranslateAdd(method, instance)) { - var timezone = (instance.TypeMapping as ClickHouseDateTimeOffsetTypeMapping)?.Timezone; + var timezone = (instance.TypeMapping as IClickHouseTimezoneTypeMapping)?.Timezone; var timezoneDescription = timezone is null ? "no declared timezone" : $"timezone '{timezone}'"; - return $"The '{displayName}' method cannot be translated for a DateTimeOffset column with " - + $"{timezoneDescription}. .NET preserves the instance offset, while ClickHouse applies " - + "the column timezone's calendar rules and may change the offset across a daylight-saving " - + "transition. Use a UTC or Fixed/UTC offset store type, or perform the addition on the client."; + return method.DeclaringType == typeof(DateTimeOffset) + ? $"The '{displayName}' method cannot be translated for a DateTimeOffset column with " + + $"{timezoneDescription}. .NET preserves the instance offset, while ClickHouse applies " + + "the column timezone's calendar rules and may change the offset across a daylight-saving " + + "transition. Use a UTC or Fixed/UTC offset store type, or perform the addition on the client." + : $"The '{displayName}' method cannot be translated for a column with {timezoneDescription}, " + + "because that timezone changes offset. ClickHouse calendar arithmetic keeps the wall clock " + + "but cannot produce the hour the clocks skip, and its absolute arithmetic shifts the wall " + + "clock by an hour across a transition; .NET does neither. Use a UTC or Fixed/UTC offset " + + "store type, or perform the addition on the client."; + } + + if (IntegralAddMethods.TryGetValue(method, out var integral) + && !IsWithinIntegralBound(value, integral.MaxUnitCount)) + { + return $"The '{displayName}' argument is outside the range that .NET accepts for that unit, so " + + "it cannot be translated safely. ClickHouse saturates at the end of its own range where " + + ".NET raises ArgumentOutOfRangeException. Let .NET evaluate the call to preserve it."; } if (!FractionalAddMethods.TryGetValue(method, out var fractional)) @@ -429,9 +466,27 @@ internal static bool IsAddMethod(MethodInfo method) return null; } - private static bool CanTranslateDateTimeOffsetAdd(MethodInfo method, SqlExpression instance) - => method.DeclaringType != typeof(DateTimeOffset) - || instance.TypeMapping is ClickHouseDateTimeOffsetTypeMapping { HasFixedOffset: true }; + /// + /// Whether the source column's declared timezone lets an Add* method keep .NET semantics. + /// + /// + /// + /// A source must declare a fixed offset. .NET preserves the instance + /// offset, which a named zone's calendar rules can change, and a store type that declares no + /// timezone leaves the rendering to the server. + /// + /// + /// A or source is read as a wall clock, so it only has + /// to avoid a zone that changes offset — see + /// for the two ways that breaks. A store + /// type with no declared timezone is read as a UTC wall clock and stays translatable. + /// + /// + private static bool CanTranslateAdd(MethodInfo method, SqlExpression instance) + => method.DeclaringType == typeof(DateTimeOffset) + ? instance.TypeMapping is ClickHouseDateTimeOffsetTypeMapping { HasFixedOffset: true } + : !ClickHouseTimezones.MayObserveDaylightSaving( + (instance.TypeMapping as IClickHouseTimezoneTypeMapping)?.Timezone); private SqlExpression AddFunction(string function, SqlExpression instance, SqlExpression value, Type returnType) => _sqlExpressionFactory.Function( diff --git a/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs b/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs index 899abfd..f31eb74 100644 --- a/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs +++ b/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs @@ -9,6 +9,13 @@ public class ClickHouseSqlTranslatingExpressionVisitor : RelationalSqlTranslatin { private readonly ClickHouseArrayLinqTranslator _arrayLinqTranslator; + /// + /// Reasons already reported for this translation. A single unsupported call can be reached more + /// than once — the same expression may appear twice in a predicate, and building the reason for an + /// Add* call re-visits its operands — so the set keeps the message from repeating. + /// + private readonly HashSet _reportedTranslationErrors = new(StringComparer.Ordinal); + public ClickHouseSqlTranslatingExpressionVisitor( RelationalSqlTranslatingExpressionVisitorDependencies dependencies, QueryCompilationContext queryCompilationContext, @@ -59,12 +66,21 @@ protected override Expression VisitMethodCall(MethodCallExpression methodCallExp && ClickHouseDateTimeMethodTranslator.GetUnsupportedAddTranslationErrorDetails( methodCallExpression.Method, sqlInstance, sqlArgument) is { } errorDetails) { - AddTranslationErrorDetails(errorDetails); + ReportTranslationError(errorDetails); } return translated; } + /// Attaches a reason to the translation failure, at most once per distinct reason. + private void ReportTranslationError(string details) + { + if (_reportedTranslationErrors.Add(details)) + { + AddTranslationErrorDetails(details); + } + } + /// /// Reports a clear reason when two date/time values are added or subtracted. /// @@ -99,10 +115,16 @@ protected override Expression VisitBinary(BinaryExpression binaryExpression) && IsDateOrTimeType(binaryExpression.Left.Type) && IsDateOrTimeType(binaryExpression.Right.Type)) { - AddTranslationErrorDetails( - "Arithmetic on two date or time values is not supported, because ClickHouse has no " - + "operator that matches the .NET result. Compare the two values directly, or project " - + "them and do the arithmetic on the client."); + // Shifting a date by a span has a translatable equivalent, so point at it rather than + // sending the reader to the client. Subtracting two dates has none. + ReportTranslationError( + IsSpanType(binaryExpression.Right.Type) && !IsSpanType(binaryExpression.Left.Type) + ? "Adding or subtracting a TimeSpan is not supported, because ClickHouse rejects the " + + "mixed operands. Use the Add* methods instead — 'x.AddDays(-7)' translates where " + + "'x - TimeSpan.FromDays(7)' does not." + : "Arithmetic on two date or time values is not supported, because ClickHouse has no " + + "operator that matches the .NET result. Compare the two values directly, or project " + + "them and do the arithmetic on the client."); return QueryCompilationContext.NotTranslatedExpression; } @@ -110,6 +132,10 @@ protected override Expression VisitBinary(BinaryExpression binaryExpression) return base.VisitBinary(binaryExpression); } + /// Whether the type is a length of time rather than a point in time. + private static bool IsSpanType(Type type) + => (Nullable.GetUnderlyingType(type) ?? type) == typeof(TimeSpan); + private static bool IsDateOrTimeType(Type type) { var unwrapped = Nullable.GetUnderlyingType(type) ?? type; diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTime64TypeMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTime64TypeMapping.cs index f94811b..90beff9 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTime64TypeMapping.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTime64TypeMapping.cs @@ -3,7 +3,7 @@ namespace ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping; -public class ClickHouseDateTime64TypeMapping : RelationalTypeMapping +public class ClickHouseDateTime64TypeMapping : RelationalTypeMapping, IClickHouseTimezoneTypeMapping { private const int DefaultPrecision = 3; diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeOffsetTypeMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeOffsetTypeMapping.cs index 0d8d922..2114e29 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeOffsetTypeMapping.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeOffsetTypeMapping.cs @@ -46,7 +46,8 @@ namespace ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping; /// On write, refuses a value the store type cannot hold. ClickHouse /// wraps such a value rather than reporting it. /// -public class ClickHouseDateTimeOffsetTypeMapping : RelationalTypeMapping, IClickHouseWriteValidatingTypeMapping +public class ClickHouseDateTimeOffsetTypeMapping + : RelationalTypeMapping, IClickHouseWriteValidatingTypeMapping, IClickHouseTimezoneTypeMapping { /// /// One .NET tick is 100 ns, which is precision 7. This makes the round trip exact, so a @@ -54,7 +55,7 @@ public class ClickHouseDateTimeOffsetTypeMapping : RelationalTypeMapping, IClick /// public const int DefaultPrecision = 7; - public const string DefaultTimezone = "UTC"; + public const string DefaultTimezone = ClickHouseTimezones.Utc; /// .NET cannot render more than 7 fractional digits, because a tick is its smallest unit. private const int MaxFractionalDigits = 7; @@ -68,11 +69,10 @@ public class ClickHouseDateTimeOffsetTypeMapping : RelationalTypeMapping, IClick /// /// ClickHouse spells a fixed-offset timezone Fixed/UTC±HH:MM:SS. See - /// for why the pattern is this strict. + /// for why the pattern is this strict. It is shared with the + /// query translators, which ask the same question about a source column. /// - private static readonly Regex FixedOffsetRegex = new( - @"^Fixed/UTC([+-])(\d{2}):(\d{2}):(\d{2})$", - RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex FixedOffsetRegex = ClickHouseTimezones.FixedOffsetRegex; // DateTimeOffset holds an offset only within ±14 hours, and only in whole minutes. ClickHouse // accepts both a larger magnitude and a finer granularity, for example 'Fixed/UTC+00:00:42'. @@ -101,9 +101,7 @@ public class ClickHouseDateTimeOffsetTypeMapping : RelationalTypeMapping, IClick /// preserve semantics for these mappings: named zones with daylight /// saving may change offset while .NET deliberately keeps the instance offset. /// - internal bool HasFixedOffset - => Timezone == DefaultTimezone - || Timezone is not null && FixedOffsetRegex.IsMatch(Timezone); + internal bool HasFixedOffset => ClickHouseTimezones.IsFixedOffset(Timezone); public ClickHouseDateTimeOffsetTypeMapping() : this(DefaultPrecision, DefaultTimezone) diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeTypeMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeTypeMapping.cs index 6a2aa85..52c7d31 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeTypeMapping.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeTypeMapping.cs @@ -3,7 +3,7 @@ namespace ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping; -public class ClickHouseDateTimeTypeMapping : RelationalTypeMapping +public class ClickHouseDateTimeTypeMapping : RelationalTypeMapping, IClickHouseTimezoneTypeMapping { public string? Timezone { get; } diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/IClickHouseTimezoneTypeMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/IClickHouseTimezoneTypeMapping.cs new file mode 100644 index 0000000..6a91ee3 --- /dev/null +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/IClickHouseTimezoneTypeMapping.cs @@ -0,0 +1,68 @@ +using System.Text.RegularExpressions; + +namespace ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping; + +/// +/// A date/time mapping whose ClickHouse store type can declare a timezone, such as +/// DateTime64(3, 'Europe/London'). +/// +/// +/// The declared timezone decides how ClickHouse renders an instant, and therefore how its date/time +/// functions behave. classifies the name. +/// +public interface IClickHouseTimezoneTypeMapping +{ + /// + /// The timezone the store type declares, or when it declares none. + /// + string? Timezone { get; } +} + +/// +/// Classifies the timezone name in a ClickHouse date/time store type. +/// +public static class ClickHouseTimezones +{ + /// The one named timezone that is known to have no daylight saving. + public const string Utc = "UTC"; + + /// + /// ClickHouse spells a fixed-offset timezone Fixed/UTC±HH:MM:SS. The pattern is strict + /// because the offset is read back from the captured groups, and .NET has no timezone of that name. + /// + public static readonly Regex FixedOffsetRegex = new( + @"^Fixed/UTC([+-])(\d{2}):(\d{2}):(\d{2})$", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + + /// + /// Whether the timezone has one offset for every instant, so that adding a calendar unit and adding + /// the matching span of absolute time always agree. + /// + public static bool IsFixedOffset(string? timezone) + => timezone == Utc || (timezone is not null && FixedOffsetRegex.IsMatch(timezone)); + + /// + /// Whether the store type names a timezone that may change offset, which makes ClickHouse date/time + /// arithmetic disagree with .NET. + /// + /// + /// + /// A named zone with daylight saving breaks both halves of the ClickHouse add* family, in + /// different ways. The calendar functions (addDays, addMonths, addYears) keep + /// the wall clock, which normally matches .NET, but a result that lands in the hour the clocks skip + /// does not exist: measured on ClickHouse 26.7, + /// addDays(toDateTime64('2024-03-30 01:30:00', 3, 'Europe/London'), 1) gives + /// 2024-03-31 00:30, whereas .NET gives 01:30. The absolute functions + /// (addHours … addMilliseconds) move the instant, so any interval that crosses a + /// transition shifts the wall clock by an hour against .NET. + /// + /// + /// A store type that declares no timezone is not one of these. The driver reads such a column as a + /// UTC wall clock, so absolute arithmetic agrees with .NET. Its calendar arithmetic still follows the + /// server's session_timezone, which cannot be seen from here — that residual limit is + /// documented rather than translated around, because refusing it would give up the default mapping. + /// + /// + public static bool MayObserveDaylightSaving(string? timezone) + => timezone is not null && !IsFixedOffset(timezone); +} diff --git a/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs b/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs index f27d9dc..e79357c 100644 --- a/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs +++ b/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs @@ -17,6 +17,16 @@ public class DateTimeMemberEntity /// Mapped to ClickHouse Date32. public DateOnly Date { get; set; } + /// + /// A on a named timezone with daylight-saving transitions. The driver reads + /// this column as a wall clock in that zone, which is what makes ClickHouse arithmetic on it + /// disagree with .NET. + /// + public DateTime TimestampLondon { get; set; } + + /// A on a timezone that declares one offset for every instant. + public DateTime TimestampUtc { get; set; } + /// Mapped to ClickHouse DateTime64(7, 'UTC') by the DateTimeOffset mapping. public DateTimeOffset Offset { get; set; } @@ -56,6 +66,10 @@ protected override void OnModelCreating(ModelBuilder modelBuilder) entity.Property(e => e.Timestamp).HasColumnName("ts"); entity.Property(e => e.Timestamp64).HasColumnName("ts64").HasColumnType("DateTime64(7)"); entity.Property(e => e.Date).HasColumnName("d"); + entity.Property(e => e.TimestampLondon).HasColumnName("ts_london") + .HasColumnType("DateTime64(7, 'Europe/London')"); + entity.Property(e => e.TimestampUtc).HasColumnName("ts_utc") + .HasColumnType("DateTime64(7, 'UTC')"); entity.Property(e => e.Offset).HasColumnName("off"); entity.Property(e => e.OffsetLondon).HasColumnName("off_london") .HasColumnType("DateTime64(7, 'Europe/London')"); @@ -81,6 +95,14 @@ public class DateTimeMemberFixture : IAsyncLifetime /// Row 1's time of day, to one tick. public static readonly TimeSpan InstantTimeOfDay = TimeSpan.FromTicks(496_521_234_567); + /// + /// Row 3's wall clock in Europe/London. The UK moves from +00:00 to +01:00 at 01:00 UTC on + /// 2026-03-29, so 01:30 on the following day does not exist. Both halves of the ClickHouse + /// add* family therefore disagree with .NET here: addDays(x, 1) keeps the wall clock + /// but cannot produce 01:30, and addHours(x, 24) moves the instant and lands on 02:30. + /// + public static readonly DateTime LondonBeforeTransition = new(2026, 3, 28, 1, 30, 0); + public async Task InitializeAsync() { ConnectionString = await SharedContainer.GetConnectionStringAsync(); @@ -95,6 +117,8 @@ CREATE TABLE datetime_member_test ( ts DateTime, ts64 DateTime64(7), d Date32, + ts_london DateTime64(7, 'Europe/London'), + ts_utc DateTime64(7, 'UTC'), off DateTime64(7, 'UTC'), off_london DateTime64(7, 'Europe/London'), off_fixed DateTime64(7, 'Fixed/UTC+05:30:00'), @@ -106,15 +130,24 @@ ORDER BY id using var insertCmd = connection.CreateCommand(); // Row 2 is the last day of a month, so AddMonths and AddYears have a day to clamp. + // Row 3 sits just before a daylight-saving transition, so its Add* results land on a wall clock + // that ClickHouse and .NET disagree about. See DateTimeMemberFixture.LondonBeforeTransition. insertCmd.CommandText = """ INSERT INTO datetime_member_test - (id, ts, ts64, d, off, off_london, off_fixed, off_naive) VALUES + (id, ts, ts64, d, ts_london, ts_utc, + off, off_london, off_fixed, off_naive) VALUES (1, '2026-08-16 13:47:32', '2026-08-16 13:47:32.1234567', '2026-08-16', + '2026-08-16 13:47:32.1234567', '2026-08-16 13:47:32.1234567', '2026-08-16 13:47:32.1234567', '2026-08-16 13:47:32.1234567+00:00', '2026-08-16 13:47:32.1234567+00:00', '2026-08-16 13:47:32.1234567'), (2, '2026-01-31 00:00:00', '2026-01-31 00:00:00.0000000', '2026-01-31', + '2026-01-31 00:00:00.0000000', '2026-01-31 00:00:00.0000000', '2026-01-31 00:00:00.0000000', '2026-03-28 12:00:00.0000000+00:00', - '2026-01-31 00:00:00.0000000+00:00', '2026-01-31 00:00:00.0000000') + '2026-01-31 00:00:00.0000000+00:00', '2026-01-31 00:00:00.0000000'), + (3, '2026-03-28 01:30:00', '2026-03-28 01:30:00.0000000', '2026-03-28', + '2026-03-28 01:30:00.0000000', '2026-03-28 01:30:00.0000000', + '2026-03-28 01:30:00.0000000', '2026-03-28 01:30:00.0000000+00:00', + '2026-03-28 01:30:00.0000000+00:00', '2026-03-28 01:30:00.0000000') """; await insertCmd.ExecuteNonQueryAsync(); } @@ -407,7 +440,7 @@ public async Task DateTimeOffset_UtcNow_runs_on_the_server() .Select(e => e.Id); Assert.Contains("now64", query.ToQueryString()); - Assert.Equal([1L, 2L], await query.OrderBy(id => id).ToListAsync()); + Assert.Equal([1L, 2L, 3L], await query.OrderBy(id => id).ToListAsync()); } // ---------------------------------------------------------------- Add* @@ -634,6 +667,163 @@ public async Task AddDays_beyond_the_dotnet_range_throws_the_dotnet_exception() await Assert.ThrowsAsync(() => query.SingleAsync()); } + // ------------------------------------------------- Add* on a daylight-saving DateTime column + + [Fact] + public async Task AddDays_on_a_dst_mapping_uses_client_semantics_through_the_skipped_hour() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var source = await context.Events.AsNoTracking().Where(e => e.Id == 3) + .Select(e => e.TimestampLondon).SingleAsync(); + var query = context.Events.AsNoTracking().Where(e => e.Id == 3) + .Select(e => e.TimestampLondon.AddDays(1)); + + // 2026-03-29 01:30 does not exist in London: the clocks go straight from 01:00 to 02:00. + // ClickHouse addDays keeps the wall clock but cannot land there, and answers 00:30 instead. + Assert.Equal(DateTimeMemberFixture.LondonBeforeTransition, source); + Assert.DoesNotContain("addDays", query.ToQueryString()); + Assert.Equal(new DateTime(2026, 3, 29, 1, 30, 0), await query.SingleAsync()); + } + + [Fact] + public async Task AddHours_on_a_dst_mapping_uses_client_semantics_across_the_transition() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 3) + .Select(e => e.TimestampLondon.AddHours(24)); + + // addHours moves the instant, so the server would render 02:30 where .NET keeps the wall + // clock and gives 01:30. + Assert.DoesNotContain("addHours", query.ToQueryString()); + Assert.Equal(new DateTime(2026, 3, 29, 1, 30, 0), await query.SingleAsync()); + } + + [Fact] + public async Task AddDays_with_a_fraction_on_a_dst_mapping_uses_client_semantics() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 3) + .Select(e => e.TimestampLondon.AddDays(1.5)); + + // The addMilliseconds fallback is absolute too, so the whole family stays on the client. + Assert.DoesNotContain("addMilliseconds", query.ToQueryString()); + Assert.Equal( + DateTimeMemberFixture.LondonBeforeTransition.AddDays(1.5), await query.SingleAsync()); + } + + [Fact] + public async Task AddDays_on_a_dst_mapping_in_a_predicate_reports_the_reason() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking() + .Where(e => e.TimestampLondon.AddDays(1) > e.TimestampLondon); + + var exception = await Assert.ThrowsAsync(() => query.ToListAsync()); + + Assert.Contains("timezone 'Europe/London'", exception.Message); + Assert.Contains("changes offset", exception.Message); + } + + [Fact] + public async Task AddHours_on_a_utc_mapping_still_translates() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 3) + .Select(e => e.TimestampUtc.AddHours(24)); + + // A declared UTC zone has one offset for every instant, so the gate must not catch it. + Assert.Contains("addHours", query.ToQueryString()); + Assert.Equal(new DateTime(2026, 3, 29, 1, 30, 0), await query.SingleAsync()); + } + + [Fact] + public async Task AddDays_on_a_timezone_less_mapping_still_translates() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 3) + .Select(e => e.Timestamp64.AddDays(1)); + + // The driver reads a timezone-less column as a UTC wall clock, so this stays translatable. + Assert.Contains("addDays", query.ToQueryString()); + Assert.Equal(new DateTime(2026, 3, 29, 1, 30, 0), await query.SingleAsync()); + } + + // ------------------------------------------------- integral Add* range + + [Fact] + public async Task AddYears_above_the_dotnet_bound_is_not_translated() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddYears(20_000)); + + // ClickHouse saturates at the year 9999; .NET raises instead, and that is what must survive. + Assert.DoesNotContain("addYears", query.ToQueryString()); + await Assert.ThrowsAsync(() => query.SingleAsync()); + } + + [Fact] + public async Task AddMonths_above_the_dotnet_bound_is_not_translated() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddMonths(500_000)); + + Assert.DoesNotContain("addMonths", query.ToQueryString()); + await Assert.ThrowsAsync(() => query.SingleAsync()); + } + + [Fact] + public async Task DateOnly_AddDays_above_the_dotnet_bound_is_not_translated() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Date.AddDays(4_000_000)); + + Assert.DoesNotContain("addDays", query.ToQueryString()); + await Assert.ThrowsAsync(() => query.SingleAsync()); + } + + [Fact] + public async Task AddYears_above_the_dotnet_bound_in_a_predicate_reports_the_reason() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking() + .Where(e => e.Timestamp64.AddYears(20_000) > e.Timestamp64); + + var exception = await Assert.ThrowsAsync(() => query.ToListAsync()); + + Assert.Contains("outside the range that .NET accepts", exception.Message); + } + + [Fact] + public async Task AddYears_at_the_dotnet_bound_still_translates() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // The bound is inclusive, and only the argument is checked here — the instance decides whether + // the result also fits, and that cannot be known during translation. + Assert.Contains( + "addYears", + context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddYears(10_000)).ToQueryString()); + } + + [Fact] + public async Task AddYears_with_a_parameter_still_translates() + { + var years = 1; + + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddYears(years)); + + // An int needs no exactness check, so a parameter is translated even though its magnitude + // cannot be checked. + Assert.Contains("addYears", query.ToQueryString()); + Assert.Equal(DateTimeMemberFixture.Instant.AddYears(1), await query.SingleAsync()); + } + [Fact] public async Task Add_composes_with_a_component_member() => Assert.Equal(2027, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddYears(1).Year))); @@ -651,7 +841,7 @@ public async Task UtcNow_runs_on_the_server() // If EF Core evaluated DateTime.UtcNow on the client, the SQL would carry a literal instead. Assert.Contains("now64", query.ToQueryString()); - Assert.Equal([1L, 2L], await query.OrderBy(id => id).ToListAsync()); + Assert.Equal([1L, 2L, 3L], await query.OrderBy(id => id).ToListAsync()); var none = await context.Events.AsNoTracking() .Where(e => e.Timestamp64 < DateTime.UtcNow.AddYears(-100)) @@ -707,6 +897,51 @@ public async Task Subtracting_two_date_times_in_a_predicate_reports_a_clear_reas Assert.Contains("Arithmetic on two date or time values", exception.Message); } + [Fact] + public async Task Subtracting_a_TimeSpan_in_a_predicate_points_at_the_Add_methods() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // 'x - TimeSpan.FromDays(7)' is a common way to write a rolling window, and the advice for + // subtracting two dates does not fit it: AddDays(-7) translates. + var query = context.Events.AsNoTracking() + .Where(e => e.Timestamp64 > DateTime.UtcNow - TimeSpan.FromDays(7)); + + var exception = await Assert.ThrowsAsync(() => query.ToListAsync()); + + Assert.Contains("Use the Add* methods instead", exception.Message); + Assert.Contains("AddDays(-7)", exception.Message); + } + + [Fact] + public async Task A_repeated_untranslatable_call_reports_its_reason_once() + { + var days = 1.5; + + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking() + .Where(e => e.Timestamp64.AddDays(days) > e.Timestamp + && e.Timestamp64.AddDays(days) < e.Timestamp); + + var exception = await Assert.ThrowsAsync(() => query.ToListAsync()); + + const string reason = "argument must be a constant"; + Assert.Equal(1, CountOccurrences(exception.Message, reason)); + } + + private static int CountOccurrences(string haystack, string needle) + { + var count = 0; + for (var i = haystack.IndexOf(needle, StringComparison.Ordinal); + i >= 0; + i = haystack.IndexOf(needle, i + needle.Length, StringComparison.Ordinal)) + { + count++; + } + + return count; + } + [Fact] public async Task Subtracting_two_times_of_day_in_a_projection_works() {