You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/10/12 02:16:12 UTC
[PR] [python] Align BIGINT and timestamp evolution with Java [paimon]
JingsongLi opened a new pull request, #10508:
URL: https://github.com/apache/paimon/pull/10508
## What is the purpose of the change?
Align type-change paths already supported by PyPaimon with Java and verify them through Native reads. The existing Arrow cast reads timestamp physical ticks as BIGINT and treats BIGINT as timestamp ticks, so the results use the wrong units.
**Depends on https://github.com/apache/paimon-rust/pull/1107.** This PR is a draft until the Rust fix is merged because Native CI deliberately builds paimon-rust from upstream `main`.
## Brief change log
- Use epoch seconds for TIMESTAMP/TIMESTAMP_LTZ → BIGINT, including floor division before the epoch and Java's LTZ local-date-time round trip.
- Use Java's signed-long intermediate multiplication for BIGINT → ordinary TIMESTAMP and retain NULLs for Arrow storage overflow.
- Apply these conversions in the existing shared schema-evolution helper, which is already used by scalar, recursive nested and leaf-projection readers.
- Add 30 unit cases for sliced/chunked arrays, NULLs, Java boundary values and Arrow units.
- Add 43 REST Native parity cases for append, primary-key and data-evolution tables, covering mixed schema versions, single-value file statistics, predicate-only fields, NULLs, ROW/ARRAY/MAP evolution and nested/MAP key projections. Native checks explicitly forbid Python read/plan fallback.
## Verifying this change
- Reproduced the old Native result `42001ms → 42001`, where Java and the corrected Python path return `42`.
- Rebuilt the Native extension using the dependent Rust branch.
- `schema_evolution_cast_test.py` + `native_schema_evolution_cast_test.py`: 166 passed with Native plan/read/write/commit enabled (296 Native plans/reads and 176 Native writes/commits).
- Existing schema-evolution, PK reads, MAP key projection and nested projection suites: 112 passed, 21 subtests passed with Native enabled.
- Final nested epoch/MAP key projection checks: 3 passed across all table modes.
- Changed-file Flake8, `git diff --check` and Python 3.6 grammar parsing passed.
- Java `CastExecutorTest#testTimestampToNumeric+testTimestampToNumericPreEpoch`: 2 passed. Direct Java CastExecutors calls confirmed the overflow golden values.
No new Python API, dependency or unsupported Python type-conversion capability is introduced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Align BIGINT and timestamp evolution with Java [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #10508:
URL: https://github.com/apache/paimon/pull/10508#discussion_r4244336368
##########
paimon-python/pypaimon/casting/schema_evolution.py:
##########
@@ -77,19 +77,60 @@ def _truncate_timestamp(array, target_type, source_paimon_type, target_paimon_ty
return pa.array(values, type=pa.int64()).cast(target_type)
+def _timestamp_to_bigint(array):
+ ticks = {'s': 1, 'ms': 1000, 'us': 1000000, 'ns': 1000000000}[array.type.unit]
+ values = []
+ for value in array.cast(pa.int64()).to_pylist():
+ if value is None:
+ values.append(None)
+ elif array.type.tz is None:
+ values.append(value // ticks)
+ else:
+ # Java's LTZ rule round-trips Timestamp through LocalDateTime.
+ # Timestamp stores its epoch day as an int in that conversion.
+ millis = value * 1000 if array.type.unit == 's' else value // (ticks // 1000)
+ if not -(1 << 63) <= millis < (1 << 63):
+ values.append(None)
+ continue
+ day = _narrow_integer(millis // 86400000, 32)
+ values.append((day * 86400000 + millis % 86400000) // 1000)
+ return pa.array(values, type=pa.int64())
+
+
+def _bigint_to_timestamp(array, target_type):
+ ticks = {'s': 1, 'ms': 1000, 'us': 1000000, 'ns': 1000000000}[target_type.unit]
+ values = []
+ for value in array.to_pylist():
+ if value is None:
+ values.append(None)
+ continue
+ # NumericPrimitiveToTimestamp performs value.longValue() * 1000
+ # before conversion. Java's signed long multiplication wraps.
+ millis = _narrow_integer(value * 1000, 64)
+ value = millis // 1000 if target_type.unit == 's' else millis * (ticks // 1000)
+ values.append(value if -(1 << 63) <= value < (1 << 63) else None)
+ return pa.array(values, type=pa.int64()).cast(target_type)
+
+
def cast_array_for_schema_evolution(array, target_type, source_paimon_type=None, target_paimon_type=None):
"""Materialize existing type changes using Java numeric and timestamp casts.
Declared Paimon types are needed for precision changes within one Arrow
- timestamp unit. Numeric narrowing preserves low bits, while floating-point
- values truncate and saturate to int/long before byte/short narrowing.
+ timestamp unit. BIGINT/timestamp conversions use epoch seconds, including
+ pre-epoch flooring and Java's millis multiplication. Numeric narrowing
+ preserves low bits, while floating-point values truncate and saturate to
+ int/long before byte/short narrowing.
DECIMAL scale reduction uses HALF_UP and returns NULL on precision overflow.
"""
timestamp = _truncate_timestamp(array, target_type, source_paimon_type, target_paimon_type)
if timestamp is not None:
return timestamp
if array.type == target_type:
return array
+ if pa.types.is_timestamp(array.type) and target_type == pa.int64():
+ return _timestamp_to_bigint(array)
+ if array.type == pa.int64() and pa.types.is_timestamp(target_type) and target_type.tz is None:
Review Comment:
Correction after checking the baseline: `INT -> TIMESTAMP` already raises this error before this PR, and the PR explicitly scopes the change to conversions already supported by Python. This is not a regression or a blocker for this PR; I have dismissed my earlier change request. The BIGINT/timestamp changes remain consistent with Java and Rust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Align INT/BIGINT and timestamp evolution with Java [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10508:
URL: https://github.com/apache/paimon/pull/10508
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Align BIGINT and timestamp evolution with Java [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #10508:
URL: https://github.com/apache/paimon/pull/10508#discussion_r4244336368
##########
paimon-python/pypaimon/casting/schema_evolution.py:
##########
@@ -77,19 +77,60 @@ def _truncate_timestamp(array, target_type, source_paimon_type, target_paimon_ty
return pa.array(values, type=pa.int64()).cast(target_type)
+def _timestamp_to_bigint(array):
+ ticks = {'s': 1, 'ms': 1000, 'us': 1000000, 'ns': 1000000000}[array.type.unit]
+ values = []
+ for value in array.cast(pa.int64()).to_pylist():
+ if value is None:
+ values.append(None)
+ elif array.type.tz is None:
+ values.append(value // ticks)
+ else:
+ # Java's LTZ rule round-trips Timestamp through LocalDateTime.
+ # Timestamp stores its epoch day as an int in that conversion.
+ millis = value * 1000 if array.type.unit == 's' else value // (ticks // 1000)
+ if not -(1 << 63) <= millis < (1 << 63):
+ values.append(None)
+ continue
+ day = _narrow_integer(millis // 86400000, 32)
+ values.append((day * 86400000 + millis % 86400000) // 1000)
+ return pa.array(values, type=pa.int64())
+
+
+def _bigint_to_timestamp(array, target_type):
+ ticks = {'s': 1, 'ms': 1000, 'us': 1000000, 'ns': 1000000000}[target_type.unit]
+ values = []
+ for value in array.to_pylist():
+ if value is None:
+ values.append(None)
+ continue
+ # NumericPrimitiveToTimestamp performs value.longValue() * 1000
+ # before conversion. Java's signed long multiplication wraps.
+ millis = _narrow_integer(value * 1000, 64)
+ value = millis // 1000 if target_type.unit == 's' else millis * (ticks // 1000)
+ values.append(value if -(1 << 63) <= value < (1 << 63) else None)
+ return pa.array(values, type=pa.int64()).cast(target_type)
+
+
def cast_array_for_schema_evolution(array, target_type, source_paimon_type=None, target_paimon_type=None):
"""Materialize existing type changes using Java numeric and timestamp casts.
Declared Paimon types are needed for precision changes within one Arrow
- timestamp unit. Numeric narrowing preserves low bits, while floating-point
- values truncate and saturate to int/long before byte/short narrowing.
+ timestamp unit. BIGINT/timestamp conversions use epoch seconds, including
+ pre-epoch flooring and Java's millis multiplication. Numeric narrowing
+ preserves low bits, while floating-point values truncate and saturate to
+ int/long before byte/short narrowing.
DECIMAL scale reduction uses HALF_UP and returns NULL on precision overflow.
"""
timestamp = _truncate_timestamp(array, target_type, source_paimon_type, target_paimon_type)
if timestamp is not None:
return timestamp
if array.type == target_type:
return array
+ if pa.types.is_timestamp(array.type) and target_type == pa.int64():
+ return _timestamp_to_bigint(array)
+ if array.type == pa.int64() and pa.types.is_timestamp(target_type) and target_type.tz is None:
Review Comment:
This condition only handles `pa.int64()`, but Java `NumericPrimitiveToTimestamp.create()` supports both `INTEGER` and `BIGINT`. Paimon `INT` is represented as `pa.int32()`, so an existing `INT -> TIMESTAMP(...)` evolution falls through to `array.cast(target_type)` and PyArrow raises `ArrowNotImplementedError` (`Unsupported cast from int32 to timestamp`). Please include the `pa.int32()` source path (and add a regression case for INT -> TIMESTAMP) so this matches the Rust/Java behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org