You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "HuaHuaY (via GitHub)" <gi...@apache.org> on 2025/11/25 09:25:56 UTC
[PR] GH-48246: [C++][Parquet] fix pre-1970 timestamps roundtrip test [arrow]
HuaHuaY opened a new pull request, #48247:
URL: https://github.com/apache/arrow/pull/48247
### Rationale for this change
Parquet should return the same timestamps as written.
### What changes are included in this PR?
`ArrowTimestampToImpalaTimestamp` will adjust the calculation based on whether the timestamp is negative
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48246: [C++][Parquet] fix pre-1970 timestamps roundtrip test [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48247:
URL: https://github.com/apache/arrow/pull/48247#issuecomment-3574638864
:warning: GitHub issue #48246 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48246: [C++][Parquet] Fix pre-1970 INT96 timestamps roundtrip [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac merged PR #48247:
URL: https://github.com/apache/arrow/pull/48247
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48246: [C++][Parquet] Fix pre-1970 INT96 timestamps roundtrip [arrow]
Posted by "HuaHuaY (via GitHub)" <gi...@apache.org>.
HuaHuaY commented on code in PR #48247:
URL: https://github.com/apache/arrow/pull/48247#discussion_r2562750226
##########
cpp/src/parquet/column_writer.h:
##########
@@ -259,14 +259,15 @@ constexpr int64_t kJulianEpochOffsetDays = INT64_C(2440588);
template <int64_t UnitPerDay, int64_t NanosecondsPerUnit>
inline void ArrowTimestampToImpalaTimestamp(const int64_t time, Int96* impala_timestamp) {
- int64_t julian_days = (time / UnitPerDay) + kJulianEpochOffsetDays;
- (*impala_timestamp).value[2] = (uint32_t)julian_days;
+ int32_t julian_days = static_cast<int32_t>(time / UnitPerDay + kJulianEpochOffsetDays) +
Review Comment:
You are right. I have fixed it and added more tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48246: [C++][Parquet] fix pre-1970 timestamps roundtrip test [arrow]
Posted by "HuaHuaY (via GitHub)" <gi...@apache.org>.
HuaHuaY commented on PR #48247:
URL: https://github.com/apache/arrow/pull/48247#issuecomment-3574640811
@wgtmac @pitrou Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48246: [C++][Parquet] Fix pre-1970 INT96 timestamps roundtrip [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #48247:
URL: https://github.com/apache/arrow/pull/48247#discussion_r2560637334
##########
cpp/src/parquet/column_writer.h:
##########
@@ -259,14 +259,15 @@ constexpr int64_t kJulianEpochOffsetDays = INT64_C(2440588);
template <int64_t UnitPerDay, int64_t NanosecondsPerUnit>
inline void ArrowTimestampToImpalaTimestamp(const int64_t time, Int96* impala_timestamp) {
- int64_t julian_days = (time / UnitPerDay) + kJulianEpochOffsetDays;
- (*impala_timestamp).value[2] = (uint32_t)julian_days;
+ int32_t julian_days = static_cast<int32_t>(time / UnitPerDay + kJulianEpochOffsetDays) +
Review Comment:
I just did some research on how to adjust negative values and it seems that following code looks more correct:
```cpp
int64_t q = time / UnitPerDay;
int64_t r = time % UnitPerDay;
if (r < 0) {
q -= 1;
r += UnitPerDay;
}
uint32_t julian_days = static_cast<uint32_t>(q + kJulianEpochOffsetDays);
impala_timestamp->value[2] = julian_days;
uint64_t last_day_units = static_cast<uint64_t>(r);
uint64_t last_day_nanos = last_day_units * NanosecondsPerUnit;
std::memcpy(impala_timestamp, &last_day_nanos, sizeof(uint64_t));
```
The main difference is when time is an exact multiple of UnitPerDay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48246: [C++][Parquet] Fix pre-1970 INT96 timestamps roundtrip [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #48247:
URL: https://github.com/apache/arrow/pull/48247#discussion_r2560637334
##########
cpp/src/parquet/column_writer.h:
##########
@@ -259,14 +259,15 @@ constexpr int64_t kJulianEpochOffsetDays = INT64_C(2440588);
template <int64_t UnitPerDay, int64_t NanosecondsPerUnit>
inline void ArrowTimestampToImpalaTimestamp(const int64_t time, Int96* impala_timestamp) {
- int64_t julian_days = (time / UnitPerDay) + kJulianEpochOffsetDays;
- (*impala_timestamp).value[2] = (uint32_t)julian_days;
+ int32_t julian_days = static_cast<int32_t>(time / UnitPerDay + kJulianEpochOffsetDays) +
Review Comment:
I just did some research to adjust negative values and seems that following code loos more correct:
```cpp
int64_t q = time / UnitPerDay;
int64_t r = time % UnitPerDay;
if (r < 0) {
q -= 1;
r += UnitPerDay;
}
uint32_t julian_days = static_cast<uint32_t>(q + kJulianEpochOffsetDays);
impala_timestamp->value[2] = julian_days;
uint64_t last_day_units = static_cast<uint64_t>(r);
uint64_t last_day_nanos = last_day_units * NanosecondsPerUnit;
std::memcpy(impala_timestamp, &last_day_nanos, sizeof(uint64_t));
```
The main difference is when time is an exact multiple of UnitPerDay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org