You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/03/17 04:50:37 UTC
[GitHub] [arrow-rs] anliakho2 opened a new pull request #1460: Integrate fixes for Timestamp[MICROS] and infinite loop hang when reading parquet
anliakho2 opened a new pull request #1460:
URL: https://github.com/apache/arrow-rs/pull/1460
# Which issue does this PR close?
Closes #1459 , #1458 .
# Rationale for this change
Fixing few issues experiences reading parquet files
# What changes are included in this PR?
Add handling for the Timestamp(TimeUnit::MICROS) and Timestamp(TimeUnit::MILLIS) logical types.
Also handle RLE bit packed runs that don't have proper full packing.
# Are there any user-facing changes?
no, except that more files would be read correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a change in pull request #1460: Integrate fixes for Timestamp[MICROS] and infinite loop hang when reading parquet
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #1460:
URL: https://github.com/apache/arrow-rs/pull/1460#discussion_r830342143
##########
File path: parquet/src/arrow/array_reader.rs
##########
@@ -428,6 +430,24 @@ where
}
Arc::new(builder.finish()) as ArrayRef
}
+ ArrowType::Timestamp(ArrowTimeUnit::Nanosecond, ref tz) => {
+ if let Some(LogicalType::TIMESTAMP(t)) = self.column_desc.logical_type() {
+ match t.unit {
+ TimeUnit::MICROS(_) => {
+ let a = arrow::compute::cast(&array, &ArrowType::Timestamp(Microsecond, tz.clone()))?;
+ arrow::compute::cast(&a, &ArrowType::Timestamp(Nanosecond, tz.clone()))?
+ }
+ TimeUnit::MILLIS(_) => {
+ let a = arrow::compute::cast(&array, &ArrowType::Timestamp(Millisecond, tz.clone()))?;
+ arrow::compute::cast(&a, &ArrowType::Timestamp(Nanosecond, tz.clone()))?
+ }
+ _ => arrow::compute::cast(&array, &target_type.clone())?
+ }
+ }
Review comment:
The logic looks correct. But we need some tests to verify them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] anliakho2 commented on pull request #1460: Integrate fixes for Timestamp[MICROS] and infinite loop hang when reading parquet
Posted by GitBox <gi...@apache.org>.
anliakho2 commented on pull request #1460:
URL: https://github.com/apache/arrow-rs/pull/1460#issuecomment-1072903583
@viirya @alamb Will do tests shortly, need some time to finish up tasks at work :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org