You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "xinlifoobar (via GitHub)" <gi...@apache.org> on 2024/05/29 10:56:31 UTC

[PR] Try to Improve performance of extracting statistics from parquet files [datafusion]

xinlifoobar opened a new pull request, #10711:
URL: https://github.com/apache/datafusion/pull/10711

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Closes #10626
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   ## What changes are included in this PR?
   
   Replace the `get_statatistics` macro by `get_statistics_iter` to pass the whole iterator as argument and avoid the usage of intermediate `ScalarValue`. Some improvements comparing to current `main` branch.
   
   ```bash
   $ cargo bench --bench parquet_statistic -- --baseline main
   
   Extract statistics for UInt64/extract_statistics/UInt64
                           time:   [698.93 ns 699.44 ns 700.00 ns]
                           change: [-22.340% -22.130% -21.921%] (p = 0.00 < 0.05)
                           Performance has improved.
   Found 8 outliers among 100 measurements (8.00%)
     1 (1.00%) low severe
     4 (4.00%) high mild
     3 (3.00%) high severe
   
   Extract statistics for F64/extract_statistics/F64
                           time:   [1.0565 µs 1.0578 µs 1.0591 µs]
                           change: [-18.722% -18.049% -17.495%] (p = 0.00 < 0.05)
                           Performance has improved.
   Found 4 outliers among 100 measurements (4.00%)
     1 (1.00%) low mild
     3 (3.00%) high mild
   
   Extract statistics for String/extract_statistics/String
                           time:   [1.4269 µs 1.4282 µs 1.4297 µs]
                           change: [-14.782% -13.611% -12.786%] (p = 0.00 < 0.05)
                           Performance has improved.
   Found 6 outliers among 100 measurements (6.00%)
     1 (1.00%) low severe
     1 (1.00%) low mild
     1 (1.00%) high mild
     3 (3.00%) high severe
   
   Extract statistics for Dictionary(Int32, String)/extract_statistics/Dictionary(Int32, String)
                           time:   [1.6035 µs 1.6051 µs 1.6066 µs]
                           change: [+0.5829% +3.2782% +5.6259%] (p = 0.01 < 0.05)
                           Change within noise threshold.
   Found 10 outliers among 100 measurements (10.00%)
     7 (7.00%) low mild
     3 (3.00%) high severe
   
   ```
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ## Are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ## Are there any user-facing changes?
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "xinlifoobar (via GitHub)" <gi...@apache.org>.
xinlifoobar commented on code in PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#discussion_r1618850434


##########
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs:
##########
@@ -211,32 +394,25 @@ pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics
     data_type: &DataType,
     iterator: I,
 ) -> Result<ArrayRef> {
-    let scalars = iterator
-        .map(|x| x.and_then(|s| get_statistic!(s, min, min_bytes, Some(data_type))));
-    collect_scalars(data_type, scalars)
+    match get_statistics_iter!(iterator, min, min_bytes, Some(data_type)) {

Review Comment:
   This looks good to me but without macros the repetitions between `min` and `max` are  still there ... Do you suggest to have another macro to wrap all above and reuse it in min statistic and max statistic?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#issuecomment-2140671404

   🤔  I seem to have broken a test -- I will review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#issuecomment-2142182674

   the other obvious thing to do in this PR might be to use a macro to avoid the copy/paste between min_statistics and max_statistics...
   
   So I guess the question is "shall we keep working on this PR or shall we plan to work on it as a follow on PR" 🤔 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "xinlifoobar (via GitHub)" <gi...@apache.org>.
xinlifoobar commented on PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#issuecomment-2148814728

   Seems there is a CI issue, not related to this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files, support correct timestamp extraction [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#issuecomment-2148381401

   Converting to draft as we figure out next steps


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files, support correct timestamp extraction [datafusion]

Posted by "xinlifoobar (via GitHub)" <gi...@apache.org>.
xinlifoobar commented on PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#issuecomment-2143614517

   > Thanks again @xinlifoobar -- I went through this PR and I don't think we can merge it yet as is because it has a regression.
   > 
   > The current code will extract the values as the underlying parquet type (e.g. as an Int32Array) and then the predicate pruning code will cast it as necessary
   > 
   > However, when I pushed one more commit to make the types that are not yet handled explicit (rather than relying on a `_`) I see a bunch of types that aren't yet covered that are important (like `Interval` for example)
   > 
   > Thus I suggest:
   > 
   > 1. Break out your change to support extracting timestamps with/without timezones as its own PR (it is quite good)
   > 2. We then work on filling out tests for the other types (I'll file tickets)
   > 3. Then we can come back to this PR (or maybe we want to work on it in parallel)
   > 
   > I am sorry I didn't see this before and I am sorry that we clearly don't have adequate test coverage
   
   LGTM. It was clear to me what it is missing. Thanks!
   
   This PR is mixed with too many items I think... I will move out some of the code, e.g., for timestamps, later in separated PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#issuecomment-2140862882

   🤔  I think what is happening is that statistics for dictionary types are not being handled correctly. Investigting


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "xinlifoobar (via GitHub)" <gi...@apache.org>.
xinlifoobar commented on PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#issuecomment-2142516238

   > the other obvious thing to do in this PR might be to use a macro to avoid the copy/paste between min_statistics and max_statistics...
   > 
   > So I guess the question is "shall we keep working on this PR or shall we plan to work on it as a follow on PR" 🤔
   
   How about we merge this and do all the listings in later PRs. Personally, I think to keep a long history of PR may not a good practice. Some following I may thing:
   
   1. Merging the make_statistics_iter and make_decimal_statistics_iter using `&[&ParquetStatistics]`
   2. Avoid the duplicate code of min_statistics and max_statistcs, could be another macro to concat `Min`/`Max` and `StatisticIterator`?
   3. Complete the implementations of statistics for Timestamp and Dictionary.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files, support correct timestamp extraction [datafusion]

Posted by "xinlifoobar (via GitHub)" <gi...@apache.org>.
xinlifoobar commented on PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#issuecomment-2148806273

   > I think we have now filled out support for the other data types, so perhaps we can revisit the code in this PR again. What do you think @xinlifoobar ?
   
   Hey @alamb, I just republished this PR with some major changes including:
   - Add types we missed in previous there.
   - Play around with the [`paste! `](https://crates.io/crates/paste) macro which I found it may useful here to remove the duplicate codes. (Feel free to revert the change though).
   
   I also play around the `make_stats_iterator` and `make_decimal_stats_iterator` with the paste macro... which made the code difficult to read. Here I recommend keeping them separately and clean. What do you think?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "xinlifoobar (via GitHub)" <gi...@apache.org>.
xinlifoobar commented on PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#issuecomment-2141115385

   > This is awesome -- thank you so much @xinlifoobar -- good team effort ✋
   > 
   > I took the liberty of also porting over the Decimal statistics extraction and removed the old `get_statistic` code and pushed that to your branch
   
   I'd like the idea to use the partial specialization for Decimal, I am actually thinking of passing an array to the `get_statistic` macro to make it more generic. However, in opposite, we will lose some validation abilities.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#discussion_r1621096433


##########
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs:
##########
@@ -556,8 +885,8 @@ mod test {
                 None,
                 None,
             ]),
-            expected_min: timestamp_array([Some(1), Some(5), None]),
-            expected_max: timestamp_array([Some(3), Some(9), None]),
+            expected_min: i64_array([Some(1), Some(5), None]),

Review Comment:
   I see this is consistent with the existing code 👍 



##########
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs:
##########
@@ -211,19 +323,239 @@ pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics
     data_type: &DataType,
     iterator: I,
 ) -> Result<ArrayRef> {
-    let scalars = iterator
-        .map(|x| x.and_then(|s| get_statistic!(s, min, min_bytes, Some(data_type))));
-    collect_scalars(data_type, scalars)
+    match data_type {
+        DataType::Boolean => Ok(Arc::new(BooleanArray::from_iter(
+            MinBooleanStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Int8 => Ok(Arc::new(Int8Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = i8::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::Int16 => Ok(Arc::new(Int16Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = i16::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::Int32 => Ok(Arc::new(Int32Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Int64 => Ok(Arc::new(Int64Array::from_iter(
+            MinInt64StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::UInt8 => Ok(Arc::new(UInt8Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = u8::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::UInt16 => Ok(Arc::new(UInt16Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = u16::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::UInt32 => Ok(Arc::new(UInt32Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| x.map(|x| *x as u32)),
+        ))),
+        DataType::UInt64 => Ok(Arc::new(UInt64Array::from_iter(
+            MinInt64StatsIterator::new(iterator).map(|x| x.map(|x| *x as u64)),
+        ))),
+        DataType::Float32 => Ok(Arc::new(Float32Array::from_iter(
+            MinFloatStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Float64 => Ok(Arc::new(Float64Array::from_iter(
+            MinDoubleStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Date32 => Ok(Arc::new(Date32Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Date64 => Ok(Arc::new(Date64Array::from_iter(
+            MinInt32StatsIterator::new(iterator)
+                .map(|x| x.map(|x| i64::from(*x) * 24 * 60 * 60 * 1000)),
+        ))),
+        DataType::Timestamp(_, _) => Ok(Arc::new(Int64Array::from_iter(
+            MinInt64StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Binary => Ok(Arc::new(BinaryArray::from_iter(
+            MinByteArrayStatsIterator::new(iterator).map(|x| x.map(|x| x.to_vec())),
+        ))),
+        DataType::Utf8 => Ok(Arc::new(StringArray::from_iter(
+            MinByteArrayStatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    let res = std::str::from_utf8(x).map(|s| s.to_string()).ok();
+                    if res.is_none() {
+                        log::debug!("Utf8 statistics is a non-UTF8 value, ignoring it.");
+                    }
+                    res
+                })
+            }),
+        ))),
+        DataType::FixedSizeBinary(size) => Ok(Arc::new(FixedSizeBinaryArray::from(
+            MinFixedLenByteArrayStatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if x.len().try_into() == Ok(*size) {
+                        Some(x)
+                    } else {
+                        log::debug!(
+                            "FixedSizeBinary({}) statistics is a binary of size {}, ignoring it.",
+                            size,
+                            x.len(),
+                        );
+                        None
+                    }
+                })
+            }).collect::<Vec<_>>(),
+        ))),
+        _ => {
+            let scalars = iterator.map(|x| {
+                x.and_then(|s| get_statistic!(s, min, min_bytes, Some(data_type)))
+            });
+            collect_scalars(data_type, scalars)
+        }
+    }
 }
 
 /// Extracts the max statistics from an iterator of [`ParquetStatistics`] to an [`ArrayRef`]
 pub(crate) fn max_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics>>>(
     data_type: &DataType,
     iterator: I,
 ) -> Result<ArrayRef> {
-    let scalars = iterator
-        .map(|x| x.and_then(|s| get_statistic!(s, max, max_bytes, Some(data_type))));
-    collect_scalars(data_type, scalars)
+    match data_type {
+        DataType::Boolean => Ok(Arc::new(BooleanArray::from_iter(
+            MaxBooleanStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Int8 => Ok(Arc::new(Int8Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = i8::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::Int16 => Ok(Arc::new(Int16Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = i16::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::Int32 => Ok(Arc::new(Int32Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Int64 => Ok(Arc::new(Int64Array::from_iter(
+            MaxInt64StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::UInt8 => Ok(Arc::new(UInt8Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = u8::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::UInt16 => Ok(Arc::new(UInt16Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = u16::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::UInt32 => Ok(Arc::new(UInt32Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| x.map(|x| *x as u32)),
+        ))),
+        DataType::UInt64 => Ok(Arc::new(UInt64Array::from_iter(
+            MaxInt64StatsIterator::new(iterator).map(|x| x.map(|x| *x as u64)),
+        ))),
+        DataType::Float32 => Ok(Arc::new(Float32Array::from_iter(
+            MaxFloatStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Float64 => Ok(Arc::new(Float64Array::from_iter(
+            MaxDoubleStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Date32 => Ok(Arc::new(Date32Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Date64 => Ok(Arc::new(Date64Array::from_iter(
+            MaxInt32StatsIterator::new(iterator)
+                .map(|x| x.map(|x| i64::from(*x) * 24 * 60 * 60 * 1000)),
+        ))),
+        DataType::Timestamp(_, _) => Ok(Arc::new(Int64Array::from_iter(

Review Comment:
   It is consistent with the existing code



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#discussion_r1621096433


##########
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs:
##########
@@ -556,8 +885,8 @@ mod test {
                 None,
                 None,
             ]),
-            expected_min: timestamp_array([Some(1), Some(5), None]),
-            expected_max: timestamp_array([Some(3), Some(9), None]),
+            expected_min: i64_array([Some(1), Some(5), None]),

Review Comment:
   I see this is consistent with the existing code 👍 
   
   (note to other reviewers is that this test resulted in errors previously)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#discussion_r1619384338


##########
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs:
##########
@@ -211,32 +394,25 @@ pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics
     data_type: &DataType,
     iterator: I,
 ) -> Result<ArrayRef> {
-    let scalars = iterator
-        .map(|x| x.and_then(|s| get_statistic!(s, min, min_bytes, Some(data_type))));
-    collect_scalars(data_type, scalars)
+    match get_statistics_iter!(iterator, min, min_bytes, Some(data_type)) {

Review Comment:
   I played around with this code this afternoon and here is what I came up with: https://github.com/apache/datafusion/pull/10715
   
   What do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#issuecomment-2142179438

   > I'd like the idea to use the partial specialization for Decimal, I am actually thinking of passing an array to the make_stats_iterator macro to make it more generic. However, in opposite, we will lose some validation abilities.
   
   I think using a `&[&ParquetStatistics]`  (aka an array of references to statistics) would be good.
   
   Let's ensure it doesn't need `&[ParquetStatistics]` (an array of references to owned statistics) as I would like very much to be able to only evaluate this code for row groups that we haven't already filtered out (aka I want to be able to filter the statistics prior to passing down here)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan merged PR #10711:
URL: https://github.com/apache/datafusion/pull/10711


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#discussion_r1618800964


##########
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs:
##########
@@ -52,131 +55,311 @@ fn sign_extend_be(b: &[u8]) -> [u8; 16] {
     result
 }
 
-/// Extract a single min/max statistics from a [`ParquetStatistics`] object
-///
-/// * `$column_statistics` is the `ParquetStatistics` object
-/// * `$func is the function` (`min`/`max`) to call to get the value
-/// * `$bytes_func` is the function (`min_bytes`/`max_bytes`) to call to get the value as bytes
-/// * `$target_arrow_type` is the [`DataType`] of the target statistics
-macro_rules! get_statistic {
-    ($column_statistics:expr, $func:ident, $bytes_func:ident, $target_arrow_type:expr) => {{
-        if !$column_statistics.has_min_max_set() {
-            return None;
-        }
-        match $column_statistics {
-            ParquetStatistics::Boolean(s) => Some(ScalarValue::Boolean(Some(*s.$func()))),
-            ParquetStatistics::Int32(s) => {
-                match $target_arrow_type {
-                    // int32 to decimal with the precision and scale
-                    Some(DataType::Decimal128(precision, scale)) => {
-                        Some(ScalarValue::Decimal128(
-                            Some(*s.$func() as i128),
-                            *precision,
-                            *scale,
-                        ))
-                    }
-                    Some(DataType::Int8) => {
-                        Some(ScalarValue::Int8(Some((*s.$func()).try_into().unwrap())))
-                    }
-                    Some(DataType::Int16) => {
-                        Some(ScalarValue::Int16(Some((*s.$func()).try_into().unwrap())))
-                    }
-                    Some(DataType::UInt8) => {
-                        Some(ScalarValue::UInt8(Some((*s.$func()).try_into().unwrap())))
-                    }
-                    Some(DataType::UInt16) => {
-                        Some(ScalarValue::UInt16(Some((*s.$func()).try_into().unwrap())))
-                    }
-                    Some(DataType::UInt32) => {
-                        Some(ScalarValue::UInt32(Some((*s.$func()) as u32)))
-                    }
-                    Some(DataType::Date32) => {
-                        Some(ScalarValue::Date32(Some(*s.$func())))
-                    }
-                    Some(DataType::Date64) => {
-                        Some(ScalarValue::Date64(Some(i64::from(*s.$func()) * 24 * 60 * 60 * 1000)))
-                    }
-                    _ => Some(ScalarValue::Int32(Some(*s.$func()))),
-                }
+macro_rules! get_statistics_iter {
+    ($column_statistics_iter:expr, $func:ident, $bytes_func:ident, $target_arrow_type:expr) => {{
+        match $target_arrow_type {
+            Some(DataType::Boolean) => Some(Arc::new(BooleanArray::from_iter(
+                $column_statistics_iter.map(|statistic| {
+                    statistic.and_then(|x| {
+                        if !x.has_min_max_set() {
+                            return None;
+                        }
+                        match x {
+                            ParquetStatistics::Boolean(s) => Some(*s.$func()),
+                            _ => None,
+                        }
+                    })
+                }),
+            )) as ArrayRef),
+            Some(DataType::Int8) => Some(Arc::new(Int8Array::from_iter(
+                $column_statistics_iter.map(|statistic| {
+                    statistic.and_then(|x| {
+                        if !x.has_min_max_set() {
+                            return None;
+                        }
+                        match x {
+                            ParquetStatistics::Int32(s) => {
+                                Some((*s.$func()).try_into().unwrap())

Review Comment:
   I think these unwraps are not great (though I see you didn't add them) -- as they may panic on invalid parquet statistics



##########
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs:
##########
@@ -52,131 +55,311 @@ fn sign_extend_be(b: &[u8]) -> [u8; 16] {
     result
 }
 
-/// Extract a single min/max statistics from a [`ParquetStatistics`] object
-///
-/// * `$column_statistics` is the `ParquetStatistics` object
-/// * `$func is the function` (`min`/`max`) to call to get the value
-/// * `$bytes_func` is the function (`min_bytes`/`max_bytes`) to call to get the value as bytes
-/// * `$target_arrow_type` is the [`DataType`] of the target statistics
-macro_rules! get_statistic {
-    ($column_statistics:expr, $func:ident, $bytes_func:ident, $target_arrow_type:expr) => {{
-        if !$column_statistics.has_min_max_set() {
-            return None;
-        }
-        match $column_statistics {
-            ParquetStatistics::Boolean(s) => Some(ScalarValue::Boolean(Some(*s.$func()))),
-            ParquetStatistics::Int32(s) => {
-                match $target_arrow_type {
-                    // int32 to decimal with the precision and scale
-                    Some(DataType::Decimal128(precision, scale)) => {
-                        Some(ScalarValue::Decimal128(
-                            Some(*s.$func() as i128),
-                            *precision,
-                            *scale,
-                        ))
-                    }
-                    Some(DataType::Int8) => {
-                        Some(ScalarValue::Int8(Some((*s.$func()).try_into().unwrap())))
-                    }
-                    Some(DataType::Int16) => {
-                        Some(ScalarValue::Int16(Some((*s.$func()).try_into().unwrap())))
-                    }
-                    Some(DataType::UInt8) => {
-                        Some(ScalarValue::UInt8(Some((*s.$func()).try_into().unwrap())))
-                    }
-                    Some(DataType::UInt16) => {
-                        Some(ScalarValue::UInt16(Some((*s.$func()).try_into().unwrap())))
-                    }
-                    Some(DataType::UInt32) => {
-                        Some(ScalarValue::UInt32(Some((*s.$func()) as u32)))
-                    }
-                    Some(DataType::Date32) => {
-                        Some(ScalarValue::Date32(Some(*s.$func())))
-                    }
-                    Some(DataType::Date64) => {
-                        Some(ScalarValue::Date64(Some(i64::from(*s.$func()) * 24 * 60 * 60 * 1000)))
-                    }
-                    _ => Some(ScalarValue::Int32(Some(*s.$func()))),
-                }
+macro_rules! get_statistics_iter {

Review Comment:
   Could you please update the doc strings to explain this macro a little more -- I think especially without type infomration doc strings on macros makes them easier to understand



##########
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs:
##########
@@ -211,32 +394,25 @@ pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics
     data_type: &DataType,
     iterator: I,
 ) -> Result<ArrayRef> {
-    let scalars = iterator
-        .map(|x| x.and_then(|s| get_statistic!(s, min, min_bytes, Some(data_type))));
-    collect_scalars(data_type, scalars)
+    match get_statistics_iter!(iterator, min, min_bytes, Some(data_type)) {

Review Comment:
   This is pretty neat @xinlifoobar 
   
   As this is currently implemented, it has quite a bit of repetition (e.g. the clauses for                             `ParquetStatistics::ByteArray(s)` in the macro above
   
   What do you think of a slightly different mode where the iteration over the statistics values and the array creation looked different
   
   Something like
   ```rust
   match data_type {
     DataType::Int8 => {
       // get an iterator over int32 stored in 
       let int32_iter = get_statistics_iter!(iterator, min, ParquetStatistics::Int32)
       // convert int32 to int8, ignoring 
       let int8_iter = int32_iter.map(|v| if let Ok(v) = v.try_into:<i8>() { Some(v) } else { None });
       Int8Array::from_iter(int8_iter)
     } 
     // similarly for other types
    ....
     DataType::Utf8 => {
       // get an iterator over bytes stored
       let byte_iter = get_statistics_iter!(iterator, min_bytes, ParquetStatistics::ByteArray)
       // convert bytes to utf8 strings
       let string_array = byte_iter.map(|v| if let Ok(v) = str::from_utf8....);
       StringArray::from_iter(int8_iter)
     } 
    ...
   ```
   
   Another reason I suggest this structure is that I think it would make it easier to support extracting statistics from multiple files in the future more easily



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "xinlifoobar (via GitHub)" <gi...@apache.org>.
xinlifoobar commented on code in PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#discussion_r1618850434


##########
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs:
##########
@@ -211,32 +394,25 @@ pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics
     data_type: &DataType,
     iterator: I,
 ) -> Result<ArrayRef> {
-    let scalars = iterator
-        .map(|x| x.and_then(|s| get_statistic!(s, min, min_bytes, Some(data_type))));
-    collect_scalars(data_type, scalars)
+    match get_statistics_iter!(iterator, min, min_bytes, Some(data_type)) {

Review Comment:
   This looks good to me but without macros the repetitions between `min` and `max` are  still there ... Do you think it will be better to have another macro to wrap all above and reuse it in min statistic and max statistic?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "xinlifoobar (via GitHub)" <gi...@apache.org>.
xinlifoobar commented on code in PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#discussion_r1618850434


##########
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs:
##########
@@ -211,32 +394,25 @@ pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics
     data_type: &DataType,
     iterator: I,
 ) -> Result<ArrayRef> {
-    let scalars = iterator
-        .map(|x| x.and_then(|s| get_statistic!(s, min, min_bytes, Some(data_type))));
-    collect_scalars(data_type, scalars)
+    match get_statistics_iter!(iterator, min, min_bytes, Some(data_type)) {

Review Comment:
   This looks good to me but without macros the repetations, between `min` are `max`  still... Do you suggest to have another macro to wrap all above and reuse it in min statistic and max statistic?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "xinlifoobar (via GitHub)" <gi...@apache.org>.
xinlifoobar commented on PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#issuecomment-2139329183

   Update the benchmark to reflect the results of newer commit.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "xinlifoobar (via GitHub)" <gi...@apache.org>.
xinlifoobar commented on code in PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#discussion_r1621882434


##########
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs:
##########
@@ -211,19 +323,239 @@ pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics
     data_type: &DataType,
     iterator: I,
 ) -> Result<ArrayRef> {
-    let scalars = iterator
-        .map(|x| x.and_then(|s| get_statistic!(s, min, min_bytes, Some(data_type))));
-    collect_scalars(data_type, scalars)
+    match data_type {
+        DataType::Boolean => Ok(Arc::new(BooleanArray::from_iter(
+            MinBooleanStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Int8 => Ok(Arc::new(Int8Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = i8::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::Int16 => Ok(Arc::new(Int16Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = i16::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::Int32 => Ok(Arc::new(Int32Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Int64 => Ok(Arc::new(Int64Array::from_iter(
+            MinInt64StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::UInt8 => Ok(Arc::new(UInt8Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = u8::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::UInt16 => Ok(Arc::new(UInt16Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = u16::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::UInt32 => Ok(Arc::new(UInt32Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| x.map(|x| *x as u32)),
+        ))),
+        DataType::UInt64 => Ok(Arc::new(UInt64Array::from_iter(
+            MinInt64StatsIterator::new(iterator).map(|x| x.map(|x| *x as u64)),
+        ))),
+        DataType::Float32 => Ok(Arc::new(Float32Array::from_iter(
+            MinFloatStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Float64 => Ok(Arc::new(Float64Array::from_iter(
+            MinDoubleStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Date32 => Ok(Arc::new(Date32Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Date64 => Ok(Arc::new(Date64Array::from_iter(
+            MinInt32StatsIterator::new(iterator)
+                .map(|x| x.map(|x| i64::from(*x) * 24 * 60 * 60 * 1000)),
+        ))),
+        DataType::Timestamp(_, _) => Ok(Arc::new(Int64Array::from_iter(
+            MinInt64StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Binary => Ok(Arc::new(BinaryArray::from_iter(
+            MinByteArrayStatsIterator::new(iterator).map(|x| x.map(|x| x.to_vec())),
+        ))),
+        DataType::Utf8 => Ok(Arc::new(StringArray::from_iter(
+            MinByteArrayStatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    let res = std::str::from_utf8(x).map(|s| s.to_string()).ok();
+                    if res.is_none() {
+                        log::debug!("Utf8 statistics is a non-UTF8 value, ignoring it.");
+                    }
+                    res
+                })
+            }),
+        ))),
+        DataType::FixedSizeBinary(size) => Ok(Arc::new(FixedSizeBinaryArray::from(
+            MinFixedLenByteArrayStatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if x.len().try_into() == Ok(*size) {
+                        Some(x)
+                    } else {
+                        log::debug!(
+                            "FixedSizeBinary({}) statistics is a binary of size {}, ignoring it.",
+                            size,
+                            x.len(),
+                        );
+                        None
+                    }
+                })
+            }).collect::<Vec<_>>(),
+        ))),
+        _ => {
+            let scalars = iterator.map(|x| {
+                x.and_then(|s| get_statistic!(s, min, min_bytes, Some(data_type)))
+            });
+            collect_scalars(data_type, scalars)
+        }
+    }
 }
 
 /// Extracts the max statistics from an iterator of [`ParquetStatistics`] to an [`ArrayRef`]
 pub(crate) fn max_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics>>>(
     data_type: &DataType,
     iterator: I,
 ) -> Result<ArrayRef> {
-    let scalars = iterator
-        .map(|x| x.and_then(|s| get_statistic!(s, max, max_bytes, Some(data_type))));
-    collect_scalars(data_type, scalars)
+    match data_type {
+        DataType::Boolean => Ok(Arc::new(BooleanArray::from_iter(
+            MaxBooleanStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Int8 => Ok(Arc::new(Int8Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = i8::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::Int16 => Ok(Arc::new(Int16Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = i16::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::Int32 => Ok(Arc::new(Int32Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Int64 => Ok(Arc::new(Int64Array::from_iter(
+            MaxInt64StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::UInt8 => Ok(Arc::new(UInt8Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = u8::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::UInt16 => Ok(Arc::new(UInt16Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = u16::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::UInt32 => Ok(Arc::new(UInt32Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| x.map(|x| *x as u32)),
+        ))),
+        DataType::UInt64 => Ok(Arc::new(UInt64Array::from_iter(
+            MaxInt64StatsIterator::new(iterator).map(|x| x.map(|x| *x as u64)),
+        ))),
+        DataType::Float32 => Ok(Arc::new(Float32Array::from_iter(
+            MaxFloatStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Float64 => Ok(Arc::new(Float64Array::from_iter(
+            MaxDoubleStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Date32 => Ok(Arc::new(Date32Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Date64 => Ok(Arc::new(Date64Array::from_iter(
+            MaxInt32StatsIterator::new(iterator)
+                .map(|x| x.map(|x| i64::from(*x) * 24 * 60 * 60 * 1000)),
+        ))),
+        DataType::Timestamp(_, _) => Ok(Arc::new(Int64Array::from_iter(

Review Comment:
   I created another PR to fix the timestamp statistic types. Let me know if you want it to be part of this PR or a following PR.
   
   https://github.com/xinlifoobar/datafusion/pull/1/files



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "xinlifoobar (via GitHub)" <gi...@apache.org>.
xinlifoobar commented on PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#issuecomment-2141111458

   Thanks @alamb for the extensive help. I have updated the latest benchmark from my local machine to the descriptions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#discussion_r1620851754


##########
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs:
##########
@@ -536,9 +868,6 @@ mod test {
     }
 
     #[test]
-    #[should_panic(

Review Comment:
   🎉 



##########
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs:
##########
@@ -211,19 +323,239 @@ pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics
     data_type: &DataType,
     iterator: I,
 ) -> Result<ArrayRef> {
-    let scalars = iterator
-        .map(|x| x.and_then(|s| get_statistic!(s, min, min_bytes, Some(data_type))));
-    collect_scalars(data_type, scalars)
+    match data_type {
+        DataType::Boolean => Ok(Arc::new(BooleanArray::from_iter(
+            MinBooleanStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Int8 => Ok(Arc::new(Int8Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = i8::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::Int16 => Ok(Arc::new(Int16Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = i16::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::Int32 => Ok(Arc::new(Int32Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Int64 => Ok(Arc::new(Int64Array::from_iter(
+            MinInt64StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::UInt8 => Ok(Arc::new(UInt8Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = u8::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::UInt16 => Ok(Arc::new(UInt16Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = u16::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::UInt32 => Ok(Arc::new(UInt32Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| x.map(|x| *x as u32)),
+        ))),
+        DataType::UInt64 => Ok(Arc::new(UInt64Array::from_iter(
+            MinInt64StatsIterator::new(iterator).map(|x| x.map(|x| *x as u64)),
+        ))),
+        DataType::Float32 => Ok(Arc::new(Float32Array::from_iter(
+            MinFloatStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Float64 => Ok(Arc::new(Float64Array::from_iter(
+            MinDoubleStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Date32 => Ok(Arc::new(Date32Array::from_iter(
+            MinInt32StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Date64 => Ok(Arc::new(Date64Array::from_iter(
+            MinInt32StatsIterator::new(iterator)
+                .map(|x| x.map(|x| i64::from(*x) * 24 * 60 * 60 * 1000)),
+        ))),
+        DataType::Timestamp(_, _) => Ok(Arc::new(Int64Array::from_iter(
+            MinInt64StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Binary => Ok(Arc::new(BinaryArray::from_iter(
+            MinByteArrayStatsIterator::new(iterator).map(|x| x.map(|x| x.to_vec())),
+        ))),
+        DataType::Utf8 => Ok(Arc::new(StringArray::from_iter(
+            MinByteArrayStatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    let res = std::str::from_utf8(x).map(|s| s.to_string()).ok();
+                    if res.is_none() {
+                        log::debug!("Utf8 statistics is a non-UTF8 value, ignoring it.");
+                    }
+                    res
+                })
+            }),
+        ))),
+        DataType::FixedSizeBinary(size) => Ok(Arc::new(FixedSizeBinaryArray::from(
+            MinFixedLenByteArrayStatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if x.len().try_into() == Ok(*size) {
+                        Some(x)
+                    } else {
+                        log::debug!(
+                            "FixedSizeBinary({}) statistics is a binary of size {}, ignoring it.",
+                            size,
+                            x.len(),
+                        );
+                        None
+                    }
+                })
+            }).collect::<Vec<_>>(),
+        ))),
+        _ => {
+            let scalars = iterator.map(|x| {
+                x.and_then(|s| get_statistic!(s, min, min_bytes, Some(data_type)))
+            });
+            collect_scalars(data_type, scalars)
+        }
+    }
 }
 
 /// Extracts the max statistics from an iterator of [`ParquetStatistics`] to an [`ArrayRef`]
 pub(crate) fn max_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics>>>(
     data_type: &DataType,
     iterator: I,
 ) -> Result<ArrayRef> {
-    let scalars = iterator
-        .map(|x| x.and_then(|s| get_statistic!(s, max, max_bytes, Some(data_type))));
-    collect_scalars(data_type, scalars)
+    match data_type {
+        DataType::Boolean => Ok(Arc::new(BooleanArray::from_iter(
+            MaxBooleanStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Int8 => Ok(Arc::new(Int8Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = i8::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::Int16 => Ok(Arc::new(Int16Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = i16::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::Int32 => Ok(Arc::new(Int32Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Int64 => Ok(Arc::new(Int64Array::from_iter(
+            MaxInt64StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::UInt8 => Ok(Arc::new(UInt8Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = u8::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::UInt16 => Ok(Arc::new(UInt16Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| {
+                x.and_then(|x| {
+                    if let Ok(v) = u16::try_from(*x) {
+                        Some(v)
+                    } else {
+                        None
+                    }
+                })
+            }),
+        ))),
+        DataType::UInt32 => Ok(Arc::new(UInt32Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| x.map(|x| *x as u32)),
+        ))),
+        DataType::UInt64 => Ok(Arc::new(UInt64Array::from_iter(
+            MaxInt64StatsIterator::new(iterator).map(|x| x.map(|x| *x as u64)),
+        ))),
+        DataType::Float32 => Ok(Arc::new(Float32Array::from_iter(
+            MaxFloatStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Float64 => Ok(Arc::new(Float64Array::from_iter(
+            MaxDoubleStatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Date32 => Ok(Arc::new(Date32Array::from_iter(
+            MaxInt32StatsIterator::new(iterator).map(|x| x.copied()),
+        ))),
+        DataType::Date64 => Ok(Arc::new(Date64Array::from_iter(
+            MaxInt32StatsIterator::new(iterator)
+                .map(|x| x.map(|x| i64::from(*x) * 24 * 60 * 60 * 1000)),
+        ))),
+        DataType::Timestamp(_, _) => Ok(Arc::new(Int64Array::from_iter(

Review Comment:
   I would have expected this to be a Timestamp array rather than an Int64 array 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files [datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #10711:
URL: https://github.com/apache/datafusion/pull/10711#issuecomment-2150262054

   Thank you @xinlifoobar and @alamb 🚀 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org