You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/06/01 04:05:02 UTC

[GitHub] [arrow-datafusion] AssHero opened a new pull request, #2671: If statistics of column Max/Min value does not exists in parquet file…

AssHero opened a new pull request, #2671:
URL: https://github.com/apache/arrow-datafusion/pull/2671

   # Which issue does this PR close?
   
   Closes #2176 
   
    # Rationale for this change
   If statistics of column Max/Min value does not exists in parquet file, set the Max/Min Accumulator to None.
   
   # What changes are included in this PR?
   set Max/Min Accumulator to None in parquet.rs and datasource/mod.rs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] AssHero commented on a diff in pull request #2671: If statistics of column Max/Min value does not exists in parquet file…

Posted by GitBox <gi...@apache.org>.
AssHero commented on code in PR #2671:
URL: https://github.com/apache/arrow-datafusion/pull/2671#discussion_r886753238


##########
datafusion/core/src/datasource/file_format/parquet.rs:
##########
@@ -344,6 +362,10 @@ fn fetch_statistics(
                             table_idx,
                             stats,
                         )
+                    } else {
+                        // If none statistics of current column exists, set the Max/Min Accumulator to None.

Review Comment:
   Once max_values and min_values are set to none,  even if the column statistics exists in next row_group, summarize_min_max will do nothing because the max_values and min_values are none.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] AssHero commented on a diff in pull request #2671: If statistics of column Max/Min value does not exists in parquet file, sent Min/Max to None

Posted by GitBox <gi...@apache.org>.
AssHero commented on code in PR #2671:
URL: https://github.com/apache/arrow-datafusion/pull/2671#discussion_r887638150


##########
datafusion/core/src/datasource/file_format/parquet.rs:
##########
@@ -344,6 +362,10 @@ fn fetch_statistics(
                             table_idx,
                             stats,
                         )
+                    } else {
+                        // If none statistics of current column exists, set the Max/Min Accumulator to None.

Review Comment:
   > If there are no statistics in one of the row groups, I wonder if it is correct to keep going 🤔
   
   Are there any statistics for other columns in current row groups?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] Ted-Jiang commented on a diff in pull request #2671: If statistics of column Max/Min value does not exists in parquet file…

Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #2671:
URL: https://github.com/apache/arrow-datafusion/pull/2671#discussion_r886622334


##########
datafusion/core/src/datasource/file_format/parquet.rs:
##########
@@ -344,6 +362,10 @@ fn fetch_statistics(
                             table_idx,
                             stats,
                         )
+                    } else {
+                        // If none statistics of current column exists, set the Max/Min Accumulator to None.

Review Comment:
   If i miss something, plz tell me



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #2671: If statistics of column Max/Min value does not exists in parquet file, sent Min/Max to None

Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #2671:
URL: https://github.com/apache/arrow-datafusion/pull/2671#discussion_r887296508


##########
datafusion/core/src/datasource/file_format/parquet.rs:
##########
@@ -344,6 +362,10 @@ fn fetch_statistics(
                             table_idx,
                             stats,
                         )
+                    } else {
+                        // If none statistics of current column exists, set the Max/Min Accumulator to None.

Review Comment:
   If there are no statistics in one of the row groups, I wonder if it is correct to keep going 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] Ted-Jiang commented on pull request #2671: If statistics of column Max/Min value does not exists in parquet file…

Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on PR #2671:
URL: https://github.com/apache/arrow-datafusion/pull/2671#issuecomment-1143335133

   @AssHero Thanks for your work! 👍  I will review this later today seems it's related to parquet static.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] Ted-Jiang commented on a diff in pull request #2671: If statistics of column Max/Min value does not exists in parquet file, sent Min/Max to None

Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #2671:
URL: https://github.com/apache/arrow-datafusion/pull/2671#discussion_r888051780


##########
datafusion/core/src/datasource/file_format/parquet.rs:
##########
@@ -344,6 +362,10 @@ fn fetch_statistics(
                             table_idx,
                             stats,
                         )
+                    } else {
+                        // If none statistics of current column exists, set the Max/Min Accumulator to None.

Review Comment:
   > > If i miss something, plz tell me
   > 
   > Once max_values and min_values are set to none, even if the column statistics exists in next row_group, summarize_min_max will do nothing because the max_values and min_values are none.
   Ohh, thanks i will check later.
   



##########
datafusion/core/src/datasource/file_format/parquet.rs:
##########
@@ -344,6 +362,10 @@ fn fetch_statistics(
                             table_idx,
                             stats,
                         )
+                    } else {
+                        // If none statistics of current column exists, set the Max/Min Accumulator to None.

Review Comment:
   > > If i miss something, plz tell me
   > 
   > Once max_values and min_values are set to none, even if the column statistics exists in next row_group, summarize_min_max will do nothing because the max_values and min_values are none.
   
   Ohh, thanks i will check later.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on pull request #2671: If statistics of column Max/Min value does not exists in parquet file, sent Min/Max to None

Posted by GitBox <gi...@apache.org>.
alamb commented on PR #2671:
URL: https://github.com/apache/arrow-datafusion/pull/2671#issuecomment-1144715296

   I retriggered the CI checks that were failing -- they appeared due to some network glitch


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold merged pull request #2671: If statistics of column Max/Min value does not exists in parquet file, sent Min/Max to None

Posted by GitBox <gi...@apache.org>.
tustvold merged PR #2671:
URL: https://github.com/apache/arrow-datafusion/pull/2671


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] codecov-commenter commented on pull request #2671: If statistics of column Max/Min value does not exists in parquet file, sent Min/Max to None

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #2671:
URL: https://github.com/apache/arrow-datafusion/pull/2671#issuecomment-1144581716

   # [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/2671?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#2671](https://codecov.io/gh/apache/arrow-datafusion/pull/2671?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c7bc84a) into [master](https://codecov.io/gh/apache/arrow-datafusion/commit/4a55364448c81182731f4fa4c3e65aef5b31fa0f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4a55364) will **increase** coverage by `8.58%`.
   > The diff coverage is `37.50%`.
   
   > :exclamation: Current head c7bc84a differs from pull request most recent head d220c38. Consider uploading reports for the commit d220c38 to get more accurate results
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #2671      +/-   ##
   ==========================================
   + Coverage   76.06%   84.64%   +8.58%     
   ==========================================
     Files         156      267     +111     
     Lines       27124    46926   +19802     
   ==========================================
   + Hits        20631    39720   +19089     
   - Misses       6493     7206     +713     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/arrow-datafusion/pull/2671?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [benchmarks/src/bin/nyctaxi.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/2671/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmVuY2htYXJrcy9zcmMvYmluL255Y3RheGkucnM=) | `0.00% <0.00%> (ø)` | |
   | [datafusion/common/src/column.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/2671/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb21tb24vc3JjL2NvbHVtbi5ycw==) | `95.12% <ø> (ø)` | |
   | [datafusion/common/src/dfschema.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/2671/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb21tb24vc3JjL2Rmc2NoZW1hLnJz) | `97.77% <ø> (ø)` | |
   | [datafusion/common/src/error.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/2671/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb21tb24vc3JjL2Vycm9yLnJz) | `82.27% <ø> (ø)` | |
   | [datafusion/common/src/pyarrow.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/2671/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb21tb24vc3JjL3B5YXJyb3cucnM=) | `0.00% <ø> (ø)` | |
   | [datafusion/common/src/scalar.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/2671/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb21tb24vc3JjL3NjYWxhci5ycw==) | `74.94% <ø> (ø)` | |
   | [datafusion/core/benches/parquet\_query\_sql.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/2671/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL2JlbmNoZXMvcGFycXVldF9xdWVyeV9zcWwucnM=) | `0.00% <ø> (ø)` | |
   | [datafusion/core/fuzz-utils/src/lib.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/2671/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL2Z1enotdXRpbHMvc3JjL2xpYi5ycw==) | `100.00% <ø> (ø)` | |
   | [...usion/core/src/avro\_to\_arrow/arrow\_array\_reader.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/2671/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9hdnJvX3RvX2Fycm93L2Fycm93X2FycmF5X3JlYWRlci5ycw==) | `0.00% <ø> (ø)` | |
   | [datafusion/core/src/avro\_to\_arrow/mod.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/2671/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9hdnJvX3RvX2Fycm93L21vZC5ycw==) | `42.85% <ø> (ø)` | |
   | ... and [378 more](https://codecov.io/gh/apache/arrow-datafusion/pull/2671/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/2671?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/2671?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0806d85...d220c38](https://codecov.io/gh/apache/arrow-datafusion/pull/2671?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] AssHero commented on a diff in pull request #2671: If statistics of column Max/Min value does not exists in parquet file…

Posted by GitBox <gi...@apache.org>.
AssHero commented on code in PR #2671:
URL: https://github.com/apache/arrow-datafusion/pull/2671#discussion_r886758574


##########
datafusion/core/src/datasource/file_format/parquet.rs:
##########
@@ -344,6 +362,10 @@ fn fetch_statistics(
                             table_idx,
                             stats,
                         )
+                    } else {
+                        // If none statistics of current column exists, set the Max/Min Accumulator to None.

Review Comment:
   > If i miss something, plz tell me
   
   Once max_values and min_values are set to none, even if the column statistics exists in next row_group, summarize_min_max will do nothing because the max_values and min_values are none.



##########
datafusion/core/src/datasource/file_format/parquet.rs:
##########
@@ -344,6 +362,10 @@ fn fetch_statistics(
                             table_idx,
                             stats,
                         )
+                    } else {
+                        // If none statistics of current column exists, set the Max/Min Accumulator to None.

Review Comment:
   Once max_values and min_values are set to none,  even if the column statistics exists in next row_group, summarize_min_max will do nothing because the max_values and min_values are none.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] Ted-Jiang commented on a diff in pull request #2671: If statistics of column Max/Min value does not exists in parquet file…

Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #2671:
URL: https://github.com/apache/arrow-datafusion/pull/2671#discussion_r886621951


##########
datafusion/core/src/datasource/file_format/parquet.rs:
##########
@@ -344,6 +362,10 @@ fn fetch_statistics(
                             table_idx,
                             stats,
                         )
+                    } else {
+                        // If none statistics of current column exists, set the Max/Min Accumulator to None.

Review Comment:
   https://github.com/apache/arrow-datafusion/blob/807b7a5f7eb858e9f7162e1f00ffeeedd0bf2050/datafusion/core/src/datasource/file_format/parquet.rs#L312
   Here both `max_values` and `min_values ` are accumulator for all row_groups
   
   Set  `max_values[i] = None` in loop `for row_group_meta in meta_data.row_groups() {}` assume that  we will no longer need this col static. So we need a short cut (No need read next row_group metadata for this col avoid set another value) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #2671: If statistics of column Max/Min value does not exists in parquet file, sent Min/Max to None

Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #2671:
URL: https://github.com/apache/arrow-datafusion/pull/2671#discussion_r887292893


##########
datafusion/core/tests/sql/parquet.rs:
##########
@@ -219,13 +219,76 @@ async fn schema_merge_ignores_metadata() {
     // (no errors)
     let ctx = SessionContext::new();
     let df = ctx
-        .read_parquet(
-            table_dir.to_str().unwrap().to_string(),
-            ParquetReadOptions::default(),
-        )
+        .read_parquet(table_dir.to_str().unwrap(), ParquetReadOptions::default())
         .await
         .unwrap();
     let result = df.collect().await.unwrap();
 
     assert_eq!(result[0].schema().metadata(), result[1].schema().metadata());
 }
+
+#[tokio::test]
+async fn parquet_query_with_max_min() {
+    let tmp_dir = TempDir::new().unwrap();
+    let table_dir = tmp_dir.path().join("parquet_test");
+    let table_path = Path::new(&table_dir);
+
+    let fields = vec![
+        Field::new("c1", DataType::Int32, true),
+        Field::new("c2", DataType::Utf8, true),
+    ];
+
+    let schema = Arc::new(Schema::new(fields.clone()));
+
+    if let Ok(()) = fs::create_dir(table_path) {
+        let filename = "foo.parquet";
+        let path = table_path.join(&filename);
+        let file = fs::File::create(path).unwrap();
+        let mut writer =
+            ArrowWriter::try_new(file.try_clone().unwrap(), schema.clone(), None)
+                .unwrap();
+
+        // create mock record batch
+        let c1s = Arc::new(Int32Array::from_slice(&[1, 2, 3]));
+        let c2s = Arc::new(StringArray::from_slice(&["aaa", "bbb", "ccc"]));

Review Comment:
   Since there is code for other data types (e.g. int64) can you please also add such columns to this test?



##########
datafusion/core/tests/sql/parquet.rs:
##########
@@ -219,13 +219,76 @@ async fn schema_merge_ignores_metadata() {
     // (no errors)
     let ctx = SessionContext::new();
     let df = ctx
-        .read_parquet(
-            table_dir.to_str().unwrap().to_string(),
-            ParquetReadOptions::default(),
-        )
+        .read_parquet(table_dir.to_str().unwrap(), ParquetReadOptions::default())
         .await
         .unwrap();
     let result = df.collect().await.unwrap();
 
     assert_eq!(result[0].schema().metadata(), result[1].schema().metadata());
 }
+
+#[tokio::test]
+async fn parquet_query_with_max_min() {

Review Comment:
   I verified that this test fails without the other changes in this PR 
   
   ```
   failures:
   
   ---- sql::parquet::parquet_query_with_max_min stdout ----
   thread 'sql::parquet::parquet_query_with_max_min' panicked at 'assertion failed: `(left == right)`
     left: `["+-------------+", "| MIN(foo.c2) |", "+-------------+", "| aaa         |", "+-------------+"]`,
    right: `["+-------------+", "| MIN(foo.c2) |", "+-------------+", "|             |", "+-------------+"]`: 
   
   expected:
   
   [
       "+-------------+",
       "| MIN(foo.c2) |",
       "+-------------+",
       "| aaa         |",
       "+-------------+",
   ]
   actual:
   
   [
       "+-------------+",
       "| MIN(foo.c2) |",
       "+-------------+",
       "|             |",
       "+-------------+",
   ]
   ```



##########
datafusion/core/tests/sql/parquet.rs:
##########
@@ -219,13 +219,76 @@ async fn schema_merge_ignores_metadata() {
     // (no errors)
     let ctx = SessionContext::new();
     let df = ctx
-        .read_parquet(
-            table_dir.to_str().unwrap().to_string(),
-            ParquetReadOptions::default(),
-        )
+        .read_parquet(table_dir.to_str().unwrap(), ParquetReadOptions::default())
         .await
         .unwrap();
     let result = df.collect().await.unwrap();
 
     assert_eq!(result[0].schema().metadata(), result[1].schema().metadata());
 }
+
+#[tokio::test]
+async fn parquet_query_with_max_min() {
+    let tmp_dir = TempDir::new().unwrap();
+    let table_dir = tmp_dir.path().join("parquet_test");
+    let table_path = Path::new(&table_dir);
+
+    let fields = vec![
+        Field::new("c1", DataType::Int32, true),
+        Field::new("c2", DataType::Utf8, true),
+    ];
+
+    let schema = Arc::new(Schema::new(fields.clone()));
+
+    if let Ok(()) = fs::create_dir(table_path) {

Review Comment:
   I recommend failing fast here if we can't make the directory:
   
   ```suggestion
        fs::create_dir(table_path).unwrap()
   ```
   
   Thus if that fails then the test will immediately stop rather whatever happens to be in the directory



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] AssHero commented on a diff in pull request #2671: If statistics of column Max/Min value does not exists in parquet file, sent Min/Max to None

Posted by GitBox <gi...@apache.org>.
AssHero commented on code in PR #2671:
URL: https://github.com/apache/arrow-datafusion/pull/2671#discussion_r887639545


##########
datafusion/core/tests/sql/parquet.rs:
##########
@@ -219,13 +219,76 @@ async fn schema_merge_ignores_metadata() {
     // (no errors)
     let ctx = SessionContext::new();
     let df = ctx
-        .read_parquet(
-            table_dir.to_str().unwrap().to_string(),
-            ParquetReadOptions::default(),
-        )
+        .read_parquet(table_dir.to_str().unwrap(), ParquetReadOptions::default())
         .await
         .unwrap();
     let result = df.collect().await.unwrap();
 
     assert_eq!(result[0].schema().metadata(), result[1].schema().metadata());
 }
+
+#[tokio::test]
+async fn parquet_query_with_max_min() {
+    let tmp_dir = TempDir::new().unwrap();
+    let table_dir = tmp_dir.path().join("parquet_test");
+    let table_path = Path::new(&table_dir);
+
+    let fields = vec![
+        Field::new("c1", DataType::Int32, true),
+        Field::new("c2", DataType::Utf8, true),
+    ];
+
+    let schema = Arc::new(Schema::new(fields.clone()));
+
+    if let Ok(()) = fs::create_dir(table_path) {
+        let filename = "foo.parquet";
+        let path = table_path.join(&filename);
+        let file = fs::File::create(path).unwrap();
+        let mut writer =
+            ArrowWriter::try_new(file.try_clone().unwrap(), schema.clone(), None)
+                .unwrap();
+
+        // create mock record batch
+        let c1s = Arc::new(Int32Array::from_slice(&[1, 2, 3]));
+        let c2s = Arc::new(StringArray::from_slice(&["aaa", "bbb", "ccc"]));

Review Comment:
   > Since there is code for other data types (e.g. int64) can you please also add such columns to this test?
   
   Yes, i can add such columns in this test. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org