You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "maxburke (via GitHub)" <gi...@apache.org> on 2023/11/22 17:59:44 UTC

[I] Possible bug in Parquet pruning code? [arrow-datafusion]

maxburke opened a new issue, #8309:
URL: https://github.com/apache/arrow-datafusion/issues/8309

   ### Describe the bug
   
   I'm running into a weird issue with a datafusion query on a parquet file where if I select with a condition testing certain values, I get no results back. 
   
   ### To Reproduce
   
   For example:
   ```
   ❯ create external table t0 stored as parquet location 'a7546b6b206d882e928a1325f8cbcce4.parquet';
   0 rows in set. Query took 0.019 seconds.
   ❯ select distinct direction from t0;
   +-----------+
   | direction |
   +-----------+
   | Merged    |
   | Two Way   |
   | Outgoing  |
   | Incoming  |
   +-----------+
   4 rows in set. Query took 0.015 seconds.
   ❯ select * from t0 where "direction" = 'Outgoing';
   +--------------------------+-----------+-------+
   | ul_node_id               | direction | adt   |
   +--------------------------+-----------+-------+
   | XRLDMlrMwNT4jrCE2hzvbA== | Outgoing  | 222   |
   | uuzYwO69bCHsouZHOeG8Wg== | Outgoing  | 178   |
   ...snip...
   | YlaylK0fQqkfvKbIONzkJQ== | Outgoing  | 115   |
   | AUsoU3ojLVrmgRJhuypwMA== | Outgoing  | 649   |
   +--------------------------+-----------+-------+
   100 rows in set. Query took 0.013 seconds.
   ❯ select * from t0 where "direction" = 'Merged';
   +--------------------------+-----------+------+
   | ul_node_id               | direction | adt  |
   +--------------------------+-----------+------+
   | YxRwadNzW7uJht9xp4g46Q== | Merged    | 5605 |
   +--------------------------+-----------+------+
   1 row in set. Query took 0.010 seconds.
   ❯ select * from t0 where "direction" = 'Incoming';
   0 rows in set. Query took 0.007 seconds.
   ❯ select * from t0 where "direction" = 'Two Way';
   0 rows in set. Query took 0.006 seconds.
   ```
   
   I've checked to see if there's trailing whitespace in the values "Incoming" and "Two Way in the table and there isn't. pandas seems to be able to do similar queries just fine:
   ```
   >>> df[df['direction'] == "Two Way"]
                      ul_node_id direction    adt
   1    XRLDMlrMwNT4jrCE2hzvbA==   Two Way    420
   5    uuzYwO69bCHsouZHOeG8Wg==   Two Way    337
   8    crHf6lzksh5sVFw8/Sf5Ew==   Two Way    324
   11   OSS+4pM008wnvNH/c19uHg==   Two Way   4674
   12   NF20BBZ0OjDvIbcGxQuhwQ==   Two Way  23116
   ..                        ...       ...    ...
   288  kvQm8I+WxR9C1Il1XxyvbQ==   Two Way    204
   291  RCYo+XLiUXnvL7tb8pml3w==   Two Way   5777
   294  YTsSJkf0qT7tvaosC0gRhA==   Two Way   4409
   297  YlaylK0fQqkfvKbIONzkJQ==   Two Way    336
   298  AUsoU3ojLVrmgRJhuypwMA==   Two Way   1049
   
   [101 rows x 3 columns]
   ```
   
   The parquet file above is attached.
   [a7546b6b206d882e928a1325f8cbcce4.parquet.zip](https://github.com/apache/arrow-datafusion/files/13443023/a7546b6b206d882e928a1325f8cbcce4.parquet.zip)
   
   
   ### Expected behavior
   
   The query should return values.
   
   ### Additional context
   
   Andy Grove demonstrated that disabling Parquet pruning causes the query to return the correct values: https://the-asf.slack.com/archives/C01QUFS30TD/p1700671664714069


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Possible bug in Parquet pruning code? [arrow-datafusion]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #8309:
URL: https://github.com/apache/arrow-datafusion/issues/8309#issuecomment-1823270849

   This still seems incorrect. I would expect min of `Incoming` and max of `Two Way`.
   
   ```
   +-----------+
   | direction |
   +-----------+
   | Merged    |
   | Two Way   |
   | Outgoing  |
   | Incoming  |
   +-----------+
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Possible bug in Parquet pruning code? [arrow-datafusion]

Posted by "maxburke (via GitHub)" <gi...@apache.org>.
maxburke closed issue #8309: Possible bug in Parquet pruning code?
URL: https://github.com/apache/arrow-datafusion/issues/8309


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Possible bug in Parquet pruning code? [arrow-datafusion]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #8309:
URL: https://github.com/apache/arrow-datafusion/issues/8309#issuecomment-1823257066

   nm, it is a bug in bdt


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Possible bug in Parquet pruning code? [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8309:
URL: https://github.com/apache/arrow-datafusion/issues/8309#issuecomment-1823282875

   However, the calculated min / maxes are different:
   
   ```
   ❯ select min(direction), max(direction) from 'a7546b6b206d882e928a1325f8cbcce4.parquet';
   +---------------------------------------------------------+---------------------------------------------------------+
   | MIN(a7546b6b206d882e928a1325f8cbcce4.parquet.direction) | MAX(a7546b6b206d882e928a1325f8cbcce4.parquet.direction) |
   +---------------------------------------------------------+---------------------------------------------------------+
   | Incoming                                                | Two Way                                                 |
   +---------------------------------------------------------+---------------------------------------------------------+
   1 row in set. Query took 0.003 seconds.
   ❯ select arrow_typeof(direction) from 'a7546b6b206d882e928a1325f8cbcce4.parquet';
   +------------------------------------------------------------------+
   | arrow_typeof(a7546b6b206d882e928a1325f8cbcce4.parquet.direction) |
   +------------------------------------------------------------------+
   | Utf8                                                             |
   | Utf8                                                             |
   ...
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Possible bug in Parquet pruning code? [arrow-datafusion]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #8309:
URL: https://github.com/apache/arrow-datafusion/issues/8309#issuecomment-1823268912

   Here is the corrected version:
   
   ```
   $ bdt view-parquet-meta test.parquet 
   +------------+------------+
   | Key        | Value      |
   +------------+------------+
   | Version    | 1          |
   | Created By | UrbanLogiq |
   | Rows       | 301        |
   | Row Groups | 1          |
   +------------+------------+
   
   Row Group 0 of 1 contains 301 rows and has 3384 bytes:
   
   +-------------+--------------+---------------+-----------------+-------+--------------------------+--------------------------+
   | Column Name | Logical Type | Physical Type | Distinct Values | Nulls | Min                      | Max                      |
   +-------------+--------------+---------------+-----------------+-------+--------------------------+--------------------------+
   | ul_node_id  | String       | BYTE_ARRAY    | N/A             | 0     | /ehIvdei+UGfkQ4Gy5fr1w== | zThqpswvY6fa3VHF4BKWfw== |
   | direction   | String       | BYTE_ARRAY    | N/A             | 0     | Merged                   | Outgoing                 |
   | adt         | N/A          | INT32         | N/A             | 0     | 15                       | 23116                    |
   +-------------+--------------+---------------+-----------------+-------+--------------------------+--------------------------+
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Possible bug in Parquet pruning code? [arrow-datafusion]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #8309:
URL: https://github.com/apache/arrow-datafusion/issues/8309#issuecomment-1823256133

   I don't know if this helps, but the min and max values seem to be incorrect in the parquet file (or maybe I have a bug in bdt).
   
   ```
   $ bdt view-parquet-meta test.parquet 
   +------------+------------+
   | Key        | Value      |
   +------------+------------+
   | Version    | 1          |
   | Created By | UrbanLogiq |
   | Rows       | 301        |
   | Row Groups | 1          |
   +------------+------------+
   
   Row Group 0 of 1 contains 301 rows and has 3384 bytes:
   
   +-------------+--------------+---------------+-----------------+-------+--------------------------+--------------------------+
   | Column Name | Logical Type | Physical Type | Distinct Values | Nulls | Min                      | Max                      |
   +-------------+--------------+---------------+-----------------+-------+--------------------------+--------------------------+
   | ul_node_id  | String       | BYTE_ARRAY    | N/A             | 0     | /ehIvdei+UGfkQ4Gy5fr1w== | /ehIvdei+UGfkQ4Gy5fr1w== |
   | direction   | String       | BYTE_ARRAY    | N/A             | 0     | Merged                   | Merged                   |
   | adt         | N/A          | INT32         | N/A             | 0     | 15                       | 23116                    |
   +-------------+--------------+---------------+-----------------+-------+--------------------------+--------------------------+
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Possible bug in Parquet pruning code? [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8309:
URL: https://github.com/apache/arrow-datafusion/issues/8309#issuecomment-1823251117

   Added to https://github.com/apache/arrow-datafusion/issues/8227
   
   I agree this looks like a bug in the pruning code. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Possible bug in Parquet pruning code? [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8309:
URL: https://github.com/apache/arrow-datafusion/issues/8309#issuecomment-1823278802

   The metadata in the file appears to claim that `direction` has a min value of `Merged` and a max value of `Outgoing` 🤔 
   
   ```shell
   alamb@MacBook-Pro-8:~/Downloads$ parquet-tools meta  a7546b6b206d882e928a1325f8cbcce4.parquet
   file:        file:/Users/alamb/Downloads/a7546b6b206d882e928a1325f8cbcce4.parquet
   creator:     UrbanLogiq
   extra:       ARROW:schema = /////+gAAAAQAAAAAAAKAA4ADAALAAQACgAAABQAAAAAAAABBAAKAAwAAAAIAAQACgAAAAgAAAAIAAAAAAAAAAMAAAB8AAAAPAAAAAQAAACg////GAAAACAAAAAAAAACHAAAAAgADAAEAAsACAAAACAAAAAAAAABAAAAAAMAAABhZHQA1P///xQAAAAMAAAAAAAABQwAAAAAAAAAxP///wkAAABkaXJlY3Rpb24AAAAQABQAEAAAAA8ABAAAAAgAEAAAABgAAAAMAAAAAAAABRAAAAAAAAAABAAEAAQAAAAKAAAAdWxfbm9kZV9pZAAA
   
   file schema: arrow_schema
   --------------------------------------------------------------------------------
   ul_node_id:  REQUIRED BINARY L:STRING R:0 D:0
   direction:   REQUIRED BINARY L:STRING R:0 D:0
   adt:         REQUIRED INT32 R:0 D:0
   
   row group 1: RC:301 TS:3384 OFFSET:4
   --------------------------------------------------------------------------------
   ul_node_id:   BINARY ZSTD DO:4 FPO:1796 SZ:2143/3187/1.49 VC:301 ENC:RLE_DICTIONARY,RLE,PLAIN ST:[min: /ehIvdei+UGfkQ4Gy5fr1w==, max: zThqpswvY6fa3VHF4BKWfw==, num_nulls not defined]
   direction:    BINARY ZSTD DO:2243 FPO:2311 SZ:195/177/0.91 VC:301 ENC:RLE_DICTIONARY,RLE,PLAIN ST:[min: Merged, max: Outgoing, num_nulls not defined]
   adt:          INT32 ZSTD DO:2500 FPO:3159 SZ:1046/1503/1.44 VC:301 ENC:RLE_DICTIONARY,RLE,PLAIN ST:[min: 15, max: 23116, num_nulls not defined]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org