You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/06/22 13:40:55 UTC

[GitHub] [arrow-datafusion] alamb opened a new pull request, #2767: minor: panic rather than fail silently on bad dictionary in hash join

alamb opened a new pull request, #2767:
URL: https://github.com/apache/arrow-datafusion/pull/2767

   # Which issue does this PR close?
   
   Follow on to https://github.com/apache/arrow-datafusion/pull/2721 from @AssHero  
   
    # Rationale for this change
   If a `DictionaryArray` contains an index that can't be converted to `usize` (e.g. it is negative) it is invalid. Previously if this happened the error would be silently ignored. 
   
   # What changes are included in this PR?
   Change the code to panic if an invalid index was provided so any such error will be clearer
   
   # Are there any user-facing changes?
   no 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] xudong963 merged pull request #2767: minor: panic rather than fail silently on bad dictionary in hash join

Posted by GitBox <gi...@apache.org>.
xudong963 merged PR #2767:
URL: https://github.com/apache/arrow-datafusion/pull/2767


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] codecov-commenter commented on pull request #2767: minor: panic rather than fail silently on bad dictionary in hash join

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #2767:
URL: https://github.com/apache/arrow-datafusion/pull/2767#issuecomment-1163151177

   # [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/2767?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#2767](https://codecov.io/gh/apache/arrow-datafusion/pull/2767?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9973c69) into [master](https://codecov.io/gh/apache/arrow-datafusion/commit/80e6e75200bfd58a9b8fdaf5f4c3af7160eb9722?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (80e6e75) will **increase** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #2767      +/-   ##
   ==========================================
   + Coverage   84.93%   84.95%   +0.01%     
   ==========================================
     Files         271      271              
     Lines       48146    48140       -6     
   ==========================================
   + Hits        40892    40896       +4     
   + Misses       7254     7244      -10     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/arrow-datafusion/pull/2767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [datafusion/core/src/physical\_plan/hash\_join.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/2767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9waHlzaWNhbF9wbGFuL2hhc2hfam9pbi5ycw==) | `94.68% <ø> (+0.57%)` | :arrow_up: |
   | [datafusion/common/src/scalar.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/2767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb21tb24vc3JjL3NjYWxhci5ycw==) | `74.94% <0.00%> (+0.11%)` | :arrow_up: |
   | [datafusion/expr/src/logical\_plan/plan.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/2767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy9sb2dpY2FsX3BsYW4vcGxhbi5ycw==) | `74.11% <0.00%> (+0.39%)` | :arrow_up: |
   | [datafusion/expr/src/window\_frame.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/2767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy93aW5kb3dfZnJhbWUucnM=) | `93.27% <0.00%> (+0.84%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/2767?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/2767?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [80e6e75...9973c69](https://codecov.io/gh/apache/arrow-datafusion/pull/2767?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #2767: minor: panic rather than fail silently on bad dictionary in hash join

Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #2767:
URL: https://github.com/apache/arrow-datafusion/pull/2767#discussion_r903761910


##########
datafusion/core/src/physical_plan/hash_join.rs:
##########
@@ -961,42 +961,35 @@ macro_rules! equal_rows_elem_with_string_dict {
         let (left_values, left_values_index) = {
             let keys_col = left_array.keys();
             if keys_col.is_valid($left) {
-                let values_index = keys_col.value($left).to_usize().ok_or_else(|| {
-                    DataFusionError::Internal(format!(
-                    "Can not convert index to usize in dictionary of type creating group by value {:?}",
-                    keys_col.data_type()
-                ))
-                });
-
-                match values_index {
-                    Ok(index) => (as_string_array(left_array.values()), Some(index)),
-                    _ => (as_string_array(left_array.values()), None)

Review Comment:
   this ignores invalid data in DictionaryArrays (treats them as though they were NULLs) -- I think `panic` is better



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] xudong963 commented on pull request #2767: minor: panic rather than fail silently on bad dictionary in hash join

Posted by GitBox <gi...@apache.org>.
xudong963 commented on PR #2767:
URL: https://github.com/apache/arrow-datafusion/pull/2767#issuecomment-1163176717

   Thank you @alamb , let's merge it!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org