You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "llama90 (via GitHub)" <gi...@apache.org> on 2023/10/21 12:23:15 UTC
[PR] GH-35785: [C++] Support null type columns for join operation [arrow]
llama90 opened a new pull request, #38383:
URL: https://github.com/apache/arrow/pull/38383
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
Supports join on null types for non-key columns.
* Issue raised: 35785
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
### What changes are included in this PR?
* Separated the check of supported types for key and non-key columns during join execution
* Updated the logic associated with Null types for non-key columns.
* Modified the random tests for Hash Join (TEST(HashJoin, Random)) to allow the creation of Null as non-key column values.
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
### Are these changes tested?
Yes
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
### Are there any user-facing changes?
For users utilizing the Join operation
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
-->
<!-- **This PR includes breaking changes to public APIs.** -->
<!--
Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
-->
<!-- **This PR contains a "Critical Fix".** -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35785: [C++] Support null type non-key columns for join operation [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38383:
URL: https://github.com/apache/arrow/pull/38383#issuecomment-1773781617
:warning: GitHub issue #35785 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35785: [C++] Support null type non-key columns for join operation [arrow]
Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #38383:
URL: https://github.com/apache/arrow/pull/38383#issuecomment-1773775211
There were policy considerations and a personal lack of understanding regarding the code when it came to supporting `Null` types for `key` field columns.
Therefore, although there was an intent to support `key` columns as well, the initial improvements were made to enable support for `non-key` columns.
```cpp
// Constraints
RandomDataTypeConstraints key_type_constraints;
key_type_constraints.Default();
key_type_constraints.withoutNullColumn();
```
In the code, when the // key_type_constraints.withoutNullColumn(); part is commented out, the following error can be encountered.
<details>
<summary>error message</summary>
Test 55: RIGHT_OUTER EQ_IS parallel = true bloom_filter = false
/Users/lama/workspace/arrow-2/cpp/src/arrow/acero/hash_join_node_test.cc:1231: Failure
Failed
'_error_or_value45.status()' failed with NotImplemented: Function 'equal' has no kernel matching input types (null, null)
/Users/lama/workspace/arrow-2/cpp/src/arrow/compute/expression.cc:551 call.function->DispatchBest(&types)
/Users/lama/workspace/arrow-2/cpp/src/arrow/acero/hash_join_node.cc:386 filter.Bind(filter_schema, exec_context)
/Users/lama/workspace/arrow-2/cpp/src/arrow/acero/hash_join_node.cc:737 schema_mgr->BindFilter(join_options.filter, left_schema, right_schema, plan->query_context()->exec_context())
/Users/lama/workspace/arrow-2/cpp/src/arrow/acero/exec_plan.cc:587 MakeExecNode(this->factory_name, plan, std::move(inputs), *this->options, registry)
/Users/lama/workspace/arrow-2/cpp/src/arrow/acero/exec_plan.cc:582 std::get<Declaration>(input).AddToPlan(plan, registry)
/Users/lama/workspace/arrow-2/cpp/src/arrow/acero/exec_plan.cc:686 with_sink.AddToPlan(exec_plan.get())
/Users/lama/workspace/arrow-2/cpp/src/arrow/acero/hash_join_node_test.cc:945 DeclarationToExecBatches(std::move(join), parallel)
Google Test trace:
/Users/lama/workspace/arrow-2/cpp/src/arrow/acero/hash_join_node_test.cc:1120: RIGHT_OUTER EQ_IS parallel = true bloom_filter = false
</details>
It mentions that an "equal" operator for `null`, `null` is needed, which seems to require further investigation (however, I'm actually not sure why a join operation on null key columns would be necessary)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35785: [C++] Support null type non-key columns for join operation [arrow]
Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on PR #38383:
URL: https://github.com/apache/arrow/pull/38383#issuecomment-3592897555
Hello, I don't have enough time to work on this issue/PR at the moment.
So I will close this PR. If anyone is interested, please feel free to continue the work.
Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35785: [C++] Support null type non-key columns for join operation [arrow]
Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 closed pull request #38383: GH-35785: [C++] Support null type non-key columns for join operation
URL: https://github.com/apache/arrow/pull/38383
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35785: [C++] Support null type non-key columns for join operation [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38383:
URL: https://github.com/apache/arrow/pull/38383#issuecomment-1773774303
:warning: GitHub issue #35785 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35785: [C++] Support null type non-key columns for join operation [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on PR #38383:
URL: https://github.com/apache/arrow/pull/38383#issuecomment-3547443541
Thank you for your contribution. Unfortunately, this
pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label
or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you
do not have repository permissions to reopen the PR, please tag a maintainer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org