You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "David Li (Jira)" <ji...@apache.org> on 2021/03/29 15:19:00 UTC
[jira] [Commented] (ARROW-11762) [C++][Dataset] Refactor
Partitioning to explicitly treat null and absent fields identically
[ https://issues.apache.org/jira/browse/ARROW-11762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17310723#comment-17310723 ]
David Li commented on ARROW-11762:
----------------------------------
I'm a little confused here. It seems the two expressions are not equivalent, as {{a == 0}} should match both the paths {{/a=0/b=__HIVE_DEFAULT_PARTITION__}} and {{/a=0/b=1}} but {{a == 0 && is_null(b)}} should match only {{/a=0/b=__HIVE_DEFAULT_PARTITION__}}.
> [C++][Dataset] Refactor Partitioning to explicitly treat null and absent fields identically
> -------------------------------------------------------------------------------------------
>
> Key: ARROW-11762
> URL: https://issues.apache.org/jira/browse/ARROW-11762
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Affects Versions: 3.0.0
> Reporter: Ben Kietzman
> Assignee: Weston Pace
> Priority: Major
> Fix For: 4.0.0
>
>
> ARROW-10438 adds support for partition expressions with explicit absence of a partition key by including an {{is_null(field_ref("absent key field name"))}} in the conjunction. Whenever possible, this should be preferred to an equivalent conjunction which simply omits an equality expression for the missing field.
> Additionally since an absent partition key and a null partition key is semantically equivalent to a null valued partition key, we should ensure there is no difference in behavior. Currently, {{equal(field_ref("a"), literal(0))}} and {{and_(equal(field_ref("a"), literal(0)), is_null("b"))}} are formatted differently
--
This message was sent by Atlassian Jira
(v8.3.4#803005)