You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2019/04/26 05:51:48 UTC
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Hello Vihang Karajgaonkar, Sudhanshu Arora,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/13127
to review the following change.
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
IMPALA-8454 (part 3): enable recursive file listing by default
This enables recursive listing of files within partition directories by
default. This is a behavior change, but in fact makes Impala consistent
with modern versions of Hive, Spark, Presto, etc.
In fact, this is necessary for querying certain Hive tables which have
been written out by a query containing a UNION ALL clause if that query
is executed by Tez (see HIVE-12812 for example).
Technically, this is an incompatible change. Although it's unlikely
people were relying on the non-recursive listing, this patch offers two
escape hatches:
- an individual table may be marked with the
'impala.disable.recursive.listing' property
- impala may be globally configured with
--recursively_list_partitions=false
Given that we know this behavior is inconsistent with other SQL engines,
and that there is no performance benefit to not recursing in the common
case that there _are_ no subdirectories, I made the flag "hidden" and
did not document the new table property. These are only "chicken bit"
flags.
Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A tests/metadata/test_recursive_listing.py
8 files changed, 150 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/13127/1
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
Patch Set 2: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4087/
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 23:21:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Vihang Karajgaonkar, Sudhanshu Arora, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13127
to look at the new patch set (#3).
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
IMPALA-8454 (part 3): enable recursive file listing by default
This enables recursive listing of files within partition directories by
default. This is a behavior change, but in fact makes Impala consistent
with modern versions of Hive, Spark, Presto, etc.
In fact, this is necessary for querying certain Hive tables which have
been written out by a query containing a UNION ALL clause if that query
is executed by Tez (see HIVE-12812 for example).
Technically, this is an incompatible change. Although it's unlikely
people were relying on the non-recursive listing, this patch offers two
escape hatches:
- an individual table may be marked with the
'impala.disable.recursive.listing' property
- impala may be globally configured with
--recursively_list_partitions=false
Given that we know this behavior is inconsistent with other SQL engines,
and that there is no performance benefit to not recursing in the common
case that there _are_ no subdirectories, I made the flag "hidden" and
did not document the new table property. These are only "chicken bit"
flags.
Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A tests/metadata/test_recursive_listing.py
8 files changed, 151 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/13127/3
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
Patch Set 1: Code-Review+2
(1 comment)
Looks pretty good to me.
http://gerrit.cloudera.org:8080/#/c/13127/1/tests/metadata/test_recursive_listing.py
File tests/metadata/test_recursive_listing.py:
http://gerrit.cloudera.org:8080/#/c/13127/1/tests/metadata/test_recursive_listing.py@59
PS1, Line 59: execute
nit: execute_query_expect_success in multiple places? (does an extra assert on result.success)
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 06:38:10 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
Patch Set 1:
(1 comment)
> Patch Set 1: Code-Review+2
>
> (1 comment)
>
> Looks pretty good to me.
http://gerrit.cloudera.org:8080/#/c/13127/1/tests/metadata/test_recursive_listing.py
File tests/metadata/test_recursive_listing.py:
http://gerrit.cloudera.org:8080/#/c/13127/1/tests/metadata/test_recursive_listing.py@59
PS1, Line 59: execute
> nit: execute_query_expect_success in multiple places? (does an extra assert
Done
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 17:17:49 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 04:25:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 11:43:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has removed a vote on this change.
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
Removed Verified-1 by Impala Public Jenkins <im...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/2929/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 06:16:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
IMPALA-8454 (part 3): enable recursive file listing by default
This enables recursive listing of files within partition directories by
default. This is a behavior change, but in fact makes Impala consistent
with modern versions of Hive, Spark, Presto, etc.
In fact, this is necessary for querying certain Hive tables which have
been written out by a query containing a UNION ALL clause if that query
is executed by Tez (see HIVE-12812 for example).
Technically, this is an incompatible change. Although it's unlikely
people were relying on the non-recursive listing, this patch offers two
escape hatches:
- an individual table may be marked with the
'impala.disable.recursive.listing' property
- impala may be globally configured with
--recursively_list_partitions=false
Given that we know this behavior is inconsistent with other SQL engines,
and that there is no performance benefit to not recursing in the common
case that there _are_ no subdirectories, I made the flag "hidden" and
did not document the new table property. These are only "chicken bit"
flags.
Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Reviewed-on: http://gerrit.cloudera.org:8080/13127
Reviewed-by: Bharath Vissapragada <bh...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A tests/metadata/test_recursive_listing.py
8 files changed, 151 insertions(+), 2 deletions(-)
Approvals:
Bharath Vissapragada: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 4
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4081/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 06:10:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4087/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 17:53:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13127/3/common/thrift/BackendGflags.thrift
File common/thrift/BackendGflags.thrift:
http://gerrit.cloudera.org:8080/#/c/13127/3/common/thrift/BackendGflags.thrift@131
PS3, Line 131: 52: required bool recursively_list_partitions
had to resolve a conflict here with the newly-added authorization_provider flag above.
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 18:20:45 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/2973/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 19:29:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Vihang Karajgaonkar, Sudhanshu Arora, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13127
to look at the new patch set (#2).
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
IMPALA-8454 (part 3): enable recursive file listing by default
This enables recursive listing of files within partition directories by
default. This is a behavior change, but in fact makes Impala consistent
with modern versions of Hive, Spark, Presto, etc.
In fact, this is necessary for querying certain Hive tables which have
been written out by a query containing a UNION ALL clause if that query
is executed by Tez (see HIVE-12812 for example).
Technically, this is an incompatible change. Although it's unlikely
people were relying on the non-recursive listing, this patch offers two
escape hatches:
- an individual table may be marked with the
'impala.disable.recursive.listing' property
- impala may be globally configured with
--recursively_list_partitions=false
Given that we know this behavior is inconsistent with other SQL engines,
and that there is no performance benefit to not recursing in the common
case that there _are_ no subdirectories, I made the flag "hidden" and
did not document the new table property. These are only "chicken bit"
flags.
Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A tests/metadata/test_recursive_listing.py
8 files changed, 151 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/13127/2
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/2936/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 18:21:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 21:31:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4106/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 23:09:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13127 )
Change subject: IMPALA-8454 (part 3): enable recursive file listing by default
......................................................................
Patch Set 2: Verified+1 Code-Review+2
Overriding -1 due to unrelated IMPALA-8466 and forwarding Bharath's +2.
--
To view, visit http://gerrit.cloudera.org:8080/13127
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8
Gerrit-Change-Number: 13127
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <su...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 18:15:58 +0000
Gerrit-HasComments: No