You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2018/03/06 16:11:42 UTC
[1/2] impala git commit: IMPALA-6599: fixes return for
NativeLibCacheSetNeedsRefresh
Repository: impala
Updated Branches:
refs/heads/2.x a73b8f833 -> ff38d6299
IMPALA-6599: fixes return for NativeLibCacheSetNeedsRefresh
Current fe_support.cc:[..]_FeSupport_NativeLibCacheSetNeedsRefresh
always returns false. In the frontend, this is logged, which causes
unneeded, incorrect, and confusing spam.
This method returns false if unable to manage the string input
argument (path). It then invokes the lib_cache's SetNeedsRefresh
which always succeeds (either the path does not exist or, if it exists,
needs refresh is set). This change modifies the return value after
this call to be true instead of false.
Testing:
- verified the spam without the change by looking at the logs from
query_test/test_udfs.py (~4000 log messages)
- verified that none of these log messages show up with the change
applied.
Change-Id: I11f34a63a25f5ab6acabcc2f52b7e8f22d8a4da3
Reviewed-on: http://gerrit.cloudera.org:8080/9497
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/ff38d629
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/ff38d629
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/ff38d629
Branch: refs/heads/2.x
Commit: ff38d6299e36a3ed6b7ab2aee130f7b7e5e2e09c
Parents: fc1578e
Author: Vuk Ercegovac <ve...@cloudera.com>
Authored: Mon Mar 5 11:16:37 2018 -0800
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Tue Mar 6 01:10:15 2018 +0000
----------------------------------------------------------------------
be/src/service/fe-support.cc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/ff38d629/be/src/service/fe-support.cc
----------------------------------------------------------------------
diff --git a/be/src/service/fe-support.cc b/be/src/service/fe-support.cc
index 2b6e2fd..2d48d73 100644
--- a/be/src/service/fe-support.cc
+++ b/be/src/service/fe-support.cc
@@ -469,7 +469,7 @@ Java_org_apache_impala_service_FeSupport_NativeLibCacheSetNeedsRefresh(JNIEnv* e
str.assign(hdfs_location_data.get());
}
LibCache::instance()->SetNeedsRefresh(str);
- return static_cast<jboolean>(false);
+ return static_cast<jboolean>(true);
}
extern "C"
[2/2] impala git commit: IMPALA-6553: [DOCS]
load_catalog_in_background default change
Posted by ta...@apache.org.
IMPALA-6553: [DOCS] load_catalog_in_background default change
Change-Id: I548b2d1532c12f8d3c795a940b7f980482ecf09b
Reviewed-on: http://gerrit.cloudera.org:8080/9389
Reviewed-by: John Russell <jr...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/fc1578e1
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/fc1578e1
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/fc1578e1
Branch: refs/heads/2.x
Commit: fc1578e1a3a58be78c000c725170737decb7d50e
Parents: a73b8f8
Author: Alex Rodoni <ar...@cloudera.com>
Authored: Wed Feb 21 17:20:28 2018 -0800
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Tue Mar 6 01:10:15 2018 +0000
----------------------------------------------------------------------
docs/shared/impala_common.xml | 38 +++++++++++++++++++++----
docs/topics/impala_invalidate_metadata.xml | 10 +++----
2 files changed, 38 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/fc1578e1/docs/shared/impala_common.xml
----------------------------------------------------------------------
diff --git a/docs/shared/impala_common.xml b/docs/shared/impala_common.xml
index a052879..c64d18b 100644
--- a/docs/shared/impala_common.xml
+++ b/docs/shared/impala_common.xml
@@ -3443,10 +3443,39 @@ select * from header_line limit 10;
</p>
<p id="load_catalog_in_background">
- By default, the metadata loading and caching on startup happens asynchronously, so Impala can begin
- accepting requests promptly. To enable the original behavior, where Impala waited until all metadata was
- loaded before accepting any requests, set the <cmdname>catalogd</cmdname> configuration option
- <codeph>--load_catalog_in_background=false</codeph>.
+ Use <codeph>--load_catalog_in_background</codeph> option to control when
+ the metadata of a table is loaded.
+ <ul>
+ <li>
+ If set to <codeph>false</codeph>, the metadata of a table is
+ loaded when it is referenced for the first time. This means that the
+ first run of a particular query can be slower than subsequent runs.
+ Starting in Impala 2.2, the default for
+ <codeph>load_catalog_in_background</codeph> is
+ <codeph>false</codeph>.
+ </li>
+ <li>
+ If set to <codeph>true</codeph>, the catalog service attempts to
+ load metadata for a table even if no query needed that metadata. So
+ metadata will possibly be already loaded when the first query that
+ would need it is run. However, for the following reasons, we
+ recommend not to set the option to <codeph>true</codeph>.
+ <ul>
+ <li>
+ Background load can interfere with query-specific metadata
+ loading. This can happen on startup or after invalidating
+ metadata, with a duration depending on the amount of metadata,
+ and can lead to a seemingly random long running queries that are
+ difficult to diagnose.
+ </li>
+ <li>
+ Impala may load metadata for tables that are possibly never
+ used, potentially increasing catalog size and consequently memory
+ usage for both catalog service and Impala Daemon.
+ </li>
+ </ul>
+ </li>
+ </ul>
</p>
<ul id="catalogd_xrefs">
@@ -3458,7 +3487,6 @@ select * from header_line limit 10;
<cmdname>catalogd</cmdname> daemon.
</p>
</li>
-
<li>
<p>
The <codeph>REFRESH</codeph> and <codeph>INVALIDATE METADATA</codeph> statements are no longer needed
http://git-wip-us.apache.org/repos/asf/impala/blob/fc1578e1/docs/topics/impala_invalidate_metadata.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_invalidate_metadata.xml b/docs/topics/impala_invalidate_metadata.xml
index 4f63d34..ddd79d5 100644
--- a/docs/topics/impala_invalidate_metadata.xml
+++ b/docs/topics/impala_invalidate_metadata.xml
@@ -192,11 +192,11 @@ under the License.
By default, the <codeph>INVALIDATE METADATA</codeph> command checks HDFS permissions of the underlying data
files and directories, caching this information so that a statement can be cancelled immediately if for
example the <codeph>impala</codeph> user does not have permission to write to the data directory for the
- table. (This checking does not apply if you have set the <cmdname>catalogd</cmdname> configuration option
- <codeph>--load_catalog_in_background=false</codeph>.) Impala reports any lack of write permissions as an
- <codeph>INFO</codeph> message in the log file, in case that represents an oversight. If you change HDFS
- permissions to make data readable or writeable by the Impala user, issue another <codeph>INVALIDATE
- METADATA</codeph> to make Impala aware of the change.
+ table. (This checking does not apply when the <cmdname>catalogd</cmdname> configuration option
+ <codeph>--load_catalog_in_background</codeph> is set to <codeph>false</codeph>, which it is by default.)
+ Impala reports any lack of write permissions as an <codeph>INFO</codeph> message in the log file, in case
+ that represents an oversight. If you change HDFS permissions to make data readable or writeable by the Impala
+ user, issue another <codeph>INVALIDATE METADATA</codeph> to make Impala aware of the change.
</p>
<p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>