You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2018/03/06 01:23:30 UTC
[1/2] impala git commit: IMPALA-6553: [DOCS]
load_catalog_in_background default change
Repository: impala
Updated Branches:
refs/heads/master 77d07f806 -> 5f2f445e7
IMPALA-6553: [DOCS] load_catalog_in_background default change
Change-Id: I548b2d1532c12f8d3c795a940b7f980482ecf09b
Reviewed-on: http://gerrit.cloudera.org:8080/9389
Reviewed-by: John Russell <jr...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/3a1d802e
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/3a1d802e
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/3a1d802e
Branch: refs/heads/master
Commit: 3a1d802eada1b4ab77f5fe1a95fc6e8d2a02d0f4
Parents: 77d07f8
Author: Alex Rodoni <ar...@cloudera.com>
Authored: Wed Feb 21 17:20:28 2018 -0800
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Tue Mar 6 00:48:00 2018 +0000
----------------------------------------------------------------------
docs/shared/impala_common.xml | 38 +++++++++++++++++++++----
docs/topics/impala_invalidate_metadata.xml | 10 +++----
2 files changed, 38 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/3a1d802e/docs/shared/impala_common.xml
----------------------------------------------------------------------
diff --git a/docs/shared/impala_common.xml b/docs/shared/impala_common.xml
index 03892eb..4c5e57c 100644
--- a/docs/shared/impala_common.xml
+++ b/docs/shared/impala_common.xml
@@ -3443,10 +3443,39 @@ select * from header_line limit 10;
</p>
<p id="load_catalog_in_background">
- By default, the metadata loading and caching on startup happens asynchronously, so Impala can begin
- accepting requests promptly. To enable the original behavior, where Impala waited until all metadata was
- loaded before accepting any requests, set the <cmdname>catalogd</cmdname> configuration option
- <codeph>--load_catalog_in_background=false</codeph>.
+ Use <codeph>--load_catalog_in_background</codeph> option to control when
+ the metadata of a table is loaded.
+ <ul>
+ <li>
+ If set to <codeph>false</codeph>, the metadata of a table is
+ loaded when it is referenced for the first time. This means that the
+ first run of a particular query can be slower than subsequent runs.
+ Starting in Impala 2.2, the default for
+ <codeph>load_catalog_in_background</codeph> is
+ <codeph>false</codeph>.
+ </li>
+ <li>
+ If set to <codeph>true</codeph>, the catalog service attempts to
+ load metadata for a table even if no query needed that metadata. So
+ metadata will possibly be already loaded when the first query that
+ would need it is run. However, for the following reasons, we
+ recommend not to set the option to <codeph>true</codeph>.
+ <ul>
+ <li>
+ Background load can interfere with query-specific metadata
+ loading. This can happen on startup or after invalidating
+ metadata, with a duration depending on the amount of metadata,
+ and can lead to a seemingly random long running queries that are
+ difficult to diagnose.
+ </li>
+ <li>
+ Impala may load metadata for tables that are possibly never
+ used, potentially increasing catalog size and consequently memory
+ usage for both catalog service and Impala Daemon.
+ </li>
+ </ul>
+ </li>
+ </ul>
</p>
<ul id="catalogd_xrefs">
@@ -3458,7 +3487,6 @@ select * from header_line limit 10;
<cmdname>catalogd</cmdname> daemon.
</p>
</li>
-
<li>
<p>
The <codeph>REFRESH</codeph> and <codeph>INVALIDATE METADATA</codeph> statements are no longer needed
http://git-wip-us.apache.org/repos/asf/impala/blob/3a1d802e/docs/topics/impala_invalidate_metadata.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_invalidate_metadata.xml b/docs/topics/impala_invalidate_metadata.xml
index 4f63d34..ddd79d5 100644
--- a/docs/topics/impala_invalidate_metadata.xml
+++ b/docs/topics/impala_invalidate_metadata.xml
@@ -192,11 +192,11 @@ under the License.
By default, the <codeph>INVALIDATE METADATA</codeph> command checks HDFS permissions of the underlying data
files and directories, caching this information so that a statement can be cancelled immediately if for
example the <codeph>impala</codeph> user does not have permission to write to the data directory for the
- table. (This checking does not apply if you have set the <cmdname>catalogd</cmdname> configuration option
- <codeph>--load_catalog_in_background=false</codeph>.) Impala reports any lack of write permissions as an
- <codeph>INFO</codeph> message in the log file, in case that represents an oversight. If you change HDFS
- permissions to make data readable or writeable by the Impala user, issue another <codeph>INVALIDATE
- METADATA</codeph> to make Impala aware of the change.
+ table. (This checking does not apply when the <cmdname>catalogd</cmdname> configuration option
+ <codeph>--load_catalog_in_background</codeph> is set to <codeph>false</codeph>, which it is by default.)
+ Impala reports any lack of write permissions as an <codeph>INFO</codeph> message in the log file, in case
+ that represents an oversight. If you change HDFS permissions to make data readable or writeable by the Impala
+ user, issue another <codeph>INVALIDATE METADATA</codeph> to make Impala aware of the change.
</p>
<p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
[2/2] impala git commit: IMPALA-6599: fixes return for
NativeLibCacheSetNeedsRefresh
Posted by ta...@apache.org.
IMPALA-6599: fixes return for NativeLibCacheSetNeedsRefresh
Current fe_support.cc:[..]_FeSupport_NativeLibCacheSetNeedsRefresh
always returns false. In the frontend, this is logged, which causes
unneeded, incorrect, and confusing spam.
This method returns false if unable to manage the string input
argument (path). It then invokes the lib_cache's SetNeedsRefresh
which always succeeds (either the path does not exist or, if it exists,
needs refresh is set). This change modifies the return value after
this call to be true instead of false.
Testing:
- verified the spam without the change by looking at the logs from
query_test/test_udfs.py (~4000 log messages)
- verified that none of these log messages show up with the change
applied.
Change-Id: I11f34a63a25f5ab6acabcc2f52b7e8f22d8a4da3
Reviewed-on: http://gerrit.cloudera.org:8080/9497
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/5f2f445e
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/5f2f445e
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/5f2f445e
Branch: refs/heads/master
Commit: 5f2f445e7d29ed26f6818b5c41edda2fe7c49b59
Parents: 3a1d802
Author: Vuk Ercegovac <ve...@cloudera.com>
Authored: Mon Mar 5 11:16:37 2018 -0800
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Tue Mar 6 00:53:30 2018 +0000
----------------------------------------------------------------------
be/src/service/fe-support.cc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/5f2f445e/be/src/service/fe-support.cc
----------------------------------------------------------------------
diff --git a/be/src/service/fe-support.cc b/be/src/service/fe-support.cc
index 2b6e2fd..2d48d73 100644
--- a/be/src/service/fe-support.cc
+++ b/be/src/service/fe-support.cc
@@ -469,7 +469,7 @@ Java_org_apache_impala_service_FeSupport_NativeLibCacheSetNeedsRefresh(JNIEnv* e
str.assign(hdfs_location_data.get());
}
LibCache::instance()->SetNeedsRefresh(str);
- return static_cast<jboolean>(false);
+ return static_cast<jboolean>(true);
}
extern "C"