You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2024/08/07 02:32:13 UTC
(impala) branch master updated (c4230cff2 -> b1941c8f1)
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from c4230cff2 IMPALA-13272: Analytic function of collections can lead to crash
new ed94f31a2 IMPALA-13279: Upgrade gcovr to 7.2
new 72e38f0e7 IMPALA-13252: (Addendum) PrintId cancel query
new b1941c8f1 IMPALA-13071: Update the doc of Impala components
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/runtime/query-exec-mgr.cc | 2 +-
bin/coverage_helper.sh | 3 ++-
bin/impala-config.sh | 2 +-
bin/impala-gcovr | 2 +-
docs/topics/impala_components.xml | 33 +++++++++++++++++++--------------
infra/python/deps/py3-requirements.txt | 10 ++++++++++
infra/python/deps/requirements.txt | 4 ----
7 files changed, 34 insertions(+), 22 deletions(-)
(impala) 01/03: IMPALA-13279: Upgrade gcovr to 7.2
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit ed94f31a25de86ef768d98fd3e4a27e0aedc5cca
Author: Joe McDonnell <jo...@cloudera.com>
AuthorDate: Sat Aug 3 13:20:30 2024 -0700
IMPALA-13279: Upgrade gcovr to 7.2
In some environments, the code coverage report is empty even
though the tests ran successfully and gcno/gcda files are
written properly.
This upgrades to gcovr 7.2, which does not show the same
problem. gcovr 7.2 requires Python 3.8, so this switches to use
Python 3.8 from the toolchain and installs gcovr in the Python 3
virtualenv.
gcovr 7.2 outputs logging to stderr, so this also modifies
bin/coverage_helper.sh to redirect stderr to stdout.
Testing:
- Verified that this can generate a report locally and on
the affected environment
Change-Id: I5b1aaa92c65f54149a3e7230cbe56d5286f1051a
Reviewed-on: http://gerrit.cloudera.org:8080/21647
Reviewed-by: Michael Smith <mi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
bin/coverage_helper.sh | 3 ++-
bin/impala-config.sh | 2 +-
bin/impala-gcovr | 2 +-
infra/python/deps/py3-requirements.txt | 10 ++++++++++
infra/python/deps/requirements.txt | 4 ----
5 files changed, 14 insertions(+), 7 deletions(-)
diff --git a/bin/coverage_helper.sh b/bin/coverage_helper.sh
index 408c28989..2a50e3bbc 100755
--- a/bin/coverage_helper.sh
+++ b/bin/coverage_helper.sh
@@ -79,7 +79,8 @@ if [ ${REPORT_ACTION} -eq 1 ]; then
--exclude=".*src/benchmarks.*" \
--exclude=".*generated-sources/gen-cpp.*" \
--exclude=".*src/util/bit-packing.inline.h.*" \
- --html --html-details -o "${REPORT_DIRECTORY}/index.html" > "${REPORT_DIRECTORY}/gcovr.out"
+ --html --html-details -o "${REPORT_DIRECTORY}/index.html" \
+ > "${REPORT_DIRECTORY}/gcovr.out" 2>&1
fi
if [ ${ZERO_COUNTERS_ACTION} -eq 1 ]; then
diff --git a/bin/impala-config.sh b/bin/impala-config.sh
index 7c23c8006..68ae68c6d 100755
--- a/bin/impala-config.sh
+++ b/bin/impala-config.sh
@@ -183,7 +183,7 @@ export IMPALA_POSTGRES_JDBC_DRIVER_VERSION=42.5.6
unset IMPALA_POSTGRES_JDBC_DRIVER_URL
export IMPALA_PYTHON_VERSION=2.7.16
unset IMPALA_PYTHON_URL
-export IMPALA_PYTHON3_VERSION=3.7.16
+export IMPALA_PYTHON3_VERSION=3.8.18
export IMPALA_RAPIDJSON_VERSION=1.1.0
unset IMPALA_RAPIDJSON_URL
export IMPALA_RE2_VERSION=20190301
diff --git a/bin/impala-gcovr b/bin/impala-gcovr
index 57b86c32e..7619723ed 100755
--- a/bin/impala-gcovr
+++ b/bin/impala-gcovr
@@ -17,5 +17,5 @@
# specific language governing permissions and limitations
# under the License.
-source "$(dirname "$0")/impala-python-common.sh"
+source "$(dirname "$0")/impala-python3-common.sh"
exec "$PY_ENV_DIR/bin/gcovr" "$@"
diff --git a/infra/python/deps/py3-requirements.txt b/infra/python/deps/py3-requirements.txt
index b61bc461c..902304489 100644
--- a/infra/python/deps/py3-requirements.txt
+++ b/infra/python/deps/py3-requirements.txt
@@ -30,3 +30,13 @@ pylint == 2.10.2
platformdirs == 2.4.1
typing-extensions == 3.10.0.2
k5test==0.10.3
+gcovr == 7.2
+ Jinja2 == 3.1.4
+ flit-core == 3.9.0
+ lxml == 5.2.2
+ Cython == 3.0.10
+ colorlog == 6.8.2
+ Pygments == 2.13.0
+ MarkupSafe == 2.1.5
+ tomli == 2.0.1
+ packaging == 24.1
diff --git a/infra/python/deps/requirements.txt b/infra/python/deps/requirements.txt
index c73adcfcc..e70ef69e4 100644
--- a/infra/python/deps/requirements.txt
+++ b/infra/python/deps/requirements.txt
@@ -23,10 +23,6 @@
allpairspy == 2.5.0
argparse == 1.4.0
future == 0.18.3
-gcovr == 4.2
- Jinja2 == 2.11.3
- MarkupSafe == 1.1.1
- lxml == 4.9.1
hdfs == 2.0.2
docopt == 0.6.2
execnet == 1.4.0
(impala) 02/03: IMPALA-13252: (Addendum) PrintId cancel query
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 72e38f0e7517f1e140bb833278832a72fb1e0501
Author: Michael Smith <mi...@cloudera.com>
AuthorDate: Tue Aug 6 11:17:32 2024 -0700
IMPALA-13252: (Addendum) PrintId cancel query
Use PrintId in log message "CancelFromThreadPool(): cancel query <id>".
Fixes test_no_tuniqueid in exhaustive test runs.
Change-Id: Id17af7bfd51264ce147878072b739845e3a6cfec
Reviewed-on: http://gerrit.cloudera.org:8080/21649
Reviewed-by: Yida Wu <wy...@gmail.com>
Reviewed-by: Riza Suminto <ri...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/runtime/query-exec-mgr.cc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/runtime/query-exec-mgr.cc b/be/src/runtime/query-exec-mgr.cc
index 262f47a6f..d9b82461d 100644
--- a/be/src/runtime/query-exec-mgr.cc
+++ b/be/src/runtime/query-exec-mgr.cc
@@ -267,7 +267,7 @@ void QueryExecMgr::CancelQueriesForFailedCoordinators(
void QueryExecMgr::CancelFromThreadPool(const QueryCancellationTask& cancellation_task) {
QueryState* qs = cancellation_task.GetQueryState();
- VLOG(1) << "CancelFromThreadPool(): cancel query " << qs->query_id();
+ VLOG(1) << "CancelFromThreadPool(): cancel query " << PrintId(qs->query_id());
qs->Cancel();
qs->is_coord_active_.Store(false);
ReleaseQueryState(qs);
(impala) 03/03: IMPALA-13071: Update the doc of Impala components
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit b1941c8f17d1e258b37a2883ebd7433eac4046f3
Author: m-sanjana19 <ma...@gmail.com>
AuthorDate: Mon Jul 29 20:49:29 2024 +0530
IMPALA-13071: Update the doc of Impala components
Change-Id: I83192110d29c4d44529d1276a17c9da4a91435aa
Reviewed-on: http://gerrit.cloudera.org:8080/21621
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
---
docs/topics/impala_components.xml | 33 +++++++++++++++++++--------------
1 file changed, 19 insertions(+), 14 deletions(-)
diff --git a/docs/topics/impala_components.xml b/docs/topics/impala_components.xml
index cb70eb84f..6ce2ebe87 100644
--- a/docs/topics/impala_components.xml
+++ b/docs/topics/impala_components.xml
@@ -141,20 +141,25 @@ under the License.
<conbody>
- <p> The Impala component known as the Catalog Service relays the metadata
- changes from Impala SQL statements to all the Impala daemons in a
- cluster. It is physically represented by a daemon process named
- <codeph>catalogd</codeph>. You only need such a process on one host in
- a cluster. Because the requests are passed through the StateStore
- daemon, it makes sense to run the <cmdname>statestored</cmdname> and
- <cmdname>catalogd</cmdname> services on the same host. </p>
-
- <p> The catalog service avoids the need to issue <codeph>REFRESH</codeph>
- and <codeph>INVALIDATE METADATA</codeph> statements when the metadata
- changes are performed by statements issued through Impala. When you
- create a table, load data, and so on through Hive, you do need to issue
- <codeph>REFRESH</codeph> or <codeph>INVALIDATE METADATA</codeph> on an
- Impala daemon before executing a query there. </p>
+ <p> The Impala component known as the Catalog Service relays the metadata changes from Impala
+ SQL statements to all the Impala coordinators in a cluster. It is physically represented by
+ a daemon process named <codeph>catalogd</codeph>. You only need such a process on one host
+ in a cluster. Because the requests are passed through the StateStore daemon, it makes sense
+ to run the <cmdname>statestored</cmdname> and <cmdname>catalogd</cmdname> services on the
+ same host. </p>
+
+ <p> The catalog service avoids the need to issue <codeph>REFRESH</codeph> and
+ <codeph>INVALIDATE METADATA</codeph> statements when the metadata changes are performed by
+ statements issued through Impala.
+ </p>
+ <p> When you create a table, load data, and so on through Hive, you do need to issue
+ <codeph>REFRESH</codeph> or <codeph>INVALIDATE METADATA</codeph> on an Impala daemon
+ before executing a query. Performing <codeph>REFRESH</codeph> or <codeph>INVALIDATE
+ METADATA</codeph> is not required when <cite>Automatic Invalidation/Refresh of
+ Metadata</cite> is enabled. See <xref href="impala_metadata.xml#impala_metadata">Automatic
+ Invalidation/Refresh of Metadata</xref> also known as the Hive Metastore (HMS) event
+ processor.<note id="note_eyx_qcp_fcc" type="note">From Impala 4.1, Automatic
+ Invalidation/Refresh of Metadata is enabled by default.</note></p>
<p>
This feature touches a number of aspects of Impala: