You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2021/05/20 14:20:10 UTC
[impala] branch master updated (26eaa43 -> 71ccad2)
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from 26eaa43 IMPALA-10695: add dedicated thread pool for OSS/JindoFS.
new 47d868f IMPALA-10702: Add warning logs for slow or large catalogd response
new 7440765 IMPALA-8769: [DOCS] Change the shell default
new 71ccad2 IMPALA-10701: Switch to use TByteBuffer from thrift
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/util/backend-gflag-util.cc | 8 ++
common/thrift/BackendGflags.thrift | 4 +
docs/topics/impala_shell_options.xml | 6 +-
.../org/apache/impala/catalog/ImpaladCatalog.java | 4 +-
.../impala/catalog/local/CatalogdMetaProvider.java | 2 +-
.../java/org/apache/impala/common/JniUtil.java | 36 ++++++++-
.../org/apache/impala/service/BackendConfig.java | 13 ++++
.../java/org/apache/impala/service/JniCatalog.java | 91 +++++++++++++++++-----
.../java/org/apache/impala/util/TByteBuffer.java | 60 --------------
9 files changed, 134 insertions(+), 90 deletions(-)
delete mode 100644 fe/src/main/java/org/apache/impala/util/TByteBuffer.java
[impala] 02/03: IMPALA-8769: [DOCS] Change the shell default
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 74407651427cdca574b49a4d57a83d8b552659ee
Author: Shajini Thayasingh <st...@cloudera.com>
AuthorDate: Tue Apr 27 11:10:35 2021 -0700
IMPALA-8769: [DOCS] Change the shell default
Changed the shell default from Beeswax to HS2.
Also changed the default port number from 21000 to 21050.
Change-Id: Ia695a01c28bd6350645394d2bbaded731039189c
Reviewed-on: http://gerrit.cloudera.org:8080/17349
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
docs/topics/impala_shell_options.xml | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/docs/topics/impala_shell_options.xml b/docs/topics/impala_shell_options.xml
index c47f3de..87496f4 100644
--- a/docs/topics/impala_shell_options.xml
+++ b/docs/topics/impala_shell_options.xml
@@ -250,7 +250,7 @@ under the License.
</entry>
<entry>
<p>
- Connects to the <codeph>impalad</codeph> daemon on the specified host. The default port of 21000
+ Connects to the <codeph>impalad</codeph> daemon on the specified host. The default port of 21050
is assumed unless you provide another value. You can connect to any host in your cluster that is
running <codeph>impalad</codeph>. If you connect to an instance of <codeph>impalad</codeph> that
was started with an alternate port specified by the <codeph>--fe_port</codeph> flag, provide that
@@ -574,13 +574,13 @@ under the License.
<varname>protocol</varname> values are:<ul>
<li><codeph>'hs2'</codeph>: Impala-shell uses the binary TCP
based transport to speak to the Impala Daemon via the
- HiveServer2 protocol.</li>
+ HiveServer2 protocol. This is the current default setting.</li>
<li><codeph>'hs2-http'</codeph>: Impala-shell uses HTTP
transport to speak to the Impala Daemon via the
HiveServer2 protocol.</li>
<li><codeph>'beeswax'</codeph>: Impala-shell uses the binary
TCP based transport to speak to the Impala Daemon via
- Beeswax. This is the current default setting.</li>
+ Beeswax.</li>
</ul></p><p>You cannot connect to the 3.2 or earlier versions
of Impala using the <codeph>'hs2'</codeph> or
<codeph>'hs2-http'</codeph> option.</p><p>Beeswax support is
[impala] 03/03: IMPALA-10701: Switch to use TByteBuffer from thrift
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 71ccad2f415aa2e3f1cac35514c2d6b9c6db3ebc
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Wed May 12 09:55:34 2021 +0800
IMPALA-10701: Switch to use TByteBuffer from thrift
We have a copy of the TByteBuffer when we are using thrift versions
prior to 0.10. After IMPALA-7825 bumps the thrift version to 0.11.0, we
can use it directly.
Tests
- Run CORE tests
Change-Id: Ia0c7834253a16e440204264b0462a1590dea2463
Reviewed-on: http://gerrit.cloudera.org:8080/17428
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
.../org/apache/impala/catalog/ImpaladCatalog.java | 4 +-
.../impala/catalog/local/CatalogdMetaProvider.java | 2 +-
.../java/org/apache/impala/util/TByteBuffer.java | 60 ----------------------
3 files changed, 2 insertions(+), 64 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java b/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
index edbfee2..a033934 100644
--- a/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
+++ b/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
@@ -27,9 +27,7 @@ import java.util.Map;
import java.util.Set;
import java.util.concurrent.atomic.AtomicLong;
import java.util.concurrent.atomic.AtomicReference;
-import java.util.stream.Collectors;
-import org.apache.commons.collections.CollectionUtils;
import org.apache.impala.analysis.TableName;
import org.apache.impala.authorization.AuthorizationChecker;
import org.apache.impala.authorization.AuthorizationPolicy;
@@ -50,10 +48,10 @@ import org.apache.impala.thrift.TUniqueId;
import org.apache.impala.thrift.TUpdateCatalogCacheRequest;
import org.apache.impala.thrift.TUpdateCatalogCacheResponse;
import org.apache.impala.util.PatternMatcher;
-import org.apache.impala.util.TByteBuffer;
import org.apache.impala.util.TUniqueIdUtil;
import org.apache.thrift.TException;
import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TByteBuffer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java b/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
index 46db9c3..1ee1041 100644
--- a/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
+++ b/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
@@ -89,11 +89,11 @@ import org.apache.impala.thrift.TUpdateCatalogCacheRequest;
import org.apache.impala.thrift.TUpdateCatalogCacheResponse;
import org.apache.impala.thrift.TValidWriteIdList;
import org.apache.impala.util.ListMap;
-import org.apache.impala.util.TByteBuffer;
import org.apache.thrift.TDeserializer;
import org.apache.thrift.TException;
import org.apache.thrift.TSerializer;
import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TByteBuffer;
import org.ehcache.sizeof.SizeOf;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/fe/src/main/java/org/apache/impala/util/TByteBuffer.java b/fe/src/main/java/org/apache/impala/util/TByteBuffer.java
deleted file mode 100644
index 28d05aa..0000000
--- a/fe/src/main/java/org/apache/impala/util/TByteBuffer.java
+++ /dev/null
@@ -1,60 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.impala.util;
-
-import org.apache.thrift.transport.TTransport;
-import org.apache.thrift.transport.TTransportException;
-import java.nio.BufferUnderflowException;
-import java.nio.ByteBuffer;
-
-/**
- * ByteBuffer-backed implementation of TTransport. This is copied from thrift 0.10.0.
- * TODO: Upgrade thrift to 0.10.0 or higher and remove this file.
- */
-public final class TByteBuffer extends TTransport {
- private final ByteBuffer byteBuffer;
-
- public TByteBuffer(ByteBuffer byteBuffer) { this.byteBuffer = byteBuffer; }
-
- @Override
- public boolean isOpen() { return true; }
-
- @Override
- public void open() {}
-
- @Override
- public void close() {}
-
- @Override
- public int read(byte[] buf, int off, int len) throws TTransportException {
- final int n = Math.min(byteBuffer.remaining(), len);
- if (n > 0) {
- try {
- byteBuffer.get(buf, off, n);
- } catch (BufferUnderflowException e) {
- throw new TTransportException("Unexpected end of input buffer", e);
- }
- }
- return n;
- }
-
- @Override
- public void write(byte[] buf, int off, int len) throws TTransportException {
- throw new TTransportException("Write is not supported by TByteBuffer");
- }
-}
[impala] 01/03: IMPALA-10702: Add warning logs for slow or large
catalogd response
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 47d868fe586905b2f4ea347a97ce582a45aac405
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Wed May 12 16:20:05 2021 +0800
IMPALA-10702: Add warning logs for slow or large catalogd response
It'd be helpful to log the slow or large responses of catalogd in
debugging scalability issues. This patch adds these warning logs in
JniCatalog, where we serialize thrift responses. See some example
outputs in the jira description.
Responses that have size larger than 50MB or take more than 60s to
finish will be logged with the request. Add flags for these two
thredshold in case users found the warnings too verbose and want to
increase the thresholds.
Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Reviewed-on: http://gerrit.cloudera.org:8080/17427
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/util/backend-gflag-util.cc | 8 ++
common/thrift/BackendGflags.thrift | 4 +
.../java/org/apache/impala/common/JniUtil.java | 36 ++++++++-
.../org/apache/impala/service/BackendConfig.java | 13 ++++
.../java/org/apache/impala/service/JniCatalog.java | 91 +++++++++++++++++-----
5 files changed, 129 insertions(+), 23 deletions(-)
diff --git a/be/src/util/backend-gflag-util.cc b/be/src/util/backend-gflag-util.cc
index de87481..2cf44fd 100644
--- a/be/src/util/backend-gflag-util.cc
+++ b/be/src/util/backend-gflag-util.cc
@@ -170,6 +170,12 @@ DEFINE_bool(allow_ordinals_in_having, false,
"supported in Impala 3.x and earlier. We intend to disable it since 4.0. So it "
"defaults to be false. See IMPALA-7844.");
+DEFINE_int32(warn_catalog_response_size_mb, 50,
+ "Threshold in MB to log a warning for large catalogd response size.");
+
+DEFINE_int32(warn_catalog_response_duration_s, 60,
+ "Threshold in seconds to log a warning for slow catalogd response.");
+
namespace impala {
Status GetConfigFromCommand(const string& flag_cmd, string& result) {
@@ -295,6 +301,8 @@ Status PopulateThriftBackendGflags(TBackendGflags& cfg) {
cfg.__set_fallback_to_hms_on_errors(FLAGS_fallback_to_hms_on_errors);
cfg.__set_enable_catalogd_hms_cache(FLAGS_enable_catalogd_hms_cache);
cfg.__set_kudu_sasl_protocol_name(FLAGS_kudu_sasl_protocol_name);
+ cfg.__set_warn_catalog_response_size_mb(FLAGS_warn_catalog_response_size_mb);
+ cfg.__set_warn_catalog_response_duration_s(FLAGS_warn_catalog_response_duration_s);
return Status::OK();
}
diff --git a/common/thrift/BackendGflags.thrift b/common/thrift/BackendGflags.thrift
index a2e5680..f59aabe 100644
--- a/common/thrift/BackendGflags.thrift
+++ b/common/thrift/BackendGflags.thrift
@@ -207,4 +207,8 @@ struct TBackendGflags {
91: required bool enable_catalogd_hms_cache
92: required string kudu_sasl_protocol_name
+
+ 93: required i32 warn_catalog_response_size_mb
+
+ 94: required i32 warn_catalog_response_duration_s
}
diff --git a/fe/src/main/java/org/apache/impala/common/JniUtil.java b/fe/src/main/java/org/apache/impala/common/JniUtil.java
index 3e1e7df..2c718c3 100644
--- a/fe/src/main/java/org/apache/impala/common/JniUtil.java
+++ b/fe/src/main/java/org/apache/impala/common/JniUtil.java
@@ -31,6 +31,8 @@ import java.lang.management.ThreadInfo;
import java.util.ArrayList;
import java.util.Map;
+import org.apache.commons.lang.StringUtils;
+import org.apache.impala.service.BackendConfig;
import org.apache.impala.thrift.TGetJMXJsonResponse;
import org.apache.impala.util.JMXJsonUtil;
import org.apache.thrift.TBase;
@@ -49,7 +51,9 @@ import org.apache.impala.thrift.TJvmMemoryPool;
import org.apache.impala.thrift.TJvmThreadInfo;
import org.apache.impala.util.JvmPauseMonitor;
-import org.apache.log4j.Logger;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
/**
* Utility class with methods intended for JNI clients
*/
@@ -57,7 +61,7 @@ public class JniUtil {
private final static TBinaryProtocol.Factory protocolFactory_ =
new TBinaryProtocol.Factory();
- private static final Logger LOG = Logger.getLogger(JniUtil.class);
+ private static final Logger LOG = LoggerFactory.getLogger(JniUtil.class);
/**
* Initializes the JvmPauseMonitor instance.
@@ -140,6 +144,34 @@ public class JniUtil {
}
/**
+ * Warn if the result size or the response time exceeds thresholds.
+ */
+ public static void logResponse(long resultSize, long startTime, TBase<?, ?> thriftReq,
+ String methodName) {
+ long duration = System.currentTimeMillis() - startTime;
+ boolean tooLarge = (resultSize > BackendConfig.INSTANCE.getWarnCatalogResponseSize());
+ boolean tooSlow =
+ (duration > BackendConfig.INSTANCE.getWarnCatalogResponseDurationMs());
+ if (tooLarge || tooSlow) {
+ String header = (tooLarge && tooSlow) ? "Response too large and too slow" :
+ (tooLarge ? "Response too large" : "Response too slow");
+ String request = (thriftReq == null) ? "" :
+ ", request: " + StringUtils.abbreviate(thriftReq.toString(), 1000);
+ LOG.warn("{}: size={} ({}), duration={}ms ({}), method: {}{}",
+ header, resultSize, PrintUtils.printBytes(resultSize),
+ duration, PrintUtils.printTimeMs(duration), methodName, request);
+ }
+ }
+
+ public static void logResponse(long startTime, TBase<?, ?> thriftReq, String method) {
+ logResponse(0, startTime, thriftReq, method);
+ }
+
+ public static void logResponse(long startTime, String method) {
+ logResponse(startTime, null, method);
+ }
+
+ /**
* Collect the JVM's memory statistics into a thrift structure for translation into
* Impala metrics by the backend. A synthetic 'total' memory pool is included with
* aggregate statistics for all real pools. Metrics for the JvmPauseMonitor
diff --git a/fe/src/main/java/org/apache/impala/service/BackendConfig.java b/fe/src/main/java/org/apache/impala/service/BackendConfig.java
index 8b0525c..3e1fe82 100644
--- a/fe/src/main/java/org/apache/impala/service/BackendConfig.java
+++ b/fe/src/main/java/org/apache/impala/service/BackendConfig.java
@@ -23,6 +23,7 @@ import static org.apache.hadoop.fs.CommonConfigurationKeysPublic
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.security.authentication.util.KerberosName;
import org.apache.impala.analysis.SqlScanner;
+import org.apache.impala.common.PrintUtils;
import org.apache.impala.thrift.TBackendGflags;
import com.google.common.annotations.VisibleForTesting;
@@ -38,8 +39,14 @@ public class BackendConfig {
private TBackendGflags backendCfg_;
+ // Thresholds in warning slow or large catalog responses.
+ private long warnCatalogResponseSize_;
+ private long warnCatalogResponseDurationMs_;
+
private BackendConfig(TBackendGflags cfg) {
backendCfg_ = cfg;
+ warnCatalogResponseSize_ = cfg.warn_catalog_response_size_mb * 1024L * 1024L;
+ warnCatalogResponseDurationMs_ = cfg.warn_catalog_response_duration_s * 1000L;
}
public static void create(TBackendGflags cfg) {
@@ -264,6 +271,12 @@ public class BackendConfig {
backendCfg_.allow_ordinals_in_having = allow_ordinals_in_having;
}
+ public long getWarnCatalogResponseSize() { return warnCatalogResponseSize_; }
+
+ public long getWarnCatalogResponseDurationMs() {
+ return warnCatalogResponseDurationMs_;
+ }
+
// Inits the auth_to_local configuration in the static KerberosName class.
private static void initAuthToLocal() {
// If auth_to_local is enabled, we read the configuration hadoop.security.auth_to_local
diff --git a/fe/src/main/java/org/apache/impala/service/JniCatalog.java b/fe/src/main/java/org/apache/impala/service/JniCatalog.java
index 3b0fbd4..cd4ae88 100644
--- a/fe/src/main/java/org/apache/impala/service/JniCatalog.java
+++ b/fe/src/main/java/org/apache/impala/service/JniCatalog.java
@@ -28,7 +28,6 @@ import java.util.UUID;
import org.apache.impala.authorization.AuthorizationConfig;
import org.apache.impala.authorization.AuthorizationFactory;
import org.apache.impala.authorization.AuthorizationManager;
-import org.apache.impala.authorization.User;
import org.apache.impala.catalog.CatalogException;
import org.apache.impala.catalog.CatalogServiceCatalog;
import org.apache.impala.catalog.Db;
@@ -146,11 +145,14 @@ public class JniCatalog {
public byte[] getCatalogDelta(byte[] thriftGetCatalogDeltaReq) throws
ImpalaException, TException {
+ long start = System.currentTimeMillis();
TGetCatalogDeltaRequest params = new TGetCatalogDeltaRequest();
JniUtil.deserializeThrift(protocolFactory_, params, thriftGetCatalogDeltaReq);
- return new TSerializer(protocolFactory_).serialize(new TGetCatalogDeltaResponse(
+ byte[] res = new TSerializer(protocolFactory_).serialize(new TGetCatalogDeltaResponse(
catalog_.getCatalogDelta(params.getNative_catalog_server_ptr(),
params.getFrom_version())));
+ JniUtil.logResponse(res.length, start, params, "getCatalogDelta");
+ return res;
}
/**
@@ -164,11 +166,14 @@ public class JniCatalog {
* Executes the given DDL request and returns the result.
*/
public byte[] execDdl(byte[] thriftDdlExecReq) throws ImpalaException {
+ long start = System.currentTimeMillis();
TDdlExecRequest params = new TDdlExecRequest();
JniUtil.deserializeThrift(protocolFactory_, params, thriftDdlExecReq);
TSerializer serializer = new TSerializer(protocolFactory_);
try {
- return serializer.serialize(catalogOpExecutor_.execDdlRequest(params));
+ byte[] res = serializer.serialize(catalogOpExecutor_.execDdlRequest(params));
+ JniUtil.logResponse(res.length, start, params, "execDdl");
+ return res;
} catch (TException e) {
throw new InternalException(e.getMessage());
}
@@ -179,12 +184,15 @@ public class JniCatalog {
*/
public byte[] resetMetadata(byte[] thriftResetMetadataReq)
throws ImpalaException, TException {
+ long start = System.currentTimeMillis();
TResetMetadataRequest req = new TResetMetadataRequest();
JniUtil.deserializeThrift(protocolFactory_, req, thriftResetMetadataReq);
TSerializer serializer = new TSerializer(protocolFactory_);
catalogOperationUsage.increment(req);
try {
- return serializer.serialize(catalogOpExecutor_.execResetMetadata(req));
+ byte[] res = serializer.serialize(catalogOpExecutor_.execResetMetadata(req));
+ JniUtil.logResponse(res.length, start, req, "resetMetadata");
+ return res;
} finally {
catalogOperationUsage.decrement(req);
}
@@ -197,6 +205,7 @@ public class JniCatalog {
*/
public byte[] getDbs(byte[] thriftGetTablesParams) throws ImpalaException,
TException {
+ long start = System.currentTimeMillis();
TGetDbsParams params = new TGetDbsParams();
JniUtil.deserializeThrift(protocolFactory_, params, thriftGetTablesParams);
List<Db> dbs = catalog_.getDbs(PatternMatcher.MATCHER_MATCH_ALL);
@@ -205,7 +214,9 @@ public class JniCatalog {
for (FeDb db: dbs) tDbs.add(db.toThrift());
result.setDbs(tDbs);
TSerializer serializer = new TSerializer(protocolFactory_);
- return serializer.serialize(result);
+ byte[] res = serializer.serialize(result);
+ JniUtil.logResponse(res.length, start, params, "getDbs");
+ return res;
}
/**
@@ -215,6 +226,7 @@ public class JniCatalog {
*/
public byte[] getTableNames(byte[] thriftGetTablesParams) throws ImpalaException,
TException {
+ long start = System.currentTimeMillis();
TGetTablesParams params = new TGetTablesParams();
JniUtil.deserializeThrift(protocolFactory_, params, thriftGetTablesParams);
List<String> tables = catalog_.getTableNames(params.db,
@@ -222,17 +234,21 @@ public class JniCatalog {
TGetTablesResult result = new TGetTablesResult();
result.setTables(tables);
TSerializer serializer = new TSerializer(protocolFactory_);
- return serializer.serialize(result);
+ byte[] res = serializer.serialize(result);
+ JniUtil.logResponse(res.length, start, params, "getTableNames");
+ return res;
}
/**
* Returns the collected metrics of a table.
*/
- public String getTableMetrics(byte[] getTableMetricsParams) throws ImpalaException,
- TException {
+ public String getTableMetrics(byte[] getTableMetricsParams) throws ImpalaException {
+ long start = System.currentTimeMillis();
TGetTableMetricsParams params = new TGetTableMetricsParams();
JniUtil.deserializeThrift(protocolFactory_, params, getTableMetricsParams);
- return catalog_.getTableMetrics(params.table_name);
+ String res = catalog_.getTableMetrics(params.table_name);
+ JniUtil.logResponse(res.length(), start, params, "getTableMetrics");
+ return res;
}
/**
@@ -240,10 +256,13 @@ public class JniCatalog {
*/
public byte[] getCatalogObject(byte[] thriftParams) throws ImpalaException,
TException {
+ long start = System.currentTimeMillis();
TCatalogObject objectDescription = new TCatalogObject();
JniUtil.deserializeThrift(protocolFactory_, objectDescription, thriftParams);
TSerializer serializer = new TSerializer(protocolFactory_);
- return serializer.serialize(catalog_.getTCatalogObject(objectDescription));
+ byte[] res = serializer.serialize(catalog_.getTCatalogObject(objectDescription));
+ JniUtil.logResponse(res.length, start, objectDescription, "getCatalogObject");
+ return res;
}
/**
@@ -252,19 +271,25 @@ public class JniCatalog {
*/
public String getJsonCatalogObject(byte[] thriftParams) throws ImpalaException,
TException {
+ long start = System.currentTimeMillis();
TCatalogObject objectDescription = new TCatalogObject();
JniUtil.deserializeThrift(protocolFactory_, objectDescription, thriftParams);
TSerializer jsonSerializer = new TSerializer(new TSimpleJSONProtocol.Factory());
- return jsonSerializer.toString(catalog_.getTCatalogObject(objectDescription));
+ String res = jsonSerializer.toString(catalog_.getTCatalogObject(objectDescription));
+ JniUtil.logResponse(res.length(), start, objectDescription, "getJsonCatalogObject");
+ return res;
}
public byte[] getPartialCatalogObject(byte[] thriftParams) throws ImpalaException,
TException {
+ long start = System.currentTimeMillis();
TGetPartialCatalogObjectRequest req =
new TGetPartialCatalogObjectRequest();
JniUtil.deserializeThrift(protocolFactory_, req, thriftParams);
TSerializer serializer = new TSerializer(protocolFactory_);
- return serializer.serialize(catalog_.getPartialCatalogObject(req));
+ byte[] res = serializer.serialize(catalog_.getPartialCatalogObject(req));
+ JniUtil.logResponse(res.length, start, req, "getPartialCatalogObject");
+ return res;
}
/**
@@ -272,6 +297,7 @@ public class JniCatalog {
*/
public byte[] getFunctions(byte[] thriftParams) throws ImpalaException,
TException {
+ long start = System.currentTimeMillis();
TGetFunctionsRequest request = new TGetFunctionsRequest();
JniUtil.deserializeThrift(protocolFactory_, request, thriftParams);
TSerializer serializer = new TSerializer(protocolFactory_);
@@ -288,18 +314,22 @@ public class JniCatalog {
response.addToFunctions(fn.toThrift());
}
- return serializer.serialize(response);
+ byte[] res = serializer.serialize(response);
+ JniUtil.logResponse(res.length, start, request, "getFunctions");
+ return res;
}
- public void prioritizeLoad(byte[] thriftLoadReq) throws ImpalaException,
- TException {
+ public void prioritizeLoad(byte[] thriftLoadReq) throws ImpalaException {
+ long start = System.currentTimeMillis();
TPrioritizeLoadRequest request = new TPrioritizeLoadRequest();
JniUtil.deserializeThrift(protocolFactory_, request, thriftLoadReq);
catalog_.prioritizeLoad(request.getObject_descs());
+ JniUtil.logResponse(start, request, "prioritizeLoad");
}
public byte[] getPartitionStats(byte[] thriftParams)
throws ImpalaException, TException {
+ long start = System.currentTimeMillis();
TGetPartitionStatsRequest request = new TGetPartitionStatsRequest();
JniUtil.deserializeThrift(protocolFactory_, request, thriftParams);
TSerializer serializer = new TSerializer(protocolFactory_);
@@ -310,7 +340,9 @@ public class JniCatalog {
response.setStatus(
new TStatus(TErrorCode.INTERNAL_ERROR, ImmutableList.of(e.getMessage())));
}
- return serializer.serialize(response);
+ byte[] res = serializer.serialize(response);
+ JniUtil.logResponse(res.length, start, request, "getPartitionStats");
+ return res;
}
/**
@@ -319,12 +351,15 @@ public class JniCatalog {
*/
public byte[] updateCatalog(byte[] thriftUpdateCatalog) throws ImpalaException,
TException {
+ long start = System.currentTimeMillis();
TUpdateCatalogRequest request = new TUpdateCatalogRequest();
JniUtil.deserializeThrift(protocolFactory_, request, thriftUpdateCatalog);
TSerializer serializer = new TSerializer(protocolFactory_);
catalogOperationUsage.increment(request);
try {
- return serializer.serialize(catalogOpExecutor_.updateCatalog(request));
+ byte[] res = serializer.serialize(catalogOpExecutor_.updateCatalog(request));
+ JniUtil.logResponse(res.length, start, request, "updateCatalog");
+ return res;
} finally {
catalogOperationUsage.decrement(request);
}
@@ -334,35 +369,49 @@ public class JniCatalog {
* Returns information about the current catalog usage.
*/
public byte[] getCatalogUsage() throws ImpalaException, TException {
+ long start = System.currentTimeMillis();
TSerializer serializer = new TSerializer(protocolFactory_);
- return serializer.serialize(catalog_.getCatalogUsage());
+ byte[] res = serializer.serialize(catalog_.getCatalogUsage());
+ JniUtil.logResponse(res.length, start, /*thriftReq*/null, "getCatalogUsage");
+ return res;
}
/**
* Returns information about the current catalog operation metrics.
*/
public byte[] getOperationUsage() throws ImpalaException, TException {
+ long start = System.currentTimeMillis();
TSerializer serializer = new TSerializer(protocolFactory_);
- return serializer.serialize(catalog_.getOperationUsage());
+ byte[] res = serializer.serialize(catalog_.getOperationUsage());
+ JniUtil.logResponse(res.length, start, /*thriftReq*/null, "getOperationUsage");
+ return res;
}
public byte[] getEventProcessorSummary() throws TException {
+ long start = System.currentTimeMillis();
TSerializer serializer = new TSerializer(protocolFactory_);
- return serializer.serialize(catalog_.getEventProcessorSummary());
+ byte[] res = serializer.serialize(catalog_.getEventProcessorSummary());
+ JniUtil.logResponse(res.length, start, /*thriftReq*/null, "getEventProcessorSummary");
+ return res;
}
public void updateTableUsage(byte[] req) throws ImpalaException {
+ long start = System.currentTimeMillis();
TUpdateTableUsageRequest thriftReq = new TUpdateTableUsageRequest();
JniUtil.deserializeThrift(protocolFactory_, thriftReq, req);
catalog_.updateTableUsage(thriftReq);
+ JniUtil.logResponse(start, thriftReq, "updateTableUsage");
}
public byte[] getCatalogServerMetrics() throws ImpalaException, TException {
+ long start = System.currentTimeMillis();
TGetCatalogServerMetricsResponse response = new TGetCatalogServerMetricsResponse();
response.setCatalog_partial_fetch_rpc_queue_len(
catalog_.getPartialFetchRpcQueueLength());
response.setEvent_metrics(catalog_.getEventProcessorMetrics());
TSerializer serializer = new TSerializer(protocolFactory_);
- return serializer.serialize(response);
+ byte[] res = serializer.serialize(response);
+ JniUtil.logResponse(res.length, start, /*thriftReq*/null, "getCatalogServerMetrics");
+ return res;
}
}