You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/04/11 07:14:00 UTC
[jira] [Commented] (IMPALA-11669) Make Thrift max message size configuration
[ https://issues.apache.org/jira/browse/IMPALA-11669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710433#comment-17710433 ]
ASF subversion and git services commented on IMPALA-11669:
----------------------------------------------------------
Commit 3333af5d97c9367474fca6177a6dd415d747793f in impala's branch refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3333af5d9 ]
IMPALA-12043: Fix "MaxMessageSize reached" exception in Thrift JNI
Large results from something like COMPUTE INCREMENTAL STATS can result
in "TTransportException: MaxMessageSize reached". This happens when
CatalogdMetaProvider.updateCatalogCache() receives a buffer through the
JNI from NativeGetNextCatalogObjectUpdate that exceeds 100MB.
TByteBuffer inherits from TEndpointTransport, which in Thrift 0.16.0
adds a MaxMessageSize limit. TMemoryBuffer adds a constructor that
allows passing in a TConfiguration object to customize the limit, which
we make use of in IMPALA-11669, but TByteBuffer does not provide a
similar interface (THRIFT-5696) and was overlooked.
This patch fix the issue by copying
org.apache.thrift.transport.TByteBuffer from thrift-0.16.0 to
org.apache.impala.util.TByteBuffer and patch it with fix from
THRIFT-5696. It then pass thrift_rpc_max_message_size value as a
TConfiguration to the TByteBuffer constructor.
Testing:
- Pass CatalogdMetaProviderTest tests.
Change-Id: I105db332cd312d80bac0313090576bc47064ee16
Reviewed-on: http://gerrit.cloudera.org:8080/19712
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Make Thrift max message size configuration
> ------------------------------------------
>
> Key: IMPALA-11669
> URL: https://issues.apache.org/jira/browse/IMPALA-11669
> Project: IMPALA
> Issue Type: Task
> Components: Backend
> Affects Versions: Impala 4.2.0
> Reporter: Joe McDonnell
> Assignee: Riza Suminto
> Priority: Critical
> Fix For: Impala 4.2.0
>
>
> With the upgrade to Thrift 0.16, Thrift now has a protection against malicious message in the form of a maximum size for messages. This is currently set to 100MB by default. Impala should add the ability to override this default value. In particular, it seems like communication between coordinators and the catalogd may need a larger value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org