You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/03/01 05:53:00 UTC
[jira] [Commented] (KUDU-3450) SubprocessProtocol has a hard-coded limit on message size, but RangerClient doesn't honor that while generating requests
[ https://issues.apache.org/jira/browse/KUDU-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17694867#comment-17694867 ]
ASF subversion and git services commented on KUDU-3450:
-------------------------------------------------------
Commit 7b593aedaded2f01fbf8a6094524e6012ad542e7 in kudu's branch refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=7b593aeda ]
KUDU-3450 handling of oversized messages in Ranger process wrapper
This patch updates the Ranger client process wrapper to handle
oversized messages. With this patch, upon receiving an oversized
message, the wrapper discards the message and logs about the error.
The input stream is ready to receive next messages after that.
A corresponding unit test for MessageIO.readBytes() is added as well.
In addition, now the --subprocess_max_message_size_bytes master's flag
controls the maximum message size at the Ranger client process wrapper
as well. SubprocessConfiguration.MAX_MESSAGE_BYTES_DEFAULT has been
updated to match the default value of the flag.
This is a follow-up to ae22d32ef5895a02a763665cfd2812aa61be5ab3.
Change-Id: I9fe57bdeeb8a5515578e3feb7cacbbb0d7692e3e
Reviewed-on: http://gerrit.cloudera.org:8080/19516
Tested-by: Alexey Serbin <al...@apache.org>
Reviewed-by: Attila Bukor <ab...@apache.org>
> SubprocessProtocol has a hard-coded limit on message size, but RangerClient doesn't honor that while generating requests
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: KUDU-3450
> URL: https://issues.apache.org/jira/browse/KUDU-3450
> Project: Kudu
> Issue Type: Bug
> Components: subprocess
> Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0
> Reporter: Alexey Serbin
> Priority: Major
>
> As one can see from the code in the [SubprocessProtocol|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/subprocess/subprocess_protocol.cc#L106-L110] and in [MessageIO utility Java class|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java#L69-L73], it's not possible to receive messages in PB format that are larger than a pre-defined limit. Also, there isn't a way to pass {{SubprocessRequestPB::request}} and {{SubprocessResponsePB::response}} data in multiple chunks in the SubprocessProtocol by design.
> The default setting for the maximum message size is 1MiByte in the implementation of [the relevant C++|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/subprocess/subprocess_protocol.cc#L46] and [Java subprocess components|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java#L54].
> The implementation of the related C++ components has the setting effectively hard-coded and there isn't a way to change it. The implementation of the related Java components has this setting configurable via the {{maxMsgBytes}} configuration property.
> Anyways, nether the code in {{RangerClient}} nor the code in {{RangerProtocolHandler}} honors the limit:
> * [RangerClient::AuthorizeActionMultipleColumns()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/ranger/ranger_client.cc#L435-L448]
> * [RangerClient::AuthorizeActionMultipleTables()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/ranger/ranger_client.cc#L468-L492]
> * [ProtocolHandler.unpackAndExecuteRequest()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java#L47-L56]
> With that, it's possible to get into an impasse situation from the both sides. For example, if a large request has been generated by {{RangerClient::AuthorizeActionMultipleColumns()}} or {{RangerClient::AuthorizeActionMultipleTables()}}, the following stack trace might be observed in the output of the subprocess that runs the Ranger client plugin:
> {noformat}
> Exception in thread "main" java.util.concurrent.CompletionException: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the protobuf message
> at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1629)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the protobuf message
> at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:74)
> at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> ... 3 more
> Caused by: java.io.IOException: message size (2763197) exceeds maximum message size (1048576)
> at org.apache.kudu.subprocess.MessageIO.readBytes(MessageIO.java:71)
> at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:68)
> ... 4 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)