[jira] [Updated] (KUDU-3450) SubprocessProtocol has a hard-coded limit on message size, but RangerClient doesn't honor that while generating requests

["Alexey Serbin (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KUDU-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexey Serbin updated KUDU-3450:
--------------------------------
    Affects Version/s: 1.16.0
                       1.15.0
                       1.14.0
                       1.13.0
                       1.12.0

> SubprocessProtocol has a hard-coded limit on message size, but RangerClient doesn't honor that while generating requests
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: KUDU-3450
>                 URL: https://issues.apache.org/jira/browse/KUDU-3450
>             Project: Kudu
>          Issue Type: Bug
>          Components: subprocess
>    Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0
>            Reporter: Alexey Serbin
>            Priority: Major
>
> As one can see from the code in the [SubprocessProtocol|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/subprocess/subprocess_protocol.cc#L106-L110] and in [MessageIO utility Java class|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageIO.java#L69-L73], it's not possible to receive messages in PB format that are larger than a pre-defined limit.  Also, there isn't a way to pass {{SubprocessRequestPB::request}} and {{SubprocessResponsePB::response}} data in multiple chunks in the SubprocessProtocol by design.
> The default setting for the maximum message size is as 1MiByte for the implementation of [the relevant C++|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/subprocess/subprocess_protocol.cc#L46] and [Java subprocess components|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessConfiguration.java#L54].
> The implementation of the related C++ components has the setting effectively hard-coded and there isn't a way to change it.   The implementation of the related Java components has this setting configurable via the {{maxMsgBytes}} configuration property.
> Anyways, nether the code in {{RangerClient}} nor the code in {{RangerProtocolHandler}} honors that limit:
> * [RangerClient::AuthorizeActionMultipleColumns()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/ranger/ranger_client.cc#L435-L448]
> * [RangerClient::AuthorizeActionMultipleTables()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/src/kudu/ranger/ranger_client.cc#L468-L492]
> * [ProtocolHandler.unpackAndExecuteRequest()|https://github.com/apache/kudu/blob/9684200713b5e1bf258437527127bd98acfa1e42/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java#L47-L56]
> With that, it's possible to get into an impasse situation from the both sides.  For example, if a large request has been generated by {{RangerClient::AuthorizeActionMultipleColumns()}} or {{RangerClient::AuthorizeActionMultipleTables()}}, the following stack trace might be observed in the output of the subprocess that runs the Ranger client plugin:
> {noformat}
> Exception in thread "main" java.util.concurrent.CompletionException: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the protobuf message
>         at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
>         at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
>         at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1629)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.kudu.subprocess.KuduSubprocessException: Unable to read the protobuf message
>         at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:74)
>         at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
>         ... 3 more
> Caused by: java.io.IOException: message size (2763197) exceeds maximum message size (1048576)
>         at org.apache.kudu.subprocess.MessageIO.readBytes(MessageIO.java:71)
>         at org.apache.kudu.subprocess.MessageReader.run(MessageReader.java:68)
>         ... 4 more
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)