[jira] [Commented] (MESOS-4210) Investigate increasing protobuf protocol message size limit.

["Rob Russo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MESOS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15269826#comment-15269826 ] 

Rob Russo commented on MESOS-4210:
----------------------------------

This does affect our actual production use case. In the latest versions of spark when using the parquet relation they attempt to speed up file discovery by parallelizing it into multiple tasks. Unfortunately if using fine grained mode they tend to do this using a few number of tasks (always seemed to be 8) and the smaller the number of tasks, the larger the result that needs to be sent back to the driver from each task to list the discovered files. With our file setup this exceeds the max protobuf length once the matched files grows into the tens of thousands (which is not that much for us).

For now we are getting around this using a custom built protobuf jar with a higher limit set and building mesos with that jar supplied but it would be great if we could at least have a option to adjust this limit as a mesos config since the limit can be set at runtime.

I realize that protobufs are not designed to handle large messages but given this very standard use case of spark on mesos it seems that there should be some easy workaround here.

> Investigate increasing protobuf protocol message size limit.
> ------------------------------------------------------------
>
>                 Key: MESOS-4210
>                 URL: https://issues.apache.org/jira/browse/MESOS-4210
>             Project: Mesos
>          Issue Type: Bug
>            Reporter: Artem Harutyunyan
>
> {noformat}
> [libprotobuf ERROR google/protobuf/io/coded_stream.cc:171] A protocol message was rejected because it was too big (more than 67108864 bytes). To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h. 
> F20151217 16:33:44.832834 4076 construct.cpp:48] Check failed: parsed Unexpected failure while parsing protobuf
> Check failure stack trace: ***
> @ 0x2b9bab353b68 (unknown) 
> @ 0x2b9bab353ac4 (unknown) 
> @ 0x2b9bab3534ba (unknown) 
> @ 0x2b9bab356274 (unknown) 
> @ 0x2b9bab339d09 (unknown) 
> @ 0x2b9bab338917 (unknown) 
> @ 0x2b9bab33f404 (unknown) 
> @ 0x2b9b68350e18 (unknown) 
> {noformat}
> The error is presumably caused by a "user sending a very large command line".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)