You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Michael Yzvenn Wolanski (Jira)" <ji...@apache.org> on 2019/09/04 00:53:00 UTC
[jira] [Created] (BEAM-8145) Pubsub message size limit not taking
size increase from base64 encoding into account
Michael Yzvenn Wolanski created BEAM-8145:
---------------------------------------------
Summary: Pubsub message size limit not taking size increase from base64 encoding into account
Key: BEAM-8145
URL: https://issues.apache.org/jira/browse/BEAM-8145
Project: Beam
Issue Type: Bug
Components: io-java-gcp
Reporter: Michael Yzvenn Wolanski
In the PubSubIO, the default max batch size is set to `10 * 1024 * 1024` bytes. This however does not take into account the size increase of base64 encoding the messages after the flush. Base64 encodes each set of three bytes into four bytes.
Therefore the 'true' size limit placed on the unencoded batch should be
x = ((10 * 1024 * 1024) / 4) * 3 = 7864320 bytes
--
This message was sent by Atlassian Jira
(v8.3.2#803003)