You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2019/09/04 01:00:33 UTC
[jira] [Work logged] (BEAM-8145) Pubsub message size limit not
taking size increase from base64 encoding into account
[ https://issues.apache.org/jira/browse/BEAM-8145?focusedWorklogId=305998&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305998 ]
ASF GitHub Bot logged work on BEAM-8145:
----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Sep/19 00:59
Start Date: 04/Sep/19 00:59
Worklog Time Spent: 10m
Work Description: Primevenn commented on issue #8184: [BEAM-8145] Max publish batch byte size set to a value which takes base64 encodin…
URL: https://github.com/apache/beam/pull/8184#issuecomment-527695960
R: @chamikaramj Done - https://issues.apache.org/jira/browse/BEAM-8145. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 305998)
Remaining Estimate: 0h
Time Spent: 10m
> Pubsub message size limit not taking size increase from base64 encoding into account
> ------------------------------------------------------------------------------------
>
> Key: BEAM-8145
> URL: https://issues.apache.org/jira/browse/BEAM-8145
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Reporter: Michael Yzvenn Wolanski
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In the PubSubIO, the default max batch size is set to `10 * 1024 * 1024` bytes. This however does not take into account the size increase of base64 encoding the messages after the flush. Base64 encodes each set of three bytes into four bytes.
> Therefore the 'true' size limit placed on the unencoded batch should be
> x = ((10 * 1024 * 1024) / 4) * 3 = 7864320 bytes
--
This message was sent by Atlassian Jira
(v8.3.2#803003)