[GitHub] [beam] kennknowles opened a new issue, #19384: PubsubIO may exceed maximum payload size

[GitBox <gi...@apache.org>]

kennknowles opened a new issue, #19384:
URL: https://github.com/apache/beam/issues/19384

   In a batch job on Dataflow that reads payload and metadata from a Bigquery table and publishes them to PubsubIO, I sometimes experience errors:
   ```
   
   com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
   "message" : "Request
   payload size exceeds the limit: 10485760 bytes.",
   
   
   ```
   
    
   
   PubsubIO Javadoc says it will use the the global limit of 10 MiB by default but it seems that doesn't work in all circumstances. I'm handling relatively large records here, up to 600 KiB per message.
    
    Adding
   ```
   
   .withMaxBatchBytesSize(5242880)
   ```
   
   after
   ```
   
   PubsubIO.writeMessages().to(topic)
   ```
   
   fixes this issue.
   
    
   
   Imported from Jira [BEAM-7107](https://issues.apache.org/jira/browse/BEAM-7107). Original Jira may contain additional context.
   Reported by: MadEgg.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org