You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "jonathan-lemos (via GitHub)" <gi...@apache.org> on 2023/06/02 20:20:17 UTC
[GitHub] [beam] jonathan-lemos opened a new issue, #27000: [Bug]: PubSubIO writes cannot be more than 100 bytes
jonathan-lemos opened a new issue, #27000:
URL: https://github.com/apache/beam/issues/27000
### What happened?
Writing to Pub/Sub currently results in the following issue:
```
javax.naming.SizeLimitExceededException: Pubsub message of length 220 exceeds maximum of 100 bytes, when considering the payload and attributes. See https://cloud.google.com/pubsub/quotas#resource_limits
```
This happens because [this call](https://github.com/apache/beam/blob/c0b23a7ea0bd7faf0bdecb2d16e8aed8d8af24a3/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java#L1413) passes in the maximum number of *messages* allowed in a batch, when it should pass in the maximum number of *bytes* allowed in a batch.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jonathan-lemos commented on issue #27000: [Bug]: PubSubIO writes cannot be more than 100 bytes
Posted by "jonathan-lemos (via GitHub)" <gi...@apache.org>.
jonathan-lemos commented on issue #27000:
URL: https://github.com/apache/beam/issues/27000#issuecomment-1574270860
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey closed issue #27000: [Bug]: PubSubIO writes from bounded PCollection cannot be more than 100 bytes
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey closed issue #27000: [Bug]: PubSubIO writes from bounded PCollection cannot be more than 100 bytes
URL: https://github.com/apache/beam/issues/27000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bvolpato commented on issue #27000: [Bug]: PubSubIO writes from bounded PCollection cannot be more than 100 bytes
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on issue #27000:
URL: https://github.com/apache/beam/issues/27000#issuecomment-1601402593
The issue was fixed and will be available in Beam 2.49.0.
The impact is specific to batch jobs that use PubsubIO, as the wrong validation is part of `PubsubBoundedWriter`.
Thanks for the fix @jonathan-lemos.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org