You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/07/15 18:00:04 UTC
[GitHub] [beam] johnjcasey opened a new issue, #22303: [Task]: Add tests to Kafka SDF and fix known and discovered issues
johnjcasey opened a new issue, #22303:
URL: https://github.com/apache/beam/issues/22303
### What needs to happen?
Kafka IO SDF currently has an issue where it is not restarting properly: https://github.com/apache/beam/issues/21730
In addition, there was a separate issue where it would not commit back to Kafka correctly.
As such, it appears that Kafka SDF has not been sufficiently tested, and the scope of this task is to test it more robustly, and fix both the linked issue, and any other issues discovered.
### Issue Priority
Priority: 1
### Issue Component
Component: io-java-kafka
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #22303: [Task]: Add tests to Kafka SDF and fix known and discovered issues
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #22303:
URL: https://github.com/apache/beam/issues/22303#issuecomment-1205278111
@Abacn this issue can be used to track the kafka testing improvements
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on issue #22303: [Task]: Add tests to Kafka SDF and fix known and discovered issues
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #22303:
URL: https://github.com/apache/beam/issues/22303#issuecomment-1281221881
how should we triage this? should we assign it to @johnjcasey or @Abacn ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #22303: [Task]: Add tests to Kafka SDF and fix known and discovered issues
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #22303:
URL: https://github.com/apache/beam/issues/22303#issuecomment-1225773890
https://github.com/apache/beam/issues/22715 also
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #22303: [Task]: Add tests to Kafka SDF and fix known and discovered issues
Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #22303:
URL: https://github.com/apache/beam/issues/22303#issuecomment-1374949407
Apparently Kafka performance test is failing again after #24879 in. This is because we removed `shuffle=appliance` there and then read from shuffle becomes faster. There are some reports that runner v2 is putting more pressure on kafka, and resulting in higher error rate. Now kafka server is overwhelmed with write requests and making write pipeline fail.
This reveals real issue of the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org