You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2023/03/17 17:18:05 UTC
[GitHub] [beam] Abacn opened a new pull request, #25886: Fix Illegal to use destination for jms producer
Abacn opened a new pull request, #25886:
URL: https://github.com/apache/beam/pull/25886
Fixes #25849
The issue is that the producer was created with a destination in #25849. This kind of producer cannot set destination again in send()
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #25886: Fix Illegal to use destination for jms producer
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25886:
URL: https://github.com/apache/beam/pull/25886#issuecomment-1477972347
@Amraneze is both pipeline (with/without excessive port occupying) running on same version of Dataflow runner (v1 or v2) ?
Anyway this is another issue than that reported in #25849. @sriram23kmm are you able to try the fix of this PR, that is compile your own jmsio artifact and replace that shipped with Beam v2.46.0 to validate it fixes your issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amraneze commented on pull request #25886: Fix Illegal to use destination for jms producer
Posted by "Amraneze (via GitHub)" <gi...@apache.org>.
Amraneze commented on PR #25886:
URL: https://github.com/apache/beam/pull/25886#issuecomment-1477437564
For the IT tests with other connectors, we don't have access to the library TIBCO. What should we do ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #25886: Fix Illegal to use destination for jms producer
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25886:
URL: https://github.com/apache/beam/pull/25886#issuecomment-1481176291
@sriram23kmm that's great. Let us get this in first then. The port usage is another issue than the original bug and could be fixed separately. The next version v2.47.0 which is expected be released in late April.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amraneze commented on pull request #25886: Fix Illegal to use destination for jms producer
Posted by "Amraneze (via GitHub)" <gi...@apache.org>.
Amraneze commented on PR #25886:
URL: https://github.com/apache/beam/pull/25886#issuecomment-1478220381
@Abacn It's the same pipeline one was running with the version 2.45 + the new code for JmsIO and one was with the version 2.46. Both of them have the same configuration (same number of workers, ....) and they use Dataflow Runner v1 (we had some drawbacks with v2)
Here is a screenshot of the port usage after running the pipeline (using only beam 2.46) with `numberOfWorkerHarnessThreads` = 32. It stabilizes at 2k open ports.
<img width="1664" alt="image" src="https://user-images.githubusercontent.com/28459763/226658971-75509571-43a5-4880-96c2-a6e7ea999eee.png">
A screenshot of the port usage after running the pipeline (using beam 2.46 + the fix of this PR). It stabilizes at 2k open ports.
<img width="1672" alt="image" src="https://user-images.githubusercontent.com/28459763/226668678-f87aca1f-dbeb-4504-909c-c94da306af19.png">
A screenshot of the port usage after running the pipeline (using beam 2.45 + the fix of this PR + new JmsIO code). It stabilizes at 2k open ports.
<img width="1666" alt="image" src="https://user-images.githubusercontent.com/28459763/226680600-37804041-fd74-4359-a187-3b5ac7457aac.png">
>Note: The pipelines are running with the same configuration, `18` workers and `n2-highmem-4` without `autoscaling`
I guess, we will need to setup the connection at the `setup cycle` instead of `startBundle cycle`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #25886: Fix Illegal to use destination for jms producer
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25886:
URL: https://github.com/apache/beam/pull/25886#issuecomment-1476352594
>
Hi @Amraneze, thanks for the work. Is this small fix work or there are other things need fix? If this suffices we can get this in and have a nightly snapshot version that can be used as a workaround.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #25886: Fix Illegal to use destination for jms producer
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25886:
URL: https://github.com/apache/beam/pull/25886#issuecomment-1477833593
> There are two things with Apache Qpid connector, we can't use static topic. The workaround was to use dynamic topic and give a function that returns always the same value. Also, with the same connector, we found out that the number of ports opened is around 5k (18 workers * 300) which consume the majority of the ports in our NAT gateway. But, using the same code of JmsIO without the new version of Apache beam works fine.
>
> >Note: We don't want to use the option `numberOfWorkerHarnessThreads`
What is 300 coming from? Is it the number of thread per worker, or total number of bundles? This sounds like connection leaking.
Also, what is the timeline for the fix? If not before the cut of 2.47 (beginning of April we should revert #25887 at the moment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn merged pull request #25886: Fix Illegal to use destination for jms producer
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #25886:
URL: https://github.com/apache/beam/pull/25886
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amraneze commented on pull request #25886: Fix Illegal to use destination for jms producer
Posted by "Amraneze (via GitHub)" <gi...@apache.org>.
Amraneze commented on PR #25886:
URL: https://github.com/apache/beam/pull/25886#issuecomment-1477436811
There are two things with Apache Qpid connector, we can't use static topic. The workaround was to use dynamic topic and give a function that returns always the same value. Also, with the same connector, we found out that the number of ports opened is around 5k (18 workers * 300) which consume the majority of the ports in our NAT gateway. But, using the same code of JmsIO without the new version of Apache beam works fine.
>Note: We don't want to use the option `numberOfWorkerHarnessThreads`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] sriram23kmm commented on pull request #25886: Fix Illegal to use destination for jms producer
Posted by "sriram23kmm (via GitHub)" <gi...@apache.org>.
sriram23kmm commented on PR #25886:
URL: https://github.com/apache/beam/pull/25886#issuecomment-1480729343
@Abacn I tried this PR fix + 2.46 beam it solves our problem.
Also, when it is going release in the next version(2.47.0)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #25886: Fix Illegal to use destination for jms producer
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25886:
URL: https://github.com/apache/beam/pull/25886#issuecomment-1481261746
done. The nightly snapshot is in the directory https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-io-jms/2.47.0-SNAPSHOT/ and will be updated to include the fix in the next snapshot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] sriram23kmm commented on pull request #25886: Fix Illegal to use destination for jms producer
Posted by "sriram23kmm (via GitHub)" <gi...@apache.org>.
sriram23kmm commented on PR #25886:
URL: https://github.com/apache/beam/pull/25886#issuecomment-1481232875
Thanks, @Abacn so then can you merge this PR and get this immediately since it was affecting our production
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amraneze commented on pull request #25886: Fix Illegal to use destination for jms producer
Posted by "Amraneze (via GitHub)" <gi...@apache.org>.
Amraneze commented on PR #25886:
URL: https://github.com/apache/beam/pull/25886#issuecomment-1475827267
Hey @Abacn, We got the same issue in our pipeline too. We added some workaround. But, I will create a new PR to add more IT tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amraneze commented on pull request #25886: Fix Illegal to use destination for jms producer
Posted by "Amraneze (via GitHub)" <gi...@apache.org>.
Amraneze commented on PR #25886:
URL: https://github.com/apache/beam/pull/25886#issuecomment-1477953566
> What is 300 coming from? Is it the number of thread per worker, or total number of bundles? This sounds like connection leaking.
For now, we are testing with `numberOfWorkerHarnessThreads` to confirm the hypothesis that it's related to number of thread per worker. I will let you know about it.
> Also, what is the timeline for the fix? If not before the cut of 2.47 (beginning of April we should revert #25887 at the moment
The problem is that we tested the same code (JmsIO) that we merged in our pipelines with the version 2.45 and we didn't have any issues (for the last 2 weeks). It's just when we upgraded the apache beam to 2.46, so I'm not sure if it's related to JmsIO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #25886: Fix Illegal to use destination for jms producer
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25886:
URL: https://github.com/apache/beam/pull/25886#issuecomment-1474173591
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #25886: Fix Illegal to use destination for jms producer
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25886:
URL: https://github.com/apache/beam/pull/25886#issuecomment-1474172239
R: @sriram23kmm
CC: @Amraneze
This indicates we need some integration test for JmsIO (entered #25887). Would help if you are interested in adding integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org