You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/12 18:49:53 UTC
[GitHub] [beam] ronoaldo opened a new issue, #23189: [Bug]: Calling a Java custom PTransform from Go using xlang fails on Dataflow Runner
ronoaldo opened a new issue, #23189:
URL: https://github.com/apache/beam/issues/23189
### What happened?
As mentioned in #22931 - I am testing xlang support for the Go Runtime. I first was doing it wrong, using the Go direct runner, but then I moved on to test with the Dataflow runner.
I noticed that on Dataflow Runner, I could call a custom [Java Ptransform](https://github.com/ronoaldo/micro-beam/blob/main/05_xlang/java/src/main/java/com/ronoaldo/SplitWordsFromJava.java) from [Python](https://github.com/ronoaldo/micro-beam/blob/main/05_xlang/pipeline.py):
However,I could not call the same PTransform [from Go](https://github.com/ronoaldo/micro-beam/blob/main/05_xlang/go/pipeline.go). To submit the job, I started the Java expansion service as I did for the Python run, and called my Go pipeline. I see that the expansions service is called from Go, that the Go runtime stages the .jar files into Cloud Storage, but then the pipeline fails:
The only error message available is:
```
S03:e5/ProcessElementAndRestrictionWithSizing+External/FlatMapElements/FlatMap/ParMultiDo(Anonymous)+External/Filter/ParDo(Anonymous)/ParMultiDo(Anonymous)+stats.Count/stats.keyedCountFn+stats.Count/stats.SumPerKey/CombinePerKey/CoGBK+stats.Count/stats.SumPerKey/CombinePerKey/stats.sumIntFn/Partial+stats.Count/stats.SumPerKey/CombinePerKey/CoGBK/Write failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers:
go-job-1-1662153285384063-09021416-tqko-harness-x8dv
Root cause: The worker lost contact with the service.,
go-job-1-1662153285384063-09021416-tqko-harness-x8dv
Root cause: The worker lost contact with the service.,
go-job-1-1662153285384063-09021416-tqko-harness-x8dv
Root cause: The worker lost contact with the service.,
go-job-1-1662153285384063-09021416-tqko-harness-x8dv
Root cause: The worker lost contact with the service.
```
Here is a full [pipeline log](https://docs.google.com/spreadsheets/d/1OFKBdDiHsz2ZAu5K_DjP4g15Wr2tr3rroyLN3qmts8o/edit?usp=sharing) and [worker logs](https://docs.google.com/spreadsheets/d/1aN8ycCBvs3tJljZ2gN2J0rv8GypQxebIcfM7tygGEeU/edit?usp=sharing) downloaded from Logs explorer.
Is this an actual bug or a unsupported workflow? I am trying to follow the Multi language pipelines section of the [Beam Programming Model docs](https://beam.apache.org/documentation/programming-guide/#multi-language-pipelines), regarding expose the Java PTransform (which I assume is correct since calling from Python works) and consuming it from Go, which I'm not sure if I missed any important steps.
### Issue Priority
Priority: 2
### Issue Component
Component: sdk-go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on issue #23189: [Bug]: Calling a Java custom PTransform from Go using xlang fails on Dataflow Runner
Posted by GitBox <gi...@apache.org>.
lostluck commented on issue #23189:
URL: https://github.com/apache/beam/issues/23189#issuecomment-1247348202
I can confirm that the same should have worked for both, but I don't have the experience with Xlang to understand what's going wrong with what you have here.
The logs indicate that both the Go and the Java containers are starting up, but then the workers simply crash/die.
It would be useful to confirm whether the Python pipeline and the Go pipeline are uploading the same java artifacts however. Dumps of the pipeline protos (which should also be uploaded to the GCS folders) would also be valuable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on issue #23189: [Bug]: Calling a Java custom PTransform from Go using xlang fails on Dataflow Runner
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #23189:
URL: https://github.com/apache/beam/issues/23189#issuecomment-1247395845
I don't see workitems actually executing in the worker log you provided.
Can you check other logs (for example, kubelet) to see if there are any errors related to pipeline setup ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org