You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Chamikara Madhusanka Jayalath (Jira)" <ji...@apache.org> on 2020/05/20 22:52:00 UTC
[jira] [Comment Edited] (BEAM-9383) Staging Dataflow artifacts from
environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112645#comment-17112645 ]
Chamikara Madhusanka Jayalath edited comment on BEAM-9383 at 5/20/20, 10:51 PM:
--------------------------------------------------------------------------------
I tried running a Kafka pipeline on Dataflow and I see a lot of jars being staged during pipeline submission.
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/596ab8b3-840a-43ff-accb-8f6815e1a302.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/596ab8b3-840a-43ff-accb-8f6815e1a302.jar in 24 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/6f80255b-453f-4ad8-aa28-7e40fdfeedac.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/6f80255b-453f-4ad8-aa28-7e40fdfeedac.jar in 22 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/2704b169-8874-4163-9f3c-ab8765f3c330.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/2704b169-8874-4163-9f3c-ab8765f3c330.jar in 69 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/40bd912f-ce2f-45a8-9625-019b85c46cc7.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/40bd912f-ce2f-45a8-9625-019b85c46cc7.jar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/9d1bfb42-518d-4cc7-9a3a-7a8ea792ce6f.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/9d1bfb42-518d-4cc7-9a3a-7a8ea792ce6f.jar in 8 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/7e1e7095-32d6-4ea6-b9a0-aa5e2ffdbb31.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/7e1e7095-32d6-4ea6-b9a0-aa5e2ffdbb31.jar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/926b735c-552f-4a3a-9e81-f0fe8162ce26.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/926b735c-552f-4a3a-9e81-f0fe8162ce26.jar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/e9f829ba-eadf-4ae4-98c4-492238cb9998.jar...
...
Ideally there should be only one jar,
beam-sdks-java-io-expansion-service-2.22.0-SNAPSHOT.jar
Any idea where additional jars are coming from. Also can we use names of jars instread of URLs so that we can easily identify what these are ?
cc: [~robertwb] [~lcwik]
was (Author: chamikara):
I tried running a Kafka pipeline on Dataflow and I see a lot of jars being staged during pipeline submission.
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/596ab8b3-840a-43ff-accb-8f6815e1a302.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/596ab8b3-840a-43ff-accb-8f6815e1a302.jar in 24 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/6f80255b-453f-4ad8-aa28-7e40fdfeedac.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/6f80255b-453f-4ad8-aa28-7e40fdfeedac.jar in 22 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/2704b169-8874-4163-9f3c-ab8765f3c330.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/2704b169-8874-4163-9f3c-ab8765f3c330.jar in 69 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/40bd912f-ce2f-45a8-9625-019b85c46cc7.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/40bd912f-ce2f-45a8-9625-019b85c46cc7.jar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/9d1bfb42-518d-4cc7-9a3a-7a8ea792ce6f.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/9d1bfb42-518d-4cc7-9a3a-7a8ea792ce6f.jar in 8 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/7e1e7095-32d6-4ea6-b9a0-aa5e2ffdbb31.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/7e1e7095-32d6-4ea6-b9a0-aa5e2ffdbb31.jar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/926b735c-552f-4a3a-9e81-f0fe8162ce26.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/926b735c-552f-4a3a-9e81-f0fe8162ce26.jar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/e9f829ba-eadf-4ae4-98c4-492238cb9998.jar...
...
Ideally there should be only one jar,
beam-sdks-java-io-expansion-service-2.22.0-SNAPSHOT.ja
Any idea where additional jars are coming from. Also can we use names of jars instread of URLs so that we can easily identify what these are ?
cc: [~robertwb] [~lcwik]
> Staging Dataflow artifacts from environment
> -------------------------------------------
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
> Reporter: Heejong Lee
> Assignee: Heejong Lee
> Priority: P0
> Fix For: 2.22.0
>
> Time Spent: 12h
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)