You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/12/05 17:03:32 UTC
[GitHub] [beam] max-lepikhin opened a new pull request, #24527: adding azure IO
max-lepikhin opened a new pull request, #24527:
URL: https://github.com/apache/beam/pull/24527
Flink job-server does not include Azure Blob Storage filesystem. This change adds it.
R: @phoerious @kennknowles
fixes #24367
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] max-lepikhin commented on pull request #24527: adding azure IO
Posted by GitBox <gi...@apache.org>.
max-lepikhin commented on PR #24527:
URL: https://github.com/apache/beam/pull/24527#issuecomment-1338676154
Note that this change will enable azure file system but it is not sufficient for using "--artifacts_dir" in azfs from python flink_runner.
See: https://github.com/apache/beam/issues/24537
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] max-lepikhin closed pull request #24527: adding azure IO
Posted by GitBox <gi...@apache.org>.
max-lepikhin closed pull request #24527: adding azure IO
URL: https://github.com/apache/beam/pull/24527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] max-lepikhin commented on pull request #24527: adding azure IO
Posted by GitBox <gi...@apache.org>.
max-lepikhin commented on PR #24527:
URL: https://github.com/apache/beam/pull/24527#issuecomment-1371504056
or beam.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #24527: adding azure IO
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #24527:
URL: https://github.com/apache/beam/pull/24527#issuecomment-1350194994
Could you clarify what you mean by "will enable azure file system" ?
This seems to be adding this to the job server (not the SDK container). Or do you need this to read/write from Azure with non-portable Flink runner somehow ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #24527: adding azure IO
Posted by GitBox <gi...@apache.org>.
kennknowles commented on PR #24527:
URL: https://github.com/apache/beam/pull/24527#issuecomment-1338098705
Thanks! I wonder what @chamikaramj and @apilloud think about how to manage plugins for job servers so we can stage files to all the filesystems. Perhaps simply including all the "major" clouds is enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] max-lepikhin commented on pull request #24527: adding azure IO
Posted by GitBox <gi...@apache.org>.
max-lepikhin commented on PR #24527:
URL: https://github.com/apache/beam/pull/24527#issuecomment-1350207932
> This seems to be adding this to the job server (not the SDK container). Or do you need this to read/write from Azure with non-portable Flink runner somehow ?
Not sure about the differences between portable and non-portable. Python beam requires a beam flink job-server, we used the jar from maven "org.apache.beam:beam-runners-flink-1.14-job-server:2.43.0". The jar was passed to the python beam using [flink_job_server_jar](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L1479) flag. Then, the job server failed to read artifacts from either:
- /tmp (default) when --artifacts_dir is not set. In this case it fails on the flink worker side as artifact's path is passed unchanged to it. Guessing the artifacts must be staged in a cloud storage not in local /tmp on the job VM.
- azfs://. In this case it fails in the job-server as azure file system is not on the class path when the server starts.
This change adds azure file system to the jar so that the job-server didn't fail when azfs:// is used in --artifacts_dir.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] max-lepikhin commented on pull request #24527: adding azure IO
Posted by GitBox <gi...@apache.org>.
max-lepikhin commented on PR #24527:
URL: https://github.com/apache/beam/pull/24527#issuecomment-1371503894
We no longer use flink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org