You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/08/31 03:46:58 UTC
[GitHub] [beam] AldairCoronel opened a new pull request #12734: Add authentication story via pipeline options and tests using Azurite.
AldairCoronel opened a new pull request #12734:
URL: https://github.com/apache/beam/pull/12734
**Please** add a meaningful description for your change here
Add azure pipeline options functionality to the azureblobstoragefilesystem and blobstorageio tests using Azurite.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace
--- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12734: Add authentication story via pipeline options and tests using Azurite.
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12734:
URL: https://github.com/apache/beam/pull/12734#issuecomment-698645238
@AldairCoronel - What is the next step on this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12734: Add authentication story via pipeline options and tests using Azurite.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12734:
URL: https://github.com/apache/beam/pull/12734#issuecomment-686792340
can you fix lint, spotless and any precommit errors?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #12734: Add authentication story via pipeline options and tests using Azurite.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12734:
URL: https://github.com/apache/beam/pull/12734#issuecomment-1041455024
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12734: Add authentication story via pipeline options and tests using Azurite.
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12734:
URL: https://github.com/apache/beam/pull/12734#issuecomment-698645238
@AldairCoronel - What is the next step on this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AldairCoronel commented on pull request #12734: Add authentication story via pipeline options and tests using Azurite.
Posted by GitBox <gi...@apache.org>.
AldairCoronel commented on pull request #12734:
URL: https://github.com/apache/beam/pull/12734#issuecomment-723372503
> @AldairCoronel With this PR, will it possible to connect to several different storage accounts using different connection strings?
At the moment you can only connect to a single account.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12734: Add authentication story via pipeline options and tests using Azurite.
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12734:
URL: https://github.com/apache/beam/pull/12734#issuecomment-730711056
@AldairCoronel - What is the next step on this PR? Do you need help?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tanya-borisova commented on pull request #12734: Add authentication story via pipeline options and tests using Azurite.
Posted by GitBox <gi...@apache.org>.
tanya-borisova commented on pull request #12734:
URL: https://github.com/apache/beam/pull/12734#issuecomment-692740987
@AldairCoronel With this PR, will it possible to connect to several different storage accounts using different connection strings?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #12734: Add authentication story via pipeline options and tests using Azurite.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12734:
URL: https://github.com/apache/beam/pull/12734#issuecomment-1032573386
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AldairCoronel commented on pull request #12734: Add authentication story via pipeline options and tests using Azurite.
Posted by GitBox <gi...@apache.org>.
AldairCoronel commented on pull request #12734:
URL: https://github.com/apache/beam/pull/12734#issuecomment-683534181
R: @pabloem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on a change in pull request #12734: Add authentication story via pipeline options and tests using Azurite.
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #12734:
URL: https://github.com/apache/beam/pull/12734#discussion_r519216957
##########
File path: sdks/python/setup.py
##########
@@ -154,18 +154,18 @@ def get_version():
'oauth2client>=2.0.1,<4',
'protobuf>=3.12.2,<4',
# [BEAM-6287] pyarrow is not supported on Windows for Python 2
- ('pyarrow>=0.15.1,<0.18.0; python_version >= "3.0" or '
- 'platform_system != "Windows"'),
+ (
+ 'pyarrow>=0.15.1,<0.18.0; python_version >= "3.0" or '
+ 'platform_system != "Windows"'),
'pydot>=1.2.0,<2',
'python-dateutil>=2.8.0,<3',
- 'pytz>=2018.3',
- # [BEAM-5628] Beam VCF IO is not supported in Python 3.
+ 'pytz>=2018.3', # [BEAM-5628] Beam VCF IO is not supported in Python 3.
Review comment:
This comment applies to the next dependency, not here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12734: Add authentication story via pipeline options and tests using Azurite.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12734:
URL: https://github.com/apache/beam/pull/12734#issuecomment-699192199
I have asked @AldairCoronel to please push this through to the end. He's busy this week, but we'll try to finalize it starting next week.
@tanya-borisova thanks for the interest. I hope we'll be able to wrap this up soon, and have it available for Beam 2.26 or 2.27. Will that work for you? : )
It should be possible to have a relative shortcut to simply pass the connection string via options sometimes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem edited a comment on pull request #12734: Add authentication story via pipeline options and tests using Azurite.
Posted by GitBox <gi...@apache.org>.
pabloem edited a comment on pull request #12734:
URL: https://github.com/apache/beam/pull/12734#issuecomment-699192199
I have asked @AldairCoronel to please push this through to the end. He's busy this week, but we'll try to finalize it starting next week.
@tanya-borisova thanks for the interest. I hope we'll be able to wrap this up soon, and have it available for Beam 2.26 or 2.27. Will that work for you? : )
It should be possible to have a relative shortcut to simply pass the connection string via pipeline options, and have that available soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AldairCoronel commented on pull request #12734: Add authentication story via pipeline options and tests using Azurite.
Posted by GitBox <gi...@apache.org>.
AldairCoronel commented on pull request #12734:
URL: https://github.com/apache/beam/pull/12734#issuecomment-723372785
> @AldairCoronel - What is the next step on this PR?
The only thing missing is to be able to run the azurite tests with gradle.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tanya-borisova commented on pull request #12734: Add authentication story via pipeline options and tests using Azurite.
Posted by GitBox <gi...@apache.org>.
tanya-borisova commented on pull request #12734:
URL: https://github.com/apache/beam/pull/12734#issuecomment-692744673
FYI we at Wayve are going to use Beam for our data pipelines, and we store our data in Azure. Naturally, we're very interested in these features getting merged and getting to the new Beam release :)
Please do let me know if I can help in any way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] closed pull request #12734: Add authentication story via pipeline options and tests using Azurite.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #12734:
URL: https://github.com/apache/beam/pull/12734
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org