You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/12/06 14:24:43 UTC
[GitHub] [beam] ilya-kozyrev opened a new pull request #16140: [BEAM-13377][Playground] Update CI/CD python scripts to work with specific SDK
ilya-kozyrev opened a new pull request #16140:
URL: https://github.com/apache/beam/pull/16140
- Add argparser.
- Add SDK as input argument.
- Small refactoring
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] daria-malkova commented on a change in pull request #16140: [BEAM-13377][Playground] Update CI/CD python scripts to work with specific SDK
Posted by GitBox <gi...@apache.org>.
daria-malkova commented on a change in pull request #16140:
URL: https://github.com/apache/beam/pull/16140#discussion_r763062547
##########
File path: playground/infrastructure/ci_cd.py
##########
@@ -16,37 +16,76 @@
"""
Module implements CI/CD steps for Beam Playground examples
"""
-
+import argparse
import asyncio
import os
+from typing import List, Literal
+import config
+from api.v1.api_pb2 import Sdk
from cd_helper import CDHelper
from ci_helper import CIHelper
-from helper import find_examples, get_supported_categories
+from helper import find_examples, get_supported_categories, Example
from logger import setup_logger
+parser = argparse.ArgumentParser(
+ description="CI/CD Steps for Playground objects")
+parser.add_argument(
+ "--step",
+ dest="step",
+ required=True,
+ help="CI step to verify all beam examples/tests/katas. CD step to save all "
+ "beam examples/tests/katas and their outputs on the GCS",
+ choices=["CI", "CD"])
Review comment:
Can't we do it like below: config.Config.SUPPORTED_SDK?
Please move available steps to the config
##########
File path: playground/infrastructure/ci_cd.py
##########
@@ -16,37 +16,76 @@
"""
Module implements CI/CD steps for Beam Playground examples
"""
-
+import argparse
Review comment:
I don't see it in requirements, shouldn't this be added?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ilya-kozyrev commented on a change in pull request #16140: [BEAM-13377][Playground] Update CI/CD python scripts to work with specific SDK
Posted by GitBox <gi...@apache.org>.
ilya-kozyrev commented on a change in pull request #16140:
URL: https://github.com/apache/beam/pull/16140#discussion_r763076099
##########
File path: playground/infrastructure/ci_cd.py
##########
@@ -16,37 +16,76 @@
"""
Module implements CI/CD steps for Beam Playground examples
"""
-
+import argparse
import asyncio
import os
+from typing import List, Literal
+import config
+from api.v1.api_pb2 import Sdk
from cd_helper import CDHelper
from ci_helper import CIHelper
-from helper import find_examples, get_supported_categories
+from helper import find_examples, get_supported_categories, Example
from logger import setup_logger
+parser = argparse.ArgumentParser(
+ description="CI/CD Steps for Playground objects")
+parser.add_argument(
+ "--step",
+ dest="step",
+ required=True,
+ help="CI step to verify all beam examples/tests/katas. CD step to save all "
+ "beam examples/tests/katas and their outputs on the GCS",
+ choices=["CI", "CD"])
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ilya-kozyrev commented on a change in pull request #16140: [BEAM-13377][Playground] Update CI/CD python scripts to work with specific SDK
Posted by GitBox <gi...@apache.org>.
ilya-kozyrev commented on a change in pull request #16140:
URL: https://github.com/apache/beam/pull/16140#discussion_r763127467
##########
File path: playground/infrastructure/helper.py
##########
@@ -199,7 +212,9 @@ def get_supported_categories(categories_path: str) -> List[str]:
return yaml_object[TagFields.categories]
-def _get_example(filepath: str, filename: str, tag: dict) -> Example:
+def _get_example(
+ filepath: str, filename: str, tag: Dict[str, Union[str,
Review comment:
That's not my formatting :) yapf think that this one is beautiful :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KhaninArtur commented on a change in pull request #16140: [BEAM-13377][Playground] Update CI/CD python scripts to work with specific SDK
Posted by GitBox <gi...@apache.org>.
KhaninArtur commented on a change in pull request #16140:
URL: https://github.com/apache/beam/pull/16140#discussion_r763147287
##########
File path: playground/infrastructure/ci_cd.py
##########
@@ -16,37 +16,76 @@
"""
Module implements CI/CD steps for Beam Playground examples
"""
-
+import argparse
import asyncio
import os
+from typing import List, Literal
+import config
+from api.v1.api_pb2 import Sdk
from cd_helper import CDHelper
from ci_helper import CIHelper
-from helper import find_examples, get_supported_categories
+from helper import find_examples, get_supported_categories, Example
from logger import setup_logger
+parser = argparse.ArgumentParser(
+ description="CI/CD Steps for Playground objects")
+parser.add_argument(
+ "--step",
+ dest="step",
+ required=True,
+ help="CI step to verify all beam examples/tests/katas. CD step to save all "
+ "beam examples/tests/katas and their outputs on the GCS",
+ choices=["CI", "CD"])
+parser.add_argument(
+ "--sdk",
+ dest="sdk",
+ required=True,
+ help="Supported SDKs",
+ choices=config.Config.SUPPORTED_SDK)
+
+root_dir = os.getenv("BEAM_ROOT_DIR")
+categories_file = os.getenv("BEAM_EXAMPLE_CATEGORIES")
-def ci_step():
+
+def ci_step(examples: List[Example]):
"""
CI step to verify all beam examples/tests/katas
"""
- setup_logger()
- root_dir = os.getenv("BEAM_ROOT_DIR")
- categories_file = os.getenv("BEAM_EXAMPLE_CATEGORIES")
- supported_categories = get_supported_categories(categories_file)
+
ci_helper = CIHelper()
- examples = find_examples(root_dir, supported_categories)
asyncio.run(ci_helper.verify_examples(examples))
-def cd_step():
+def cd_step(examples: List[Example]):
"""
CD step to save all beam examples/tests/katas and their outputs on the GCS
"""
- setup_logger()
- root_dir = os.getenv("BEAM_ROOT_DIR")
- categories_file = os.getenv("BEAM_EXAMPLE_CATEGORIES")
- supported_categories = get_supported_categories(categories_file)
cd_helper = CDHelper()
- examples = find_examples(root_dir, supported_categories)
cd_helper.store_examples(examples)
+
+
+def _check_envs():
+ if root_dir is None:
+ raise KeyError(
+ "BEAM_ROOT_DIR environment variable should be specified in os")
+ if categories_file is None:
+ raise KeyError(
+ "BEAM_EXAMPLE_CATEGORIES environment variable should be specified in os"
+ )
+
+
+def run_ci_cd(step: Literal["CI", "CD"], sdk: Sdk):
+ supported_categories = get_supported_categories(categories_file)
+ examples = find_examples(root_dir, supported_categories, sdk)
+ if step == "CI":
+ ci_step(examples=examples)
+ if step == "CD":
+ cd_step(examples=examples)
Review comment:
It wouldn't be if we would use argparser inside of the function, but it is used only in the main and the function accepts arbitrary things, that is why it will be better to catch incorrect steps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ilya-kozyrev commented on a change in pull request #16140: [BEAM-13377][Playground] Update CI/CD python scripts to work with specific SDK
Posted by GitBox <gi...@apache.org>.
ilya-kozyrev commented on a change in pull request #16140:
URL: https://github.com/apache/beam/pull/16140#discussion_r763073955
##########
File path: playground/infrastructure/ci_cd.py
##########
@@ -16,37 +16,76 @@
"""
Module implements CI/CD steps for Beam Playground examples
"""
-
+import argparse
Review comment:
This import from standard lib :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ilya-kozyrev commented on a change in pull request #16140: [BEAM-13377][Playground] Update CI/CD python scripts to work with specific SDK
Posted by GitBox <gi...@apache.org>.
ilya-kozyrev commented on a change in pull request #16140:
URL: https://github.com/apache/beam/pull/16140#discussion_r763128242
##########
File path: playground/infrastructure/ci_cd.py
##########
@@ -16,37 +16,76 @@
"""
Module implements CI/CD steps for Beam Playground examples
"""
-
+import argparse
import asyncio
import os
+from typing import List, Literal
+import config
+from api.v1.api_pb2 import Sdk
from cd_helper import CDHelper
from ci_helper import CIHelper
-from helper import find_examples, get_supported_categories
+from helper import find_examples, get_supported_categories, Example
from logger import setup_logger
+parser = argparse.ArgumentParser(
+ description="CI/CD Steps for Playground objects")
+parser.add_argument(
+ "--step",
+ dest="step",
+ required=True,
+ help="CI step to verify all beam examples/tests/katas. CD step to save all "
+ "beam examples/tests/katas and their outputs on the GCS",
+ choices=["CI", "CD"])
+parser.add_argument(
+ "--sdk",
+ dest="sdk",
+ required=True,
+ help="Supported SDKs",
+ choices=config.Config.SUPPORTED_SDK)
+
+root_dir = os.getenv("BEAM_ROOT_DIR")
+categories_file = os.getenv("BEAM_EXAMPLE_CATEGORIES")
-def ci_step():
+
+def ci_step(examples: List[Example]):
"""
CI step to verify all beam examples/tests/katas
"""
- setup_logger()
- root_dir = os.getenv("BEAM_ROOT_DIR")
- categories_file = os.getenv("BEAM_EXAMPLE_CATEGORIES")
- supported_categories = get_supported_categories(categories_file)
+
ci_helper = CIHelper()
- examples = find_examples(root_dir, supported_categories)
asyncio.run(ci_helper.verify_examples(examples))
-def cd_step():
+def cd_step(examples: List[Example]):
"""
CD step to save all beam examples/tests/katas and their outputs on the GCS
"""
- setup_logger()
- root_dir = os.getenv("BEAM_ROOT_DIR")
- categories_file = os.getenv("BEAM_EXAMPLE_CATEGORIES")
- supported_categories = get_supported_categories(categories_file)
cd_helper = CDHelper()
- examples = find_examples(root_dir, supported_categories)
cd_helper.store_examples(examples)
+
+
+def _check_envs():
+ if root_dir is None:
+ raise KeyError(
+ "BEAM_ROOT_DIR environment variable should be specified in os")
+ if categories_file is None:
+ raise KeyError(
+ "BEAM_EXAMPLE_CATEGORIES environment variable should be specified in os"
+ )
+
+
+def run_ci_cd(step: Literal["CI", "CD"], sdk: Sdk):
+ supported_categories = get_supported_categories(categories_file)
+ examples = find_examples(root_dir, supported_categories, sdk)
+ if step == "CI":
+ ci_step(examples=examples)
+ if step == "CD":
+ cd_step(examples=examples)
Review comment:
that's not our business, I've passed choices to argparser
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #16140: [BEAM-13377][Playground] Update CI/CD python scripts to work with specific SDK
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #16140:
URL: https://github.com/apache/beam/pull/16140
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ilya-kozyrev commented on a change in pull request #16140: [BEAM-13377][Playground] Update CI/CD python scripts to work with specific SDK
Posted by GitBox <gi...@apache.org>.
ilya-kozyrev commented on a change in pull request #16140:
URL: https://github.com/apache/beam/pull/16140#discussion_r763753257
##########
File path: playground/infrastructure/ci_cd.py
##########
@@ -16,37 +16,76 @@
"""
Module implements CI/CD steps for Beam Playground examples
"""
-
+import argparse
import asyncio
import os
+from typing import List, Literal
+import config
+from api.v1.api_pb2 import Sdk
from cd_helper import CDHelper
from ci_helper import CIHelper
-from helper import find_examples, get_supported_categories
+from helper import find_examples, get_supported_categories, Example
from logger import setup_logger
+parser = argparse.ArgumentParser(
+ description="CI/CD Steps for Playground objects")
+parser.add_argument(
+ "--step",
+ dest="step",
+ required=True,
+ help="CI step to verify all beam examples/tests/katas. CD step to save all "
+ "beam examples/tests/katas and their outputs on the GCS",
+ choices=["CI", "CD"])
+parser.add_argument(
+ "--sdk",
+ dest="sdk",
+ required=True,
+ help="Supported SDKs",
+ choices=config.Config.SUPPORTED_SDK)
+
+root_dir = os.getenv("BEAM_ROOT_DIR")
+categories_file = os.getenv("BEAM_EXAMPLE_CATEGORIES")
-def ci_step():
+
+def ci_step(examples: List[Example]):
"""
CI step to verify all beam examples/tests/katas
"""
- setup_logger()
- root_dir = os.getenv("BEAM_ROOT_DIR")
- categories_file = os.getenv("BEAM_EXAMPLE_CATEGORIES")
- supported_categories = get_supported_categories(categories_file)
+
ci_helper = CIHelper()
- examples = find_examples(root_dir, supported_categories)
asyncio.run(ci_helper.verify_examples(examples))
-def cd_step():
+def cd_step(examples: List[Example]):
"""
CD step to save all beam examples/tests/katas and their outputs on the GCS
"""
- setup_logger()
- root_dir = os.getenv("BEAM_ROOT_DIR")
- categories_file = os.getenv("BEAM_EXAMPLE_CATEGORIES")
- supported_categories = get_supported_categories(categories_file)
cd_helper = CDHelper()
- examples = find_examples(root_dir, supported_categories)
cd_helper.store_examples(examples)
+
+
+def _check_envs():
+ if root_dir is None:
+ raise KeyError(
+ "BEAM_ROOT_DIR environment variable should be specified in os")
+ if categories_file is None:
+ raise KeyError(
+ "BEAM_EXAMPLE_CATEGORIES environment variable should be specified in os"
+ )
+
+
+def run_ci_cd(step: Literal["CI", "CD"], sdk: Sdk):
+ supported_categories = get_supported_categories(categories_file)
+ examples = find_examples(root_dir, supported_categories, sdk)
+ if step == "CI":
+ ci_step(examples=examples)
+ if step == "CD":
+ cd_step(examples=examples)
Review comment:
I'll add _ prefix to mark it 'protected'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pavel-avilov commented on pull request #16140: [BEAM-13377][Playground] Update CI/CD python scripts to work with specific SDK
Posted by GitBox <gi...@apache.org>.
pavel-avilov commented on pull request #16140:
URL: https://github.com/apache/beam/pull/16140#issuecomment-986846391
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #16140: [BEAM-13377][Playground] Update CI/CD python scripts to work with specific SDK
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #16140:
URL: https://github.com/apache/beam/pull/16140#issuecomment-992516654
LGTM merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KhaninArtur commented on a change in pull request #16140: [BEAM-13377][Playground] Update CI/CD python scripts to work with specific SDK
Posted by GitBox <gi...@apache.org>.
KhaninArtur commented on a change in pull request #16140:
URL: https://github.com/apache/beam/pull/16140#discussion_r763076662
##########
File path: playground/infrastructure/ci_cd.py
##########
@@ -16,37 +16,76 @@
"""
Module implements CI/CD steps for Beam Playground examples
"""
-
+import argparse
import asyncio
import os
+from typing import List, Literal
+import config
+from api.v1.api_pb2 import Sdk
from cd_helper import CDHelper
from ci_helper import CIHelper
-from helper import find_examples, get_supported_categories
+from helper import find_examples, get_supported_categories, Example
from logger import setup_logger
+parser = argparse.ArgumentParser(
+ description="CI/CD Steps for Playground objects")
+parser.add_argument(
+ "--step",
+ dest="step",
+ required=True,
+ help="CI step to verify all beam examples/tests/katas. CD step to save all "
+ "beam examples/tests/katas and their outputs on the GCS",
+ choices=["CI", "CD"])
+parser.add_argument(
+ "--sdk",
+ dest="sdk",
+ required=True,
+ help="Supported SDKs",
+ choices=config.Config.SUPPORTED_SDK)
+
+root_dir = os.getenv("BEAM_ROOT_DIR")
+categories_file = os.getenv("BEAM_EXAMPLE_CATEGORIES")
-def ci_step():
+
+def ci_step(examples: List[Example]):
"""
CI step to verify all beam examples/tests/katas
"""
- setup_logger()
- root_dir = os.getenv("BEAM_ROOT_DIR")
- categories_file = os.getenv("BEAM_EXAMPLE_CATEGORIES")
- supported_categories = get_supported_categories(categories_file)
+
ci_helper = CIHelper()
- examples = find_examples(root_dir, supported_categories)
asyncio.run(ci_helper.verify_examples(examples))
-def cd_step():
+def cd_step(examples: List[Example]):
"""
CD step to save all beam examples/tests/katas and their outputs on the GCS
"""
- setup_logger()
- root_dir = os.getenv("BEAM_ROOT_DIR")
- categories_file = os.getenv("BEAM_EXAMPLE_CATEGORIES")
- supported_categories = get_supported_categories(categories_file)
cd_helper = CDHelper()
- examples = find_examples(root_dir, supported_categories)
cd_helper.store_examples(examples)
+
+
+def _check_envs():
+ if root_dir is None:
+ raise KeyError(
+ "BEAM_ROOT_DIR environment variable should be specified in os")
+ if categories_file is None:
+ raise KeyError(
+ "BEAM_EXAMPLE_CATEGORIES environment variable should be specified in os"
+ )
+
+
+def run_ci_cd(step: Literal["CI", "CD"], sdk: Sdk):
+ supported_categories = get_supported_categories(categories_file)
+ examples = find_examples(root_dir, supported_categories, sdk)
+ if step == "CI":
+ ci_step(examples=examples)
+ if step == "CD":
+ cd_step(examples=examples)
Review comment:
Shouldn't we print or log something in case if the step value is incorrect?
##########
File path: playground/infrastructure/helper.py
##########
@@ -199,7 +212,9 @@ def get_supported_categories(categories_path: str) -> List[str]:
return yaml_object[TagFields.categories]
-def _get_example(filepath: str, filename: str, tag: dict) -> Example:
+def _get_example(
+ filepath: str, filename: str, tag: Dict[str, Union[str,
Review comment:
This formatting is a bit ugly, could you please move `tag` to the next line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ilya-kozyrev commented on pull request #16140: [BEAM-13377][Playground] Update CI/CD python scripts to work with specific SDK
Posted by GitBox <gi...@apache.org>.
ilya-kozyrev commented on pull request #16140:
URL: https://github.com/apache/beam/pull/16140#issuecomment-986854941
R: @pabloem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org