You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/12/03 16:37:14 UTC
[GitHub] [beam] pavel-avilov opened a new pull request #16127: [Beam-13366] [Playground] Add support Pipeline Options for Backend
pavel-avilov opened a new pull request #16127:
URL: https://github.com/apache/beam/pull/16127
[[BEAM-13366]](https://issues.apache.org/jira/browse/BEAM-13366)
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] daria-malkova commented on a change in pull request #16127: [BEAM-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
daria-malkova commented on a change in pull request #16127:
URL: https://github.com/apache/beam/pull/16127#discussion_r764081703
##########
File path: playground/backend/internal/executors/executor.go
##########
@@ -102,7 +104,9 @@ func (ex *Executor) Compile(ctx context.Context) *exec.Cmd {
// Returns Cmd instance
func (ex *Executor) Run(ctx context.Context) *exec.Cmd {
args := append(ex.runArgs.commandArgs, ex.runArgs.fileName)
- cmd := exec.CommandContext(ctx, ex.runArgs.commandName, args...)
+ argsWithOptions := append(args, ex.runArgs.pipelineOptions)
+ argsWithOptions = utils.RemoveEmptyValue(argsWithOptions)
Review comment:
Don't need to remove, please ensure that you don't add empty strings before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pavel-avilov commented on a change in pull request #16127: [BEAM-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
pavel-avilov commented on a change in pull request #16127:
URL: https://github.com/apache/beam/pull/16127#discussion_r762975703
##########
File path: playground/backend/internal/utils/preparators_utils.go
##########
@@ -36,3 +37,16 @@ func GetPreparators(sdk pb.Sdk, filepath string) (*[]preparators.Preparator, err
}
return prep, nil
}
+
+//SpacesToEqualsOption prepares pipelineOptions by replacing spaces to equals
+func SpacesToEqualsOption(pipelineOptions string) string {
+ preparedOptions := strings.Split(pipelineOptions, " ")
+ for index, str := range preparedOptions {
+ if index%2 == 0 && index != 0 {
+ preparedOptions[index] = fmt.Sprintf(" %s", str)
+ } else if index%2 != 0 {
+ preparedOptions[index] = fmt.Sprintf("=%s", str)
+ }
+ }
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ilya-kozyrev commented on a change in pull request #16127: [BEAM-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
ilya-kozyrev commented on a change in pull request #16127:
URL: https://github.com/apache/beam/pull/16127#discussion_r764093061
##########
File path: playground/backend/internal/executors/executor.go
##########
@@ -102,7 +104,9 @@ func (ex *Executor) Compile(ctx context.Context) *exec.Cmd {
// Returns Cmd instance
func (ex *Executor) Run(ctx context.Context) *exec.Cmd {
args := append(ex.runArgs.commandArgs, ex.runArgs.fileName)
- cmd := exec.CommandContext(ctx, ex.runArgs.commandName, args...)
+ argsWithOptions := append(args, ex.runArgs.pipelineOptions)
+ argsWithOptions = utils.RemoveEmptyValue(argsWithOptions)
Review comment:
it might come from the user side, i.e. --option1 option (2 whitespaces) and that valid command line way, however when we split to go cmd we should remove extra spaces :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ilya-kozyrev commented on a change in pull request #16127: [BEAM-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
ilya-kozyrev commented on a change in pull request #16127:
URL: https://github.com/apache/beam/pull/16127#discussion_r764093061
##########
File path: playground/backend/internal/executors/executor.go
##########
@@ -102,7 +104,9 @@ func (ex *Executor) Compile(ctx context.Context) *exec.Cmd {
// Returns Cmd instance
func (ex *Executor) Run(ctx context.Context) *exec.Cmd {
args := append(ex.runArgs.commandArgs, ex.runArgs.fileName)
- cmd := exec.CommandContext(ctx, ex.runArgs.commandName, args...)
+ argsWithOptions := append(args, ex.runArgs.pipelineOptions)
+ argsWithOptions = utils.RemoveEmptyValue(argsWithOptions)
Review comment:
it might come from the user side, i.e. --option1 \s\s option (multiple whitespace) and that valid command line way, however when we split to go cmd we should remove extra spaces :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #16127: [BEAM-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #16127:
URL: https://github.com/apache/beam/pull/16127#discussion_r764197513
##########
File path: playground/api/v1/api.proto
##########
@@ -56,6 +56,7 @@ enum PrecompiledObjectType {
message RunCodeRequest {
string code = 1;
Sdk sdk = 2;
+ string pipeline_options = 3;
Review comment:
```suggestion
// The pipeline options as they would be passed to the program (e.g. "--option1=value1 --option2=value2")
string pipeline_options = 3;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ilya-kozyrev commented on a change in pull request #16127: [BEAM-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
ilya-kozyrev commented on a change in pull request #16127:
URL: https://github.com/apache/beam/pull/16127#discussion_r764093061
##########
File path: playground/backend/internal/executors/executor.go
##########
@@ -102,7 +104,9 @@ func (ex *Executor) Compile(ctx context.Context) *exec.Cmd {
// Returns Cmd instance
func (ex *Executor) Run(ctx context.Context) *exec.Cmd {
args := append(ex.runArgs.commandArgs, ex.runArgs.fileName)
- cmd := exec.CommandContext(ctx, ex.runArgs.commandName, args...)
+ argsWithOptions := append(args, ex.runArgs.pipelineOptions)
+ argsWithOptions = utils.RemoveEmptyValue(argsWithOptions)
Review comment:
it might come from the user side, i.e. --option1 \s\s option (2 whitespaces) and that valid command line way, however when we split to go cmd we should remove extra spaces :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pavel-avilov commented on a change in pull request #16127: [BEAM-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
pavel-avilov commented on a change in pull request #16127:
URL: https://github.com/apache/beam/pull/16127#discussion_r764020306
##########
File path: playground/backend/internal/utils/preparators_utils.go
##########
@@ -36,3 +37,18 @@ func GetPreparators(sdk pb.Sdk, filepath string) (*[]preparators.Preparator, err
}
return prep, nil
}
+
+// SpacesToEqualsOption prepares pipelineOptions by replacing spaces to equals.
+// Initially, the string is converted to an array of options and their values.
+// Then, a space is added to the odd indexed words, except for the first, and an equal is added to each even word.
+func SpacesToEqualsOption(pipelineOptions string) string {
+ preparedOptions := strings.Split(pipelineOptions, " ")
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ilya-kozyrev commented on a change in pull request #16127: [BEAM-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
ilya-kozyrev commented on a change in pull request #16127:
URL: https://github.com/apache/beam/pull/16127#discussion_r764184570
##########
File path: playground/backend/internal/executors/executor.go
##########
@@ -102,7 +104,9 @@ func (ex *Executor) Compile(ctx context.Context) *exec.Cmd {
// Returns Cmd instance
func (ex *Executor) Run(ctx context.Context) *exec.Cmd {
args := append(ex.runArgs.commandArgs, ex.runArgs.fileName)
- cmd := exec.CommandContext(ctx, ex.runArgs.commandName, args...)
+ argsWithOptions := append(args, ex.runArgs.pipelineOptions)
+ argsWithOptions = utils.RemoveEmptyValue(argsWithOptions)
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ilya-kozyrev commented on pull request #16127: [BEAM-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
ilya-kozyrev commented on pull request #16127:
URL: https://github.com/apache/beam/pull/16127#issuecomment-988101159
R: @pabloem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KhaninArtur commented on a change in pull request #16127: [BEAM-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
KhaninArtur commented on a change in pull request #16127:
URL: https://github.com/apache/beam/pull/16127#discussion_r762854599
##########
File path: playground/backend/internal/utils/preparators_utils.go
##########
@@ -36,3 +37,16 @@ func GetPreparators(sdk pb.Sdk, filepath string) (*[]preparators.Preparator, err
}
return prep, nil
}
+
+//SpacesToEqualsOption prepares pipelineOptions by replacing spaces to equals
Review comment:
```suggestion
// SpacesToEqualsOption prepares pipelineOptions by replacing spaces to equals
```
##########
File path: playground/backend/internal/utils/preparators_utils.go
##########
@@ -36,3 +37,16 @@ func GetPreparators(sdk pb.Sdk, filepath string) (*[]preparators.Preparator, err
}
return prep, nil
}
+
+//SpacesToEqualsOption prepares pipelineOptions by replacing spaces to equals
+func SpacesToEqualsOption(pipelineOptions string) string {
+ preparedOptions := strings.Split(pipelineOptions, " ")
+ for index, str := range preparedOptions {
+ if index%2 == 0 && index != 0 {
+ preparedOptions[index] = fmt.Sprintf(" %s", str)
+ } else if index%2 != 0 {
+ preparedOptions[index] = fmt.Sprintf("=%s", str)
+ }
+ }
Review comment:
This logic is a bit implicit, let's add a comment that explains what is happening here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pavel-avilov commented on a change in pull request #16127: [BEAM-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
pavel-avilov commented on a change in pull request #16127:
URL: https://github.com/apache/beam/pull/16127#discussion_r762975179
##########
File path: playground/backend/internal/utils/preparators_utils.go
##########
@@ -36,3 +37,16 @@ func GetPreparators(sdk pb.Sdk, filepath string) (*[]preparators.Preparator, err
}
return prep, nil
}
+
+//SpacesToEqualsOption prepares pipelineOptions by replacing spaces to equals
Review comment:
Done
##########
File path: playground/backend/internal/utils/preparators_utils.go
##########
@@ -36,3 +37,16 @@ func GetPreparators(sdk pb.Sdk, filepath string) (*[]preparators.Preparator, err
}
return prep, nil
}
+
+//SpacesToEqualsOption prepares pipelineOptions by replacing spaces to equals
+func SpacesToEqualsOption(pipelineOptions string) string {
+ preparedOptions := strings.Split(pipelineOptions, " ")
+ for index, str := range preparedOptions {
+ if index%2 == 0 && index != 0 {
+ preparedOptions[index] = fmt.Sprintf(" %s", str)
+ } else if index%2 != 0 {
+ preparedOptions[index] = fmt.Sprintf("=%s", str)
+ }
+ }
Review comment:
Done.
##########
File path: playground/backend/internal/utils/preparators_utils.go
##########
@@ -36,3 +37,16 @@ func GetPreparators(sdk pb.Sdk, filepath string) (*[]preparators.Preparator, err
}
return prep, nil
}
+
+//SpacesToEqualsOption prepares pipelineOptions by replacing spaces to equals
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ilya-kozyrev commented on a change in pull request #16127: [BEAM-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
ilya-kozyrev commented on a change in pull request #16127:
URL: https://github.com/apache/beam/pull/16127#discussion_r763963644
##########
File path: playground/backend/internal/utils/preparators_utils.go
##########
@@ -36,3 +37,18 @@ func GetPreparators(sdk pb.Sdk, filepath string) (*[]preparators.Preparator, err
}
return prep, nil
}
+
+// SpacesToEqualsOption prepares pipelineOptions by replacing spaces to equals.
+// Initially, the string is converted to an array of options and their values.
+// Then, a space is added to the odd indexed words, except for the first, and an equal is added to each even word.
+func SpacesToEqualsOption(pipelineOptions string) string {
+ preparedOptions := strings.Split(pipelineOptions, " ")
Review comment:
btw, why we can't use strings.Replace()?
##########
File path: playground/backend/internal/utils/preparators_utils.go
##########
@@ -36,3 +37,16 @@ func GetPreparators(sdk pb.Sdk, filepath string) (*[]preparators.Preparator, err
}
return prep, nil
}
+
+//SpacesToEqualsOption prepares pipelineOptions by replacing spaces to equals
+func SpacesToEqualsOption(pipelineOptions string) string {
+ preparedOptions := strings.Split(pipelineOptions, " ")
+ for index, str := range preparedOptions {
+ if index%2 == 0 && index != 0 {
+ preparedOptions[index] = fmt.Sprintf(" %s", str)
+ } else if index%2 != 0 {
+ preparedOptions[index] = fmt.Sprintf("=%s", str)
+ }
+ }
Review comment:
Could we rename this function to `ReplaceSpacesWithEquals` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] daria-malkova commented on a change in pull request #16127: [BEAM-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
daria-malkova commented on a change in pull request #16127:
URL: https://github.com/apache/beam/pull/16127#discussion_r764099935
##########
File path: playground/backend/internal/executors/executor.go
##########
@@ -102,7 +104,9 @@ func (ex *Executor) Compile(ctx context.Context) *exec.Cmd {
// Returns Cmd instance
func (ex *Executor) Run(ctx context.Context) *exec.Cmd {
args := append(ex.runArgs.commandArgs, ex.runArgs.fileName)
- cmd := exec.CommandContext(ctx, ex.runArgs.commandName, args...)
+ argsWithOptions := append(args, ex.runArgs.pipelineOptions)
+ argsWithOptions = utils.RemoveEmptyValue(argsWithOptions)
Review comment:
Please replace with regex and move this part to the executor builder setup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pavel-avilov commented on pull request #16127: [Beam-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
pavel-avilov commented on pull request #16127:
URL: https://github.com/apache/beam/pull/16127#issuecomment-985712082
R: @ilya-kozyrev @AydarZaynutdinov @daria-malkova @KhaninArtur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #16127: [BEAM-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #16127:
URL: https://github.com/apache/beam/pull/16127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ilya-kozyrev commented on a change in pull request #16127: [BEAM-13366] [Playground] Add support Pipeline Options for Backend
Posted by GitBox <gi...@apache.org>.
ilya-kozyrev commented on a change in pull request #16127:
URL: https://github.com/apache/beam/pull/16127#discussion_r764093061
##########
File path: playground/backend/internal/executors/executor.go
##########
@@ -102,7 +104,9 @@ func (ex *Executor) Compile(ctx context.Context) *exec.Cmd {
// Returns Cmd instance
func (ex *Executor) Run(ctx context.Context) *exec.Cmd {
args := append(ex.runArgs.commandArgs, ex.runArgs.fileName)
- cmd := exec.CommandContext(ctx, ex.runArgs.commandName, args...)
+ argsWithOptions := append(args, ex.runArgs.pipelineOptions)
+ argsWithOptions = utils.RemoveEmptyValue(argsWithOptions)
Review comment:
it might come from the user side, i.e. --option1 option (2 whitespaces) and that valid command line way, however when we split to go cmd we should remove extra spaces :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org