You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/10/26 16:00:30 UTC
[GitHub] [beam] AydarZaynutdinov opened a new pull request #15803: [BEAM-12970][Playground] Implement gRPC API - RunCode
AydarZaynutdinov opened a new pull request #15803:
URL: https://github.com/apache/beam/pull/15803
add `setupCache()` method;
implement `RunCode` API method;
change `ExecutableName` to a variable of `LifeCycle` instead of method;
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AydarZaynutdinov commented on a change in pull request #15803: [BEAM-12970][Playground] Implement gRPC API - RunCode
Posted by GitBox <gi...@apache.org>.
AydarZaynutdinov commented on a change in pull request #15803:
URL: https://github.com/apache/beam/pull/15803#discussion_r737213734
##########
File path: playground/backend/cmd/server/controller.go
##########
@@ -15,21 +15,88 @@
package main
import (
- "context"
-
pb "beam.apache.org/playground/backend/internal/api/v1"
+ "beam.apache.org/playground/backend/internal/cache"
+ "beam.apache.org/playground/backend/internal/environment"
+ "beam.apache.org/playground/backend/internal/errors"
+ "beam.apache.org/playground/backend/internal/executors"
+ "beam.apache.org/playground/backend/internal/fs_tool"
+ "beam.apache.org/playground/backend/internal/validators"
+ "context"
"github.com/google/uuid"
+ "log"
)
type playgroundController struct {
+ env *environment.Environment
Review comment:
Added documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #15803: [BEAM-12970][Playground] Implement gRPC API - RunCode
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #15803:
URL: https://github.com/apache/beam/pull/15803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AydarZaynutdinov commented on a change in pull request #15803: [BEAM-12970][Playground] Implement gRPC API - RunCode
Posted by GitBox <gi...@apache.org>.
AydarZaynutdinov commented on a change in pull request #15803:
URL: https://github.com/apache/beam/pull/15803#discussion_r737213343
##########
File path: playground/backend/cmd/server/controller.go
##########
@@ -15,21 +15,88 @@
package main
import (
- "context"
-
pb "beam.apache.org/playground/backend/internal/api/v1"
+ "beam.apache.org/playground/backend/internal/cache"
+ "beam.apache.org/playground/backend/internal/environment"
+ "beam.apache.org/playground/backend/internal/errors"
+ "beam.apache.org/playground/backend/internal/executors"
+ "beam.apache.org/playground/backend/internal/fs_tool"
+ "beam.apache.org/playground/backend/internal/validators"
+ "context"
"github.com/google/uuid"
+ "log"
Review comment:
Added using of our logger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15803: [BEAM-12970][Playground] Implement gRPC API - RunCode
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15803:
URL: https://github.com/apache/beam/pull/15803#discussion_r737760119
##########
File path: playground/backend/cmd/server/controller_test.go
##########
@@ -49,28 +101,86 @@ func setup() *grpc.Server {
func teardown(server *grpc.Server) {
server.Stop()
+
+ err := os.RemoveAll("configs")
+ if err != nil {
+ fmt.Errorf("error during test setup: %s", err.Error())
+ }
}
func bufDialer(context.Context, string) (net.Conn, error) {
return lis.Dial()
}
+
func TestPlaygroundController_RunCode(t *testing.T) {
- ctx := context.Background()
- conn, err := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithInsecure())
- if err != nil {
- t.Fatalf("Failed to dial bufnet: %v", err)
+ type args struct {
+ ctx context.Context
+ request *pb.RunCodeRequest
}
- defer conn.Close()
- client := pb.NewPlaygroundServiceClient(conn)
- code := pb.RunCodeRequest{
- Code: "test",
- Sdk: pb.Sdk_SDK_JAVA,
+ tests := []struct {
+ name string
+ args args
+ wantStatus pb.Status
+ wantErr bool
+ }{
+ {
+ name: "RunCode with incorrect sdk",
+ args: args{
+ ctx: context.Background(),
+ request: &pb.RunCodeRequest{
+ Code: "MOCK_CODE",
+ Sdk: pb.Sdk_SDK_UNSPECIFIED,
+ },
+ },
+ wantErr: true,
+ },
+ {
+ name: "RunCode with correct sdk",
+ args: args{
+ ctx: context.Background(),
+ request: &pb.RunCodeRequest{
+ Code: "MOCK_CODE",
Review comment:
I'm just a little confused - does this actually run some Java code? I think it will run `java` and `javac`, right?
##########
File path: playground/backend/cmd/server/controller_test.go
##########
@@ -49,28 +101,86 @@ func setup() *grpc.Server {
func teardown(server *grpc.Server) {
server.Stop()
+
+ err := os.RemoveAll("configs")
+ if err != nil {
+ fmt.Errorf("error during test setup: %s", err.Error())
+ }
}
func bufDialer(context.Context, string) (net.Conn, error) {
return lis.Dial()
}
+
func TestPlaygroundController_RunCode(t *testing.T) {
- ctx := context.Background()
- conn, err := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithInsecure())
- if err != nil {
- t.Fatalf("Failed to dial bufnet: %v", err)
+ type args struct {
+ ctx context.Context
+ request *pb.RunCodeRequest
}
- defer conn.Close()
- client := pb.NewPlaygroundServiceClient(conn)
- code := pb.RunCodeRequest{
- Code: "test",
- Sdk: pb.Sdk_SDK_JAVA,
+ tests := []struct {
+ name string
+ args args
+ wantStatus pb.Status
+ wantErr bool
+ }{
+ {
+ name: "RunCode with incorrect sdk",
+ args: args{
+ ctx: context.Background(),
+ request: &pb.RunCodeRequest{
+ Code: "MOCK_CODE",
+ Sdk: pb.Sdk_SDK_UNSPECIFIED,
+ },
+ },
+ wantErr: true,
+ },
+ {
+ name: "RunCode with correct sdk",
+ args: args{
+ ctx: context.Background(),
+ request: &pb.RunCodeRequest{
+ Code: "MOCK_CODE",
Review comment:
oh I guess this test does not actually run it - can you add a comment about what this test checks so that it's easy to read for others? thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AydarZaynutdinov commented on a change in pull request #15803: [BEAM-12970][Playground] Implement gRPC API - RunCode
Posted by GitBox <gi...@apache.org>.
AydarZaynutdinov commented on a change in pull request #15803:
URL: https://github.com/apache/beam/pull/15803#discussion_r737224526
##########
File path: playground/backend/cmd/server/controller.go
##########
@@ -54,3 +121,125 @@ func (controller *playgroundController) GetCompileOutput(ctx context.Context, in
compileOutput := pb.GetCompileOutputResponse{Output: "test compile output"}
return &compileOutput, nil
}
+
+// setupValidators returns validators based on sdk
+func setupValidators(sdk pb.Sdk, filepath string) *[]validators.Validator {
+ var val *[]validators.Validator
+ switch sdk {
+ case pb.Sdk_SDK_JAVA:
+ val = validators.GetJavaValidators(filepath)
+ }
+ return val
+}
+
+// processCode validates, compiles and runs code by pipelineId.
+// During each operation updates status of execution and saves it into cache.
+// In case of some step is failed saves output logs to cache.
+// After success code running saves output to cache.
+// At the end of this method deletes all created folders
+func processCode(ctx context.Context, cacheService cache.Cache, lc *fs_tool.LifeCycle, execBuilder *executors.CompileBuilder, pipelineId uuid.UUID, env *environment.Environment) {
+ defer cleanUp(pipelineId, lc)
+
+ exec := execBuilder.Build()
+
+ // validate
+ log.Printf("%s: Validate() ...\n", pipelineId)
+
+ validateFunc := exec.Validate()
+ if err := validateFunc(); err != nil {
+ // error during validation
+ // TODO move to processError when status for validation error will be added
+ log.Printf("%s: Validate: %s\n", pipelineId, err.Error())
+ setToCache(ctx, cacheService, pipelineId, cache.Status, pb.Status_STATUS_ERROR)
+ return
+ }
+ log.Printf("%s: Validate() finish\n", pipelineId)
+
+ // compile
+ log.Printf("%s: Compile() ...\n", pipelineId)
+ compileCmd := exec.Compile()
+ if data, err := compileCmd.CombinedOutput(); err != nil {
+ processError(ctx, err, data, pipelineId, cacheService, pb.Status_STATUS_COMPILE_ERROR)
+ return
+ }
+ log.Printf("%s: Compile() finish\n", pipelineId)
+
+ // set empty value to pipelineId: cache.SubKey_CompileOutput
+ setToCache(ctx, cacheService, pipelineId, cache.CompileOutput, "")
+
+ className, err := lc.ExecutableName(pipelineId, env.ApplicationEnvs.WorkingDir())
+ if err != nil {
+ log.Printf("%s: get executable file name: %s\n", pipelineId, err.Error())
+ setToCache(ctx, cacheService, pipelineId, cache.Status, pb.Status_STATUS_ERROR)
+ return
+ }
+
+ exec = execBuilder.
+ WithRunner().
+ WithCommand(env.BeamSdkEnvs.ExecutorConfig.RunCmd).
+ WithArgs(env.BeamSdkEnvs.ExecutorConfig.RunArgs).
+ WithClassName(className).
Review comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15803: [BEAM-12970][Playground] Implement gRPC API - RunCode
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15803:
URL: https://github.com/apache/beam/pull/15803#discussion_r737769010
##########
File path: playground/backend/internal/fs_tool/java_fs.go
##########
@@ -45,6 +48,21 @@ func newJavaLifeCycle(pipelineId uuid.UUID, workingDir string) *LifeCycle {
ExecutableExtension: javaExecutableFileExtension,
CompiledExtension: javaCompiledFileExtension,
},
- pipelineId: pipelineId,
+ ExecutableName: executableName,
+ pipelineId: pipelineId,
}
}
+
+// executableName returns name that should be executed (HelloWorld for HelloWorld.class for java SDK)
+func executableName(pipelineId uuid.UUID, workingDir string) (string, error) {
+ baseFileFolder := filepath.Join(workingDir, javaBaseFileFolder, pipelineId.String())
+ binFileFolder := filepath.Join(baseFileFolder, javaCompiledFolderName)
+ dirEntries, err := os.ReadDir(binFileFolder)
+ if err != nil {
+ return "", err
+ }
+ if len(dirEntries) < 1 {
+ return "", errors.New("number of executable files should be more than one")
+ }
+ return strings.Split(dirEntries[len(dirEntries)-1].Name(), ".")[0], nil
Review comment:
we need to find the class with `main` method, right? How do we do that? Will it always be the last one?
##########
File path: playground/backend/internal/fs_tool/java_fs.go
##########
@@ -45,6 +48,21 @@ func newJavaLifeCycle(pipelineId uuid.UUID, workingDir string) *LifeCycle {
ExecutableExtension: javaExecutableFileExtension,
CompiledExtension: javaCompiledFileExtension,
},
- pipelineId: pipelineId,
+ ExecutableName: executableName,
+ pipelineId: pipelineId,
}
}
+
+// executableName returns name that should be executed (HelloWorld for HelloWorld.class for java SDK)
+func executableName(pipelineId uuid.UUID, workingDir string) (string, error) {
+ baseFileFolder := filepath.Join(workingDir, javaBaseFileFolder, pipelineId.String())
+ binFileFolder := filepath.Join(baseFileFolder, javaCompiledFolderName)
+ dirEntries, err := os.ReadDir(binFileFolder)
+ if err != nil {
+ return "", err
+ }
+ if len(dirEntries) < 1 {
+ return "", errors.New("number of executable files should be more than one")
Review comment:
```suggestion
return "", errors.New("number of executable files should be at least one")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AydarZaynutdinov commented on pull request #15803: [BEAM-12970][Playground] Implement gRPC API - RunCode
Posted by GitBox <gi...@apache.org>.
AydarZaynutdinov commented on pull request #15803:
URL: https://github.com/apache/beam/pull/15803#issuecomment-952084964
R: @pabloem
R: @damondouglas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AydarZaynutdinov commented on a change in pull request #15803: [BEAM-12970][Playground] Implement gRPC API - RunCode
Posted by GitBox <gi...@apache.org>.
AydarZaynutdinov commented on a change in pull request #15803:
URL: https://github.com/apache/beam/pull/15803#discussion_r737378553
##########
File path: playground/backend/cmd/server/controller.go
##########
@@ -54,3 +121,125 @@ func (controller *playgroundController) GetCompileOutput(ctx context.Context, in
compileOutput := pb.GetCompileOutputResponse{Output: "test compile output"}
return &compileOutput, nil
}
+
+// setupValidators returns validators based on sdk
+func setupValidators(sdk pb.Sdk, filepath string) *[]validators.Validator {
+ var val *[]validators.Validator
+ switch sdk {
+ case pb.Sdk_SDK_JAVA:
+ val = validators.GetJavaValidators(filepath)
+ }
+ return val
+}
+
+// processCode validates, compiles and runs code by pipelineId.
+// During each operation updates status of execution and saves it into cache.
+// In case of some step is failed saves output logs to cache.
+// After success code running saves output to cache.
+// At the end of this method deletes all created folders
+func processCode(ctx context.Context, cacheService cache.Cache, lc *fs_tool.LifeCycle, execBuilder *executors.CompileBuilder, pipelineId uuid.UUID, env *environment.Environment) {
+ defer cleanUp(pipelineId, lc)
+
+ exec := execBuilder.Build()
+
+ // validate
+ log.Printf("%s: Validate() ...\n", pipelineId)
+
+ validateFunc := exec.Validate()
+ if err := validateFunc(); err != nil {
+ // error during validation
+ // TODO move to processError when status for validation error will be added
+ log.Printf("%s: Validate: %s\n", pipelineId, err.Error())
+ setToCache(ctx, cacheService, pipelineId, cache.Status, pb.Status_STATUS_ERROR)
+ return
+ }
+ log.Printf("%s: Validate() finish\n", pipelineId)
+
+ // compile
+ log.Printf("%s: Compile() ...\n", pipelineId)
+ compileCmd := exec.Compile()
+ if data, err := compileCmd.CombinedOutput(); err != nil {
+ processError(ctx, err, data, pipelineId, cacheService, pb.Status_STATUS_COMPILE_ERROR)
+ return
+ }
+ log.Printf("%s: Compile() finish\n", pipelineId)
+
+ // set empty value to pipelineId: cache.SubKey_CompileOutput
+ setToCache(ctx, cacheService, pipelineId, cache.CompileOutput, "")
+
+ className, err := lc.ExecutableName(pipelineId, env.ApplicationEnvs.WorkingDir())
+ if err != nil {
+ log.Printf("%s: get executable file name: %s\n", pipelineId, err.Error())
+ setToCache(ctx, cacheService, pipelineId, cache.Status, pb.Status_STATUS_ERROR)
+ return
+ }
+
+ exec = execBuilder.
+ WithRunner().
+ WithCommand(env.BeamSdkEnvs.ExecutorConfig.RunCmd).
+ WithArgs(env.BeamSdkEnvs.ExecutorConfig.RunArgs).
+ WithClassName(className).
+ WithWorkingDir(lc.GetAbsoluteExecutableFilesFolderPath()).
+ Build()
+
+ log.Printf("%s: Run() ...\n", pipelineId)
+ runCmd := exec.Run()
+ data, err := runCmd.CombinedOutput()
+ if err != nil {
+ // error during run code
+ processError(ctx, err, data, pipelineId, cacheService, pb.Status_STATUS_ERROR)
+ return
+ }
+ log.Printf("%s: Run() finish\n", pipelineId)
+ processSuccess(ctx, data, pipelineId, cacheService)
+}
+
+// cleanUp removes all prepared folders for received LifeCycle
+func cleanUp(pipelineId uuid.UUID, lc *fs_tool.LifeCycle) {
+ log.Printf("%s: DeleteFolders() ...\n", pipelineId)
+ err := lc.DeleteFolders()
+ if err != nil {
+ log.Printf("%s: DeleteFolders(): %s\n", pipelineId, err.Error())
+ }
+ log.Printf("%s: DeleteFolders() complete\n", pipelineId)
+ log.Printf("%s: complete\n", pipelineId)
+}
+
+// processError processes error received during processing code via setting a corresponding status and output to cache
+func processError(ctx context.Context, err error, data []byte, pipelineId uuid.UUID, cacheService cache.Cache, status pb.Status) {
+ switch status {
+ case pb.Status_STATUS_ERROR:
+ log.Printf("%s: Run: err: %s, output: %s\n", pipelineId, err.Error(), data)
+
+ // set to cache pipelineId: cache.SubKey_RunOutput: err.Error()
+ setToCache(ctx, cacheService, pipelineId, cache.RunOutput, "error: "+err.Error()+", output: "+string(data))
+
+ // set to cache pipelineId: cache.SubKey_Status: pb.Status_STATUS_ERROR
+ setToCache(ctx, cacheService, pipelineId, cache.Status, pb.Status_STATUS_ERROR)
+ case pb.Status_STATUS_COMPILE_ERROR:
+ log.Printf("%s: Compile: err: %s, output: %s\n", pipelineId, err.Error(), data)
+
+ // set to cache pipelineId: cache.SubKey_CompileOutput: err.Error()
+ setToCache(ctx, cacheService, pipelineId, cache.CompileOutput, "error: "+err.Error()+", output: "+string(data))
+
+ // set to cache pipelineId: cache.SubKey_Status: pb.Status_STATUS_ERROR
+ setToCache(ctx, cacheService, pipelineId, cache.Status, pb.Status_STATUS_COMPILE_ERROR)
+ }
+}
+
+// processSuccess processes case after successful code processing via setting a corresponding status and output to cache
+func processSuccess(ctx context.Context, output []byte, pipelineId uuid.UUID, cacheService cache.Cache) {
+ // set to cache pipelineId: cache.SubKey_RunOutput: output
+ setToCache(ctx, cacheService, pipelineId, cache.RunOutput, string(output))
Review comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AydarZaynutdinov commented on a change in pull request #15803: [BEAM-12970][Playground] Implement gRPC API - RunCode
Posted by GitBox <gi...@apache.org>.
AydarZaynutdinov commented on a change in pull request #15803:
URL: https://github.com/apache/beam/pull/15803#discussion_r738032201
##########
File path: playground/backend/cmd/server/controller_test.go
##########
@@ -49,28 +101,86 @@ func setup() *grpc.Server {
func teardown(server *grpc.Server) {
server.Stop()
+
+ err := os.RemoveAll("configs")
+ if err != nil {
+ fmt.Errorf("error during test setup: %s", err.Error())
+ }
}
func bufDialer(context.Context, string) (net.Conn, error) {
return lis.Dial()
}
+
func TestPlaygroundController_RunCode(t *testing.T) {
- ctx := context.Background()
- conn, err := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithInsecure())
- if err != nil {
- t.Fatalf("Failed to dial bufnet: %v", err)
+ type args struct {
+ ctx context.Context
+ request *pb.RunCodeRequest
}
- defer conn.Close()
- client := pb.NewPlaygroundServiceClient(conn)
- code := pb.RunCodeRequest{
- Code: "test",
- Sdk: pb.Sdk_SDK_JAVA,
+ tests := []struct {
+ name string
+ args args
+ wantStatus pb.Status
+ wantErr bool
+ }{
+ {
+ name: "RunCode with incorrect sdk",
+ args: args{
+ ctx: context.Background(),
+ request: &pb.RunCodeRequest{
+ Code: "MOCK_CODE",
+ Sdk: pb.Sdk_SDK_UNSPECIFIED,
+ },
+ },
+ wantErr: true,
+ },
+ {
+ name: "RunCode with correct sdk",
+ args: args{
+ ctx: context.Background(),
+ request: &pb.RunCodeRequest{
+ Code: "MOCK_CODE",
Review comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AydarZaynutdinov commented on a change in pull request #15803: [BEAM-12970][Playground] Implement gRPC API - RunCode
Posted by GitBox <gi...@apache.org>.
AydarZaynutdinov commented on a change in pull request #15803:
URL: https://github.com/apache/beam/pull/15803#discussion_r738032426
##########
File path: playground/backend/internal/fs_tool/java_fs.go
##########
@@ -45,6 +48,21 @@ func newJavaLifeCycle(pipelineId uuid.UUID, workingDir string) *LifeCycle {
ExecutableExtension: javaExecutableFileExtension,
CompiledExtension: javaCompiledFileExtension,
},
- pipelineId: pipelineId,
+ ExecutableName: executableName,
+ pipelineId: pipelineId,
}
}
+
+// executableName returns name that should be executed (HelloWorld for HelloWorld.class for java SDK)
+func executableName(pipelineId uuid.UUID, workingDir string) (string, error) {
+ baseFileFolder := filepath.Join(workingDir, javaBaseFileFolder, pipelineId.String())
+ binFileFolder := filepath.Join(baseFileFolder, javaCompiledFolderName)
+ dirEntries, err := os.ReadDir(binFileFolder)
+ if err != nil {
+ return "", err
+ }
+ if len(dirEntries) < 1 {
+ return "", errors.New("number of executable files should be more than one")
Review comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15803: [BEAM-12970][Playground] Implement gRPC API - RunCode
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15803:
URL: https://github.com/apache/beam/pull/15803#discussion_r736994177
##########
File path: playground/backend/cmd/server/controller.go
##########
@@ -54,3 +121,125 @@ func (controller *playgroundController) GetCompileOutput(ctx context.Context, in
compileOutput := pb.GetCompileOutputResponse{Output: "test compile output"}
return &compileOutput, nil
}
+
+// setupValidators returns validators based on sdk
+func setupValidators(sdk pb.Sdk, filepath string) *[]validators.Validator {
+ var val *[]validators.Validator
+ switch sdk {
+ case pb.Sdk_SDK_JAVA:
+ val = validators.GetJavaValidators(filepath)
+ }
+ return val
+}
+
+// processCode validates, compiles and runs code by pipelineId.
+// During each operation updates status of execution and saves it into cache.
+// In case of some step is failed saves output logs to cache.
+// After success code running saves output to cache.
+// At the end of this method deletes all created folders
+func processCode(ctx context.Context, cacheService cache.Cache, lc *fs_tool.LifeCycle, execBuilder *executors.CompileBuilder, pipelineId uuid.UUID, env *environment.Environment) {
+ defer cleanUp(pipelineId, lc)
+
+ exec := execBuilder.Build()
+
+ // validate
+ log.Printf("%s: Validate() ...\n", pipelineId)
+
+ validateFunc := exec.Validate()
+ if err := validateFunc(); err != nil {
+ // error during validation
+ // TODO move to processError when status for validation error will be added
+ log.Printf("%s: Validate: %s\n", pipelineId, err.Error())
+ setToCache(ctx, cacheService, pipelineId, cache.Status, pb.Status_STATUS_ERROR)
+ return
+ }
+ log.Printf("%s: Validate() finish\n", pipelineId)
+
+ // compile
+ log.Printf("%s: Compile() ...\n", pipelineId)
+ compileCmd := exec.Compile()
+ if data, err := compileCmd.CombinedOutput(); err != nil {
+ processError(ctx, err, data, pipelineId, cacheService, pb.Status_STATUS_COMPILE_ERROR)
+ return
+ }
+ log.Printf("%s: Compile() finish\n", pipelineId)
+
+ // set empty value to pipelineId: cache.SubKey_CompileOutput
+ setToCache(ctx, cacheService, pipelineId, cache.CompileOutput, "")
+
+ className, err := lc.ExecutableName(pipelineId, env.ApplicationEnvs.WorkingDir())
+ if err != nil {
+ log.Printf("%s: get executable file name: %s\n", pipelineId, err.Error())
+ setToCache(ctx, cacheService, pipelineId, cache.Status, pb.Status_STATUS_ERROR)
+ return
+ }
+
+ exec = execBuilder.
+ WithRunner().
+ WithCommand(env.BeamSdkEnvs.ExecutorConfig.RunCmd).
+ WithArgs(env.BeamSdkEnvs.ExecutorConfig.RunArgs).
+ WithClassName(className).
Review comment:
I suppose this is for Java, right? or do we also expect a class name for go/python?
##########
File path: playground/backend/cmd/server/controller.go
##########
@@ -15,21 +15,88 @@
package main
import (
- "context"
-
pb "beam.apache.org/playground/backend/internal/api/v1"
+ "beam.apache.org/playground/backend/internal/cache"
+ "beam.apache.org/playground/backend/internal/environment"
+ "beam.apache.org/playground/backend/internal/errors"
+ "beam.apache.org/playground/backend/internal/executors"
+ "beam.apache.org/playground/backend/internal/fs_tool"
+ "beam.apache.org/playground/backend/internal/validators"
+ "context"
"github.com/google/uuid"
+ "log"
)
type playgroundController struct {
+ env *environment.Environment
+ cacheService cache.Cache
+
pb.UnimplementedPlaygroundServiceServer
}
//RunCode is running code from requests using a particular SDK
func (controller *playgroundController) RunCode(ctx context.Context, info *pb.RunCodeRequest) (*pb.RunCodeResponse, error) {
- // TODO implement this method
- pipelineInfo := pb.RunCodeResponse{PipelineUuid: uuid.NewString()}
+ // check for correct sdk
+ switch info.Sdk {
+ case pb.Sdk_SDK_UNSPECIFIED, pb.Sdk_SDK_GO, pb.Sdk_SDK_PYTHON, pb.Sdk_SDK_SCIO:
+ log.Printf("RunCode(): unimplemented sdk: %s\n", info.Sdk)
+ return nil, errors.InvalidArgumentError("Run code()", "unimplemented sdk: "+info.Sdk.String())
+ }
+
+ pipelineId := uuid.New()
+
+ defer func() {
+ log.Printf("RunCode() is completed for pipeline with id: %s\n", pipelineId)
+ }()
+
+ cacheExpirationTime := controller.env.ApplicationEnvs.CacheEnvs().KeyExpirationTime()
Review comment:
Is it a default expiration time configured for the environment?
##########
File path: playground/backend/cmd/server/controller.go
##########
@@ -15,21 +15,88 @@
package main
import (
- "context"
-
pb "beam.apache.org/playground/backend/internal/api/v1"
+ "beam.apache.org/playground/backend/internal/cache"
+ "beam.apache.org/playground/backend/internal/environment"
+ "beam.apache.org/playground/backend/internal/errors"
+ "beam.apache.org/playground/backend/internal/executors"
+ "beam.apache.org/playground/backend/internal/fs_tool"
+ "beam.apache.org/playground/backend/internal/validators"
+ "context"
"github.com/google/uuid"
+ "log"
Review comment:
should we be using our logger libnrary? https://github.com/apache/beam/blob/master/playground/backend/internal/logger/logger.go
##########
File path: playground/backend/cmd/server/controller.go
##########
@@ -15,21 +15,88 @@
package main
import (
- "context"
-
pb "beam.apache.org/playground/backend/internal/api/v1"
+ "beam.apache.org/playground/backend/internal/cache"
+ "beam.apache.org/playground/backend/internal/environment"
+ "beam.apache.org/playground/backend/internal/errors"
+ "beam.apache.org/playground/backend/internal/executors"
+ "beam.apache.org/playground/backend/internal/fs_tool"
+ "beam.apache.org/playground/backend/internal/validators"
+ "context"
"github.com/google/uuid"
+ "log"
)
type playgroundController struct {
+ env *environment.Environment
Review comment:
please document in detail what the playgroundController does so that maintainers can easily read the code in the future
##########
File path: playground/backend/cmd/server/controller.go
##########
@@ -15,21 +15,88 @@
package main
import (
- "context"
-
pb "beam.apache.org/playground/backend/internal/api/v1"
+ "beam.apache.org/playground/backend/internal/cache"
+ "beam.apache.org/playground/backend/internal/environment"
+ "beam.apache.org/playground/backend/internal/errors"
+ "beam.apache.org/playground/backend/internal/executors"
+ "beam.apache.org/playground/backend/internal/fs_tool"
+ "beam.apache.org/playground/backend/internal/validators"
+ "context"
"github.com/google/uuid"
+ "log"
)
type playgroundController struct {
+ env *environment.Environment
+ cacheService cache.Cache
+
pb.UnimplementedPlaygroundServiceServer
}
//RunCode is running code from requests using a particular SDK
func (controller *playgroundController) RunCode(ctx context.Context, info *pb.RunCodeRequest) (*pb.RunCodeResponse, error) {
- // TODO implement this method
- pipelineInfo := pb.RunCodeResponse{PipelineUuid: uuid.NewString()}
+ // check for correct sdk
+ switch info.Sdk {
+ case pb.Sdk_SDK_UNSPECIFIED, pb.Sdk_SDK_GO, pb.Sdk_SDK_PYTHON, pb.Sdk_SDK_SCIO:
+ log.Printf("RunCode(): unimplemented sdk: %s\n", info.Sdk)
+ return nil, errors.InvalidArgumentError("Run code()", "unimplemented sdk: "+info.Sdk.String())
+ }
+
+ pipelineId := uuid.New()
Review comment:
if we generate the uuid here without inspecting the pipeline, when will we ever have a cache hit for any given pipeline? what will be the job of the cache?
##########
File path: playground/backend/cmd/server/controller.go
##########
@@ -54,3 +121,125 @@ func (controller *playgroundController) GetCompileOutput(ctx context.Context, in
compileOutput := pb.GetCompileOutputResponse{Output: "test compile output"}
return &compileOutput, nil
}
+
+// setupValidators returns validators based on sdk
+func setupValidators(sdk pb.Sdk, filepath string) *[]validators.Validator {
+ var val *[]validators.Validator
+ switch sdk {
+ case pb.Sdk_SDK_JAVA:
+ val = validators.GetJavaValidators(filepath)
+ }
+ return val
+}
+
+// processCode validates, compiles and runs code by pipelineId.
+// During each operation updates status of execution and saves it into cache.
+// In case of some step is failed saves output logs to cache.
+// After success code running saves output to cache.
+// At the end of this method deletes all created folders
+func processCode(ctx context.Context, cacheService cache.Cache, lc *fs_tool.LifeCycle, execBuilder *executors.CompileBuilder, pipelineId uuid.UUID, env *environment.Environment) {
+ defer cleanUp(pipelineId, lc)
+
+ exec := execBuilder.Build()
+
+ // validate
+ log.Printf("%s: Validate() ...\n", pipelineId)
+
+ validateFunc := exec.Validate()
+ if err := validateFunc(); err != nil {
+ // error during validation
+ // TODO move to processError when status for validation error will be added
+ log.Printf("%s: Validate: %s\n", pipelineId, err.Error())
+ setToCache(ctx, cacheService, pipelineId, cache.Status, pb.Status_STATUS_ERROR)
+ return
+ }
+ log.Printf("%s: Validate() finish\n", pipelineId)
+
+ // compile
+ log.Printf("%s: Compile() ...\n", pipelineId)
+ compileCmd := exec.Compile()
+ if data, err := compileCmd.CombinedOutput(); err != nil {
+ processError(ctx, err, data, pipelineId, cacheService, pb.Status_STATUS_COMPILE_ERROR)
+ return
+ }
+ log.Printf("%s: Compile() finish\n", pipelineId)
+
+ // set empty value to pipelineId: cache.SubKey_CompileOutput
+ setToCache(ctx, cacheService, pipelineId, cache.CompileOutput, "")
+
+ className, err := lc.ExecutableName(pipelineId, env.ApplicationEnvs.WorkingDir())
+ if err != nil {
+ log.Printf("%s: get executable file name: %s\n", pipelineId, err.Error())
+ setToCache(ctx, cacheService, pipelineId, cache.Status, pb.Status_STATUS_ERROR)
+ return
+ }
+
+ exec = execBuilder.
+ WithRunner().
+ WithCommand(env.BeamSdkEnvs.ExecutorConfig.RunCmd).
+ WithArgs(env.BeamSdkEnvs.ExecutorConfig.RunArgs).
+ WithClassName(className).
+ WithWorkingDir(lc.GetAbsoluteExecutableFilesFolderPath()).
+ Build()
+
+ log.Printf("%s: Run() ...\n", pipelineId)
+ runCmd := exec.Run()
+ data, err := runCmd.CombinedOutput()
+ if err != nil {
+ // error during run code
+ processError(ctx, err, data, pipelineId, cacheService, pb.Status_STATUS_ERROR)
+ return
+ }
+ log.Printf("%s: Run() finish\n", pipelineId)
+ processSuccess(ctx, data, pipelineId, cacheService)
+}
+
+// cleanUp removes all prepared folders for received LifeCycle
+func cleanUp(pipelineId uuid.UUID, lc *fs_tool.LifeCycle) {
+ log.Printf("%s: DeleteFolders() ...\n", pipelineId)
+ err := lc.DeleteFolders()
+ if err != nil {
+ log.Printf("%s: DeleteFolders(): %s\n", pipelineId, err.Error())
+ }
+ log.Printf("%s: DeleteFolders() complete\n", pipelineId)
+ log.Printf("%s: complete\n", pipelineId)
+}
+
+// processError processes error received during processing code via setting a corresponding status and output to cache
+func processError(ctx context.Context, err error, data []byte, pipelineId uuid.UUID, cacheService cache.Cache, status pb.Status) {
+ switch status {
+ case pb.Status_STATUS_ERROR:
+ log.Printf("%s: Run: err: %s, output: %s\n", pipelineId, err.Error(), data)
+
+ // set to cache pipelineId: cache.SubKey_RunOutput: err.Error()
+ setToCache(ctx, cacheService, pipelineId, cache.RunOutput, "error: "+err.Error()+", output: "+string(data))
+
+ // set to cache pipelineId: cache.SubKey_Status: pb.Status_STATUS_ERROR
+ setToCache(ctx, cacheService, pipelineId, cache.Status, pb.Status_STATUS_ERROR)
+ case pb.Status_STATUS_COMPILE_ERROR:
+ log.Printf("%s: Compile: err: %s, output: %s\n", pipelineId, err.Error(), data)
+
+ // set to cache pipelineId: cache.SubKey_CompileOutput: err.Error()
+ setToCache(ctx, cacheService, pipelineId, cache.CompileOutput, "error: "+err.Error()+", output: "+string(data))
+
+ // set to cache pipelineId: cache.SubKey_Status: pb.Status_STATUS_ERROR
+ setToCache(ctx, cacheService, pipelineId, cache.Status, pb.Status_STATUS_COMPILE_ERROR)
+ }
+}
+
+// processSuccess processes case after successful code processing via setting a corresponding status and output to cache
+func processSuccess(ctx context.Context, output []byte, pipelineId uuid.UUID, cacheService cache.Cache) {
+ // set to cache pipelineId: cache.SubKey_RunOutput: output
+ setToCache(ctx, cacheService, pipelineId, cache.RunOutput, string(output))
Review comment:
does it make sense to also add compiler output to cache in case of success? (maybe not - but if it doesn't, let's document)
##########
File path: playground/backend/cmd/server/controller.go
##########
@@ -15,21 +15,88 @@
package main
import (
- "context"
-
pb "beam.apache.org/playground/backend/internal/api/v1"
+ "beam.apache.org/playground/backend/internal/cache"
+ "beam.apache.org/playground/backend/internal/environment"
+ "beam.apache.org/playground/backend/internal/errors"
+ "beam.apache.org/playground/backend/internal/executors"
+ "beam.apache.org/playground/backend/internal/fs_tool"
+ "beam.apache.org/playground/backend/internal/validators"
+ "context"
"github.com/google/uuid"
+ "log"
)
type playgroundController struct {
+ env *environment.Environment
+ cacheService cache.Cache
+
pb.UnimplementedPlaygroundServiceServer
}
//RunCode is running code from requests using a particular SDK
func (controller *playgroundController) RunCode(ctx context.Context, info *pb.RunCodeRequest) (*pb.RunCodeResponse, error) {
- // TODO implement this method
- pipelineInfo := pb.RunCodeResponse{PipelineUuid: uuid.NewString()}
+ // check for correct sdk
+ switch info.Sdk {
+ case pb.Sdk_SDK_UNSPECIFIED, pb.Sdk_SDK_GO, pb.Sdk_SDK_PYTHON, pb.Sdk_SDK_SCIO:
+ log.Printf("RunCode(): unimplemented sdk: %s\n", info.Sdk)
+ return nil, errors.InvalidArgumentError("Run code()", "unimplemented sdk: "+info.Sdk.String())
+ }
+
+ pipelineId := uuid.New()
+
+ defer func() {
+ log.Printf("RunCode() is completed for pipeline with id: %s\n", pipelineId)
+ }()
+
+ cacheExpirationTime := controller.env.ApplicationEnvs.CacheEnvs().KeyExpirationTime()
Review comment:
please file a JIRA issue to document the `environment` package. For example, I am not sure what `KeyExpirationTime` means in this case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org