You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/04/08 15:30:00 UTC
[jira] [Work logged] (BEAM-6677) Provide Portable Flink cluster
setup
[ https://issues.apache.org/jira/browse/BEAM-6677?focusedWorklogId=224411&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224411 ]
ASF GitHub Bot logged work on BEAM-6677:
----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Apr/19 15:29
Start Date: 08/Apr/19 15:29
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #8250: [BEAM-6677] Job Server on Dataproc cluster support
URL: https://github.com/apache/beam/pull/8250
Previously we assumed that Job server instance should be set up on Jenkins.
This pr extends the existing cluster with job server setup. Thanks to that we won’t have to setup a dedicated Jenkins job step to setup the job server and manage the communication between Jenkins (job server_)and Fataproc (the cluster). We open up a tunnel with appropriate JobServer ports to make the cluster usage easier.
Of course, setting up the cluster without the job server is still possible. To do that one needs to not pass the JOB_SERVER_IMAGE variable,
Other than that, this PR includes some convenient improvements (default values for GCLOUD_ZONE and DATAPROC_VERSION) and a fix for flink init action
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) <br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 224411)
Time Spent: 10m
Remaining Estimate: 0h
> Provide Portable Flink cluster setup
> ------------------------------------
>
> Key: BEAM-6677
> URL: https://issues.apache.org/jira/browse/BEAM-6677
> Project: Beam
> Issue Type: New Feature
> Components: testing
> Reporter: Lukasz Gajowy
> Assignee: Lukasz Gajowy
> Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> A Flink cluster that can be set up for testing purposes. Currently, the easiest option seems to be using google cloud Dataproc for this purpose (ready to use init actions, scalable Flink instance, easy to configure).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)