You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/03/02 20:06:00 UTC
[jira] [Work logged] (BEAM-6685) JobServer driver with stubbed out
spark related component
[ https://issues.apache.org/jira/browse/BEAM-6685?focusedWorklogId=206782&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-206782 ]
ASF GitHub Bot logged work on BEAM-6685:
----------------------------------------
Author: ASF GitHub Bot
Created on: 02/Mar/19 20:05
Start Date: 02/Mar/19 20:05
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #7977: [BEAM-6685] [BEAM-6686] Create scaffolding for Spark portable runner
URL: https://github.com/apache/beam/pull/7977
This PR sets up the scaffolding upon which we can build a portable Spark runner. (It is very close to the structure of the Flink portable runner.)
- I chose to put this on the master branch, rather than the [structured streaming](https://github.com/apache/beam/tree/spark-runner_structured-streaming) branch. The implementation details of how a pipeline is translated and run are not the focus of this PR; in fact, they are intended to eventually be switched out.
- For now, the runner merely sets up a Spark context and attempts to translate/run a portable pipeline. It "fails as intended" because the map of translation functions is empty, left to be filled in by future PRs.
R: @angoenka, @mxm, @echauchot
------------------------
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) <br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 206782)
Time Spent: 10m
Remaining Estimate: 0h
> JobServer driver with stubbed out spark related component
> ---------------------------------------------------------
>
> Key: BEAM-6685
> URL: https://issues.apache.org/jira/browse/BEAM-6685
> Project: Beam
> Issue Type: Task
> Components: runner-spark
> Reporter: Ankur Goenka
> Assignee: Kyle Weaver
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)