You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Hung Tran (JIRA)" <ji...@apache.org> on 2018/04/03 18:47:00 UTC
[jira] [Commented] (GOBBLIN-385) Add Spark execution mode for
Gobblin
[ https://issues.apache.org/jira/browse/GOBBLIN-385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424428#comment-16424428 ]
Hung Tran commented on GOBBLIN-385:
-----------------------------------
[^samza.diff]
[~vinothchandar], I have attached the Samza work. Gobblin was plugged into Samza through the {{SystemProducer}} interface. The {{GobblinSystemProducer}} creates a {{SamzaTaskRunner}} for each source that is registered by Samza. The {{SamzaTaskRunner}} instantiates a {{JobContext}} that is configured to execute a {{SamzaSource}}. This {{SamzaSource}} consumes from a queue that is written to when Samza calls the {{SystemProducer.send()}} call.
Please see the unit tests for example configuration.
> Add Spark execution mode for Gobblin
> ------------------------------------
>
> Key: GOBBLIN-385
> URL: https://issues.apache.org/jira/browse/GOBBLIN-385
> Project: Apache Gobblin
> Issue Type: New Feature
> Components: gobblin-cluster
> Reporter: Vinoth Chandar
> Assignee: Hung Tran
> Priority: Major
> Attachments: samza.diff
>
>
> If there is interest, happy to contribute spark execution mode and eventually add support for ingesting data into [https://github.com/uber/hudi] format..
> Please provide some guidance
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)