[jira] [Commented] (BEAM-14101) [CdapIO] Design and implement Spark Receiver Builder

["Danny McCormick (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17550111#comment-17550111 ] 

Danny McCormick commented on BEAM-14101:
----------------------------------------

This issue has been migrated to https://github.com/apache/beam/issues/21673

> [CdapIO] Design and implement Spark Receiver Builder
> ----------------------------------------------------
>
>                 Key: BEAM-14101
>                 URL: https://issues.apache.org/jira/browse/BEAM-14101
>             Project: Beam
>          Issue Type: Task
>          Components: io-java-cdap
>            Reporter: Elizaveta Lomteva
>            Assignee: Elizaveta Lomteva
>            Priority: P2
>              Labels: cdap-io-sprint-4
>          Time Spent: 12h 50m
>  Remaining Estimate: 0h
>
> h3. Context:
> CDAP plugins that support streaming sources include Receiver classes (ex. [HubSpotReceiver|https://github.com/data-integrations/hubspot/blob/develop/src/main/java/io/cdap/plugin/hubspot/source/streaming/HubspotReceiver.java]) that extend {{org.apache.spark.streaming.receiver.Receiver}} abstract class. Receiver classes are used by plugin Streaming Utils classes (ex. [HubSpotStreamingUtils|https://github.com/data-integrations/hubspot/blob/develop/src/main/java/io/cdap/plugin/hubspot/source/streaming/HubspotStreamingUtil.java]) to provide {{getStream()}} method to Streaming Source classes (ex. [HubSpotStreamingSource|https://github.com/data-integrations/hubspot/blob/develop/src/main/java/io/cdap/plugin/hubspot/source/streaming/HubspotStreamingSource.java]) and usually placed in {{cdap/plugin/\{<plugin name>}/plugin/source/streaming}} folder (ex.[ HubSpot plugin repo|https://github.com/data-integrations/hubspot/tree/develop/src/main/java/io/cdap/plugin/hubspot/source/streaming]).
> Reference information:
>  * [Streaming plug-ins integration|https://docs.google.com/document/d/1T-bhd0Qk7DBePIfgHEPagYiA1oLP4z5kYEd0S1SOGxQ/edit#heading=h.o88i6p9b13o9] (Apache CDAP Connection Design Doc)
>  * [Plugin integration process description|https://docs.google.com/document/d/1T-bhd0Qk7DBePIfgHEPagYiA1oLP4z5kYEd0S1SOGxQ/edit#heading=h.1h6udb1b52xc] (Apache CDAP Connection Design Doc)
>  * [Streaming wrapper design|https://docs.google.com/document/d/1T-bhd0Qk7DBePIfgHEPagYiA1oLP4z5kYEd0S1SOGxQ/edit#heading=h.fcafz0ydsso1] (Apache CDAP Connection Design Doc)
> h3. Task Description:
> Required to design custom Spark receivers builder class so that the custom receivers can be used in Apache Beam connector via SparkReceiverIO interface (used in CDAP IO as a dependency).
> h3. Acceptance criteria:
> Design of builder class(es) that will create custom Spark receivers in Apache Beam connectors ({{{}SparkReceiverIO{}}}).
> h4. Note:
> It is necessary that this builder class be independent of CDAP receivers and can be used by any other custom Spark receiver like a part of SparkReceiverIO.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)