You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@beam.apache.org by Ilya Kozyrev <il...@akvelon.com> on 2022/01/18 19:42:27 UTC

[RFC][design/idea] CDAP plugins support in Apache Beam

TL:DR: We want to develop support for Apache CDAP batch and streaming plugins to enrich Apache Beam connectors to external applications. Please review the design[1] to help us bring CDAP plugins integrations into Apache Beam.


Hi all,


I along with a few community members thought of an idea to create an Apache Beam IO package for Apache CDAP.  CDAP IO package will enable creating integrations of Apache CDAP plugins<https://github.com/data-integrations> to Apache Beam to extend application connectors offered by Apache Beam.


CDAP IO connector will support batch source and sinks via HadoopFormatIO. CDAP IO connector will support streaming sources (SparkReceiverIO to proxy Custom Spark Receivers for usage in Apache Beam). The proposed design and implementation details are described in the design document[1].


Initially, we are thinking of creating the following integrations:

  *   Salesforce
  *   ServiceNow (batch)
  *   Zendesk (batch)
  *   Salesforce Marketing Cloud (batch)
  *   Hubspot


Please share your feedback both on the idea and the design doc.


Thanks!


[1] https://docs.google.com/document/d/1T-bhd0Qk7DBePIfgHEPagYiA1oLP4z5kYEd0S1SOGxQ/edit?usp=sharing

Re: [RFC][design/idea] CDAP plugins support in Apache Beam

Posted by Chamikara Jayalath <ch...@google.com>.

Thanks. Added some comments.

- Cham

On Tue, Jan 18, 2022 at 1:49 PM Kenneth Knowles <ke...@apache.org> wrote:

> Very cool. Thanks for sharing!
>
> On Tue, Jan 18, 2022 at 11:42 AM Ilya Kozyrev <il...@akvelon.com>
> wrote:
>
>> TL:DR: We want to develop support for Apache CDAP batch and streaming
>> plugins to enrich Apache Beam connectors to external applications. Please
>> review the design[1] to help us bring CDAP plugins integrations into Apache
>> Beam.
>>
>>
>>
>> Hi all,
>>
>>
>>
>> I along with a few community members thought of an idea to create an
>> Apache Beam IO package for Apache CDAP.  CDAP IO package will enable
>> creating integrations of Apache CDAP plugins
>> <https://github.com/data-integrations> to Apache Beam to extend
>> application connectors offered by Apache Beam.
>>
>>
>>
>> CDAP IO connector will support batch source and sinks via HadoopFormatIO.
>> CDAP IO connector will support streaming sources (SparkReceiverIO to proxy
>> Custom Spark Receivers for usage in Apache Beam). The proposed design and
>> implementation details are described in the design document[1].
>>
>>
>>
>> Initially, we are thinking of creating the following integrations:
>>
>>    - Salesforce
>>    - ServiceNow (batch)
>>    - Zendesk (batch)
>>    - Salesforce Marketing Cloud (batch)
>>    - Hubspot
>>
>>
>>
>> Please share your feedback both on the idea and the design doc.
>>
>>
>>
>> Thanks!
>>
>>
>>
>> [1]
>> https://docs.google.com/document/d/1T-bhd0Qk7DBePIfgHEPagYiA1oLP4z5kYEd0S1SOGxQ/edit?usp=sharing
>>
>>
>>
>>
>>
>

Re: [RFC][design/idea] CDAP plugins support in Apache Beam

Posted by Kenneth Knowles <ke...@apache.org>.

Very cool. Thanks for sharing!

On Tue, Jan 18, 2022 at 11:42 AM Ilya Kozyrev <il...@akvelon.com>
wrote:

> TL:DR: We want to develop support for Apache CDAP batch and streaming
> plugins to enrich Apache Beam connectors to external applications. Please
> review the design[1] to help us bring CDAP plugins integrations into Apache
> Beam.
>
>
>
> Hi all,
>
>
>
> I along with a few community members thought of an idea to create an
> Apache Beam IO package for Apache CDAP.  CDAP IO package will enable
> creating integrations of Apache CDAP plugins
> <https://github.com/data-integrations> to Apache Beam to extend
> application connectors offered by Apache Beam.
>
>
>
> CDAP IO connector will support batch source and sinks via HadoopFormatIO.
> CDAP IO connector will support streaming sources (SparkReceiverIO to proxy
> Custom Spark Receivers for usage in Apache Beam). The proposed design and
> implementation details are described in the design document[1].
>
>
>
> Initially, we are thinking of creating the following integrations:
>
>    - Salesforce
>    - ServiceNow (batch)
>    - Zendesk (batch)
>    - Salesforce Marketing Cloud (batch)
>    - Hubspot
>
>
>
> Please share your feedback both on the idea and the design doc.
>
>
>
> Thanks!
>
>
>
> [1]
> https://docs.google.com/document/d/1T-bhd0Qk7DBePIfgHEPagYiA1oLP4z5kYEd0S1SOGxQ/edit?usp=sharing
>
>
>
>
>