[jira] [Commented] (GSOC-260) [GSOC][Beam] Add connectors to Beam ManagedIO

["Miko Aro (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/GSOC-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832892#comment-17832892 ] 

Miko Aro commented on GSOC-260:
-------------------------------

Will the implementation for the integration be based on Java and Python only? Or will it involve some other programming languages supported by the Apache Beam?

> [GSOC][Beam] Add connectors to Beam ManagedIO
> ---------------------------------------------
>
>                 Key: GSOC-260
>                 URL: https://issues.apache.org/jira/browse/GSOC-260
>             Project: Comdev GSOC
>          Issue Type: New Feature
>            Reporter: Danny McCormick
>            Priority: Major
>              Labels: Beam, gsoc, gsoc2024
>
> Apache Beam is a unified model for defining both batch and streaming data-parallel processing pipelines, as well as a set of language-specific SDKs for constructing pipelines and Runners for executing them on distributed processing backends. On top of providing lower level primitives, Beam has also introduced several higher level transforms used for machine learning and some general data processing use cases. One new transform that is being actively worked on is a unified ManagedIO transform which gives runners the ability to manage (upgrade, optimize, etc...) an IO (input source or output sink) without upgrading the whole pipeline. This project will be about adding one or more IO integrations to ManagedIO
> Objectives:
> 1. Add a BigTable integration to ManagedIO
> 2. Add a Spanner integration to ManagedIO
> Useful links:
> Apache Beam repo - https://github.com/apache/beam
> Docs on ManagedIO are relatively light since this is a new project, but here are some docs on existing IOs in Beam - https://beam.apache.org/documentation/io/connectors/



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: gsoc-unsubscribe@community.apache.org
For additional commands, e-mail: gsoc-help@community.apache.org