You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org> on 2017/09/23 20:22:00 UTC
[jira] [Comment Edited] (FLINK-7673) Flink Kinesis connector -
remove dependency on the ASL code
[ https://issues.apache.org/jira/browse/FLINK-7673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16177962#comment-16177962 ]
Tzu-Li (Gordon) Tai edited comment on FLINK-7673 at 9/23/17 8:21 PM:
---------------------------------------------------------------------
Yes, it would definitely be nice to eventually have a ASL license-free Kinesis connector that we can publish.
If we were to reverse engineer the functionality that KCL / KPL provides, the amount of work would vary across the consumer and producer:
- For the consumer, we would only need to work out how to perform the record message aggregation. I think KCL uses protobuf to do this magic. It might simply be a matter of migrating some protobuf generated classes to the Kinesis connector.
- For the producer, I previously already worked on a ASL-free version here: https://github.com/tzulitai/flink/tree/FLINK-3229-RemoveKclKplRework. Its functionality probably isn't on-par with the current producer which uses KPL, so we would need to look into that. Robert and I initially did not decide to use it because we wanted to wait and see if there may be some possibilities of license changes on the AWS side.
Either way, I would be very happy to see some progress on this front. It would make our AWS users much more happier :)
was (Author: tzulitai):
Yes, it would definitely be nice to eventually have a ASL license-free Kinesis connector that we can publish.
If we were to reverse engineer this, the amount of work would vary across the consumer and producer:
- For the consumer, we would only need to work out how to perform the record message aggregation. I think KCL uses protobuf to do this magic. It might simply be a matter of migrating some protobuf generated classes to the Kinesis connector.
- For the producer, I previously already worked on a ASL-free version here: https://github.com/tzulitai/flink/tree/FLINK-3229-RemoveKclKplRework. Its functionality probably isn't on-par with the current producer which uses KPL, so we would need to look into that. Robert and I initially did not decide to use it because we wanted to wait and see if there may be some possibilities of license changes on the AWS side.
Either way, I would be very happy to see some progress on this front. It would make our AWS users much more happier :)
> Flink Kinesis connector - remove dependency on the ASL code
> -----------------------------------------------------------
>
> Key: FLINK-7673
> URL: https://issues.apache.org/jira/browse/FLINK-7673
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
> Reporter: Radoslaw Gruchalski
>
> Currently the Flink Kinesis connector depends on two artifacts which are available under Amazon Software License code. The artifacts are:
> {noformat}
> <dependency>
> <groupId>com.amazonaws</groupId>
> <artifactId>amazon-kinesis-producer</artifactId>
> <version>${aws.kinesis-kpl.version}</version>
> </dependency>
> <dependency>
> <groupId>com.amazonaws</groupId>
> <artifactId>amazon-kinesis-client</artifactId>
> <version>${aws.kinesis-kcl.version}</version>
> <!--
> We're excluding the below from the KCL since we'll only be using the
> com.amazonaws.services.kinesis.clientlibrary.types.UserRecord class, which will not need these dependencies.
> -->
> <exclusions>
> <exclusion>
> <groupId>com.amazonaws</groupId>
> <artifactId>aws-java-sdk-dynamodb</artifactId>
> </exclusion>
> <exclusion>
> <groupId>com.amazonaws</groupId>
> <artifactId>aws-java-sdk-cloudwatch</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> {noformat}
> This prevents the connector being published to Maven Central. I would like to contribute to Flink Kinesis connector by replacing the ASL licensed code with an implementation using AWS SDK for Java. For that, I'd like to understand two things:
> - what's the correct process for making the contribution (contributor agreement, etc.)
> - what's the minimum functionality the alternative implementation would have to provide in order to get accepted
> There are three classes in the connector's source requiring modification:
> - org.apache.flink.streaming.connectors.kinesis.FlinkKinesisProducer
> - org.apache.flink.streaming.connectors.kinesis.util.KinesisConfigUtil
> - org.apache.flink.streaming.connectors.kinesis.internals.ShardConsumer
> Thank you.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)