You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by bowenli86 <gi...@git.apache.org> on 2017/08/10 22:25:34 UTC
[GitHub] flink pull request #4522: [FLINK-7366][kinesis connector] Upgrade kinesis-pr...
GitHub user bowenli86 opened a pull request:
https://github.com/apache/flink/pull/4522
[FLINK-7366][kinesis connector] Upgrade kinesis-producer-library in flink-connector-kinesis from 0.10.2 to 0.12.5
## What is the purpose of the change
Upgrade kinesis-producer-library in flink-connector-kinesis from 0.10.2 to 0.12.5.
0.10.2 is released in Nov 2015, which is a bit old.
## Verifying this change
This change is already covered by existing tests
## Does this pull request potentially affect one of the following parts:
Nothing outside flink-kinesis-connector
## Documentation
- Does this pull request introduce a new feature? (no)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bowenli86/flink FLINK-7366
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4522.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4522
----
commit 7486878631e7238283eabfdd575387d32b210f91
Author: Bowen Li <bo...@gmail.com>
Date: 2017-08-10T22:09:56Z
FLINK-7366 Upgrade kinesis-producer-library in flink-connector-kinesis from 0.10.2 to 0.12.5
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4522: [FLINK-7366][kinesis connector] Upgrade kinesis pr...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/4522
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4522
oh.....this is not good....Do you mean that we cannot upgrade KPL/KCL until Flink 1.5?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/4522
@tzulitai Are we not using Java Serialisation starting from 1.2.x?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4522
@aljoscha No worries. Dropping compatibility with Flink 1.2 Kinesis consumer is great news! I reopened (Flink-7422)[https://issues.apache.org/jira/browse/FLINK-7422] to upgrade KCL
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4522
@tzulitai any more feedbacks? We have a ticket on my company for this task, and I'd like to mark it as finished if possible :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/4522
I've pushed the PR to CI again. If it passes, I'll merge it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4522: [FLINK-7366][kinesis connector] Upgrade kinesis pr...
Posted by bowenli86 <gi...@git.apache.org>.
GitHub user bowenli86 reopened a pull request:
https://github.com/apache/flink/pull/4522
[FLINK-7366][kinesis connector] Upgrade kinesis producer library in flink-connector-kinesis
## What is the purpose of the change
We need to upgrade KPL and KCL to pick up the enhanced performance and stability for Flink to work better with Kinesis. Upgrading KPL is specially necessary, because the KPL version Flink uses is old, and doesn't have good retry and error handling logic.
**Upgrade KPL:**
flink-connector-kinesis currently uses kinesis-producer-library 0.10.2, which is released in Nov 2015 by AWS. It's old. It's the fourth release, and thus problematic. It doesn't even have good retry logic, therefore Flink fails really frequently (about every 10 mins as we observed) when Flink writes too fast to Kinesis and receives RateLimitExceededException,
Quotes from https://github.com/awslabs/amazon-kinesis-producer/issues/56, "With the newer version of the KPL it uses the AWS C++ SDK which should offer additional retries." on Oct 2016. 0.12.5, the version we are upgrading to, is released in May 2017 and should have the enhanced retry logic.
**Upgrade KCL:**
Upgrade KCL from 1.6.2 to 1.8.1
**Upgrade AWS SDK:**
from 1.10.71 to 1.11.171
## Verifying this change
This change is already covered by existing tests
## Does this pull request potentially affect one of the following parts:
There should be any impact outside flink-connector-kinesis, because 1) KPL and KCL is only used in flink-connector-kinesis, and 2) AWS SDK in flink-connector-kinesis is shaded
## Documentation
- Does this pull request introduce a new feature? (no)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bowenli86/flink FLINK-7366
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4522.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4522
----
commit 7486878631e7238283eabfdd575387d32b210f91
Author: Bowen Li <bo...@gmail.com>
Date: 2017-08-10T22:09:56Z
FLINK-7366 Upgrade kinesis-producer-library in flink-connector-kinesis from 0.10.2 to 0.12.5
commit ba9fb5f1e189e31f8085bd384edbb682a69aae7f
Author: Bowen Li <bo...@gmail.com>
Date: 2017-08-10T23:46:59Z
upgrade KCL and AWS SDK
commit 88a16efcaf4c7e2addcf978a98f8ffbea4281639
Author: Bowen Li <bo...@gmail.com>
Date: 2017-08-13T05:49:26Z
revert changes to KCL
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/4522
@tzulitai Is currently on vacation but before he left he added a Backwards compatibility test for the Kinesis consumer. We also dropped compatibility with the Flink 1.2 Kinesis consumer so I think we can also update the consumer library. (I realised this is not completely related to this PR but I just noticed it now, sorry for that.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4522
Ok, so it seems like Java serialization of the Kinesis classes was removed only starting from 1.3.x.
That means unless we do some classloading trick, we can't really upgrade the Kinesis consumer dependencies until 1.2 savepoint compatibility is no longer supported.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4522
`testRestoreFromFlink11` is failing for the Kinesis consumer, since in Flink 1.1 we were directly serializing Kinesis classes in savepoints. Upgrading the Kinesis dependency would mean that the 1.1 savepoints would not be able to be restored.
I think, however, that there recently was a poll on the mailing list to remove 1.1 savepoint compatibility support starting from 1.4. That would entail we can completely remove the restore 1.1 tests in `master`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4522
Great! I can start FLINK-7508 then
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4522
can anyone from data artisan take a look at this PR please?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4522
@tzulitai @aljoscha there's another major bug in KPL that we need to fix for flink-connector-kinesis https://github.com/awslabs/amazon-kinesis-producer/issues/10 We've been seeing that bug crashing our first Flink cluster all the time. It's blocking us from making our first Flink cluster production-ready.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4522
yeah, we ran our job with this change on Flink 1.3.0 on EMR and didn't have issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4522: [FLINK-7366][kinesis connector] Upgrade kinesis pr...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 closed the pull request at:
https://github.com/apache/flink/pull/4522
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4522
@bowenli86 its fine to upgrade KPL. It's consumer related libraries version that currently cannot be touched.
Consumer related versions can currently only be upgraded until 1) Flink 1.2 is no longer supported, or 2) we have the state migration feature available.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4522
Hmm, I think Java serialization was removed from the Kinesis connector starting from 1.3.
But thinking of it now, I recall that there was some migration path from that. Let me double check ..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4522
Quick check @bowenli86:
Was this already tested in actual AWS environments? I just want to make sure there's no dependency conflicts or similar culprits before merging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4522
@tzulitai OK, I only upgraded KPL
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4522
@bowenli86 thanks for confirming! Merging this also then ..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---