You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by bowenli86 <gi...@git.apache.org> on 2017/06/20 21:01:46 UTC
[GitHub] flink pull request #4150: [FLINK-6951][docs] Incompatible versions of httpco...
GitHub user bowenli86 opened a pull request:
https://github.com/apache/flink/pull/4150
[FLINK-6951][docs] Incompatible versions of httpcomponents jars for Flink kinesis connector
The root cause being that Flink documentation is outdated on dependency versions. I updated Flink doc.
Since I run Flink on hadoop 2.7, I only tested for hadoop 2.7 on EMR, not hadoop 2.6.
We can probably open an jira ticket for hadoop 2.6.
Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bowenli86/flink FLINK-6951
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4150.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4150
----
commit cb0bc2b86ae5604bf2e919098179d0dc26a983ae
Author: Bowen Li <bo...@gmail.com>
Date: 2017-06-20T20:58:38Z
[FLINK-6951] Incompatible versions of httpcomponents jars for Flink kinesis connector
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4150: [FLINK-6951] Incompatible versions of httpcomponen...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/4150#discussion_r126274129
--- Diff: flink-connectors/flink-connector-kinesis/pom.xml ---
@@ -36,6 +36,8 @@ under the License.
<aws.sdk.version>1.10.71</aws.sdk.version>
<aws.kinesis-kcl.version>1.6.2</aws.kinesis-kcl.version>
<aws.kinesis-kpl.version>0.10.2</aws.kinesis-kpl.version>
+ <apache.httpclient.version>4.3.6</apache.httpclient.version>
+ <apache.httpcore.version>4.3.3</apache.httpcore.version>
--- End diff --
I assume these versions correspond to what the AWS SDK is using, correct?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4150
@bowenli86 To you have to actually explicitly add the http component dependencies to the pom.xml file, or does the "promoteTransitiveDependencies" config in the shading already cover that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4150: [FLINK-6951] Incompatible versions of httpcomponen...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4150#discussion_r127138994
--- Diff: flink-connectors/flink-connector-kinesis/pom.xml ---
@@ -56,6 +58,18 @@ under the License.
</dependency>
<dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ <version>${apache.httpclient.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpcore</artifactId>
+ <version>${apache.httpcore.version}</version>
+ </dependency>
--- End diff --
For `promoteTransitiveDependencies`, what does it really mean? https://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html#promoteTransitiveDependencies is kind of very confusing to understand.
To your second point, my intention is neither adding http-components 4.3.x for hadoop-aws, nor for AWS s3. It is actually for aws-java-sdk-kinesis used by flink-connector-kinesis. This way, users won't worry about versions of http-components depended by aws-java-sdk-kinesis anymore. Does it make sense?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
My Flink job checkpoints to S3, I'm configuring S3AFileSystem shown in https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html#flink-for-hadoop-27.
I doubt if this error is due to S3AFileSystem uses httpcomponents 4.2, since httpcomponents 4.3.x [has deprecated SSLSocketFactory](https://hc.apache.org/httpcomponents-client-4.3.x/httpclient/apidocs/org/apache/http/conn/ssl/SSLSocketFactory.html)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
Yes, we do shade the `com.amazonaws.*` packages. It was fixed before to address problems with running the Kinesis connector on older EMR versions. So I guess that my assumption of that there are conflicting `aws-java-sdk` versions shouldn't hold, given that the dependency is properly shaded ..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951][docs] Incompatible versions of httpcomponent...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
Thanks for the PR @bowenli86.
However, I'm not entirely sure the original version in the doc count as "outdated".
For example, would the original versions work if you're not using the Kinesis connector?
Or, if we decide to bump the `aws-java-sdk` version in the Kinesis connector, would these httpcomponent versions need to correspond to?
I'm wondering if it actually makes sense to shade the httpcomponent dependencies in the Kinesis connector, so that it doesn't incorrectly pick up other versions in the classpath?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
@bowenli86 but isn't the shading of httpcomponents in the Kinesis consumer supposed to avoid conflicts with whatever version you're using for S3AFileSystem?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
Thanks for confirming. I'll close this ticket and PR
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by casidiablo <gi...@git.apache.org>.
Github user casidiablo commented on the issue:
https://github.com/apache/flink/pull/4150
Since EMR only supports Flink 1.3 I had to checkout release-1.3 and compile the connector from there. Then I was getting this `Socket not created by this factory` error.
I then patched 1.3 to include these changes, and that fixed it.
The current master version can't be used because so many things have changed since then, and it does not seem to be compatible with EMR right now.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
From the looks of https://github.com/druid-io/druid/issues/4456, could it be that we need to update our AWS Java SDK version?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
Merging to `release-1.3` ..
---
[GitHub] flink pull request #4150: [FLINK-6951] Incompatible versions of httpcomponen...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4150#discussion_r126511039
--- Diff: flink-connectors/flink-connector-kinesis/pom.xml ---
@@ -36,6 +36,8 @@ under the License.
<aws.sdk.version>1.10.71</aws.sdk.version>
<aws.kinesis-kcl.version>1.6.2</aws.kinesis-kcl.version>
<aws.kinesis-kpl.version>0.10.2</aws.kinesis-kpl.version>
+ <apache.httpclient.version>4.3.6</apache.httpclient.version>
+ <apache.httpcore.version>4.3.3</apache.httpcore.version>
--- End diff --
yes.
Here's the dependency tree:
aws-java-sdk-kinesis 1.10.71
aws-java-sdk-core 1.10.71
httpclient 4.3.6
httpcore 4.3.3
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by xiatao123 <gi...@git.apache.org>.
Github user xiatao123 commented on the issue:
https://github.com/apache/flink/pull/4150
I ran into a similar issue in EMR. Any suggestion on how to fix it?
`Release label:emr-5.11.0
Hadoop distribution:Amazon 2.7.3
Applications:Flink 1.3.2`
`java.lang.IllegalStateException: Socket not created by this factory
at org.apache.http.util.Asserts.check(Asserts.java:34)
at org.apache.http.conn.ssl.SSLSocketFactory.isSecure(SSLSocketFactory.java:435)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:186)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:326)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:837)
at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607)
at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376)
at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338)
at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287)
at org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:1940)
at org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:1910)
at org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.describeStream(AmazonKinesisClient.java:656)
at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.describeStream(KinesisProxy.java:361)
at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getShardsOfStream(KinesisProxy.java:323)
at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getShardList(KinesisProxy.java:231)
at org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.discoverNewShardsToSubscribe(KinesisDataFetcher.java:430)
at org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.run(FlinkKinesisConsumer.java:202)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
`
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
@tzulitai I shaded httpcomponents. Does it look good?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4150: [FLINK-6951] Incompatible versions of httpcomponen...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/4150#discussion_r127135589
--- Diff: flink-connectors/flink-connector-kinesis/pom.xml ---
@@ -56,6 +58,18 @@ under the License.
</dependency>
<dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ <version>${apache.httpclient.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpcore</artifactId>
+ <version>${apache.httpcore.version}</version>
+ </dependency>
--- End diff --
As Stephan mentioned, these shouldn't be needed to be declared. They'll already be pulled in due to the `promoteTransitiveDependencies` property in the shade config.
Adding them for the sake of hadoop-aws dependency doesn't make sense, for the same reason as before; we shouldn't assume that the Kinesis connector user will also use AWS S3.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4150
@bowenli86 I think Kinesis is quite popular. The conflict only arises in combination with certain Hadoop versions, that may be the cause why not everybody experiences it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by xiatao123 <gi...@git.apache.org>.
Github user xiatao123 commented on the issue:
https://github.com/apache/flink/pull/4150
Got the issue fixed after apply this patch in
Release label:emr-5.11.0
Hadoop distribution:Amazon 2.7.3
Applications:Flink 1.3.2
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4150
@tzulitai Do we shade the `aws-sdk-java` in the Kinesis connector? We should probably, and shade it in Hadoop as well. If not, this could be a cause of the conflict...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by burkaygur <gi...@git.apache.org>.
Github user burkaygur commented on the issue:
https://github.com/apache/flink/pull/4150
similar to @bowenli86 i am receiving the same socket error. Tried the proposed fix in the PR, but still getting the error. Bumped the JDK version, didnt help either. Anyone has a working configuration for this connector?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
@casidiablo thanks for the info!
I'll merge this for `release-1.3` then, and will keep an extra eye on whether the problem still occurs for 1.4 / 1.5.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4150
Hmmm, tricky one. Do parts of the code use different configuration (one instantiates non SSL sockets, the other one assumes SSL sockets)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
On second thought, the change is actually a general improvement to the packaging of the connector, so it would make sense to merge this to `master` and `release-1.4` also.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by casidiablo <gi...@git.apache.org>.
Github user casidiablo commented on the issue:
https://github.com/apache/flink/pull/4150
This actually worked for me. I ran it on EMR 5.10, compiling the kinesis connector from the release-1.3 branch (since EMR provides Flink 1.3)
---
[GitHub] flink pull request #4150: [FLINK-6951] Incompatible versions of httpcomponen...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/4150
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
@tzulitai @tedyu
Hi guys, I'm also wondering how much percentage of Flink users are using Flink with Kinesis? Kinesis has so many limitations that we are considering to move from Kinesis to Kafka. Knowing general preference of Flink users will make us more confident in this move. I guess not a lot people are using Flink + Kinesis, otherwise I won't be the first one running into this dependency conflict bug. Do you guys have any numbers in mind?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
@casidiablo did you mean that without applying this PR's patch, the current master worked for you? Or you had to apply this patch in order for it to work?
---
[GitHub] flink pull request #4150: [FLINK-6951] Incompatible versions of httpcomponen...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/4150#discussion_r126274141
--- Diff: docs/setup/aws.md ---
@@ -224,8 +224,6 @@ Depending on which file system you use, please add the following dependencies. Y
- `S3AFileSystem`:
- `hadoop-aws-2.7.2.jar`
- `aws-java-sdk-1.7.4.jar`
- - `httpcore-4.2.5.jar`
- - `httpclient-4.2.5.jar`
--- End diff --
I'm not sure that these instructions can be removed. Could you describe a bit? Especially in the case if the user is not using the Flink Kinesis connector.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
I added http components to dependencies because Hadoop-aws 2.7 depends on http components 4.2.
```
$ jar -tf flink-connector-kinesis_2.11-1.4-SNAPSHOT.jar | grep apache/http
org/apache/flink/kinesis/shaded/org/apache/http/
org/apache/flink/kinesis/shaded/org/apache/http/impl/
org/apache/flink/kinesis/shaded/org/apache/http/impl/cookie/
org/apache/flink/kinesis/shaded/org/apache/http/impl/cookie/BestMatchSpec.class
org/apache/flink/kinesis/shaded/org/apache/http/impl/cookie/RFC2965PortAttributeHandler.class
...
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
@casidiablo @tzulitai sorry, I had a long day yesterday and misunderstood what you mean.
And thank you @casidiablo @xiatao123 both for confirming it works.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
ok, I kind of figured out what `promoteTransitiveDependencies` means.
I removed explicit dependencies of httpclient and httpcore from `pom.xml`, and made sure http-components are shaded to `org/apache/flink/kinesis/shaded/orsg/apache/http/*` in flink-connector-kinesis_2.11-1.4-SNAPSHOT.jar
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4150: [FLINK-6951] Incompatible versions of httpcomponen...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4150#discussion_r126509865
--- Diff: docs/setup/aws.md ---
@@ -224,8 +224,6 @@ Depending on which file system you use, please add the following dependencies. Y
- `S3AFileSystem`:
- `hadoop-aws-2.7.2.jar`
- `aws-java-sdk-1.7.4.jar`
- - `httpcore-4.2.5.jar`
- - `httpclient-4.2.5.jar`
--- End diff --
I see your point. Yeah, users run on AWS EMR don't necessarily use Kinesis. I'll revert this part.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951][docs] Incompatible versions of httpcomponent...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
@tedyu @tzulitai
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
During testing, I saw the following exception:
```java
java.lang.IllegalStateException: Socket not created by this factory
at org.apache.flink.kinesis.shaded.org.apache.http.util.Asserts.check(Asserts.java:34)
at org.apache.flink.kinesis.shaded.org.apache.http.conn.ssl.SSLSocketFactory.isSecure(SSLSocketFactory.java:435)
at org.apache.flink.kinesis.shaded.org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:186)
at org.apache.flink.kinesis.shaded.org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:326)
at org.apache.flink.kinesis.shaded.org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.flink.kinesis.shaded.org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.flink.kinesis.shaded.org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at org.apache.flink.kinesis.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.flink.kinesis.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:837)
at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607)
at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376)
at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338)
at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287)
at org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:1940)
at org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:1910)
at org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.describeStream(AmazonKinesisClient.java:656)
at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.describeStream(KinesisProxy.java:363)
at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getShardsOfStream(KinesisProxy.java:325)
at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getShardList(KinesisProxy.java:233)
at org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.discoverNewShardsToSubscribe(KinesisDataFetcher.java:430)
at org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.run(FlinkKinesisConsumer.java:203)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:262)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
```
According to https://github.com/Jean-Emile/org.apache.httpclient/blob/master/src/main/java/org/apache/http/conn/ssl/SSLSocketFactory.java, it seems a different kind of socket is passed around when shading httpcomponents. I'll see if I have time to dig deeper. Any insight is highly appreciated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
Thanks @bowenli86 for trying that out. Please keep my updated on this :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
@tzulitai shading httpclient/core sounder like a more sound fix.
I'll see if I have time to shade them and resubmit the PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4150: [FLINK-6951] Incompatible versions of httpcomponents jars...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
Thanks! Just for double check: have you on your side already verified that this works when you're using the connector with S3 (which caused the issue for you before)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4150: [FLINK-6951] Incompatible versions of httpcomponen...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4150#discussion_r127139879
--- Diff: flink-connectors/flink-connector-kinesis/pom.xml ---
@@ -56,6 +58,18 @@ under the License.
</dependency>
<dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ <version>${apache.httpclient.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpcore</artifactId>
+ <version>${apache.httpcore.version}</version>
+ </dependency>
--- End diff --
To clarify a bit more:
- hadoop-aws 2.7 depends on http-components 4.2.x
- aws-java-sdk-kinesis 1.10.71, a core dependency of flink-kinesis-connector, depends on http-components 4.3.x.
Thus, we should include http-components 4.3 into flink-kinesis-connector so that users won't worry about deciding http-components version if they use hadoop-aws and flink-kinesis-connector at the same time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---