You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by boneill42 <gi...@git.apache.org> on 2016/04/06 18:30:14 UTC
[GitHub] spark pull request: Upgraded KCL version to support de-aggregation...
GitHub user boneill42 opened a pull request:
https://github.com/apache/spark/pull/12209
Upgraded KCL version to support de-aggregation of Kinesis records.
## What changes were proposed in this pull request?
Upgrades KCL dependency, backported to 1.6.1.
## How was this patch tested?
Used kinesis word count example against a stream containing aggregated data.
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
See: SPARK-14421
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/boneill42/spark SPARK-14421
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12209.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12209
----
commit 5395ad01a9622175919667839a77568623859c45
Author: Brian O'Neill <bo...@monetate.com>
Date: 2016-04-06T16:21:59Z
Upgraded KCL version to support de-aggregation of Kinesis records.
See: SPARK-14421
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Upgraded KCL version to support de-aggregation...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-206451757
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207470462
build/mvn will download Maven 3.3.9. You may need --force to override your local install. 3.3.3 is the wrong version.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-218530991
OK -- I just confirmed that I still see this issue on master.
```
Caused by: java.lang.ClassNotFoundException: com.google.protobuf.ProtocolStringList
```
I'll submit a patch now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207422116
Good point. I updated to `kinesis.protobuf`. (I wanted to stay away from any potential collision with the parent, which is why I didn't re-used the spark.shade.packageName)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207446368
Jenkins test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207447217
**[Test build #55346 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55346/consoleFull)** for PR 12209 at commit [`fe98fc1`](https://github.com/apache/spark/commit/fe98fc156218fb50d9c71e2be9e45037f4ba8737).
* This patch **fails build dependency tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-206520181
Yep, that worked... but also acknowledging that there is a chance for conflict since both 2.5.0 and 2.6.1 will be in play.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-206750570
Summoning @vanzin to ask if this would indeed only affect the Kinesis artifacts? if that's all it affects and it fixes a problem, seems OK. Seems risky of course to modify protobuf any more widely.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Upgraded KCL version to support de-aggregation...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12209#discussion_r58743959
--- Diff: pom.xml ---
@@ -128,7 +128,7 @@
<slf4j.version>1.7.10</slf4j.version>
<log4j.version>1.2.17</log4j.version>
<hadoop.version>2.2.0</hadoop.version>
- <protobuf.version>2.5.0</protobuf.version>
+ <protobuf.version>2.6.1</protobuf.version>
--- End diff --
This isn't necessarily OK since it changes protobuf for the whole project, and could conflict with versions used by, say, Hadoop. 2.4 and 2.5 were terribly incompatible, and I am paranoid about 2.5 vs 2.6.
The KCL update itself is probably fine. Is that helpful at all by itself? you'll have to update the deps files to reflect the updated dependencies for tests to pass.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207061046
Ah, brilliant. I just tested out the relocation and it appeared to work. (PR updated)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 closed the pull request at:
https://github.com/apache/spark/pull/12209
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12209#discussion_r59029907
--- Diff: extras/kinesis-asl-assembly/pom.xml ---
@@ -142,6 +147,15 @@
<include>*:*</include>
</includes>
</artifactSet>
+ <relocations>
+ <relocation>
--- End diff --
Nit: this section is indented 2 spaces too far, though it won't matter
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207478285
A-ha, well, two problems here. The underlying cause is that it can't download Maven because the ASF site is in maintenance mode now. Try http://archive.apache.org/dist/maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.tar.gz
But I was surprised to see 3.3.3 since we are on 3.3.9 in master, and I see now you opened the PR vs the 1.6 branch. This should be opened vs master, and then it is back-ported as needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Upgraded KCL version to support de-aggregation...
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-206477403
I had created a JIRA:
https://issues.apache.org/jira/browse/SPARK-14421
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207047304
Adding the assembly to the classpath that way means the newer protobuf will override any older version on the Spark jars. So kinesis will work, but any code in Spark that is not compatible with the new protobuf will fail. It sounds a little risk.
Doing relocation in the kinesis assembly should be pretty straight forward. Here's an example from Spark's own build: https://github.com/apache/spark/blob/master/pom.xml#L2207
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-213436049
@boneill42 do you want to take a crack at adding this change and bumping to 1.6.2 in master, so that it gets into 2.0?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-206503634
I'm not actively working it. We'll likely just fork and build a custom kinesis-asl jar.
Does it make sense to upgrade the dependencies for kinesis-asl? (and not touch the parent pom?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-206488987
Agreed. I just wanted to get it on the radar. People might experience sporadic problems consuming from kinesis streams until we can get it in there.
(title updated)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Upgraded KCL version to support de-aggregation...
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on a diff in the pull request:
https://github.com/apache/spark/pull/12209#discussion_r58746934
--- Diff: pom.xml ---
@@ -128,7 +128,7 @@
<slf4j.version>1.7.10</slf4j.version>
<log4j.version>1.2.17</log4j.version>
<hadoop.version>2.2.0</hadoop.version>
- <protobuf.version>2.5.0</protobuf.version>
+ <protobuf.version>2.6.1</protobuf.version>
--- End diff --
Agreed... definitely needs testing.
Just upgrading the KCL didn't work. The new KCL relies on the following class from protobuf: `com/google/protobuf/ProtocolStringList.class`, which doesn't exist until 2.6.1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-213522262
yeah -- sorry, been busy with the day job.
I'll move this to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207447229
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55346/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-206506455
I looked at the KCL dependencies, and 1.4.0 already depended on protobuf 1.6.1. What was the error you got -- clearly a version mismatch problem? just checking that it now becomes unavoidable but somehow worked before.
Yes, it might be viable to explicitly depend on protobuf 1.6.1 only in the KCL assembly. Add a version and un-mark it as provided and add a comment. That could be interesting to try. Although there's a risk of conflict in theory, it sounds like it at least fixes a real problem in practice now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207447225
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Upgraded KCL version to support de-aggregation...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-206485499
Yeah, this could be tough if the problem is really a protobuf difference. You need to update the title of this PR. See the link above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207042981
That was my fear as well. Right now, we are adding the kinesis-asl-assembly jar via spark-defaults.conf.
```
spark.driver.extraClassPath /Users/brianoneill/tools/spark/spark-streaming-kinesis-asl-assembly_2.10-1.6.1-monetate.jar
spark.executor.extraClassPath /Users/brianoneill/tools/spark/spark-streaming-kinesis-asl-assembly_2.10-1.6.1-monetate.jar
```
That appears to work, but it could be luck. It could be that some of the 2.5.0 and 2.6.1 classes are compatible, or a 2.6.1-specific class is getting loaded first. I'm not familiar enough with how Spark constructs the classpaths.
Note also that we are running this from PySpark.
Here is a gist with the example that fails w/ KCL 1.4.0, but works with 1.6:
https://gist.github.com/boneill42/020dde814346c6b4ad0ba28406c3ea10
If you add the assembly to the classpath as above, you can run this gist with:
```
python dump.py boneill-2 my-stream https://kinesis.us-east-1.amazonaws.com us-east-1
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207448261
Right, you'll also have to update the build/deps files to reflect this. dev/test-dependencies.sh can help.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-206500850
If you're actively working on this, that's fine, but if ultimately we can't merge this change because of protobuf then the PR should be closed. It stays around as does the JIRA
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207033787
This would only affect the kinesis artifacts. But to actually use that at runtime, don't you need to relocate the classes in the assembly?
Otherwise, when you add the kinesis assembly to an app with "--packages", by default it will try to load protobuf from the parent class loader, which might still contain the old version.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207467133
@srowen ok, i'm able to run `dev/test-dependencies.sh`.
I had to manually download mvn and put it in the right place though.
The script didn't give me any errors.
In looking at the Jenkins output, it looks like it can't find mvn. (exactly what I ran into)
```
build/mvn: line 152: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.3.3/bin/mvn: No such file or directory
```
Is that just a red herring?
I can proceed and just add the `deps`, but I'm not sure how to test it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-210466238
@boneill42 Aha right. That indicates that updating the library actually entailed some significant dependency changes. I don't know if that's OK to back-port to 1.6, but it's already in master. Presumably master needs your additional changes here to resolve your issue, and you could bump to 1.6.2 while you're at it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Upgraded KCL version to support de-aggregation...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-206468663
If it turns out this will move forward, you need to make a JIRA and connect it. https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207446996
**[Test build #55346 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55346/consoleFull)** for PR 12209 at commit [`fe98fc1`](https://github.com/apache/spark/commit/fe98fc156218fb50d9c71e2be9e45037f4ba8737).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-206511603
Here is the full stack:
```
Caused by: java.lang.NoClassDefFoundError: com/google/protobuf/ProtocolStringList
at com.amazonaws.services.kinesis.clientlibrary.types.UserRecord.deaggregate(UserRecord.java:235)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:146)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:24)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.google.protobuf.ProtocolStringList
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
```
I'll see if I can perform surgery at the kinesis-asl level and leave the top pom alone.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12209#discussion_r58960099
--- Diff: extras/kinesis-asl-assembly/pom.xml ---
@@ -142,6 +147,15 @@
<include>*:*</include>
</includes>
</artifactSet>
+ <relocations>
+ <relocation>
+ <pattern>com.google.protobuf</pattern>
+ <shadedPattern>kcl.protobuf</shadedPattern>
--- End diff --
to be consistent, the package name should be `${spark.shade.packageName}.kcl.protobuf`(or `kinesis.protobuf`? kcl is not immediately obvious).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-208770702
@boneill42 if you'll reopen vs master I'll merge it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-218538444
New PR available here:
https://github.com/apache/spark/pull/13054
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-210462041
@srowen it looks like master is already migrated to kinesis consumer library v 1.6.2:
https://github.com/apache/spark/blob/master/pom.xml#L150
Can that be backported safely to 1.6.x?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207472451
Locally, i'm okay.
I'm seeing the 3.3.3 error in the Jenkins build:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55346/console
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by boneill42 <gi...@git.apache.org>.
Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207479286
ah, got it. I will re-open a new against master. (and reference this one)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-14421 : Upgrade Kinesis Client Library (...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207133156
Just a naming suggestion but otherwise this LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org