You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by pronix <gi...@git.apache.org> on 2015/10/22 09:59:44 UTC
[GitHub] spark pull request: update twitter4j to 4.0.4 version
GitHub user pronix opened a pull request:
https://github.com/apache/spark/pull/9221
update twitter4j to 4.0.4 version
update twitter4j to 4.0.4 version
https://issues.apache.org/jira/browse/SPARK-11245
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pronix/spark twitter4j_update
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9221.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9221
----
commit f82a64832e75da04bb2ef83f645352f5a7e93ab1
Author: dima <pr...@gmail.com>
Date: 2015-10-22T07:55:52Z
update twitter4j to 4.0.4 version
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: update twitter4j to 4.0.4 version
Posted by pronix <gi...@git.apache.org>.
Github user pronix commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-150155800
@steveloughran you mean require something change ? or just potential issue ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-151068777
No, I presume that's what led @pronix to the changes here? @pronix did you try this with some real data? Yeah we don't have tests here anyway, and I'd hope that it continues to work if the API is used in the same way, but at least observing it working once is important.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-150835772
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: update twitter4j to 4.0.4 version
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-150151459
Looking @ the twitter4j poms, their dependencies pull in a com.googlecode.kryo -given the kryo pain its a possible troublespot. However, that com.googlecode release is so old its in a different package from the com.esotericsoftware-packaged releases
Various modules seems to refer to a different junit (3.8.2 in http-2, 4.10 in core, ...), but those are test scoped only. Log4j 1.2.17; SLF4J 1.7.12, commons-logging 1.1 -nothing incompatible with the spark versions, afaik.
main [pom.xml](https://github.com/yusuke/twitter4j/blob/master/twitter4j-core/pom.xml)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: update twitter4j to 4.0.4 version
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-150177327
Yeah, we would need more info here, to update across major releases. Because this is just an isolated 'leaf' package, we can probably tolerate more change there. But, at least, can you summarize any incompatible changes or siginficant ones, and more importantly key changes in the dependency tree?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: update twitter4j to 4.0.4 version
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-150138607
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-150727437
There arent any real testing for twitter, as we cannot really send and receive data from twitter. So the tests passing does prove nothing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-151425243
you are right. I'd probably have to do it by releasing an upstream artifact which will wait until run on your jenkins instance before posting your credentials to a new s3n bucket which your client will also tag as world readable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-151103830
Adding some tests here could be an interesting little project. the main thing to do is to have some config file in test/resources, which git is set up to ignore; if set use that to trigger the test run. We do this in hadoop for testing S3 & swift. There's still the issue that jenkins skips the work, so someone has to manually do the (slow) runs before each patch, but at least there's some automation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9221
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: update twitter4j to 4.0.4 version
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-150296842
slf4j & log4j are in sync with Spark 1.6. I can't actually see an explicit ref to commons-logging in the spark poms (just lots of exclusions of the hadoop one), but some indirect references come in at v 1.1.3.
That leaves kryo. Precisely because this is a very old import with different package names, this isn't going to be as fraught as trouble as the main kryo package.
mvn repo [says its just for testing anyway](http://mvnrepository.com/artifact/org.twitter4j/twitter4j-core/4.0.4)
No sign of the other troublespots: guava, jackson, jetty...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by pronix <gi...@git.apache.org>.
Github user pronix commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-150568906
all tests are passed :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: update twitter4j to 4.0.4 version
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-150201234
Yes, but if you're proposing the change I believe you need to investigate the implications. This doesn't address dependency changes which Steve already alluded to.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-150353314
Seems pretty reasonable then. Let's see what tests say.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-151291754
...so if I submit a patch which modifies that package and runs a test which prints your env vars to stdout, I've got your keys.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: update twitter4j to 4.0.4 version
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-150177373
@pronix also you need to look at https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark and update the title, etc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-150358196
**[Test build #1943 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1943/consoleFull)** for PR 9221 at commit [`f82a648`](https://github.com/apache/spark/commit/f82a64832e75da04bb2ef83f645352f5a7e93ab1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-151276071
If you use env vars, you'd better make sure they don't ever leak into test reports (e.g no logging of env vars in an executor)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-150353292
**[Test build #1943 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1943/consoleFull)** for PR 9221 at commit [`f82a648`](https://github.com/apache/spark/commit/f82a64832e75da04bb2ef83f645352f5a7e93ab1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-151298462
There are things like permissions and limits :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-151281735
Yeah, gotta make sure of that.
And i forgot to mention, those variables are set in our Jenkins configuration as secrets which gets injected as env vars automatically. So it stays safe within the realm of Jenkins admins. There is also an environmental flag called "ENABLE_KINESIS_TESTS" which need to be explicitly set to turn on the tests. The testing harness in Spark enables it only for PRs where SPARK/external/kinesis-asl is modified in the PR. Otherwise it stays unset and no tests are run, as we DO NOT WANT Kinesis tests to be running by default (cost, etc.), ONLY when the kinesis code is changed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: update twitter4j to 4.0.4 version
Posted by pronix <gi...@git.apache.org>.
Github user pronix commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-150200410
@srowen Official documentation about upgrade http://blog.twitter4j.org/2014/03/13/69
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-151274724
Actually, Kinesis tests had the same problem, and i fixed it by having a
this env variable based enabling of the tests - if the env variable with
AWS configs is set then the test kick in. It would be good to add that for
twitter as well.
On Mon, Oct 26, 2015 at 4:18 AM, Steve Loughran <no...@github.com>
wrote:
> Adding some tests here could be an interesting little project. the main
> thing to do is to have some config file in test/resources, which git is set
> up to ignore; if set use that to trigger the test run. We do this in hadoop
> for testing S3 & swift. There's still the issue that jenkins skips the
> work, so someone has to manually do the (slow) runs before each patch, but
> at least there's some automation.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/9221#issuecomment-151103830>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11245] update twitter4j to 4.0.4 versio...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/9221#issuecomment-151025081
@srowen Did you test it by actually passing data?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org