You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by trystanleftwich <gi...@git.apache.org> on 2016/05/01 19:34:40 UTC
[GitHub] spark pull request: [SPARK-15046] When running hive-thriftserver w...
GitHub user trystanleftwich opened a pull request:
https://github.com/apache/spark/pull/12824
[SPARK-15046] When running hive-thriftserver with yarn on a secure
## What changes were proposed in this pull request?
Changed it so when you ask for the variable
"spark.yarn.token.renewal.interval" from sparkConf you get it as Time
instead of Long as thats how its stored.
## How was this patch tested?
Built from source and ran Hive-thriftserver against a secure cluster.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/trystanleftwich/spark SPARK-15046
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12824.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12824
----
commit 1c48d0714cfcd9d32692ef480b5394f4fe4d854b
Author: trystanleftwich <tr...@atscale.com>
Date: 2016-05-01T19:22:30Z
[SPARK-15046] When running hive-thriftserver with yarn on a secure
cluster the workers fail with java.lang.NumberFormatException
Changed it so when you ask for the variable
"spark.yarn.token.renewal.interval" from sparkConf you get it as Time
instead of Long as thats how its stored.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #12824: [SPARK-15046] When running hive-thriftserver with...
Posted by trystanleftwich <gi...@git.apache.org>.
Github user trystanleftwich closed the pull request at:
https://github.com/apache/spark/pull/12824
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12824: [SPARK-15046] When running hive-thriftserver with yarn o...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/12824
Could you update the title to match what the change is about? It doesn't say anything currently.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12824: [SPARK-15046] When running hive-thriftserver with yarn o...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/12824
@trystanleftwich could you make the changes suggested above? thx
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15046] When running hive-thriftserver w...
Posted by trystanleftwich <gi...@git.apache.org>.
Github user trystanleftwich commented on the pull request:
https://github.com/apache/spark/pull/12824#issuecomment-222153725
@tgravescs This was the default value that was causing this error.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15046] When running hive-thriftserver w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12824#issuecomment-221958607
**[Test build #59397 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59397/consoleFull)** for PR 12824 at commit [`1c48d07`](https://github.com/apache/spark/commit/1c48d0714cfcd9d32692ef480b5394f4fe4d854b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12824: [SPARK-15046] When running hive-thriftserver with yarn o...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/12824
@tgravescs the problem is this code in Client.scala:
sparkConf.set(TOKEN_RENEWAL_INTERVAL, renewalInterval)
That will write the value to the config with the `ms` suffix. I think it would be better to move the `TOKEN_RENEWAL_INTERVAL` definition to the core module's config list, so that the code that reads it can also just use the constant instead of using its own parsing code. It shouldn't need a default either, since that config is set when it's needed (which why the config entry is defined as optional).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15046] When running hive-thriftserver w...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/12824#issuecomment-221958301
@trystanleftwich to clarify you get this exception because you are specifying the config in ms, or the default value is causing this exception?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15046] When running hive-thriftserver w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12824#issuecomment-221987762
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59397/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15046] When running hive-thriftserver w...
Posted by wsu13 <gi...@git.apache.org>.
Github user wsu13 commented on the pull request:
https://github.com/apache/spark/pull/12824#issuecomment-221759878
I ran into the same exception. I use 2.0.0-preview releases. And I ran spark-shell on yarn.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12824: [SPARK-15046] When running hive-thriftserver with yarn o...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/12824
@trystanleftwich are you planning to update this PR? Otherwise I'll fix this up and send a separate PR. If I don't hear from you until tomorrow I'll do that, since we need to fix this for 2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12824: [SPARK-15046] When running hive-thriftserver with yarn o...
Posted by trystanleftwich <gi...@git.apache.org>.
Github user trystanleftwich commented on the issue:
https://github.com/apache/spark/pull/12824
@vanzin please take it, I don't have time to finish this off. Apologies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12824: [SPARK-15046] When running hive-thriftserver with yarn o...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/12824
sorry I haven't gotten back to this to understand it. do you know if something changed such that it now comes out with the ms (86400079ms) in 2.0? When I did a quick look it didn't look like (24 hours).toMillis would return that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15046] When running hive-thriftserver w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12824#issuecomment-216066967
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15046] When running hive-thriftserver w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12824#issuecomment-221987759
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15046] When running hive-thriftserver w...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/12824#issuecomment-221958395
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15046] When running hive-thriftserver w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12824#issuecomment-221987455
**[Test build #59397 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59397/consoleFull)** for PR 12824 at commit [`1c48d07`](https://github.com/apache/spark/commit/1c48d0714cfcd9d32692ef480b5394f4fe4d854b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org