You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by shenh062326 <gi...@git.apache.org> on 2016/12/17 06:01:55 UTC
[GitHub] spark pull request #16324: Resolve faile to use UDF that jar file in hdfs.
GitHub user shenh062326 opened a pull request:
https://github.com/apache/spark/pull/16324
Resolve faile to use UDF that jar file in hdfs.
## What changes were proposed in this pull request?
In SparkContext, setURLStreamHandlerFactory method on URL with an instance of FsUrlStreamHandlerFactory, to prevent failed to use UDF with jar file in HDFS.
## How was this patch tested?
I have test it in my cluster.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shenh062326/spark SPARK-18910
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16324.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16324
----
commit a91b08fcbdd76e02c2a244ea4ffca726339fdba8
Author: yuling <yu...@alipay.com>
Date: 2016-12-17T05:56:53Z
Resolve faile to use UDF that jar file in hdfs.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16324: [SPARK-18910][SQL]Resolve faile to use UDF that jar file...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/16324
This is to allow using jars defined using HDFS-API, not just HDFS right? In that case it sounds like a good idea too ... but we need a test case for it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16324: [SPARK-18910][SQL]Resolve faile to use UDF that jar file...
Posted by shenh062326 <gi...@git.apache.org>.
Github user shenh062326 commented on the issue:
https://github.com/apache/spark/pull/16324
Currently\uff0cwe can create a UDF with jar in HDFS, but failed to use it.
Spark driver won't download the jar from HDFS, it only add the path to the classLoader.
If we don't support reading UDF jar from HDFS, we should download the UDF jar.
I think support reading UDF jar from HDFS is better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16324: [SPARK-18910][SQL]Resolve faile to use UDF that j...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/16324
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16324: [SPARK-18910][SQL]Resolve faile to use UDF that jar file...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16324
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16324: [SPARK-18910][SQL]Resolve faile to use UDF that jar file...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16324
Please add the description in the JIRA to the PR description. FYI, you still can edit the description after you create the PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16324: [SPARK-18910][SQL]Resolve faile to use UDF that jar file...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16324
**[Test build #70296 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70296/testReport)** for PR 16324 at commit [`a91b08f`](https://github.com/apache/spark/commit/a91b08fcbdd76e02c2a244ea4ffca726339fdba8).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16324: [SPARK-18910][SQL]Resolve faile to use UDF that jar file...
Posted by melin <gi...@git.apache.org>.
Github user melin commented on the issue:
https://github.com/apache/spark/pull/16324
in spark 2.1.0
0: jdbc:hive2://localhost:10000> add jar hdfs:///user/datacompute/datacompute-udf-1.0-
Error: java.net.MalformedURLException: unknown protocol: hdfs (state=,code=0)
0: jdbc:hive2://localhost:10000>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16324: [SPARK-18910][SQL]Resolve faile to use UDF that jar file...
Posted by shenh062326 <gi...@git.apache.org>.
Github user shenh062326 commented on the issue:
https://github.com/apache/spark/pull/16324
Should we download the UDF jar from hdfs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16324: [SPARK-18910][SQL]Resolve faile to use UDF that jar file...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16324
> This method can only be called once per JVM, so it is typically executed in a static block. This limitation means that if some other part of your program\u2014perhaps a third-party component outside your control\u2014 sets a URLStreamHandlerFactory, you won\u2019t be able to use this approach for reading data from Hadoop.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16324: [SPARK-18910][SQL]Resolve faile to use UDF that jar file...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16324
This might not be the right solution to do it, as explained above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16324: Resolve faile to use UDF that jar file in hdfs.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16324
**[Test build #70296 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70296/testReport)** for PR 16324 at commit [`a91b08f`](https://github.com/apache/spark/commit/a91b08fcbdd76e02c2a244ea4ffca726339fdba8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16324: [SPARK-18910][SQL]Resolve faile to use UDF that jar file...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16324
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70296/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16324: [SPARK-18910][SQL]Resolve faile to use UDF that jar file...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16324
First, I am not sure whether we should support reading UDF jar from HDFS.
Second, if we want to support, the best reviewers are @zsxwing @tdas They added the file [HDFSMetadataLog.scala](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala) recently
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16324: [SPARK-18910][SQL]Resolve faile to use UDF that jar file...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16324
(gentle ping @henh062326)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16324: [SPARK-18910][SQL]Resolve faile to use UDF that jar file...
Posted by shenh062326 <gi...@git.apache.org>.
Github user shenh062326 commented on the issue:
https://github.com/apache/spark/pull/16324
I\u2018m sorry, @rxin, I don't understand what you mean.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16324: [SPARK-18910][SQL]Resolve faile to use UDF that jar file...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/16324
I was saying we need to create a test case for this change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org