You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Xiaoyong Zhu <xi...@microsoft.com> on 2015/04/17 15:44:03 UTC
UDF cannot be found when the query is submitted via templeton
Hi experts
I am trying to use an UDF (I have already put that in the metastore using CREATE FUNCTION) as following.
select count(FindPattern(s_sitename)) AS testcol from weblogs;
However, when I tried to use the UDF from WebHCat (i.e. submit the above command via WebHCat), the job always fails saying
Added [/tmp/2cb22c27-72d3-4b41-aea0-655df1192872_resources/HiveUDF.jar] to class path
Added resources: [hdfs://PATHTOFOLDER/Portal-Queries/HiveUDF.jar]
FAILED: SemanticException [Error 10011]: Line 1:13 Invalid function FindPattern
If I execute this command through Hive CLI (through hive -f file or execute it in the interactive shell) the statement above works. From the log I can see the jar file is added but it seems the function cannot be found. Can someone help to share some thoughts on this issue?
Btw, the create function statement is as following (changing the hdfs URI to full path does not work either):
CREATE FUNCTION FindPattern AS 'HiveUDF.FindPattern' USING JAR 'hdfs:///UDFFolder/HiveUDF.jar';
Thanks in advance!
Xiaoyong
RE: UDF cannot be found when the query is submitted via templeton
Posted by Xiaoyong Zhu <xi...@microsoft.com>.
What we do is to upload the script first into HDFS, then use the "file" option in WebHCat to submit the queries - so I think the REST call should not matter... right?
Xiaoyong
From: Eugene Koifman [mailto:ekoifman@hortonworks.com]
Sent: Monday, April 20, 2015 10:32 PM
To: user@hive.apache.org
Subject: Re: UDF cannot be found when the query is submitted via templeton
can you give the complete REST call you are making to submit the query?
From: Xiaoyong Zhu <xi...@microsoft.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Sunday, April 19, 2015 at 8:23 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: RE: UDF cannot be found when the query is submitted via templeton
No, it doesn't...
What surprises me is that for HDInsight (Hadoop on Azure) which is using Azure BLOB storage, using
ADD JAR wasb:///test/HiveUDF.jar;
CREATE TEMPORARY FUNCTION FindPat as 'HiveUDF.FindPattern'
select count(FindPat(columnname)) from table1;
would work. However, for my own cluster,
ADD JAR hdfs:///test/HiveUDF.jar;
CREATE TEMPORARY FUNCTION FindPat as 'HiveUDF.FindPattern'
select count(FindPat(columnname)) from table1;
does not work...
Xiaoyong
From: Jason Dere [mailto:jdere@hortonworks.com]
Sent: Saturday, April 18, 2015 1:37 AM
To: <us...@hive.apache.org>>
Subject: Re: UDF cannot be found when the query is submitted via templeton
Does fully qualifying the function name (HiveUDF.FindPattern()) in the query help here?
On Apr 17, 2015, at 6:44 AM, Xiaoyong Zhu <xi...@microsoft.com>> wrote:
Hi experts
I am trying to use an UDF (I have already put that in the metastore using CREATE FUNCTION) as following.
select count(FindPattern(s_sitename)) AS testcol from weblogs;
However, when I tried to use the UDF from WebHCat (i.e. submit the above command via WebHCat), the job always fails saying
Added [/tmp/2cb22c27-72d3-4b41-aea0-655df1192872_resources/HiveUDF.jar] to class path
Added resources: [hdfs://PATHTOFOLDER/Portal-Queries/HiveUDF.jar]
FAILED: SemanticException [Error 10011]: Line 1:13 Invalid function FindPattern
If I execute this command through Hive CLI (through hive -f file or execute it in the interactive shell) the statement above works. From the log I can see the jar file is added but it seems the function cannot be found. Can someone help to share some thoughts on this issue?
Btw, the create function statement is as following (changing the hdfs URI to full path does not work either):
CREATE FUNCTION FindPattern AS 'HiveUDF.FindPattern' USING JAR'hdfs:///UDFFolder/HiveUDF.jar'<hdfs://UDFFolder/HiveUDF.jar'>;
Thanks in advance!
Xiaoyong
Re: UDF cannot be found when the query is submitted via templeton
Posted by Eugene Koifman <ek...@hortonworks.com>.
can you give the complete REST call you are making to submit the query?
From: Xiaoyong Zhu <xi...@microsoft.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Sunday, April 19, 2015 at 8:23 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: RE: UDF cannot be found when the query is submitted via templeton
No, it doesn't...
What surprises me is that for HDInsight (Hadoop on Azure) which is using Azure BLOB storage, using
ADD JAR wasb:///test/HiveUDF.jar;
CREATE TEMPORARY FUNCTION FindPat as 'HiveUDF.FindPattern'
select count(FindPat(columnname)) from table1;
would work. However, for my own cluster,
ADD JAR hdfs:///test/HiveUDF.jar;
CREATE TEMPORARY FUNCTION FindPat as 'HiveUDF.FindPattern'
select count(FindPat(columnname)) from table1;
does not work...
Xiaoyong
From: Jason Dere [mailto:jdere@hortonworks.com]
Sent: Saturday, April 18, 2015 1:37 AM
To: <us...@hive.apache.org>>
Subject: Re: UDF cannot be found when the query is submitted via templeton
Does fully qualifying the function name (HiveUDF.FindPattern()) in the query help here?
On Apr 17, 2015, at 6:44 AM, Xiaoyong Zhu <xi...@microsoft.com>> wrote:
Hi experts
I am trying to use an UDF (I have already put that in the metastore using CREATE FUNCTION) as following.
select count(FindPattern(s_sitename)) AS testcol from weblogs;
However, when I tried to use the UDF from WebHCat (i.e. submit the above command via WebHCat), the job always fails saying
Added [/tmp/2cb22c27-72d3-4b41-aea0-655df1192872_resources/HiveUDF.jar] to class path
Added resources: [hdfs://PATHTOFOLDER/Portal-Queries/HiveUDF.jar]
FAILED: SemanticException [Error 10011]: Line 1:13 Invalid function FindPattern
If I execute this command through Hive CLI (through hive -f file or execute it in the interactive shell) the statement above works. From the log I can see the jar file is added but it seems the function cannot be found. Can someone help to share some thoughts on this issue?
Btw, the create function statement is as following (changing the hdfs URI to full path does not work either):
CREATE FUNCTION FindPattern AS 'HiveUDF.FindPattern' USING JAR'hdfs:///UDFFolder/HiveUDF.jar'<hdfs://UDFFolder/HiveUDF.jar'>;
Thanks in advance!
Xiaoyong
RE: UDF cannot be found when the query is submitted via templeton
Posted by Xiaoyong Zhu <xi...@microsoft.com>.
No, it doesn't...
What surprises me is that for HDInsight (Hadoop on Azure) which is using Azure BLOB storage, using
ADD JAR wasb:///test/HiveUDF.jar;
CREATE TEMPORARY FUNCTION FindPat as 'HiveUDF.FindPattern'
select count(FindPat(columnname)) from table1;
would work. However, for my own cluster,
ADD JAR hdfs:///test/HiveUDF.jar;
CREATE TEMPORARY FUNCTION FindPat as 'HiveUDF.FindPattern'
select count(FindPat(columnname)) from table1;
does not work...
Xiaoyong
From: Jason Dere [mailto:jdere@hortonworks.com]
Sent: Saturday, April 18, 2015 1:37 AM
To: <us...@hive.apache.org>
Subject: Re: UDF cannot be found when the query is submitted via templeton
Does fully qualifying the function name (HiveUDF.FindPattern()) in the query help here?
On Apr 17, 2015, at 6:44 AM, Xiaoyong Zhu <xi...@microsoft.com>> wrote:
Hi experts
I am trying to use an UDF (I have already put that in the metastore using CREATE FUNCTION) as following.
select count(FindPattern(s_sitename)) AS testcol from weblogs;
However, when I tried to use the UDF from WebHCat (i.e. submit the above command via WebHCat), the job always fails saying
Added [/tmp/2cb22c27-72d3-4b41-aea0-655df1192872_resources/HiveUDF.jar] to class path
Added resources: [hdfs://PATHTOFOLDER/Portal-Queries/HiveUDF.jar]
FAILED: SemanticException [Error 10011]: Line 1:13 Invalid function FindPattern
If I execute this command through Hive CLI (through hive -f file or execute it in the interactive shell) the statement above works. From the log I can see the jar file is added but it seems the function cannot be found. Can someone help to share some thoughts on this issue?
Btw, the create function statement is as following (changing the hdfs URI to full path does not work either):
CREATE FUNCTION FindPattern AS 'HiveUDF.FindPattern' USING JAR'hdfs:///UDFFolder/HiveUDF.jar'<hdfs://UDFFolder/HiveUDF.jar'>;
Thanks in advance!
Xiaoyong
Re: UDF cannot be found when the query is submitted via templeton
Posted by Jason Dere <jd...@hortonworks.com>.
Does fully qualifying the function name (HiveUDF.FindPattern()) in the query help here?
On Apr 17, 2015, at 6:44 AM, Xiaoyong Zhu <xi...@microsoft.com>> wrote:
Hi experts
I am trying to use an UDF (I have already put that in the metastore using CREATE FUNCTION) as following.
select count(FindPattern(s_sitename)) AS testcol from weblogs;
However, when I tried to use the UDF from WebHCat (i.e. submit the above command via WebHCat), the job always fails saying
Added [/tmp/2cb22c27-72d3-4b41-aea0-655df1192872_resources/HiveUDF.jar] to class path
Added resources: [hdfs://PATHTOFOLDER/Portal-Queries/HiveUDF.jar]
FAILED: SemanticException [Error 10011]: Line 1:13 Invalid function FindPattern
If I execute this command through Hive CLI (through hive –f file or execute it in the interactive shell) the statement above works. From the log I can see the jar file is added but it seems the function cannot be found. Can someone help to share some thoughts on this issue?
Btw, the create function statement is as following (changing the hdfs URI to full path does not work either):
CREATE FUNCTION FindPattern AS 'HiveUDF.FindPattern' USING JAR'hdfs:///UDFFolder/HiveUDF.jar';
Thanks in advance!
Xiaoyong