You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Viral Bajaria <vi...@gmail.com> on 2014/11/26 20:21:27 UTC
issue with hive wide tables/views
Hi,
I have a table which ended up having 3K+ columns. The building of the table
wasn't that painful, but the part where things suck is when creating VIEWs
on top of that table.
1 of the views that I want to create needs complex operation and references
a ton of columns or almost all of the columns.
When applying this view to hive, it takes over 25 minutes for the view
definition to get applied. Acceptable if the view didn't need frequent
updates, but not acceptable if we plan to change the view often or have
multiple such views.
So the questions:
1) Should it take so long for hive to create a view that has so many
columns ? If not, should we open a JIRA and investigate this issue ?
2) The underlying tables are CSV (raw data) or ORC (after some
processing)... would we benefit if we change it from 3K+ columns to a
single column containing List<Object> column or Map<String, Object> for all
the values and then use the required columns
We are on Hive 0.13.0 and our metastore is backed by MariaDB 10
Thanks,
Viral
Re: issue with hive wide tables/views
Posted by Viral Bajaria <vi...@gmail.com>.
Any help will be appreciate here.
This issue becomes a bigger pain when you have a VIEW referencing another
VIEW(s) which have 1000s of columns.
It seems the generation of the query plan has some un-optimized code path
when there are 1000s of columns.
A jstack of a running process ( > 30 minutes ) shows this:
https://gist.github.com/vbajaria/2b46eb015eb5f97954fc
I ran jstack multiple times on the running process and everytime the stack
trace of the SemanticAnalyzer propped up with the same results, hence I am
guessing that the underlying issue could be in there.
Let me know if any more details are needed to get any help on this. Will it
benefit if I reached out to the dev list for this ?
Thanks,
Viral
On Wed, Nov 26, 2014 at 11:21 AM, Viral Bajaria <vi...@gmail.com>
wrote:
> Hi,
>
> I have a table which ended up having 3K+ columns. The building of the
> table wasn't that painful, but the part where things suck is when creating
> VIEWs on top of that table.
>
> 1 of the views that I want to create needs complex operation and
> references a ton of columns or almost all of the columns.
>
> When applying this view to hive, it takes over 25 minutes for the view
> definition to get applied. Acceptable if the view didn't need frequent
> updates, but not acceptable if we plan to change the view often or have
> multiple such views.
>
> So the questions:
> 1) Should it take so long for hive to create a view that has so many
> columns ? If not, should we open a JIRA and investigate this issue ?
> 2) The underlying tables are CSV (raw data) or ORC (after some
> processing)... would we benefit if we change it from 3K+ columns to a
> single column containing List<Object> column or Map<String, Object> for all
> the values and then use the required columns
>
> We are on Hive 0.13.0 and our metastore is backed by MariaDB 10
>
> Thanks,
> Viral
>
>
Issue with windowing function UDAF registeration
Posted by wesley dias <we...@outlook.com>.
Hi,
While executing a simple select query using
a custom windowing UDAF I created I am constantly running into this error.
Error: java.lang.RuntimeException: Error in configuring
object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:409)
at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at
java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at
org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.reflect.InvocationTargetException
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 9 more
Caused by: java.lang.RuntimeException: Reduce operator
initialization failed
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:173)
... 14 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:647)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getWindowFunctionInfo(FunctionRegistry.java:1875)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.streamingPossible(WindowingTableFunction.java:150)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.setCanAcceptInputAsStream(WindowingTableFunction.java:221)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.initializeStreaming(WindowingTableFunction.java:266)
at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.initializeStreaming(PTFOperator.java:292)
at
org.apache.hadoop.hive.ql.exec.PTFOperator.initializeOp(PTFOperator.java:86)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:460)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:416)
at
org.apache.hadoop.hive.ql.exec.ExtractOperator.initializeOp(ExtractOperator.java:40)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:166)
... 14 more
Just wanted to check if any of you have faced this earlier. Also
when I try to run the Custom UDAF on another server it works fine. The only
difference I can see it that the hive version I am using on my local machine is
0.13.1 where it is working and on the other machine it is 0.13.0 where I see the above mentioned error. I am not sure if this was a bug
which was fixed in the later release but I just wanted to confirm the same.
Regards,
Wesley
Re: Issue with windowing function UDAF registeration
Posted by Jason Dere <jd...@hortonworks.com>.
Created HIVE-9073 (with patch) for this.
On Dec 4, 2014, at 3:40 PM, Jason Dere <jd...@hortonworks.com> wrote:
> Hi Wesley,
>
> Sorry for the late response on this .. is it true you saw this work on a different version (later) of Hive? Based on what I think should be the problem, this would still be an issue in later versions of Hive.
>
> Looks like the error may be occurring because the Hive is trying to look up the UDF by name during the UDF initialization in the reduce task. Ideally this lookup should only be happening during the compilation phase and not during the map/reduce tasks. This looks like it works ok for built-in windowing UDFs (added to the FunctionRegistry), but custom UDFs are hitting some other logic that really should only happen during compilation. We would have to fix the way WindowingTableFunction is doing its initialization for this to work with UDFs added with ADD TEMP FUNCTION.
>
> If you're building your own Hive, a workaround would be to add your custom windowing UDAF to the FunctionRegistry.
>
>
>
> On Nov 26, 2014, at 7:35 PM, wesley dias <we...@outlook.com> wrote:
>
>> Hi,
>>
>> While executing a simple select query using a custom windowing UDAF I created I am constantly running into this error.
>>
>> Error: java.lang.RuntimeException: Error in configuring object
>> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
>> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
>> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
>> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:409)
>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>> Caused by: java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
>> ... 9 more
>> Caused by: java.lang.RuntimeException: Reduce operator initialization failed
>> at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:173)
>> ... 14 more
>> Caused by: java.lang.NullPointerException
>> at org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:647)
>> at org.apache.hadoop.hive.ql.exec.FunctionRegistry.getWindowFunctionInfo(FunctionRegistry.java:1875)
>> at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.streamingPossible(WindowingTableFunction.java:150)
>> at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.setCanAcceptInputAsStream(WindowingTableFunction.java:221)
>> at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.initializeStreaming(WindowingTableFunction.java:266)
>> at org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.initializeStreaming(PTFOperator.java:292)
>> at org.apache.hadoop.hive.ql.exec.PTFOperator.initializeOp(PTFOperator.java:86)
>> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
>> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:460)
>> at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:416)
>> at org.apache.hadoop.hive.ql.exec.ExtractOperator.initializeOp(ExtractOperator.java:40)
>> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
>> at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:166)
>> ... 14 more
>>
>> Just wanted to check if any of you have faced this earlier. Also when I try to run the Custom UDAF on another server it works fine. The only difference I can see it that the hive version I am using on my local machine is 0.13.1 where it is working and on the other machine it is 0.13.0 where I see the above mentioned error. I am not sure if this was a bug which was fixed in the later release but I just wanted to confirm the same.
>>
>> Regards,
>>
>> Wesley
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Issue with windowing function UDAF registeration
Posted by Jason Dere <jd...@hortonworks.com>.
Hi Wesley,
Sorry for the late response on this .. is it true you saw this work on a different version (later) of Hive? Based on what I think should be the problem, this would still be an issue in later versions of Hive.
Looks like the error may be occurring because the Hive is trying to look up the UDF by name during the UDF initialization in the reduce task. Ideally this lookup should only be happening during the compilation phase and not during the map/reduce tasks. This looks like it works ok for built-in windowing UDFs (added to the FunctionRegistry), but custom UDFs are hitting some other logic that really should only happen during compilation. We would have to fix the way WindowingTableFunction is doing its initialization for this to work with UDFs added with ADD TEMP FUNCTION.
If you're building your own Hive, a workaround would be to add your custom windowing UDAF to the FunctionRegistry.
On Nov 26, 2014, at 7:35 PM, wesley dias <we...@outlook.com> wrote:
> Hi,
>
> While executing a simple select query using a custom windowing UDAF I created I am constantly running into this error.
>
> Error: java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:409)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 9 more
> Caused by: java.lang.RuntimeException: Reduce operator initialization failed
> at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:173)
> ... 14 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:647)
> at org.apache.hadoop.hive.ql.exec.FunctionRegistry.getWindowFunctionInfo(FunctionRegistry.java:1875)
> at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.streamingPossible(WindowingTableFunction.java:150)
> at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.setCanAcceptInputAsStream(WindowingTableFunction.java:221)
> at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.initializeStreaming(WindowingTableFunction.java:266)
> at org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.initializeStreaming(PTFOperator.java:292)
> at org.apache.hadoop.hive.ql.exec.PTFOperator.initializeOp(PTFOperator.java:86)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:460)
> at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:416)
> at org.apache.hadoop.hive.ql.exec.ExtractOperator.initializeOp(ExtractOperator.java:40)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
> at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:166)
> ... 14 more
>
> Just wanted to check if any of you have faced this earlier. Also when I try to run the Custom UDAF on another server it works fine. The only difference I can see it that the hive version I am using on my local machine is 0.13.1 where it is working and on the other machine it is 0.13.0 where I see the above mentioned error. I am not sure if this was a bug which was fixed in the later release but I just wanted to confirm the same.
>
> Regards,
>
> Wesley
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
RE: Issue with windowing function UDAF registeration
Posted by wesley dias <we...@outlook.com>.
Hi Karthik,
Thanks for the below response. But I dont think the issue is related to empty table. Even if I try tocreate an alias for a inbuilt windowing function for e.g. Last_Value
Create temporary function mylastval as 'org.apache.hadoop.hive.ql.udf.generic.GenericUDAFLastValue';
And i try using this alias in a select query I get the below mentioned error.Like mentioned below I observed this error on the hive version 0.13.0.2 on HDP 2.2
Regards,
Wesley
From: karthiksrivasthava@gmail.com
Subject: Re: Issue with windowing function UDAF registeration
Date: Wed, 26 Nov 2014 23:12:33 -0600
To: user@hive.apache.org
You might be running this query on empty table .. Windowing function throw error with null input and this is fixed in 0.14 hive version
Thanks KarthikOn Nov 26, 2014, at 21:35, wesley dias <we...@outlook.com> wrote:
Hi,
While executing a simple select query using
a custom windowing UDAF I created I am constantly running into this error.
Error: java.lang.RuntimeException: Error in configuring
object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:409)
at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at
java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at
org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.reflect.InvocationTargetException
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 9 more
Caused by: java.lang.RuntimeException: Reduce operator
initialization failed
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:173)
... 14 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:647)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getWindowFunctionInfo(FunctionRegistry.java:1875)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.streamingPossible(WindowingTableFunction.java:150)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.setCanAcceptInputAsStream(WindowingTableFunction.java:221)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.initializeStreaming(WindowingTableFunction.java:266)
at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.initializeStreaming(PTFOperator.java:292)
at
org.apache.hadoop.hive.ql.exec.PTFOperator.initializeOp(PTFOperator.java:86)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:460)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:416)
at
org.apache.hadoop.hive.ql.exec.ExtractOperator.initializeOp(ExtractOperator.java:40)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:166)
... 14 more
Just wanted to check if any of you have faced this earlier. Also
when I try to run the Custom UDAF on another server it works fine. The only
difference I can see it that the hive version I am using on my local machine is
0.13.1 where it is working and on the other machine it is 0.13.0 where I see the above mentioned error. I am not sure if this was a bug
which was fixed in the later release but I just wanted to confirm the same.
Regards,
Wesley
RE: Issue with windowing function UDAF registeration
Posted by wesley dias <we...@outlook.com>.
Hi Karthik,
Thanks for the below response. But I dont think the issue is related to empty table. Even if I try tocreate an alias for a inbuilt windowing function for e.g. Last_Value
Create temporary function mylastval as 'org.apache.hadoop.hive.ql.udf.generic.GenericUDAFLastValue';
And i try using this alias in a select query I get the below mentioned error.Like mentioned below I observed this error on the hive version 0.13.0.2 on HDP 2.2
Regards,
Wesley
From: karthiksrivasthava@gmail.com
Subject: Re: Issue with windowing function UDAF registeration
Date: Wed, 26 Nov 2014 23:12:33 -0600
To: user@hive.apache.org
You might be running this query on empty table .. Windowing function throw error with null input and this is fixed in 0.14 hive version
Thanks KarthikOn Nov 26, 2014, at 21:35, wesley dias <we...@outlook.com> wrote:
Hi,
While executing a simple select query using
a custom windowing UDAF I created I am constantly running into this error.
Error: java.lang.RuntimeException: Error in configuring
object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:409)
at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at
java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at
org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.reflect.InvocationTargetException
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 9 more
Caused by: java.lang.RuntimeException: Reduce operator
initialization failed
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:173)
... 14 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:647)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getWindowFunctionInfo(FunctionRegistry.java:1875)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.streamingPossible(WindowingTableFunction.java:150)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.setCanAcceptInputAsStream(WindowingTableFunction.java:221)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.initializeStreaming(WindowingTableFunction.java:266)
at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.initializeStreaming(PTFOperator.java:292)
at
org.apache.hadoop.hive.ql.exec.PTFOperator.initializeOp(PTFOperator.java:86)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:460)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:416)
at
org.apache.hadoop.hive.ql.exec.ExtractOperator.initializeOp(ExtractOperator.java:40)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:166)
... 14 more
Just wanted to check if any of you have faced this earlier. Also
when I try to run the Custom UDAF on another server it works fine. The only
difference I can see it that the hive version I am using on my local machine is
0.13.1 where it is working and on the other machine it is 0.13.0 where I see the above mentioned error. I am not sure if this was a bug
which was fixed in the later release but I just wanted to confirm the same.
Regards,
Wesley
Re: Issue with windowing function UDAF registeration
Posted by Karthiksrivasthava <ka...@gmail.com>.
You might be running this query on empty table .. Windowing function throw error with null input and this is fixed in 0.14 hive version
Thanks
Karthik
> On Nov 26, 2014, at 21:35, wesley dias <we...@outlook.com> wrote:
>
> Hi,
>
>
>
> While executing a simple select query using a custom windowing UDAF I created I am constantly running into this error.
>
>
>
> Error: java.lang.RuntimeException: Error in configuring object
>
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
>
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
>
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
>
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:409)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>
> Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
>
> ... 9 more
>
> Caused by: java.lang.RuntimeException: Reduce operator initialization failed
>
> at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:173)
>
> ... 14 more
>
> Caused by: java.lang.NullPointerException
>
> at org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:647)
>
> at org.apache.hadoop.hive.ql.exec.FunctionRegistry.getWindowFunctionInfo(FunctionRegistry.java:1875)
>
> at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.streamingPossible(WindowingTableFunction.java:150)
>
> at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.setCanAcceptInputAsStream(WindowingTableFunction.java:221)
>
> at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.initializeStreaming(WindowingTableFunction.java:266)
>
> at org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.initializeStreaming(PTFOperator.java:292)
>
> at org.apache.hadoop.hive.ql.exec.PTFOperator.initializeOp(PTFOperator.java:86)
>
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
>
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:460)
>
> at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:416)
>
> at org.apache.hadoop.hive.ql.exec.ExtractOperator.initializeOp(ExtractOperator.java:40)
>
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
>
> at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:166)
>
> ... 14 more
>
>
>
> Just wanted to check if any of you have faced this earlier. Also when I try to run the Custom UDAF on another server it works fine. The only difference I can see it that the hive version I am using on my local machine is 0.13.1 where it is working and on the other machine it is 0.13.0 where I see the above mentioned error. I am not sure if this was a bug which was fixed in the later release but I just wanted to confirm the same.
>
>
>
> Regards,
>
>
>
> Wesley
Issue with windowing function UDAF registeration
Posted by wesley dias <we...@outlook.com>.
Hi,
While executing a simple select query using
a custom windowing UDAF I created I am constantly running into this error.
Error: java.lang.RuntimeException: Error in configuring
object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:409)
at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at
java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at
org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.reflect.InvocationTargetException
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 9 more
Caused by: java.lang.RuntimeException: Reduce operator
initialization failed
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:173)
... 14 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:647)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getWindowFunctionInfo(FunctionRegistry.java:1875)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.streamingPossible(WindowingTableFunction.java:150)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.setCanAcceptInputAsStream(WindowingTableFunction.java:221)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.initializeStreaming(WindowingTableFunction.java:266)
at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.initializeStreaming(PTFOperator.java:292)
at
org.apache.hadoop.hive.ql.exec.PTFOperator.initializeOp(PTFOperator.java:86)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:460)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:416)
at
org.apache.hadoop.hive.ql.exec.ExtractOperator.initializeOp(ExtractOperator.java:40)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:166)
... 14 more
Just wanted to check if any of you have faced this earlier. Also
when I try to run the Custom UDAF on another server it works fine. The only
difference I can see it that the hive version I am using on my local machine is
0.13.1 where it is working and on the other machine it is 0.13.0 where I see the above mentioned error. I am not sure if this was a bug
which was fixed in the later release but I just wanted to confirm the same.
Regards,
Wesley