You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Stephen Corona <sc...@adknowledge.com> on 2009/03/17 16:47:04 UTC
NullPointerException when using join on table with map column
Hey guys. I am trying to run a query like:
select count(1), request.attributes['category_selection_method'] from request left outer join test on request.recipient_id = test.recipient_id group by request.attributes['category_selection_method'];
I can run the query WITHOUT the join and it works fine. I can also run the query with the join if I don't reference the attributes column (i.e, the map). Is there a problem with doing joins when a table has a map column?
However, when I run this query I get the following error:
java.lang.RuntimeException: org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:75)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:191)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:89)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:49)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:178)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:71)
... 3 more
Caused by: org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
at org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:178)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:189)
... 10 more
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeTypeMap.serialize(DynamicSerDeTypeMap.java:136)
at org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeFieldList.serialize(DynamicSerDeFieldList.java:249)
at org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeStructBase.serialize(DynamicSerDeStructBase.java:81)
at org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:174)
... 11 more
describe request-
request_id string
attributes map<string,string>
day string
create table for request:
create table request (
request_id
attributes map <string, string>
)
row format delimited fields terminated by '\001' collection items terminated by '\003' map keys terminated by '\002';
describe test-
recipient_id string
explain query
ABSTRACT SYNTAX TREE:
(TOK_QUERY (TOK_FROM (TOK_LEFTOUTERJOIN (TOK_TABREF request) (TOK_TABREF test) (= (TOK_COLREF request recipient_id) (TOK_COLREF test recipient_id)))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_FUNCTION count 1)) (TOK_SELEXPR ([ (TOK_COLREF request attributes) 'category_selection_method'))) (TOK_GROUPBY ([ (TOK_COLREF request attributes) 'category_selection_method'))))
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-2 depends on stages: Stage-1
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Alias -> Map Operator Tree:
test
Reduce Output Operator
key expressions:
expr: recipient_id
type: string
sort order: +
Map-reduce partition columns:
expr: recipient_id
type: string
tag: 1
value expressions:
expr: recipient_id
type: string
request
Select Operator
expressions:
expr: recipient_id
type: string
expr: attributes
type: map<string,string>
Reduce Output Operator
key expressions:
expr: 0
type: string
sort order: +
Map-reduce partition columns:
expr: 0
type: string
tag: 0
value expressions:
expr: 0
type: string
expr: 1
type: map<string,string>
Reduce Operator Tree:
Join Operator
condition map:
Left Outer Join0 to 1
condition expressions:
0 {VALUE.0} {VALUE.1}
1 {VALUE.0}
Group By Operator
aggregations:
expr: count(1)
keys:
expr: 1['category_selection_method']
type: string
mode: hash
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.mapred.SequenceFileOutputFormat
name: binary_table
Stage: Stage-2
Map Reduce
Alias -> Map Operator Tree:
/tmp/hive-root/532094399/571818099.10002
Reduce Output Operator
key expressions:
expr: 0
type: string
sort order: +
Map-reduce partition columns:
expr: 0
type: string
tag: -1
value expressions:
expr: 1
type: bigint
Reduce Operator Tree:
Group By Operator
aggregations:
expr: count(VALUE.0)
keys:
expr: KEY.0
type: string
mode: mergepartial
Select Operator
expressions:
expr: 1
type: bigint
expr: 0
type: string
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
Stage: Stage-0
Fetch Operator
limit: -1
Re: NullPointerException when using join on table with map column
Posted by Zheng Shao <zs...@gmail.com>.
Hi Stephen,
Maps are not expected to have a null value in Hive, because we use java
HashMap internally to store the Maps.
Zheng
On Tue, Mar 17, 2009 at 11:09 AM, Stephen Corona <sc...@adknowledge.com>wrote:
> Are map's allowed to have a key with a null value? Or does it expect every
> key to have a non-null value?
>
> Steve
> ________________________________________
> From: Zheng Shao [zshao9@gmail.com]
> Sent: Tuesday, March 17, 2009 1:56 PM
> To: hive-user@hadoop.apache.org
> Subject: Re: NullPointerException when using join on table with map column
>
> Hi,
>
> It seems DynamicSerde has problems with the null values somewhere. We
> are working on replacing it with LazySimpleSerDe.
>
> In the meanwhile, please use subquery and the problem should disappear.
>
> Select count(1), mykey from (Select id, mymap['mykey'] as mykey from
> tablea) tablea join tableb on tablea.id = tableb.id group by mykey;
>
> Zheng
>
>
> On 3/17/09, Stephen Corona <sc...@adknowledge.com> wrote:
> > Hey guys. I am trying to run a query like:
> >
> > select count(1), request.attributes['category_selection_method'] from
> > request left outer join test on request.recipient_id = test.recipient_id
> > group by request.attributes['category_selection_method'];
> >
> > I can run the query WITHOUT the join and it works fine. I can also run
> the
> > query with the join if I don't reference the attributes column (i.e, the
> > map). Is there a problem with doing joins when a table has a map column?
> >
> > However, when I run this query I get the following error:
> >
> > java.lang.RuntimeException: org.apache.hadoop.hive.serde2.SerDeException:
> > java.lang.NullPointerException
> > at
> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:75)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> > at org.apache.hadoop.mapred.Child.main(Child.java:155)
> > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> > org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> > at
> >
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:191)
> > at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> > at
> >
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:89)
> > at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> > at
> >
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:49)
> > at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> > at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:178)
> > at
> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:71)
> > ... 3 more
> > Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> > java.lang.NullPointerException
> > at
> >
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:178)
> > at
> >
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:189)
> > ... 10 more
> > Caused by: java.lang.NullPointerException
> > at
> >
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeTypeMap.serialize(DynamicSerDeTypeMap.java:136)
> > at
> >
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeFieldList.serialize(DynamicSerDeFieldList.java:249)
> > at
> >
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeStructBase.serialize(DynamicSerDeStructBase.java:81)
> > at
> >
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:174)
> > ... 11 more
> >
> >
> > describe request-
> >
> > request_id string
> > attributes map<string,string>
> > day string
> >
> > create table for request:
> >
> > create table request (
> > request_id
> > attributes map <string, string>
> > )
> > row format delimited fields terminated by '\001' collection items
> terminated
> > by '\003' map keys terminated by '\002';
> >
> > describe test-
> > recipient_id string
> >
> > explain query
> >
> > ABSTRACT SYNTAX TREE:
> > (TOK_QUERY (TOK_FROM (TOK_LEFTOUTERJOIN (TOK_TABREF request)
> (TOK_TABREF
> > test) (= (TOK_COLREF request recipient_id) (TOK_COLREF test
> recipient_id))))
> > (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT
> > (TOK_SELEXPR (TOK_FUNCTION count 1)) (TOK_SELEXPR ([ (TOK_COLREF request
> > attributes) 'category_selection_method'))) (TOK_GROUPBY ([ (TOK_COLREF
> > request attributes) 'category_selection_method'))))
> >
> > STAGE DEPENDENCIES:
> > Stage-1 is a root stage
> > Stage-2 depends on stages: Stage-1
> > Stage-0 is a root stage
> >
> > STAGE PLANS:
> > Stage: Stage-1
> > Map Reduce
> > Alias -> Map Operator Tree:
> > test
> > Reduce Output Operator
> > key expressions:
> > expr: recipient_id
> > type: string
> > sort order: +
> > Map-reduce partition columns:
> > expr: recipient_id
> > type: string
> > tag: 1
> > value expressions:
> > expr: recipient_id
> > type: string
> > request
> > Select Operator
> > expressions:
> > expr: recipient_id
> > type: string
> > expr: attributes
> > type: map<string,string>
> > Reduce Output Operator
> > key expressions:
> > expr: 0
> > type: string
> > sort order: +
> > Map-reduce partition columns:
> > expr: 0
> > type: string
> > tag: 0
> > value expressions:
> > expr: 0
> > type: string
> > expr: 1
> > type: map<string,string>
> > Reduce Operator Tree:
> > Join Operator
> > condition map:
> > Left Outer Join0 to 1
> > condition expressions:
> > 0 {VALUE.0} {VALUE.1}
> > 1 {VALUE.0}
> > Group By Operator
> > aggregations:
> > expr: count(1)
> > keys:
> > expr: 1['category_selection_method']
> > type: string
> > mode: hash
> > File Output Operator
> > compressed: false
> > GlobalTableId: 0
> > table:
> > input format:
> > org.apache.hadoop.mapred.SequenceFileInputFormat
> > output format:
> > org.apache.hadoop.mapred.SequenceFileOutputFormat
> > name: binary_table
> >
> > Stage: Stage-2
> > Map Reduce
> > Alias -> Map Operator Tree:
> > /tmp/hive-root/532094399/571818099.10002
> > Reduce Output Operator
> > key expressions:
> > expr: 0
> > type: string
> > sort order: +
> > Map-reduce partition columns:
> > expr: 0
> > type: string
> > tag: -1
> > value expressions:
> > expr: 1
> > type: bigint
> > Reduce Operator Tree:
> > Group By Operator
> > aggregations:
> > expr: count(VALUE.0)
> > keys:
> > expr: KEY.0
> > type: string
> > mode: mergepartial
> > Select Operator
> > expressions:
> > expr: 1
> > type: bigint
> > expr: 0
> > type: string
> > File Output Operator
> > compressed: false
> > GlobalTableId: 0
> > table:
> > input format: org.apache.hadoop.mapred.TextInputFormat
> > output format:
> > org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
> >
> > Stage: Stage-0
> > Fetch Operator
> > limit: -1
> >
> >
> >
>
> --
> Sent from Gmail for mobile | mobile.google.com
>
> Yours,
> Zheng
>
--
Yours,
Zheng
Re: NullPointerException when using join on table with map column
Posted by Zheng Shao <zs...@gmail.com>.
Hi Stephen,
Sorry for the late reply. Please do ant clean and then package. I think it's
a compilation problem.
Let us know if the problem still exists after that.
Zheng
On Tue, Apr 21, 2009 at 11:16 AM, Stephen Corona <sc...@adknowledge.com>wrote:
> Hey Zheng,
>
> I recently upgraded to the Hive trunk and now I get this error when using
> Map-type columns (even when not using joins):
>
> Example query:
>
> SELECT foo['bar'] from foobar;
>
> java.lang.AbstractMethodError:
> org.apache.hadoop.hive.ql.parse.ASTNode.getChildren()Ljava/util/List;
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:110)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:95)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:3646)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:1002)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:988)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:2940)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:3345)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:3297)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:3318)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:3297)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:3318)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:3566)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:80)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:177)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:205)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:306)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
>
> ________________________________________
> From: Zheng Shao [zshao9@gmail.com]
> Sent: Saturday, April 11, 2009 5:08 AM
> To: hive-user@hadoop.apache.org
> Subject: Re: NullPointerException when using join on table with map column
>
> Hi Stephen,
>
> I am aware of this bug - it's buried in Hive for a long time until recently
> I encountered with it again doing HIVE-266.
>
> I opened HIVE-405 for that purpose. It probably take another week or two to
> fix the problem (taking into account the time for test, code review etc).
> It's the second top priority in my list (the first one is HIVE-372).
>
> Zheng
>
> On Fri, Apr 10, 2009 at 6:37 PM, Stephen Corona <scorona@adknowledge.com
> <ma...@adknowledge.com>> wrote:
> Hey Zheng,
> I was wondering if the issue regarding map columns w/ joins has been fixed?
> I removed all null values from my map column, so all keys have a non-null
> string value. I still get an error when doing joins involving this column
> but it's in the reduce phase this time (the error message is pasted below).
> I'm wondering if there is already a JIRA for this and, if so, where is falls
> on the priority list. It's a real pain to have to use a subquery to access
> any map columns w/ a join.
>
>
> java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: Hive 2 Internal error: cannot evaluate index
> expression on string
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:169)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430)
> at org.apache.hadoop.mapred.Child.main(Child.java:155)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: Hive 2 Internal error: cannot evaluate index
> expression on string
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:451)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.createForwardJoinObject(JoinOperator.java:259)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:502)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:491)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:491)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:491)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.checkAndGenObject(JoinOperator.java:532)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:514)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:140)
> ... 2 more
> Caused by: java.lang.RuntimeException: Hive 2 Internal error: cannot
> evaluate index expression on string
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeIndexEvaluator.evaluate(ExprNodeIndexEvaluator.java:64)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:72)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:72)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:72)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:370)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:470)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:445)
> ... 11 more
>
> Steve
> ________________________________________
> From: Stephen Corona
> Sent: Tuesday, March 17, 2009 2:09 PM
> To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
> Subject: RE: NullPointerException when using join on table with map column
>
> Are map's allowed to have a key with a null value? Or does it expect every
> key to have a non-null value?
>
> Steve
> ________________________________________
> From: Zheng Shao [zshao9@gmail.com<ma...@gmail.com>]
> Sent: Tuesday, March 17, 2009 1:56 PM
> To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
> Subject: Re: NullPointerException when using join on table with map column
>
> Hi,
>
> It seems DynamicSerde has problems with the null values somewhere. We
> are working on replacing it with LazySimpleSerDe.
>
> In the meanwhile, please use subquery and the problem should disappear.
>
> Select count(1), mykey from (Select id, mymap['mykey'] as mykey from
> tablea) tablea join tableb on tablea.id<http://tablea.id> = tableb.id<
> http://tableb.id> group by mykey;
>
> Zheng
>
>
> On 3/17/09, Stephen Corona <scorona@adknowledge.com<mailto:
> scorona@adknowledge.com>> wrote:
> > Hey guys. I am trying to run a query like:
> >
> > select count(1), request.attributes['category_selection_method'] from
> > request left outer join test on request.recipient_id = test.recipient_id
> > group by request.attributes['category_selection_method'];
> >
> > I can run the query WITHOUT the join and it works fine. I can also run
> the
> > query with the join if I don't reference the attributes column (i.e, the
> > map). Is there a problem with doing joins when a table has a map column?
> >
> > However, when I run this query I get the following error:
> >
> > java.lang.RuntimeException: org.apache.hadoop.hive.serde2.SerDeException:
> > java.lang.NullPointerException
> > at
> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:75)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> > at org.apache.hadoop.mapred.Child.main(Child.java:155)
> > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> > org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> > at
> >
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:191)
> > at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> > at
> >
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:89)
> > at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> > at
> >
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:49)
> > at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> > at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:178)
> > at
> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:71)
> > ... 3 more
> > Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> > java.lang.NullPointerException
> > at
> >
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:178)
> > at
> >
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:189)
> > ... 10 more
> > Caused by: java.lang.NullPointerException
> > at
> >
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeTypeMap.serialize(DynamicSerDeTypeMap.java:136)
> > at
> >
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeFieldList.serialize(DynamicSerDeFieldList.java:249)
> > at
> >
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeStructBase.serialize(DynamicSerDeStructBase.java:81)
> > at
> >
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:174)
> > ... 11 more
> >
> >
> > describe request-
> >
> > request_id string
> > attributes map<string,string>
> > day string
> >
> > create table for request:
> >
> > create table request (
> > request_id
> > attributes map <string, string>
> > )
> > row format delimited fields terminated by '\001' collection items
> terminated
> > by '\003' map keys terminated by '\002';
> >
> > describe test-
> > recipient_id string
> >
> > explain query
> >
> > ABSTRACT SYNTAX TREE:
> > (TOK_QUERY (TOK_FROM (TOK_LEFTOUTERJOIN (TOK_TABREF request)
> (TOK_TABREF
> > test) (= (TOK_COLREF request recipient_id) (TOK_COLREF test
> recipient_id))))
> > (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT
> > (TOK_SELEXPR (TOK_FUNCTION count 1)) (TOK_SELEXPR ([ (TOK_COLREF request
> > attributes) 'category_selection_method'))) (TOK_GROUPBY ([ (TOK_COLREF
> > request attributes) 'category_selection_method'))))
> >
> > STAGE DEPENDENCIES:
> > Stage-1 is a root stage
> > Stage-2 depends on stages: Stage-1
> > Stage-0 is a root stage
> >
> > STAGE PLANS:
> > Stage: Stage-1
> > Map Reduce
> > Alias -> Map Operator Tree:
> > test
> > Reduce Output Operator
> > key expressions:
> > expr: recipient_id
> > type: string
> > sort order: +
> > Map-reduce partition columns:
> > expr: recipient_id
> > type: string
> > tag: 1
> > value expressions:
> > expr: recipient_id
> > type: string
> > request
> > Select Operator
> > expressions:
> > expr: recipient_id
> > type: string
> > expr: attributes
> > type: map<string,string>
> > Reduce Output Operator
> > key expressions:
> > expr: 0
> > type: string
> > sort order: +
> > Map-reduce partition columns:
> > expr: 0
> > type: string
> > tag: 0
> > value expressions:
> > expr: 0
> > type: string
> > expr: 1
> > type: map<string,string>
> > Reduce Operator Tree:
> > Join Operator
> > condition map:
> > Left Outer Join0 to 1
> > condition expressions:
> > 0 {VALUE.0} {VALUE.1}
> > 1 {VALUE.0}
> > Group By Operator
> > aggregations:
> > expr: count(1)
> > keys:
> > expr: 1['category_selection_method']
> > type: string
> > mode: hash
> > File Output Operator
> > compressed: false
> > GlobalTableId: 0
> > table:
> > input format:
> > org.apache.hadoop.mapred.SequenceFileInputFormat
> > output format:
> > org.apache.hadoop.mapred.SequenceFileOutputFormat
> > name: binary_table
> >
> > Stage: Stage-2
> > Map Reduce
> > Alias -> Map Operator Tree:
> > /tmp/hive-root/532094399/571818099.10002
> > Reduce Output Operator
> > key expressions:
> > expr: 0
> > type: string
> > sort order: +
> > Map-reduce partition columns:
> > expr: 0
> > type: string
> > tag: -1
> > value expressions:
> > expr: 1
> > type: bigint
> > Reduce Operator Tree:
> > Group By Operator
> > aggregations:
> > expr: count(VALUE.0)
> > keys:
> > expr: KEY.0
> > type: string
> > mode: mergepartial
> > Select Operator
> > expressions:
> > expr: 1
> > type: bigint
> > expr: 0
> > type: string
> > File Output Operator
> > compressed: false
> > GlobalTableId: 0
> > table:
> > input format: org.apache.hadoop.mapred.TextInputFormat
> > output format:
> > org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
> >
> > Stage: Stage-0
> > Fetch Operator
> > limit: -1
> >
> >
> >
>
> --
> Sent from Gmail for mobile | mobile.google.com<http://mobile.google.com>
>
> Yours,
> Zheng
>
>
>
> --
> Yours,
> Zheng
>
--
Yours,
Zheng
RE: NullPointerException when using join on table with map column
Posted by Stephen Corona <sc...@adknowledge.com>.
Hey Zheng,
I recently upgraded to the Hive trunk and now I get this error when using Map-type columns (even when not using joins):
Example query:
SELECT foo['bar'] from foobar;
java.lang.AbstractMethodError: org.apache.hadoop.hive.ql.parse.ASTNode.getChildren()Ljava/util/List;
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:110)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:95)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:3646)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:1002)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:988)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:2940)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:3345)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:3297)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:3318)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:3297)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:3318)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:3566)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:80)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:177)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:205)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:306)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
________________________________________
From: Zheng Shao [zshao9@gmail.com]
Sent: Saturday, April 11, 2009 5:08 AM
To: hive-user@hadoop.apache.org
Subject: Re: NullPointerException when using join on table with map column
Hi Stephen,
I am aware of this bug - it's buried in Hive for a long time until recently I encountered with it again doing HIVE-266.
I opened HIVE-405 for that purpose. It probably take another week or two to fix the problem (taking into account the time for test, code review etc). It's the second top priority in my list (the first one is HIVE-372).
Zheng
On Fri, Apr 10, 2009 at 6:37 PM, Stephen Corona <sc...@adknowledge.com>> wrote:
Hey Zheng,
I was wondering if the issue regarding map columns w/ joins has been fixed? I removed all null values from my map column, so all keys have a non-null string value. I still get an error when doing joins involving this column but it's in the reduce phase this time (the error message is pasted below). I'm wondering if there is already a JIRA for this and, if so, where is falls on the priority list. It's a real pain to have to use a subquery to access any map columns w/ a join.
java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Hive 2 Internal error: cannot evaluate index expression on string
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:169)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Hive 2 Internal error: cannot evaluate index expression on string
at org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:451)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
at org.apache.hadoop.hive.ql.exec.JoinOperator.createForwardJoinObject(JoinOperator.java:259)
at org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:502)
at org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:491)
at org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:491)
at org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:491)
at org.apache.hadoop.hive.ql.exec.JoinOperator.checkAndGenObject(JoinOperator.java:532)
at org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:514)
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:140)
... 2 more
Caused by: java.lang.RuntimeException: Hive 2 Internal error: cannot evaluate index expression on string
at org.apache.hadoop.hive.ql.exec.ExprNodeIndexEvaluator.evaluate(ExprNodeIndexEvaluator.java:64)
at org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:72)
at org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:72)
at org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:72)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:370)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:470)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:445)
... 11 more
Steve
________________________________________
From: Stephen Corona
Sent: Tuesday, March 17, 2009 2:09 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: NullPointerException when using join on table with map column
Are map's allowed to have a key with a null value? Or does it expect every key to have a non-null value?
Steve
________________________________________
From: Zheng Shao [zshao9@gmail.com<ma...@gmail.com>]
Sent: Tuesday, March 17, 2009 1:56 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: NullPointerException when using join on table with map column
Hi,
It seems DynamicSerde has problems with the null values somewhere. We
are working on replacing it with LazySimpleSerDe.
In the meanwhile, please use subquery and the problem should disappear.
Select count(1), mykey from (Select id, mymap['mykey'] as mykey from
tablea) tablea join tableb on tablea.id<http://tablea.id> = tableb.id<http://tableb.id> group by mykey;
Zheng
On 3/17/09, Stephen Corona <sc...@adknowledge.com>> wrote:
> Hey guys. I am trying to run a query like:
>
> select count(1), request.attributes['category_selection_method'] from
> request left outer join test on request.recipient_id = test.recipient_id
> group by request.attributes['category_selection_method'];
>
> I can run the query WITHOUT the join and it works fine. I can also run the
> query with the join if I don't reference the attributes column (i.e, the
> map). Is there a problem with doing joins when a table has a map column?
>
> However, when I run this query I get the following error:
>
> java.lang.RuntimeException: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:75)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child.main(Child.java:155)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:191)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:89)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:49)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:178)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:71)
> ... 3 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:178)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:189)
> ... 10 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeTypeMap.serialize(DynamicSerDeTypeMap.java:136)
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeFieldList.serialize(DynamicSerDeFieldList.java:249)
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeStructBase.serialize(DynamicSerDeStructBase.java:81)
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:174)
> ... 11 more
>
>
> describe request-
>
> request_id string
> attributes map<string,string>
> day string
>
> create table for request:
>
> create table request (
> request_id
> attributes map <string, string>
> )
> row format delimited fields terminated by '\001' collection items terminated
> by '\003' map keys terminated by '\002';
>
> describe test-
> recipient_id string
>
> explain query
>
> ABSTRACT SYNTAX TREE:
> (TOK_QUERY (TOK_FROM (TOK_LEFTOUTERJOIN (TOK_TABREF request) (TOK_TABREF
> test) (= (TOK_COLREF request recipient_id) (TOK_COLREF test recipient_id))))
> (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT
> (TOK_SELEXPR (TOK_FUNCTION count 1)) (TOK_SELEXPR ([ (TOK_COLREF request
> attributes) 'category_selection_method'))) (TOK_GROUPBY ([ (TOK_COLREF
> request attributes) 'category_selection_method'))))
>
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-2 depends on stages: Stage-1
> Stage-0 is a root stage
>
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Alias -> Map Operator Tree:
> test
> Reduce Output Operator
> key expressions:
> expr: recipient_id
> type: string
> sort order: +
> Map-reduce partition columns:
> expr: recipient_id
> type: string
> tag: 1
> value expressions:
> expr: recipient_id
> type: string
> request
> Select Operator
> expressions:
> expr: recipient_id
> type: string
> expr: attributes
> type: map<string,string>
> Reduce Output Operator
> key expressions:
> expr: 0
> type: string
> sort order: +
> Map-reduce partition columns:
> expr: 0
> type: string
> tag: 0
> value expressions:
> expr: 0
> type: string
> expr: 1
> type: map<string,string>
> Reduce Operator Tree:
> Join Operator
> condition map:
> Left Outer Join0 to 1
> condition expressions:
> 0 {VALUE.0} {VALUE.1}
> 1 {VALUE.0}
> Group By Operator
> aggregations:
> expr: count(1)
> keys:
> expr: 1['category_selection_method']
> type: string
> mode: hash
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format:
> org.apache.hadoop.mapred.SequenceFileInputFormat
> output format:
> org.apache.hadoop.mapred.SequenceFileOutputFormat
> name: binary_table
>
> Stage: Stage-2
> Map Reduce
> Alias -> Map Operator Tree:
> /tmp/hive-root/532094399/571818099.10002
> Reduce Output Operator
> key expressions:
> expr: 0
> type: string
> sort order: +
> Map-reduce partition columns:
> expr: 0
> type: string
> tag: -1
> value expressions:
> expr: 1
> type: bigint
> Reduce Operator Tree:
> Group By Operator
> aggregations:
> expr: count(VALUE.0)
> keys:
> expr: KEY.0
> type: string
> mode: mergepartial
> Select Operator
> expressions:
> expr: 1
> type: bigint
> expr: 0
> type: string
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
>
> Stage: Stage-0
> Fetch Operator
> limit: -1
>
>
>
--
Sent from Gmail for mobile | mobile.google.com<http://mobile.google.com>
Yours,
Zheng
--
Yours,
Zheng
Re: NullPointerException when using join on table with map column
Posted by Zheng Shao <zs...@gmail.com>.
Hi Stephen,
I am aware of this bug - it's buried in Hive for a long time until recently
I encountered with it again doing HIVE-266.
I opened HIVE-405 for that purpose. It probably take another week or two to
fix the problem (taking into account the time for test, code review etc).
It's the second top priority in my list (the first one is HIVE-372).
Zheng
On Fri, Apr 10, 2009 at 6:37 PM, Stephen Corona <sc...@adknowledge.com>wrote:
> Hey Zheng,
> I was wondering if the issue regarding map columns w/ joins has been fixed?
> I removed all null values from my map column, so all keys have a non-null
> string value. I still get an error when doing joins involving this column
> but it's in the reduce phase this time (the error message is pasted below).
> I'm wondering if there is already a JIRA for this and, if so, where is falls
> on the priority list. It's a real pain to have to use a subquery to access
> any map columns w/ a join.
>
>
> java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: Hive 2 Internal error: cannot evaluate index
> expression on string
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:169)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430)
> at org.apache.hadoop.mapred.Child.main(Child.java:155)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: Hive 2 Internal error: cannot evaluate index
> expression on string
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:451)
> at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.createForwardJoinObject(JoinOperator.java:259)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:502)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:491)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:491)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:491)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.checkAndGenObject(JoinOperator.java:532)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:514)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:140)
> ... 2 more
> Caused by: java.lang.RuntimeException: Hive 2 Internal error: cannot
> evaluate index expression on string
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeIndexEvaluator.evaluate(ExprNodeIndexEvaluator.java:64)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:72)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:72)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:72)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:370)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:470)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:445)
> ... 11 more
>
> Steve
> ________________________________________
> From: Stephen Corona
> Sent: Tuesday, March 17, 2009 2:09 PM
> To: hive-user@hadoop.apache.org
> Subject: RE: NullPointerException when using join on table with map column
>
> Are map's allowed to have a key with a null value? Or does it expect every
> key to have a non-null value?
>
> Steve
> ________________________________________
> From: Zheng Shao [zshao9@gmail.com]
> Sent: Tuesday, March 17, 2009 1:56 PM
> To: hive-user@hadoop.apache.org
> Subject: Re: NullPointerException when using join on table with map column
>
> Hi,
>
> It seems DynamicSerde has problems with the null values somewhere. We
> are working on replacing it with LazySimpleSerDe.
>
> In the meanwhile, please use subquery and the problem should disappear.
>
> Select count(1), mykey from (Select id, mymap['mykey'] as mykey from
> tablea) tablea join tableb on tablea.id = tableb.id group by mykey;
>
> Zheng
>
>
> On 3/17/09, Stephen Corona <sc...@adknowledge.com> wrote:
> > Hey guys. I am trying to run a query like:
> >
> > select count(1), request.attributes['category_selection_method'] from
> > request left outer join test on request.recipient_id = test.recipient_id
> > group by request.attributes['category_selection_method'];
> >
> > I can run the query WITHOUT the join and it works fine. I can also run
> the
> > query with the join if I don't reference the attributes column (i.e, the
> > map). Is there a problem with doing joins when a table has a map column?
> >
> > However, when I run this query I get the following error:
> >
> > java.lang.RuntimeException: org.apache.hadoop.hive.serde2.SerDeException:
> > java.lang.NullPointerException
> > at
> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:75)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> > at org.apache.hadoop.mapred.Child.main(Child.java:155)
> > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> > org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> > at
> >
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:191)
> > at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> > at
> >
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:89)
> > at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> > at
> >
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:49)
> > at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> > at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:178)
> > at
> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:71)
> > ... 3 more
> > Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> > java.lang.NullPointerException
> > at
> >
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:178)
> > at
> >
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:189)
> > ... 10 more
> > Caused by: java.lang.NullPointerException
> > at
> >
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeTypeMap.serialize(DynamicSerDeTypeMap.java:136)
> > at
> >
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeFieldList.serialize(DynamicSerDeFieldList.java:249)
> > at
> >
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeStructBase.serialize(DynamicSerDeStructBase.java:81)
> > at
> >
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:174)
> > ... 11 more
> >
> >
> > describe request-
> >
> > request_id string
> > attributes map<string,string>
> > day string
> >
> > create table for request:
> >
> > create table request (
> > request_id
> > attributes map <string, string>
> > )
> > row format delimited fields terminated by '\001' collection items
> terminated
> > by '\003' map keys terminated by '\002';
> >
> > describe test-
> > recipient_id string
> >
> > explain query
> >
> > ABSTRACT SYNTAX TREE:
> > (TOK_QUERY (TOK_FROM (TOK_LEFTOUTERJOIN (TOK_TABREF request)
> (TOK_TABREF
> > test) (= (TOK_COLREF request recipient_id) (TOK_COLREF test
> recipient_id))))
> > (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT
> > (TOK_SELEXPR (TOK_FUNCTION count 1)) (TOK_SELEXPR ([ (TOK_COLREF request
> > attributes) 'category_selection_method'))) (TOK_GROUPBY ([ (TOK_COLREF
> > request attributes) 'category_selection_method'))))
> >
> > STAGE DEPENDENCIES:
> > Stage-1 is a root stage
> > Stage-2 depends on stages: Stage-1
> > Stage-0 is a root stage
> >
> > STAGE PLANS:
> > Stage: Stage-1
> > Map Reduce
> > Alias -> Map Operator Tree:
> > test
> > Reduce Output Operator
> > key expressions:
> > expr: recipient_id
> > type: string
> > sort order: +
> > Map-reduce partition columns:
> > expr: recipient_id
> > type: string
> > tag: 1
> > value expressions:
> > expr: recipient_id
> > type: string
> > request
> > Select Operator
> > expressions:
> > expr: recipient_id
> > type: string
> > expr: attributes
> > type: map<string,string>
> > Reduce Output Operator
> > key expressions:
> > expr: 0
> > type: string
> > sort order: +
> > Map-reduce partition columns:
> > expr: 0
> > type: string
> > tag: 0
> > value expressions:
> > expr: 0
> > type: string
> > expr: 1
> > type: map<string,string>
> > Reduce Operator Tree:
> > Join Operator
> > condition map:
> > Left Outer Join0 to 1
> > condition expressions:
> > 0 {VALUE.0} {VALUE.1}
> > 1 {VALUE.0}
> > Group By Operator
> > aggregations:
> > expr: count(1)
> > keys:
> > expr: 1['category_selection_method']
> > type: string
> > mode: hash
> > File Output Operator
> > compressed: false
> > GlobalTableId: 0
> > table:
> > input format:
> > org.apache.hadoop.mapred.SequenceFileInputFormat
> > output format:
> > org.apache.hadoop.mapred.SequenceFileOutputFormat
> > name: binary_table
> >
> > Stage: Stage-2
> > Map Reduce
> > Alias -> Map Operator Tree:
> > /tmp/hive-root/532094399/571818099.10002
> > Reduce Output Operator
> > key expressions:
> > expr: 0
> > type: string
> > sort order: +
> > Map-reduce partition columns:
> > expr: 0
> > type: string
> > tag: -1
> > value expressions:
> > expr: 1
> > type: bigint
> > Reduce Operator Tree:
> > Group By Operator
> > aggregations:
> > expr: count(VALUE.0)
> > keys:
> > expr: KEY.0
> > type: string
> > mode: mergepartial
> > Select Operator
> > expressions:
> > expr: 1
> > type: bigint
> > expr: 0
> > type: string
> > File Output Operator
> > compressed: false
> > GlobalTableId: 0
> > table:
> > input format: org.apache.hadoop.mapred.TextInputFormat
> > output format:
> > org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
> >
> > Stage: Stage-0
> > Fetch Operator
> > limit: -1
> >
> >
> >
>
> --
> Sent from Gmail for mobile | mobile.google.com
>
> Yours,
> Zheng
>
--
Yours,
Zheng
RE: NullPointerException when using join on table with map column
Posted by Stephen Corona <sc...@adknowledge.com>.
Hey Zheng,
I was wondering if the issue regarding map columns w/ joins has been fixed? I removed all null values from my map column, so all keys have a non-null string value. I still get an error when doing joins involving this column but it's in the reduce phase this time (the error message is pasted below). I'm wondering if there is already a JIRA for this and, if so, where is falls on the priority list. It's a real pain to have to use a subquery to access any map columns w/ a join.
java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Hive 2 Internal error: cannot evaluate index expression on string
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:169)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Hive 2 Internal error: cannot evaluate index expression on string
at org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:451)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
at org.apache.hadoop.hive.ql.exec.JoinOperator.createForwardJoinObject(JoinOperator.java:259)
at org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:502)
at org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:491)
at org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:491)
at org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:491)
at org.apache.hadoop.hive.ql.exec.JoinOperator.checkAndGenObject(JoinOperator.java:532)
at org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:514)
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:140)
... 2 more
Caused by: java.lang.RuntimeException: Hive 2 Internal error: cannot evaluate index expression on string
at org.apache.hadoop.hive.ql.exec.ExprNodeIndexEvaluator.evaluate(ExprNodeIndexEvaluator.java:64)
at org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:72)
at org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:72)
at org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:72)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:370)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:470)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:445)
... 11 more
Steve
________________________________________
From: Stephen Corona
Sent: Tuesday, March 17, 2009 2:09 PM
To: hive-user@hadoop.apache.org
Subject: RE: NullPointerException when using join on table with map column
Are map's allowed to have a key with a null value? Or does it expect every key to have a non-null value?
Steve
________________________________________
From: Zheng Shao [zshao9@gmail.com]
Sent: Tuesday, March 17, 2009 1:56 PM
To: hive-user@hadoop.apache.org
Subject: Re: NullPointerException when using join on table with map column
Hi,
It seems DynamicSerde has problems with the null values somewhere. We
are working on replacing it with LazySimpleSerDe.
In the meanwhile, please use subquery and the problem should disappear.
Select count(1), mykey from (Select id, mymap['mykey'] as mykey from
tablea) tablea join tableb on tablea.id = tableb.id group by mykey;
Zheng
On 3/17/09, Stephen Corona <sc...@adknowledge.com> wrote:
> Hey guys. I am trying to run a query like:
>
> select count(1), request.attributes['category_selection_method'] from
> request left outer join test on request.recipient_id = test.recipient_id
> group by request.attributes['category_selection_method'];
>
> I can run the query WITHOUT the join and it works fine. I can also run the
> query with the join if I don't reference the attributes column (i.e, the
> map). Is there a problem with doing joins when a table has a map column?
>
> However, when I run this query I get the following error:
>
> java.lang.RuntimeException: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:75)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child.main(Child.java:155)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:191)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:89)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:49)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:178)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:71)
> ... 3 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:178)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:189)
> ... 10 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeTypeMap.serialize(DynamicSerDeTypeMap.java:136)
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeFieldList.serialize(DynamicSerDeFieldList.java:249)
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeStructBase.serialize(DynamicSerDeStructBase.java:81)
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:174)
> ... 11 more
>
>
> describe request-
>
> request_id string
> attributes map<string,string>
> day string
>
> create table for request:
>
> create table request (
> request_id
> attributes map <string, string>
> )
> row format delimited fields terminated by '\001' collection items terminated
> by '\003' map keys terminated by '\002';
>
> describe test-
> recipient_id string
>
> explain query
>
> ABSTRACT SYNTAX TREE:
> (TOK_QUERY (TOK_FROM (TOK_LEFTOUTERJOIN (TOK_TABREF request) (TOK_TABREF
> test) (= (TOK_COLREF request recipient_id) (TOK_COLREF test recipient_id))))
> (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT
> (TOK_SELEXPR (TOK_FUNCTION count 1)) (TOK_SELEXPR ([ (TOK_COLREF request
> attributes) 'category_selection_method'))) (TOK_GROUPBY ([ (TOK_COLREF
> request attributes) 'category_selection_method'))))
>
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-2 depends on stages: Stage-1
> Stage-0 is a root stage
>
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Alias -> Map Operator Tree:
> test
> Reduce Output Operator
> key expressions:
> expr: recipient_id
> type: string
> sort order: +
> Map-reduce partition columns:
> expr: recipient_id
> type: string
> tag: 1
> value expressions:
> expr: recipient_id
> type: string
> request
> Select Operator
> expressions:
> expr: recipient_id
> type: string
> expr: attributes
> type: map<string,string>
> Reduce Output Operator
> key expressions:
> expr: 0
> type: string
> sort order: +
> Map-reduce partition columns:
> expr: 0
> type: string
> tag: 0
> value expressions:
> expr: 0
> type: string
> expr: 1
> type: map<string,string>
> Reduce Operator Tree:
> Join Operator
> condition map:
> Left Outer Join0 to 1
> condition expressions:
> 0 {VALUE.0} {VALUE.1}
> 1 {VALUE.0}
> Group By Operator
> aggregations:
> expr: count(1)
> keys:
> expr: 1['category_selection_method']
> type: string
> mode: hash
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format:
> org.apache.hadoop.mapred.SequenceFileInputFormat
> output format:
> org.apache.hadoop.mapred.SequenceFileOutputFormat
> name: binary_table
>
> Stage: Stage-2
> Map Reduce
> Alias -> Map Operator Tree:
> /tmp/hive-root/532094399/571818099.10002
> Reduce Output Operator
> key expressions:
> expr: 0
> type: string
> sort order: +
> Map-reduce partition columns:
> expr: 0
> type: string
> tag: -1
> value expressions:
> expr: 1
> type: bigint
> Reduce Operator Tree:
> Group By Operator
> aggregations:
> expr: count(VALUE.0)
> keys:
> expr: KEY.0
> type: string
> mode: mergepartial
> Select Operator
> expressions:
> expr: 1
> type: bigint
> expr: 0
> type: string
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
>
> Stage: Stage-0
> Fetch Operator
> limit: -1
>
>
>
--
Sent from Gmail for mobile | mobile.google.com
Yours,
Zheng
RE: NullPointerException when using join on table with map column
Posted by Stephen Corona <sc...@adknowledge.com>.
Are map's allowed to have a key with a null value? Or does it expect every key to have a non-null value?
Steve
________________________________________
From: Zheng Shao [zshao9@gmail.com]
Sent: Tuesday, March 17, 2009 1:56 PM
To: hive-user@hadoop.apache.org
Subject: Re: NullPointerException when using join on table with map column
Hi,
It seems DynamicSerde has problems with the null values somewhere. We
are working on replacing it with LazySimpleSerDe.
In the meanwhile, please use subquery and the problem should disappear.
Select count(1), mykey from (Select id, mymap['mykey'] as mykey from
tablea) tablea join tableb on tablea.id = tableb.id group by mykey;
Zheng
On 3/17/09, Stephen Corona <sc...@adknowledge.com> wrote:
> Hey guys. I am trying to run a query like:
>
> select count(1), request.attributes['category_selection_method'] from
> request left outer join test on request.recipient_id = test.recipient_id
> group by request.attributes['category_selection_method'];
>
> I can run the query WITHOUT the join and it works fine. I can also run the
> query with the join if I don't reference the attributes column (i.e, the
> map). Is there a problem with doing joins when a table has a map column?
>
> However, when I run this query I get the following error:
>
> java.lang.RuntimeException: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:75)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child.main(Child.java:155)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:191)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:89)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:49)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:178)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:71)
> ... 3 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:178)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:189)
> ... 10 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeTypeMap.serialize(DynamicSerDeTypeMap.java:136)
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeFieldList.serialize(DynamicSerDeFieldList.java:249)
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeStructBase.serialize(DynamicSerDeStructBase.java:81)
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:174)
> ... 11 more
>
>
> describe request-
>
> request_id string
> attributes map<string,string>
> day string
>
> create table for request:
>
> create table request (
> request_id
> attributes map <string, string>
> )
> row format delimited fields terminated by '\001' collection items terminated
> by '\003' map keys terminated by '\002';
>
> describe test-
> recipient_id string
>
> explain query
>
> ABSTRACT SYNTAX TREE:
> (TOK_QUERY (TOK_FROM (TOK_LEFTOUTERJOIN (TOK_TABREF request) (TOK_TABREF
> test) (= (TOK_COLREF request recipient_id) (TOK_COLREF test recipient_id))))
> (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT
> (TOK_SELEXPR (TOK_FUNCTION count 1)) (TOK_SELEXPR ([ (TOK_COLREF request
> attributes) 'category_selection_method'))) (TOK_GROUPBY ([ (TOK_COLREF
> request attributes) 'category_selection_method'))))
>
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-2 depends on stages: Stage-1
> Stage-0 is a root stage
>
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Alias -> Map Operator Tree:
> test
> Reduce Output Operator
> key expressions:
> expr: recipient_id
> type: string
> sort order: +
> Map-reduce partition columns:
> expr: recipient_id
> type: string
> tag: 1
> value expressions:
> expr: recipient_id
> type: string
> request
> Select Operator
> expressions:
> expr: recipient_id
> type: string
> expr: attributes
> type: map<string,string>
> Reduce Output Operator
> key expressions:
> expr: 0
> type: string
> sort order: +
> Map-reduce partition columns:
> expr: 0
> type: string
> tag: 0
> value expressions:
> expr: 0
> type: string
> expr: 1
> type: map<string,string>
> Reduce Operator Tree:
> Join Operator
> condition map:
> Left Outer Join0 to 1
> condition expressions:
> 0 {VALUE.0} {VALUE.1}
> 1 {VALUE.0}
> Group By Operator
> aggregations:
> expr: count(1)
> keys:
> expr: 1['category_selection_method']
> type: string
> mode: hash
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format:
> org.apache.hadoop.mapred.SequenceFileInputFormat
> output format:
> org.apache.hadoop.mapred.SequenceFileOutputFormat
> name: binary_table
>
> Stage: Stage-2
> Map Reduce
> Alias -> Map Operator Tree:
> /tmp/hive-root/532094399/571818099.10002
> Reduce Output Operator
> key expressions:
> expr: 0
> type: string
> sort order: +
> Map-reduce partition columns:
> expr: 0
> type: string
> tag: -1
> value expressions:
> expr: 1
> type: bigint
> Reduce Operator Tree:
> Group By Operator
> aggregations:
> expr: count(VALUE.0)
> keys:
> expr: KEY.0
> type: string
> mode: mergepartial
> Select Operator
> expressions:
> expr: 1
> type: bigint
> expr: 0
> type: string
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
>
> Stage: Stage-0
> Fetch Operator
> limit: -1
>
>
>
--
Sent from Gmail for mobile | mobile.google.com
Yours,
Zheng
Re: NullPointerException when using join on table with map column
Posted by Zheng Shao <zs...@gmail.com>.
Hi,
It seems DynamicSerde has problems with the null values somewhere. We
are working on replacing it with LazySimpleSerDe.
In the meanwhile, please use subquery and the problem should disappear.
Select count(1), mykey from (Select id, mymap['mykey'] as mykey from
tablea) tablea join tableb on tablea.id = tableb.id group by mykey;
Zheng
On 3/17/09, Stephen Corona <sc...@adknowledge.com> wrote:
> Hey guys. I am trying to run a query like:
>
> select count(1), request.attributes['category_selection_method'] from
> request left outer join test on request.recipient_id = test.recipient_id
> group by request.attributes['category_selection_method'];
>
> I can run the query WITHOUT the join and it works fine. I can also run the
> query with the join if I don't reference the attributes column (i.e, the
> map). Is there a problem with doing joins when a table has a map column?
>
> However, when I run this query I get the following error:
>
> java.lang.RuntimeException: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:75)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child.main(Child.java:155)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:191)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:89)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:49)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:315)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:178)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:71)
> ... 3 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:178)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:189)
> ... 10 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeTypeMap.serialize(DynamicSerDeTypeMap.java:136)
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeFieldList.serialize(DynamicSerDeFieldList.java:249)
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeStructBase.serialize(DynamicSerDeStructBase.java:81)
> at
> org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:174)
> ... 11 more
>
>
> describe request-
>
> request_id string
> attributes map<string,string>
> day string
>
> create table for request:
>
> create table request (
> request_id
> attributes map <string, string>
> )
> row format delimited fields terminated by '\001' collection items terminated
> by '\003' map keys terminated by '\002';
>
> describe test-
> recipient_id string
>
> explain query
>
> ABSTRACT SYNTAX TREE:
> (TOK_QUERY (TOK_FROM (TOK_LEFTOUTERJOIN (TOK_TABREF request) (TOK_TABREF
> test) (= (TOK_COLREF request recipient_id) (TOK_COLREF test recipient_id))))
> (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT
> (TOK_SELEXPR (TOK_FUNCTION count 1)) (TOK_SELEXPR ([ (TOK_COLREF request
> attributes) 'category_selection_method'))) (TOK_GROUPBY ([ (TOK_COLREF
> request attributes) 'category_selection_method'))))
>
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-2 depends on stages: Stage-1
> Stage-0 is a root stage
>
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Alias -> Map Operator Tree:
> test
> Reduce Output Operator
> key expressions:
> expr: recipient_id
> type: string
> sort order: +
> Map-reduce partition columns:
> expr: recipient_id
> type: string
> tag: 1
> value expressions:
> expr: recipient_id
> type: string
> request
> Select Operator
> expressions:
> expr: recipient_id
> type: string
> expr: attributes
> type: map<string,string>
> Reduce Output Operator
> key expressions:
> expr: 0
> type: string
> sort order: +
> Map-reduce partition columns:
> expr: 0
> type: string
> tag: 0
> value expressions:
> expr: 0
> type: string
> expr: 1
> type: map<string,string>
> Reduce Operator Tree:
> Join Operator
> condition map:
> Left Outer Join0 to 1
> condition expressions:
> 0 {VALUE.0} {VALUE.1}
> 1 {VALUE.0}
> Group By Operator
> aggregations:
> expr: count(1)
> keys:
> expr: 1['category_selection_method']
> type: string
> mode: hash
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format:
> org.apache.hadoop.mapred.SequenceFileInputFormat
> output format:
> org.apache.hadoop.mapred.SequenceFileOutputFormat
> name: binary_table
>
> Stage: Stage-2
> Map Reduce
> Alias -> Map Operator Tree:
> /tmp/hive-root/532094399/571818099.10002
> Reduce Output Operator
> key expressions:
> expr: 0
> type: string
> sort order: +
> Map-reduce partition columns:
> expr: 0
> type: string
> tag: -1
> value expressions:
> expr: 1
> type: bigint
> Reduce Operator Tree:
> Group By Operator
> aggregations:
> expr: count(VALUE.0)
> keys:
> expr: KEY.0
> type: string
> mode: mergepartial
> Select Operator
> expressions:
> expr: 1
> type: bigint
> expr: 0
> type: string
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
>
> Stage: Stage-0
> Fetch Operator
> limit: -1
>
>
>
--
Sent from Gmail for mobile | mobile.google.com
Yours,
Zheng