You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Marc Harris (JIRA)" <ji...@apache.org> on 2011/04/12 16:31:05 UTC
[jira] [Created] (HIVE-2111) NullPointerException on select * with
table using RegexSerDe and partitions
NullPointerException on select * with table using RegexSerDe and partitions
---------------------------------------------------------------------------
Key: HIVE-2111
URL: https://issues.apache.org/jira/browse/HIVE-2111
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.7.0
Environment: Amazon Elastic Mapreduce
Reporter: Marc Harris
When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
To reproduce:
1) create a table containing the following text (notice the blank line):
====start====
fillerdatafillerdatafiller
fillerdata2fillerdata2filler
=====end=====
2) copy the file to hdfs:
hadoop dfs -put foo.txt test/part1=x/foo.txt
3) run the following hive commands to create a table:
add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
drop table test;
create external table test(col1 STRING, col2 STRING)
partitioned by (part1 STRING)
row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
with serdeproperties ( "input.regex" = "^\(.*data\)(.*data\).*$")
stored as textfile
location 'hdfs:///user/hadoop/test';
alter table test add partition (part1='x');
4) select from it with explicit columns:
select part1, col1, col2 from test;
outputs:
OK
x fillerdata fillerdata
x NULL NULL
x fillerdata 2fillerdata
5) select from it with * columns
select * from test;
outputs:
Failed with exception java.io.IOException:java.lang.NullPointerException
11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.NullPointerException
at java.util.ArrayList.addAll(ArrayList.java:472)
at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2111) NullPointerException on select *
with table using RegexSerDe and partitions
Posted by "Chinna Rao Lalam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036191#comment-13036191 ]
Chinna Rao Lalam commented on HIVE-2111:
----------------------------------------
When ever the regular expression is not matching it is returning null but it should return null row. Same thing is happening with the empty row also.
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
> Assignee: Chinna Rao Lalam
> Attachments: HIVE-2111.patch
>
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2111) NullPointerException on select * with
table using RegexSerDe and partitions
Posted by "Marc Harris (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marc Harris updated HIVE-2111:
------------------------------
Description:
When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
To reproduce:
1) create a table containing the following text (notice the blank line):
====start====
fillerdatafillerdatafiller
fillerdata2fillerdata2filler
=====end=====
2) copy the file to hdfs:
hadoop dfs -put foo.txt test/part1=x/foo.txt
3) run the following hive commands to create a table:
add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
drop table test;
create external table test(col1 STRING, col2 STRING)
partitioned by (part1 STRING)
row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
stored as textfile
location 'hdfs:///user/hadoop/test';
alter table test add partition (part1='x');
(Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
4) select from it with explicit columns:
select part1, col1, col2 from test;
outputs:
OK
x fillerdata fillerdata
x NULL NULL
x fillerdata 2fillerdata
5) select from it with * columns
select * from test;
outputs:
Failed with exception java.io.IOException:java.lang.NullPointerException
11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.NullPointerException
at java.util.ArrayList.addAll(ArrayList.java:472)
at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
... 10 more
was:
When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
To reproduce:
1) create a table containing the following text (notice the blank line):
====start====
fillerdatafillerdatafiller
fillerdata2fillerdata2filler
=====end=====
2) copy the file to hdfs:
hadoop dfs -put foo.txt test/part1=x/foo.txt
3) run the following hive commands to create a table:
add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
drop table test;
create external table test(col1 STRING, col2 STRING)
partitioned by (part1 STRING)
row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
with serdeproperties ( "input.regex" = "^\(.*data\)(.*data\).*$")
stored as textfile
location 'hdfs:///user/hadoop/test';
alter table test add partition (part1='x');
4) select from it with explicit columns:
select part1, col1, col2 from test;
outputs:
OK
x fillerdata fillerdata
x NULL NULL
x fillerdata 2fillerdata
5) select from it with * columns
select * from test;
outputs:
Failed with exception java.io.IOException:java.lang.NullPointerException
11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.NullPointerException
at java.util.ArrayList.addAll(ArrayList.java:472)
at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
... 10 more
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2111) NullPointerException on select *
with table using RegexSerDe and partitions
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065396#comment-13065396 ]
Amareshwari Sriramadasu commented on HIVE-2111:
-----------------------------------------------
Chinna, the testcase passes without the patch also. Can you verify it again?
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
> Assignee: Chinna Rao Lalam
> Attachments: HIVE-2111.1.patch, HIVE-2111.2.patch, HIVE-2111.patch
>
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2111) NullPointerException on select * with
table using RegexSerDe and partitions
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-2111:
-----------------------------
Status: Open (was: Patch Available)
Sorry, this one has gone stale...could you rebase against trunk?
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
> Assignee: Chinna Rao Lalam
> Attachments: HIVE-2111.patch
>
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2111) NullPointerException on select * with
table using RegexSerDe and partitions
Posted by "Chinna Rao Lalam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chinna Rao Lalam updated HIVE-2111:
-----------------------------------
Status: Patch Available (was: Open)
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
> Assignee: Chinna Rao Lalam
> Attachments: HIVE-2111.patch
>
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2111) NullPointerException on select *
with table using RegexSerDe and partitions
Posted by "Chinna Rao Lalam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064594#comment-13064594 ]
Chinna Rao Lalam commented on HIVE-2111:
----------------------------------------
All existing testcases passed in my environment. I have added one testcase for this scenario.
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
> Assignee: Chinna Rao Lalam
> Attachments: HIVE-2111.1.patch, HIVE-2111.2.patch, HIVE-2111.patch
>
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2111) NullPointerException on select * with
table using RegexSerDe and partitions
Posted by "Chinna Rao Lalam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chinna Rao Lalam updated HIVE-2111:
-----------------------------------
Status: Patch Available (was: Open)
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
> Assignee: Chinna Rao Lalam
> Attachments: HIVE-2111.1.patch, HIVE-2111.patch
>
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2111) NullPointerException on select *
with table using RegexSerDe and partitions
Posted by "Chinna Rao Lalam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030664#comment-13030664 ]
Chinna Rao Lalam commented on HIVE-2111:
----------------------------------------
*data
*data
*$
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2111) NullPointerException on select * with
table using RegexSerDe and partitions
Posted by "Chinna Rao Lalam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chinna Rao Lalam updated HIVE-2111:
-----------------------------------
Attachment: HIVE-2111.patch
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
> Attachments: HIVE-2111.patch
>
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2111) NullPointerException on select *
with table using RegexSerDe and partitions
Posted by "Chinna Rao Lalam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036190#comment-13036190 ]
Chinna Rao Lalam commented on HIVE-2111:
----------------------------------------
>From jira editor i got this " ^(.data)(.*data).$ " regular expression that's why i suggested like this regex may be the wrong.
But when i updated the comment i realized that what ever the regular expression is given in the defect description is correct because of the jira editor it is showing wrong regular expression.
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
> Assignee: Chinna Rao Lalam
> Attachments: HIVE-2111.patch
>
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2111) NullPointerException on select * with
table using RegexSerDe and partitions
Posted by "Chinna Rao Lalam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chinna Rao Lalam updated HIVE-2111:
-----------------------------------
Attachment: HIVE-2111.2.patch
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
> Assignee: Chinna Rao Lalam
> Attachments: HIVE-2111.1.patch, HIVE-2111.2.patch, HIVE-2111.patch
>
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2111) NullPointerException on select * with
table using RegexSerDe and partitions
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-2111:
---------------------------------
Status: Open (was: Patch Available)
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
> Assignee: Chinna Rao Lalam
> Attachments: HIVE-2111.1.patch, HIVE-2111.2.patch, HIVE-2111.patch
>
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2111) NullPointerException on select *
with table using RegexSerDe and partitions
Posted by "Marc Harris (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030831#comment-13030831 ]
Marc Harris commented on HIVE-2111:
-----------------------------------
I can't tell in which way you are suggesting the regular expression should change.
If it is about the presence of the backslash before the parentheses, well the hive command line processor seems to require these.
If it is the dot slashes, then the JIRA text editor seems to be mangling. Each Set of parentheses should contain "dot star date" and then there should be "dot star dollar" at the end.
In any case, none of this would account for the fact that "select part1, col1, col2" and "select *" produce different results.
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2111) NullPointerException on select * with
table using RegexSerDe and partitions
Posted by "Chinna Rao Lalam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chinna Rao Lalam updated HIVE-2111:
-----------------------------------
Attachment: HIVE-2111.1.patch
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
> Assignee: Chinna Rao Lalam
> Attachments: HIVE-2111.1.patch, HIVE-2111.patch
>
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2111) NullPointerException on select *
with table using RegexSerDe and partitions
Posted by "Chinna Rao Lalam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062388#comment-13062388 ]
Chinna Rao Lalam commented on HIVE-2111:
----------------------------------------
patch is rebased against trunk.
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
> Assignee: Chinna Rao Lalam
> Attachments: HIVE-2111.1.patch, HIVE-2111.patch
>
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2111) NullPointerException on select * with
table using RegexSerDe and partitions
Posted by "Chinna Rao Lalam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chinna Rao Lalam reassigned HIVE-2111:
--------------------------------------
Assignee: Chinna Rao Lalam
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
> Assignee: Chinna Rao Lalam
> Attachments: HIVE-2111.patch
>
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2111) NullPointerException on select *
with table using RegexSerDe and partitions
Posted by "Chinna Rao Lalam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030662#comment-13030662 ]
Chinna Rao Lalam commented on HIVE-2111:
----------------------------------------
Pls check for the given input data this may be the correct regex
"^(.*data)(.*data).*$"
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2111) NullPointerException on select * with
table using RegexSerDe and partitions
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HIVE-2111:
------------------------------------------
Status: Open (was: Patch Available)
Can you add a regression test (test fails without patch and passes with patch)? Because currently, serde_regex.q does 'select \*' and passes.
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
> Assignee: Chinna Rao Lalam
> Attachments: HIVE-2111.1.patch, HIVE-2111.patch
>
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2111) NullPointerException on select * with
table using RegexSerDe and partitions
Posted by "Chinna Rao Lalam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chinna Rao Lalam updated HIVE-2111:
-----------------------------------
Status: Patch Available (was: Open)
> NullPointerException on select * with table using RegexSerDe and partitions
> ---------------------------------------------------------------------------
>
> Key: HIVE-2111
> URL: https://issues.apache.org/jira/browse/HIVE-2111
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.7.0
> Environment: Amazon Elastic Mapreduce
> Reporter: Marc Harris
> Assignee: Chinna Rao Lalam
> Attachments: HIVE-2111.1.patch, HIVE-2111.2.patch, HIVE-2111.patch
>
>
> When querying against a table that is partitioned, and uses RegexSerde, select with explicit columns works, but "select *" results in a NullPointerException
> To reproduce:
> 1) create a table containing the following text (notice the blank line):
> ====start====
> fillerdatafillerdatafiller
> fillerdata2fillerdata2filler
> =====end=====
> 2) copy the file to hdfs:
> hadoop dfs -put foo.txt test/part1=x/foo.txt
> 3) run the following hive commands to create a table:
> add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar;
> drop table test;
> create external table test(col1 STRING, col2 STRING)
> partitioned by (part1 STRING)
> row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$")
> stored as textfile
> location 'hdfs:///user/hadoop/test';
> alter table test add partition (part1='x');
> (Note that the text processor seems to have mangled the regex a bit. Inside each pair of parentheses should be dot star data. After the second pair of parentheses should be dot start dollar).
> 4) select from it with explicit columns:
> select part1, col1, col2 from test;
> outputs:
> OK
> x fillerdata fillerdata
> x NULL NULL
> x fillerdata 2fillerdata
> 5) select from it with * columns
> select * from test;
> outputs:
> Failed with exception java.io.IOException:java.lang.NullPointerException
> 11/04/12 14:28:27 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
> at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.addAll(ArrayList.java:472)
> at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141)
> ... 10 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira