You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2008/09/20 04:07:44 UTC
[jira] Created: (HADOOP-4231) Hive: converting complex objects to
JSON failed.
Hive: converting complex objects to JSON failed.
------------------------------------------------
Key: HADOOP-4231
URL: https://issues.apache.org/jira/browse/HADOOP-4231
Project: Hadoop Core
Issue Type: Improvement
Reporter: Zheng Shao
Assignee: Zheng Shao
Priority: Minor
java.lang.RuntimeException: java.lang.IllegalArgumentException: Can not set int field xxx to java.lang.String at
org.apache.hadoop.hive.serde2.objectinspector.ReflectionStructObjectInspector.getStructFieldData(ReflectionStructObjectInspector.java:123)
at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:231)
at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:163)
at org.apache.hadoop.hive.serde2.MetadataTypedColumnsetSerDe.serialize(MetadataTypedColumnsetSerDe.java:181)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4231) Hive: converting complex objects to
JSON failed.
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-4231:
-------------------------------
Attachment: HADOOP-4231.0.txt
> Hive: converting complex objects to JSON failed.
> ------------------------------------------------
>
> Key: HADOOP-4231
> URL: https://issues.apache.org/jira/browse/HADOOP-4231
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Minor
> Attachments: HADOOP-4231.0.txt
>
>
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Can not set int field xxx to java.lang.String at
> org.apache.hadoop.hive.serde2.objectinspector.ReflectionStructObjectInspector.getStructFieldData(ReflectionStructObjectInspector.java:123)
> at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:231)
> at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:163)
> at org.apache.hadoop.hive.serde2.MetadataTypedColumnsetSerDe.serialize(MetadataTypedColumnsetSerDe.java:181)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4231) Hive: converting complex objects to
JSON failed.
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-4231:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Fixed as part of JIRA-4230
> Hive: converting complex objects to JSON failed.
> ------------------------------------------------
>
> Key: HADOOP-4231
> URL: https://issues.apache.org/jira/browse/HADOOP-4231
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Minor
> Attachments: HADOOP-4231.0.txt
>
>
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Can not set int field xxx to java.lang.String at
> org.apache.hadoop.hive.serde2.objectinspector.ReflectionStructObjectInspector.getStructFieldData(ReflectionStructObjectInspector.java:123)
> at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:231)
> at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:163)
> at org.apache.hadoop.hive.serde2.MetadataTypedColumnsetSerDe.serialize(MetadataTypedColumnsetSerDe.java:181)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4231) Hive: converting complex objects to
JSON failed.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634418#action_12634418 ]
Hadoop QA commented on HADOOP-4231:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12390897/HADOOP-4231.0.txt
against trunk revision 698721.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3368/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3368/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3368/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3368/console
This message is automatically generated.
> Hive: converting complex objects to JSON failed.
> ------------------------------------------------
>
> Key: HADOOP-4231
> URL: https://issues.apache.org/jira/browse/HADOOP-4231
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Minor
> Attachments: HADOOP-4231.0.txt
>
>
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Can not set int field xxx to java.lang.String at
> org.apache.hadoop.hive.serde2.objectinspector.ReflectionStructObjectInspector.getStructFieldData(ReflectionStructObjectInspector.java:123)
> at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:231)
> at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:163)
> at org.apache.hadoop.hive.serde2.MetadataTypedColumnsetSerDe.serialize(MetadataTypedColumnsetSerDe.java:181)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4231) Hive: converting complex objects to
JSON failed.
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-4231:
-------------------------------
Status: Patch Available (was: Open)
The problem was that ReduceSinkOperator was trying to convert everything to String. It should not be doing that - instead the Serializer should take care of that if necessary (like MetadataTypedColumnsetSerDe does).
In the old code reduceSinkOperator was passing <object.toString(), objectInspector> pair to Serde, it should actually pass <object, objectInspector>. There is another bug in building JSON strings, I didn't special-case for null list and null map, which is fixed in the same patch.
MetadataTypedColumnsetSerDe converts objects to JSON string only if they are NOT primitive objects, so this does not affect primitive objects like String. The logic in MetadataTypedColumnsetSerDe makes sure the data is written as "\\N" if it is a String type and it's a null.
Once we have the DynamicSerDe working, we will use that to make sure we can pass complex objects from map to reduce (and serialize complex objects).
> Hive: converting complex objects to JSON failed.
> ------------------------------------------------
>
> Key: HADOOP-4231
> URL: https://issues.apache.org/jira/browse/HADOOP-4231
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Minor
> Attachments: HADOOP-4231.0.txt
>
>
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Can not set int field xxx to java.lang.String at
> org.apache.hadoop.hive.serde2.objectinspector.ReflectionStructObjectInspector.getStructFieldData(ReflectionStructObjectInspector.java:123)
> at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:231)
> at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:163)
> at org.apache.hadoop.hive.serde2.MetadataTypedColumnsetSerDe.serialize(MetadataTypedColumnsetSerDe.java:181)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.