You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (Created) (JIRA)" <ji...@apache.org> on 2012/01/04 21:59:39 UTC
[jira] [Created] (PIG-2457) JsonLoaderStorage tests is broken for
e2e
JsonLoaderStorage tests is broken for e2e
-----------------------------------------
Key: PIG-2457
URL: https://issues.apache.org/jira/browse/PIG-2457
Project: Pig
Issue Type: Bug
Affects Versions: 0.10, 0.11
Reporter: Daniel Dai
Assignee: Daniel Dai
Fix For: 0.10, 0.11
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2457) JsonLoaderStorage tests is broken for
e2e
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180100#comment-13180100 ]
Daniel Dai commented on PIG-2457:
---------------------------------
PigStorage also fail with "-schema" option (before I thought it by default creates the schema file).
Yes, version mismatch is a bigger problem. In frontend, user can use "HADOOP_USER_CLASSPATH_FIRST" to override hadoop bundled jars, but for backend, hadoop always use bundled jars first. In our case, hadoop cannot find LRUMap in bundled jackson, so it goes to job.jar to find it. For Avro, it is not that lucky. I don't know a solution for that. We shall definitely push hadoop to bundled a newer jackson in next version (I am testing on 20.205).
> JsonLoaderStorage tests is broken for e2e
> -----------------------------------------
>
> Key: PIG-2457
> URL: https://issues.apache.org/jira/browse/PIG-2457
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10, 0.11
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2457-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2457) JsonLoaderStorage tests is broken for
e2e
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180005#comment-13180005 ]
Daniel Dai commented on PIG-2457:
---------------------------------
I think OutputCommitter is in backend. We will have to ship it or use hadoop's version of jackson. Another possible solution is to find out more specific package than org.codehaus.jackson(cuz only LRU class is not in 1.0.1), so we can ship less classes.
> JsonLoaderStorage tests is broken for e2e
> -----------------------------------------
>
> Key: PIG-2457
> URL: https://issues.apache.org/jira/browse/PIG-2457
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10, 0.11
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2457-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2457) JsonLoaderStorage tests is broken for
e2e
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180016#comment-13180016 ]
Daniel Dai commented on PIG-2457:
---------------------------------
But PigStorage works fine, seems there is still something missing. I am looking.
> JsonLoaderStorage tests is broken for e2e
> -----------------------------------------
>
> Key: PIG-2457
> URL: https://issues.apache.org/jira/browse/PIG-2457
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10, 0.11
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2457-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2457) JsonLoaderStorage tests is broken for
e2e
Posted by "Dmitriy V. Ryaboy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179901#comment-13179901 ]
Dmitriy V. Ryaboy commented on PIG-2457:
----------------------------------------
(the reason I ask is, if we are blowing up the size of our job jar and it's not necessary, I can just write my own LRUMap, it's pretty trivial)
> JsonLoaderStorage tests is broken for e2e
> -----------------------------------------
>
> Key: PIG-2457
> URL: https://issues.apache.org/jira/browse/PIG-2457
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10, 0.11
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2457-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2457) JsonLoaderStorage tests is broken for
e2e
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179910#comment-13179910 ]
Daniel Dai commented on PIG-2457:
---------------------------------
Good question. Is that possible jackon-core.jar is part of hadoop, but jackson-mapper is not? Surely large job.jar is not good.
> JsonLoaderStorage tests is broken for e2e
> -----------------------------------------
>
> Key: PIG-2457
> URL: https://issues.apache.org/jira/browse/PIG-2457
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10, 0.11
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2457-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2457) JsonLoaderStorage tests is broken for
e2e
Posted by "Dmitriy V. Ryaboy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179898#comment-13179898 ]
Dmitriy V. Ryaboy commented on PIG-2457:
----------------------------------------
How did this work before? JsonMetaData already used jackson (but not the util package).
import org.codehaus.jackson.JsonParseException;
import org.codehaus.jackson.map.JsonMappingException;
import org.codehaus.jackson.map.ObjectMapper;
> JsonLoaderStorage tests is broken for e2e
> -----------------------------------------
>
> Key: PIG-2457
> URL: https://issues.apache.org/jira/browse/PIG-2457
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10, 0.11
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2457-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2457) JsonLoaderStorage tests is broken for
e2e
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179914#comment-13179914 ]
Daniel Dai commented on PIG-2457:
---------------------------------
Here is the stack BTW:
Error: java.lang.ClassNotFoundException: org.codehaus.jackson.map.util.LRUMap at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) at org.apache.pig.builtin.JsonMetadata.<init>(JsonMetadata.java:75) at org.apache.pig.builtin.JsonStorage.storeSchema(JsonStorage.java:269) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.storeCleanup(PigOutputCommitter.java:143) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:180) at org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:1002) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:357) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:249)
> JsonLoaderStorage tests is broken for e2e
> -----------------------------------------
>
> Key: PIG-2457
> URL: https://issues.apache.org/jira/browse/PIG-2457
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10, 0.11
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2457-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2457) JsonLoaderStorage tests is broken for
e2e
Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2457:
----------------------------
Attachment: PIG-2457-1.patch
PIG-2453 introduced jackson utilities into JsonMetaData, which is use in JsonLoader/JsonStorage in backend. We need to ship these classes.
> JsonLoaderStorage tests is broken for e2e
> -----------------------------------------
>
> Key: PIG-2457
> URL: https://issues.apache.org/jira/browse/PIG-2457
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10, 0.11
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2457-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2457) JsonLoaderStorage tests is broken for
e2e
Posted by "Dmitriy V. Ryaboy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180036#comment-13180036 ]
Dmitriy V. Ryaboy commented on PIG-2457:
----------------------------------------
Like I said, I can just write an LRUMap of our own, it's not hard.
But I am worried about the version mismatch. This will affect AvroStorage too, Avro folks are the ones who requested that we bump our jackson version in the first place.
> JsonLoaderStorage tests is broken for e2e
> -----------------------------------------
>
> Key: PIG-2457
> URL: https://issues.apache.org/jira/browse/PIG-2457
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10, 0.11
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2457-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2457) JsonLoaderStorage tests is broken for
e2e
Posted by "Alan Gates (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179871#comment-13179871 ]
Alan Gates commented on PIG-2457:
---------------------------------
+1, looks good.
> JsonLoaderStorage tests is broken for e2e
> -----------------------------------------
>
> Key: PIG-2457
> URL: https://issues.apache.org/jira/browse/PIG-2457
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10, 0.11
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2457-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2457) JsonLoaderStorage tests is broken for
e2e
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179923#comment-13179923 ]
Daniel Dai commented on PIG-2457:
---------------------------------
I see the problem. Hadoop 20.205 bundles jackson 1.0.1, which does not contain LRUMap.
> JsonLoaderStorage tests is broken for e2e
> -----------------------------------------
>
> Key: PIG-2457
> URL: https://issues.apache.org/jira/browse/PIG-2457
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10, 0.11
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2457-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2457) JsonLoaderStorage tests is broken for
e2e
Posted by "Daniel Dai (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai resolved PIG-2457.
-----------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Patch committed to trunk/0.10.
> JsonLoaderStorage tests is broken for e2e
> -----------------------------------------
>
> Key: PIG-2457
> URL: https://issues.apache.org/jira/browse/PIG-2457
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10, 0.11
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2457-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2457) JsonLoaderStorage tests is broken for
e2e
Posted by "Dmitriy V. Ryaboy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179993#comment-13179993 ]
Dmitriy V. Ryaboy commented on PIG-2457:
----------------------------------------
We should probably ensure our classes are used then. I checked the size of the jars, they are about 800K:
-rw-r--r-- 1 dmitriy staff 203K Feb 14 2011 build/ivy/lib/Pig/jackson-core-asl-1.7.3.jar
-rw-r--r-- 1 dmitriy staff 611K Feb 14 2011 build/ivy/lib/Pig/jackson-mapper-asl-1.7.3.jar
Kind of painful. We do need both the mapper and core. Do Hadoop's jars override ours on the classpath?
> JsonLoaderStorage tests is broken for e2e
> -----------------------------------------
>
> Key: PIG-2457
> URL: https://issues.apache.org/jira/browse/PIG-2457
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10, 0.11
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2457-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira