You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Kevin Wilfong (Created) (JIRA)" <ji...@apache.org> on 2011/12/20 22:39:30 UTC
[jira] [Created] (HIVE-2666) StackOverflowError when using custom
UDF in map join
StackOverflowError when using custom UDF in map join
----------------------------------------------------
Key: HIVE-2666
URL: https://issues.apache.org/jira/browse/HIVE-2666
Project: Hive
Issue Type: Bug
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
When a custom UDF is used as part of a join which is converted to a map join, the XMLEncoder enters an infinite loop when serializing the map reduce task for the second time, as part of sending it to be executed. This results in a stack overflow error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2666) StackOverflowError when using custom
UDF in map join
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2666:
------------------------------
Attachment: HIVE-2666.D957.1.patch
kevinwilfong requested code review of "HIVE-2666 [jira] StackOverflowError when using custom UDF in map join".
Reviewers: JIRA
Resource files are now added to the class path as soon as they are added via the CLI. This fixes the stack overflow error mentioned in the JIRA by ensuring a consistent class loader between serializers and deserializers for the same query.
Note that now serdes which contain a static block to register themselves are now registered twice, once when adding the file to the class loader, and once when an instance of the class is created. Previously, registering a serde twice resulted in an exception, to avoid this, I have downgraded it to a warning.
When a custom UDF is used as part of a join which is converted to a map join, the XMLEncoder enters an infinite loop when serializing the map reduce task for the second time, as part of sending it to be executed. This results in a stack overflow error.
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D957
AFFECTED FILES
serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java
ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/1989/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> StackOverflowError when using custom UDF in map join
> ----------------------------------------------------
>
> Key: HIVE-2666
> URL: https://issues.apache.org/jira/browse/HIVE-2666
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2666.D957.1.patch
>
>
> When a custom UDF is used as part of a join which is converted to a map join, the XMLEncoder enters an infinite loop when serializing the map reduce task for the second time, as part of sending it to be executed. This results in a stack overflow error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2666) StackOverflowError when using custom
UDF in map join
Posted by "He Yongqiang (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
He Yongqiang resolved HIVE-2666.
--------------------------------
Resolution: Fixed
committed, thanks Kevin!
> StackOverflowError when using custom UDF in map join
> ----------------------------------------------------
>
> Key: HIVE-2666
> URL: https://issues.apache.org/jira/browse/HIVE-2666
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2666.D957.1.patch
>
>
> When a custom UDF is used as part of a join which is converted to a map join, the XMLEncoder enters an infinite loop when serializing the map reduce task for the second time, as part of sending it to be executed. This results in a stack overflow error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2666) StackOverflowError when using custom
UDF in map join
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173617#comment-13173617 ]
Phabricator commented on HIVE-2666:
-----------------------------------
kevinwilfong has commented on the revision "HIVE-2666 [jira] StackOverflowError when using custom UDF in map join".
There is a unit test query in TestNegativeCliDriver called deletejar, which verifies an error is thrown if you add a jar, delete that jar, and then attempt to use a class in the jar.
REVISION DETAIL
https://reviews.facebook.net/D957
> StackOverflowError when using custom UDF in map join
> ----------------------------------------------------
>
> Key: HIVE-2666
> URL: https://issues.apache.org/jira/browse/HIVE-2666
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2666.D957.1.patch
>
>
> When a custom UDF is used as part of a join which is converted to a map join, the XMLEncoder enters an infinite loop when serializing the map reduce task for the second time, as part of sending it to be executed. This results in a stack overflow error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2666) StackOverflowError when using custom
UDF in map join
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174279#comment-13174279 ]
Phabricator commented on HIVE-2666:
-----------------------------------
heyongqiang has committed the revision "HIVE-2666 [jira] StackOverflowError when using custom UDF in map join".
REVISION DETAIL
https://reviews.facebook.net/D957
COMMIT
https://reviews.facebook.net/rHIVE1221830
> StackOverflowError when using custom UDF in map join
> ----------------------------------------------------
>
> Key: HIVE-2666
> URL: https://issues.apache.org/jira/browse/HIVE-2666
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2666.D957.1.patch
>
>
> When a custom UDF is used as part of a join which is converted to a map join, the XMLEncoder enters an infinite loop when serializing the map reduce task for the second time, as part of sending it to be executed. This results in a stack overflow error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2666) StackOverflowError when using custom
UDF in map join
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173714#comment-13173714 ]
Phabricator commented on HIVE-2666:
-----------------------------------
heyongqiang has accepted the revision "HIVE-2666 [jira] StackOverflowError when using custom UDF in map join".
running tests
REVISION DETAIL
https://reviews.facebook.net/D957
> StackOverflowError when using custom UDF in map join
> ----------------------------------------------------
>
> Key: HIVE-2666
> URL: https://issues.apache.org/jira/browse/HIVE-2666
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2666.D957.1.patch
>
>
> When a custom UDF is used as part of a join which is converted to a map join, the XMLEncoder enters an infinite loop when serializing the map reduce task for the second time, as part of sending it to be executed. This results in a stack overflow error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2666) StackOverflowError when using custom
UDF in map join
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173609#comment-13173609 ]
Phabricator commented on HIVE-2666:
-----------------------------------
heyongqiang has commented on the revision "HIVE-2666 [jira] StackOverflowError when using custom UDF in map join".
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java:63 have u verified the delete also work fine?
REVISION DETAIL
https://reviews.facebook.net/D957
> StackOverflowError when using custom UDF in map join
> ----------------------------------------------------
>
> Key: HIVE-2666
> URL: https://issues.apache.org/jira/browse/HIVE-2666
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2666.D957.1.patch
>
>
> When a custom UDF is used as part of a join which is converted to a map join, the XMLEncoder enters an infinite loop when serializing the map reduce task for the second time, as part of sending it to be executed. This results in a stack overflow error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2666) StackOverflowError when using custom
UDF in map join
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174331#comment-13174331 ]
Hudson commented on HIVE-2666:
------------------------------
Integrated in Hive-trunk-h0.23.0 #42 (See [https://builds.apache.org/job/Hive-trunk-h0.23.0/42/])
HIVE-2666 [jira] StackOverflowError when using custom UDF in map join
(Kevin Wilfong via Yongqiang He)
Summary:
Resource files are now added to the class path as soon as they are added via the
CLI. This fixes the stack overflow error mentioned in the JIRA by ensuring a
consistent class loader between serializers and deserializers for the same
query.
Note that now serdes which contain a static block to register themselves are now
registered twice, once when adding the file to the class loader, and once when
an instance of the class is created. Previously, registering a serde twice
resulted in an exception, to avoid this, I have downgraded it to a warning.
When a custom UDF is used as part of a join which is converted to a map join,
the XMLEncoder enters an infinite loop when serializing the map reduce task for
the second time, as part of sending it to be executed. This results in a stack
overflow error.
Test Plan:
I ran the unit tests to verify nothing was broken.
I ran several queries which used custom UDFs and involved a join which was
converted to a map join. I verified these completed successfully consistently
Reviewers: JIRA, heyongqiang
Reviewed By: heyongqiang
CC: heyongqiang, kevinwilfong
Differential Revision: 957
heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1221830
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java
> StackOverflowError when using custom UDF in map join
> ----------------------------------------------------
>
> Key: HIVE-2666
> URL: https://issues.apache.org/jira/browse/HIVE-2666
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2666.D957.1.patch
>
>
> When a custom UDF is used as part of a join which is converted to a map join, the XMLEncoder enters an infinite loop when serializing the map reduce task for the second time, as part of sending it to be executed. This results in a stack overflow error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2666) StackOverflowError when using custom
UDF in map join
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174521#comment-13174521 ]
Hudson commented on HIVE-2666:
------------------------------
Integrated in Hive-trunk-h0.21 #1164 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1164/])
HIVE-2666 [jira] StackOverflowError when using custom UDF in map join
(Kevin Wilfong via Yongqiang He)
Summary:
Resource files are now added to the class path as soon as they are added via the
CLI. This fixes the stack overflow error mentioned in the JIRA by ensuring a
consistent class loader between serializers and deserializers for the same
query.
Note that now serdes which contain a static block to register themselves are now
registered twice, once when adding the file to the class loader, and once when
an instance of the class is created. Previously, registering a serde twice
resulted in an exception, to avoid this, I have downgraded it to a warning.
When a custom UDF is used as part of a join which is converted to a map join,
the XMLEncoder enters an infinite loop when serializing the map reduce task for
the second time, as part of sending it to be executed. This results in a stack
overflow error.
Test Plan:
I ran the unit tests to verify nothing was broken.
I ran several queries which used custom UDFs and involved a join which was
converted to a map join. I verified these completed successfully consistently
Reviewers: JIRA, heyongqiang
Reviewed By: heyongqiang
CC: heyongqiang, kevinwilfong
Differential Revision: 957
heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1221830
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java
> StackOverflowError when using custom UDF in map join
> ----------------------------------------------------
>
> Key: HIVE-2666
> URL: https://issues.apache.org/jira/browse/HIVE-2666
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2666.D957.1.patch
>
>
> When a custom UDF is used as part of a join which is converted to a map join, the XMLEncoder enters an infinite loop when serializing the map reduce task for the second time, as part of sending it to be executed. This results in a stack overflow error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira