You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Kent Yao (JIRA)" <ji...@apache.org> on 2018/08/21 09:00:04 UTC
[jira] [Updated] (SPARK-25174) ApplicationMaster suspends when
unregistering itself from RM with extreme large diagnostic message
[ https://issues.apache.org/jira/browse/SPARK-25174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kent Yao updated SPARK-25174:
-----------------------------
Description:
We recently ran into SPARK-18016 which has been fixed in v2.3.0. This JIRA is not about the issue in SPARK-18016 but the side-effect which it brings. When SPARK-18016 occurs, ApplicationMaster fails unregistering itself because the exception contains extreme large error information.
{code:java}
ERROR yarn.ApplicationMaster: User class threw exception: java.lang.RuntimeException: Error while decoding: java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException: Constant pool has grown past JVM limit of 0xFFFF
/* 001 */ public java.lang.Object generate(Object[] references) {
....
/* 395656 */ mutableRow.update(0, value);
/* 395657 */ }
/* 395658 */
/* 395659 */ return mutableRow;
/* 395660 */ }
/* 395661 */ }
{code}
The above codegen text is included in the final message for AM to wave goodbye to RM, while it ends up crashing the rm's ZKRMStateStore for YARN-6125 not covering the unregisterApplicationMaster's message truncation. We also create an Jira on YARN Side https://issues.apache.org/jira/browse/YARN-8691
Although SPARK-18016 fixed already, there are maybe other uncaught exception will cause this problem. I guess that we should limit the error message's size sent to RM while unregistering .
was:
We recently ran into SPARK-18016 which has been fixed in v2.3.0. This JIRA is not about the issue in SPARK-18016 but the side-effect which it brings. When SPARK-18016 occurs, ApplicationMaster fails unregistering itself because the exception contains extreme large error information.
{code:java}
ERROR yarn.ApplicationMaster: User class threw exception: java.lang.RuntimeException: Error while decoding: java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException: Constant pool has grown past JVM limit of 0xFFFF
/* 001 */ public java.lang.Object generate(Object[] references) {
....
/* 395656 */ mutableRow.update(0, value);
/* 395657 */ }
/* 395658 */
/* 395659 */ return mutableRow;
/* 395660 */ }
/* 395661 */ }
{code}
The above codegen text is included in the final message for AM to wave goodbye to RM, while it ends up crashing the rm's ZKRMStateStore for YARN-6125 not covering the unregisterApplicationMaster's message truncation. We also create an Jira on YARN Side https://issues.apache.org/jira/browse/YARN-8691
> ApplicationMaster suspends when unregistering itself from RM with extreme large diagnostic message
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-25174
> URL: https://issues.apache.org/jira/browse/SPARK-25174
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.1.1
> Reporter: Kent Yao
> Priority: Major
>
> We recently ran into SPARK-18016 which has been fixed in v2.3.0. This JIRA is not about the issue in SPARK-18016 but the side-effect which it brings. When SPARK-18016 occurs, ApplicationMaster fails unregistering itself because the exception contains extreme large error information.
> {code:java}
> ERROR yarn.ApplicationMaster: User class threw exception: java.lang.RuntimeException: Error while decoding: java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException: Constant pool has grown past JVM limit of 0xFFFF
> /* 001 */ public java.lang.Object generate(Object[] references) {
> ....
> /* 395656 */ mutableRow.update(0, value);
> /* 395657 */ }
> /* 395658 */
> /* 395659 */ return mutableRow;
> /* 395660 */ }
> /* 395661 */ }
> {code}
> The above codegen text is included in the final message for AM to wave goodbye to RM, while it ends up crashing the rm's ZKRMStateStore for YARN-6125 not covering the unregisterApplicationMaster's message truncation. We also create an Jira on YARN Side https://issues.apache.org/jira/browse/YARN-8691
> Although SPARK-18016 fixed already, there are maybe other uncaught exception will cause this problem. I guess that we should limit the error message's size sent to RM while unregistering .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org