You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by "Ganelin, Ilya" <Il...@capitalone.com> on 2014/11/05 17:02:35 UTC
Appropriate way to add a debug flag
Hello all – I am working on https://issues.apache.org/jira/browse/SPARK-3694 and would like to understand the appropriate mechanism by which to check for a debug flag before printing a graph traversal of dependencies of an RDD or Task. I understand that I can use the logging utility and use logDebug to actually print the output but the graph traversal should not be executed unless the debug output is enabled. The code changes I will be making will be in the DAGScheduler and TaskSetManager classes.
Modifying the function interfaces does not seem like the appropriate approach . Is there an existing debug flag that is set somehow within the spark config?
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
RE: Appropriate way to add a debug flag
Posted by "Ganelin, Ilya" <Il...@capitalone.com>.
That perspective - identifying the non serializable components - is actually very helpful. I'll look into figuring out a way to do this. Thanks !
-----Original Message-----
From: Josh Rosen [rosenville@gmail.com<ma...@gmail.com>]
Sent: Friday, November 07, 2014 12:51 PM Eastern Standard Time
To: Ganelin, Ilya; dev
Subject: Re: Appropriate way to add a debug flag
(Whoops, forgot to copy dev@ in my original reply; adding it back)
Yeah, the GraphViz part was mostly for fun and for understanding cyclic object graphs. In general, an object graph might contain cycles, so for understanding the overall structure it's handy to have a picture. The GraphViz thing is actually pretty fun to play with in an interactive notebook environment, since even fairly simple programs can produce really interesting object graphs.
For the purposes of debugging serialization errors, though, I guess you only need to know about some path of non-transient fields that leads from the target object to an unserializable object. For that case, you might be able to add a try-catch block that performs an object graph traversal to find a path to a non-serializable object if a serialization error occurs. Would logging this path be sufficient to debug the most common serialization issues, such as unexpected over-capture of non-serializable objects in closures?
I guess that some users might also want a more general object graph printer / debugger to help debug performance issues related to over-captures that do not lead to errors, but that might be a lower priority / could happen in a separate PR.
Another option would be to do something like http://blog.crazybob.org/2007/02/debugging-serialization.html to print a trace from the serializer's point of view, but the output from that might be hard to understand since it could obscure the chain of references / fields that led to the error.
- Josh
On Thu, Nov 6, 2014 at 12:01 PM, Ganelin, Ilya <Il...@capitalone.com>> wrote:
Hi Josh – I think this could be useful for visualizing references in RDDs but I actually wasn’t sure that this was that the original issue wanted in terms of a solution. I assumed the the more useful output would be a string output. E.g.
RDD
- Child 1
— Child 1.1
— Child 1.2
- Child 2
- Child 3
So that it’s readily integrated with the Spark logs. Would you agree?
I like the SparkConf idea, I will look into that.
From: Josh Rosen <ro...@gmail.com>>
Date: Thursday, November 6, 2014 at 2:42 PM
To: "Ganelin, Ilya" <il...@capitalone.com>>
Subject: Re: Appropriate way to add a debug flag
This is timely, since I’ve actually been hacking on some related stuff in order to debug whether unexpected objects are being pulled into closures. Here’s some code to print a graphviz DOT file that shows the graph of non-transient, non-primitive objects reachable from a given object: https://gist.github.com/JoshRosen/d6a8972c99992e97d040
For enabling / disabling automatic logging of this, I suppose that you could add a configuration option to SparkConf.
On November 5, 2014 at 8:02:35 AM, Ganelin, Ilya (ilya.ganelin@capitalone.com<ma...@capitalone.com>) wrote:
Hello all – I am working on https://issues.apache.org/jira/browse/SPARK-3694 and would like to understand the appropriate mechanism by which to check for a debug flag before printing a graph traversal of dependencies of an RDD or Task. I understand that I can use the logging utility and use logDebug to actually print the output but the graph traversal should not be executed unless the debug output is enabled. The code changes I will be making will be in the DAGScheduler and TaskSetManager classes.
Modifying the function interfaces does not seem like the appropriate approach . Is there an existing debug flag that is set somehow within the spark config?
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Appropriate way to add a debug flag
Posted by Josh Rosen <ro...@gmail.com>.
(Whoops, forgot to copy dev@ in my original reply; adding it back)
Yeah, the GraphViz part was mostly for fun and for understanding cyclic
object graphs. In general, an object graph might contain cycles, so for
understanding the overall structure it's handy to have a picture. The
GraphViz thing is actually pretty fun to play with in an interactive
notebook environment, since even fairly simple programs can produce really
interesting object graphs.
For the purposes of debugging serialization errors, though, I guess you
only need to know about some path of non-transient fields that leads from
the target object to an unserializable object. For that case, you might be
able to add a try-catch block that performs an object graph traversal to
find a path to a non-serializable object if a serialization error occurs.
Would logging this path be sufficient to debug the most common
serialization issues, such as unexpected over-capture of non-serializable
objects in closures?
I guess that some users might also want a more general object graph printer
/ debugger to help debug performance issues related to over-captures that
do not lead to errors, but that might be a lower priority / could happen in
a separate PR.
Another option would be to do something like
http://blog.crazybob.org/2007/02/debugging-serialization.html to print a
trace from the serializer's point of view, but the output from that might
be hard to understand since it could obscure the chain of references /
fields that led to the error.
- Josh
On Thu, Nov 6, 2014 at 12:01 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Hi Josh – I think this could be useful for visualizing references in RDDs
> but I actually wasn’t sure that this was that the original issue wanted in
> terms of a solution. I assumed the the more useful output would be a string
> output. E.g.
>
> RDD
> - Child 1
> — Child 1.1
> — Child 1.2
> - Child 2
> - Child 3
>
> So that it’s readily integrated with the Spark logs. Would you agree?
>
> I like the SparkConf idea, I will look into that.
> From: Josh Rosen <ro...@gmail.com>
> Date: Thursday, November 6, 2014 at 2:42 PM
> To: "Ganelin, Ilya" <il...@capitalone.com>
> Subject: Re: Appropriate way to add a debug flag
>
> This is timely, since I’ve actually been hacking on some related stuff in
> order to debug whether unexpected objects are being pulled into closures.
> Here’s some code to print a graphviz DOT file that shows the graph of
> non-transient, non-primitive objects reachable from a given object:
> https://gist.github.com/JoshRosen/d6a8972c99992e97d040
>
> For enabling / disabling automatic logging of this, I suppose that you
> could add a configuration option to SparkConf.
>
> On November 5, 2014 at 8:02:35 AM, Ganelin, Ilya (
> ilya.ganelin@capitalone.com) wrote:
>
> Hello all – I am working on
> https://issues.apache.org/jira/browse/SPARK-3694 and would like to
> understand the appropriate mechanism by which to check for a debug flag
> before printing a graph traversal of dependencies of an RDD or Task. I
> understand that I can use the logging utility and use logDebug to actually
> print the output but the graph traversal should not be executed unless the
> debug output is enabled. The code changes I will be making will be in the
> DAGScheduler and TaskSetManager classes.
>
> Modifying the function interfaces does not seem like the appropriate
> approach . Is there an existing debug flag that is set somehow within the
> spark config?
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
>
> ------------------------------
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the
> intended recipient, you are hereby notified that any review,
> retransmission, dissemination, distribution, copying or other use of, or
> taking of any action in reliance upon this information is strictly
> prohibited. If you have received this communication in error, please
> contact the sender and delete the material from your computer.
>