Deciphering spark warning "Truncated the string representation of a plan since it was too large."

[Henry M <he...@gmail.com>]

I am trying to understand if I should be concerned about this warning:

"WARN  Utils:66 - Truncated the string representation of a plan since it
was too large. This behavior can be adjusted by setting
'spark.debug.maxToStringFields' in SparkEnv.conf"

It occurs while writing a data frame to parquet.

Has any one on this list looked into this warning before and could help
explain what it means?

Thank you for your help,
Henry

Re: Deciphering spark warning "Truncated the string representation of a plan since it was too large."

["lucas.gary@gmail.com" <lu...@gmail.com>]

AFAIK the process a spark program follows is:

   1. A set of transformations are defined on a given input dataset.
   2. At some point an action is called
      1. In your case this is writing to your parquet file.
   3. When that happens spark creates a logical plan and then a physical
   plan (This is largely where your transformations are optimized) to perform
   the transformations specified.
      1. This is similar to what a sql engine does, it takes your raw SQL
      and turns it into something that it can execute to get the data you
      requested.
      2. There are a set of artifacts generated, one of those artifacts
      would be the plan that you're seeing is being truncated.

The only time I'd be concerned about this would be if I was debugging the
code and needed to see what was being truncated, it is after all a debug
setting ('spark.debug.maxToStringFields')

Good luck!

Gary

On 12 June 2017 at 15:10, Henry M <he...@gmail.com> wrote:

>
>
> I am trying to understand if I should be concerned about this warning:
>
> "WARN  Utils:66 - Truncated the string representation of a plan since it
> was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields'
> in SparkEnv.conf"
>
> It occurs while writing a data frame to parquet.
>
> Has any one on this list looked into this warning before and could help
> explain what it means?
>
> Thank you for your help,
> Henry
>