You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Muthu Jayakumar <ba...@gmail.com> on 2017/02/16 16:26:40 UTC

Pretty print a dataframe...

Hello there,

I am trying to write to log-line a dataframe/dataset queryExecution and/or
its logical plan. The current code...

def explain(extended: Boolean): Unit = {
  val explain = ExplainCommand(queryExecution.logical, extended = extended)
  sparkSession.sessionState.executePlan(explain).executedPlan.executeCollect().foreach
{
    // scalastyle:off println
    r => println(r.getString(0))
    // scalastyle:on println
  }
}

sessionState is not accessible if I were to write my own explain(log:
LoggingAdapter).

Please advice,
Muthu

Re: Pretty print a dataframe...

Posted by Muthu Jayakumar <ba...@gmail.com>.

This worked. Thanks for the tip Michael.

Thanks,
Muthu

On Thu, Feb 16, 2017 at 12:41 PM, Michael Armbrust <mi...@databricks.com>
wrote:

> The toString method of Dataset.queryExecution includes the various plans.
> I usually just log that directly.
>
> On Thu, Feb 16, 2017 at 8:26 AM, Muthu Jayakumar <ba...@gmail.com>
> wrote:
>
>> Hello there,
>>
>> I am trying to write to log-line a dataframe/dataset queryExecution
>> and/or its logical plan. The current code...
>>
>> def explain(extended: Boolean): Unit = {
>>   val explain = ExplainCommand(queryExecution.logical, extended = extended)
>>   sparkSession.sessionState.executePlan(explain).executedPlan.executeCollect().foreach {
>>     // scalastyle:off println
>>     r => println(r.getString(0))
>>     // scalastyle:on println
>>   }
>> }
>>
>> sessionState is not accessible if I were to write my own explain(log:
>> LoggingAdapter).
>>
>> Please advice,
>> Muthu
>>
>
>

Re: Pretty print a dataframe...

Posted by Michael Armbrust <mi...@databricks.com>.

The toString method of Dataset.queryExecution includes the various plans.
I usually just log that directly.

On Thu, Feb 16, 2017 at 8:26 AM, Muthu Jayakumar <ba...@gmail.com> wrote:

> Hello there,
>
> I am trying to write to log-line a dataframe/dataset queryExecution and/or
> its logical plan. The current code...
>
> def explain(extended: Boolean): Unit = {
>   val explain = ExplainCommand(queryExecution.logical, extended = extended)
>   sparkSession.sessionState.executePlan(explain).executedPlan.executeCollect().foreach {
>     // scalastyle:off println
>     r => println(r.getString(0))
>     // scalastyle:on println
>   }
> }
>
> sessionState is not accessible if I were to write my own explain(log:
> LoggingAdapter).
>
> Please advice,
> Muthu
>