You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Chris Freeman <cf...@alteryx.com> on 2015/06/12 17:05:12 UTC

Issues with `when` in Column class

I’m trying to iterate through a list of Columns and create new Columns based on a condition. However, the when method keeps giving me errors that don’t quite make sense.

If I do `when(col === “abc”, 1).otherwise(0)` I get the following error at compile time:

[error] not found: value when

However, this works in the REPL just fine after I import org.apache.spark.sql.Column.

On the other hand, if I do `col.when(col === “abc”, 1).otherwise(0)`, it will compile successfully, but then at runtime, I get this error:

java.lang.IllegalArgumentException: when() can only be applied on a Column previously generated by when() function

This appears to be pretty circular logic. How can `when` only be applied to a Column previously generated by `when`? How would I call `when` in the first place?

Re: Issues with `when` in Column class

Posted by Chris Freeman <cf...@alteryx.com>.
That did it! Thanks!

From: Yin Huai
Date: Friday, June 12, 2015 at 10:31 AM
To: Chris Freeman
Cc: "user@spark.apache.org<ma...@spark.apache.org>"
Subject: Re: Issues with `when` in Column class

Hi Chris,

Have you imported "org.apache.spark.sql.functions._"?

Thanks,

Yin

On Fri, Jun 12, 2015 at 8:05 AM, Chris Freeman <cf...@alteryx.com>> wrote:
I’m trying to iterate through a list of Columns and create new Columns based on a condition. However, the when method keeps giving me errors that don’t quite make sense.

If I do `when(col === “abc”, 1).otherwise(0)` I get the following error at compile time:

[error] not found: value when

However, this works in the REPL just fine after I import org.apache.spark.sql.Column.

On the other hand, if I do `col.when(col === “abc”, 1).otherwise(0)`, it will compile successfully, but then at runtime, I get this error:

java.lang.IllegalArgumentException: when() can only be applied on a Column previously generated by when() function

This appears to be pretty circular logic. How can `when` only be applied to a Column previously generated by `when`? How would I call `when` in the first place?


Re: Issues with `when` in Column class

Posted by Yin Huai <yh...@databricks.com>.
Hi Chris,

Have you imported "org.apache.spark.sql.functions._"?

Thanks,

Yin

On Fri, Jun 12, 2015 at 8:05 AM, Chris Freeman <cf...@alteryx.com> wrote:

>  I’m trying to iterate through a list of Columns and create new Columns
> based on a condition. However, the when method keeps giving me errors that
> don’t quite make sense.
>
>  If I do `when(col === “abc”, 1).otherwise(0)` I get the following error
> at compile time:
>
>  [error] not found: value when
>
>  However, this works in the REPL just fine after I import
> org.apache.spark.sql.Column.
>
>  On the other hand, if I do `col.when(col === “abc”, 1).otherwise(0)`, it
> will compile successfully, but then at runtime, I get this error:
>
>  java.lang.IllegalArgumentException: when() can only be applied on a
> Column previously generated by when() function
>
>  This appears to be pretty circular logic. How can `when` only be applied
> to a Column previously generated by `when`? How would I call `when` in the
> first place?
>