You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Chris Freeman <cf...@alteryx.com> on 2015/06/12 17:05:12 UTC
Issues with `when` in Column class
I’m trying to iterate through a list of Columns and create new Columns based on a condition. However, the when method keeps giving me errors that don’t quite make sense.
If I do `when(col === “abc”, 1).otherwise(0)` I get the following error at compile time:
[error] not found: value when
However, this works in the REPL just fine after I import org.apache.spark.sql.Column.
On the other hand, if I do `col.when(col === “abc”, 1).otherwise(0)`, it will compile successfully, but then at runtime, I get this error:
java.lang.IllegalArgumentException: when() can only be applied on a Column previously generated by when() function
This appears to be pretty circular logic. How can `when` only be applied to a Column previously generated by `when`? How would I call `when` in the first place?
Re: Issues with `when` in Column class
Posted by Chris Freeman <cf...@alteryx.com>.
That did it! Thanks!
From: Yin Huai
Date: Friday, June 12, 2015 at 10:31 AM
To: Chris Freeman
Cc: "user@spark.apache.org<ma...@spark.apache.org>"
Subject: Re: Issues with `when` in Column class
Hi Chris,
Have you imported "org.apache.spark.sql.functions._"?
Thanks,
Yin
On Fri, Jun 12, 2015 at 8:05 AM, Chris Freeman <cf...@alteryx.com>> wrote:
I’m trying to iterate through a list of Columns and create new Columns based on a condition. However, the when method keeps giving me errors that don’t quite make sense.
If I do `when(col === “abc”, 1).otherwise(0)` I get the following error at compile time:
[error] not found: value when
However, this works in the REPL just fine after I import org.apache.spark.sql.Column.
On the other hand, if I do `col.when(col === “abc”, 1).otherwise(0)`, it will compile successfully, but then at runtime, I get this error:
java.lang.IllegalArgumentException: when() can only be applied on a Column previously generated by when() function
This appears to be pretty circular logic. How can `when` only be applied to a Column previously generated by `when`? How would I call `when` in the first place?
Re: Issues with `when` in Column class
Posted by Yin Huai <yh...@databricks.com>.
Hi Chris,
Have you imported "org.apache.spark.sql.functions._"?
Thanks,
Yin
On Fri, Jun 12, 2015 at 8:05 AM, Chris Freeman <cf...@alteryx.com> wrote:
> I’m trying to iterate through a list of Columns and create new Columns
> based on a condition. However, the when method keeps giving me errors that
> don’t quite make sense.
>
> If I do `when(col === “abc”, 1).otherwise(0)` I get the following error
> at compile time:
>
> [error] not found: value when
>
> However, this works in the REPL just fine after I import
> org.apache.spark.sql.Column.
>
> On the other hand, if I do `col.when(col === “abc”, 1).otherwise(0)`, it
> will compile successfully, but then at runtime, I get this error:
>
> java.lang.IllegalArgumentException: when() can only be applied on a
> Column previously generated by when() function
>
> This appears to be pretty circular logic. How can `when` only be applied
> to a Column previously generated by `when`? How would I call `when` in the
> first place?
>