You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by satish chandra j <js...@gmail.com> on 2016/01/07 12:04:36 UTC
Window Functions importing issue in Spark 1.4.0
HI All,
Currently using Spark 1.4.0 version, I have a requirement to add a column
having Sequential Numbering to an existing DataFrame
I understand Window Function "rowNumber" serves my purpose
hence I have below import statements to include the same
import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions.rowNumber
But I am getting an error at the import statement itself such as
"object expressions is not a member of package org.apache.spark.sql"
"value rowNumber is not a member of object org.apache.spark.sql.functions"
Could anybody throw some light if any to fix the issue
Regards,
Satish Chandra
Re: Window Functions importing issue in Spark 1.4.0
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
According to https://github.com/apache/spark/commit/f6f2eeb17910b5d446dfd61839e37dd698d0860f
where it was first introduced, you need one of the following versions
of Spark as a dependency:
v1.6.0 v1.6.0-rc3 v1.6.0-rc2 v1.6.0-rc1 v1.6.0-preview2
v1.6.0-preview1 v1.6.0-preview v1.6-snapshot0-test v1.5.2 v1.5.1
v1.5.0
Pozdrawiam,
Jacek
Jacek Laskowski | https://medium.com/@jaceklaskowski/
Mastering Apache Spark
==> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
Follow me at https://twitter.com/jaceklaskowski
On Thu, Jan 7, 2016 at 12:04 PM, satish chandra j
<js...@gmail.com> wrote:
> HI All,
> Currently using Spark 1.4.0 version, I have a requirement to add a column
> having Sequential Numbering to an existing DataFrame
> I understand Window Function "rowNumber" serves my purpose
> hence I have below import statements to include the same
>
> import org.apache.spark.sql.expressions.Window
> import org.apache.spark.sql.functions.rowNumber
>
> But I am getting an error at the import statement itself such as
> "object expressions is not a member of package org.apache.spark.sql"
>
> "value rowNumber is not a member of object org.apache.spark.sql.functions"
>
> Could anybody throw some light if any to fix the issue
>
> Regards,
> Satish Chandra
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Window Functions importing issue in Spark 1.4.0
Posted by satish chandra j <js...@gmail.com>.
Hi Ted,
I understand there are some fixes which are applied on windows functions in
Spark 1.5.2
Hence it is not suitable to use Window function in Spark 1.4.0
Correct me if I am misleading the info in the given link below
https://github.com/apache/spark/pull/9050
On Thu, Jan 7, 2016 at 4:41 PM, Ted Yu <yu...@gmail.com> wrote:
> Please take a look at the following for sample on how rowNumber is used:
> https://github.com/apache/spark/pull/9050
>
> BTW 1.4.0 was an old release.
>
> Please consider upgrading.
>
> On Thu, Jan 7, 2016 at 3:04 AM, satish chandra j <jsatishchandra@gmail.com
> > wrote:
>
>> HI All,
>> Currently using Spark 1.4.0 version, I have a requirement to add a column
>> having Sequential Numbering to an existing DataFrame
>> I understand Window Function "rowNumber" serves my purpose
>> hence I have below import statements to include the same
>>
>> import org.apache.spark.sql.expressions.Window
>> import org.apache.spark.sql.functions.rowNumber
>>
>> But I am getting an error at the import statement itself such as
>> "object expressions is not a member of package org.apache.spark.sql"
>>
>> "value rowNumber is not a member of object org.apache.spark.sql.functions"
>>
>> Could anybody throw some light if any to fix the issue
>>
>> Regards,
>> Satish Chandra
>>
>
>
Re: Window Functions importing issue in Spark 1.4.0
Posted by satish chandra j <js...@gmail.com>.
Hi Ted,
Thanks for sharing the link on rowNumber example on usage
Could you please let me know if I could use rowNumber window function in my
currenct Spark 1.4.0 version
If yes, than why am I getting error in "import org.apache.spark.sql.
expressions.Window" and "import org.apache.spark.sql.functions.rowNumber"
Thanks for providing your valuable inputs
Regards,
Satish Chandra J
On Thu, Jan 7, 2016 at 4:41 PM, Ted Yu <yu...@gmail.com> wrote:
> Please take a look at the following for sample on how rowNumber is used:
> https://github.com/apache/spark/pull/9050
>
> BTW 1.4.0 was an old release.
>
> Please consider upgrading.
>
> On Thu, Jan 7, 2016 at 3:04 AM, satish chandra j <jsatishchandra@gmail.com
> > wrote:
>
>> HI All,
>> Currently using Spark 1.4.0 version, I have a requirement to add a column
>> having Sequential Numbering to an existing DataFrame
>> I understand Window Function "rowNumber" serves my purpose
>> hence I have below import statements to include the same
>>
>> import org.apache.spark.sql.expressions.Window
>> import org.apache.spark.sql.functions.rowNumber
>>
>> But I am getting an error at the import statement itself such as
>> "object expressions is not a member of package org.apache.spark.sql"
>>
>> "value rowNumber is not a member of object org.apache.spark.sql.functions"
>>
>> Could anybody throw some light if any to fix the issue
>>
>> Regards,
>> Satish Chandra
>>
>
>
Re: Window Functions importing issue in Spark 1.4.0
Posted by Jacek Laskowski <ja...@japila.pl>.
Ok, enuf! :) Leaving the room for now as I'm like a copycat :)
https://en.wiktionary.org/wiki/enuf
Pozdrawiam,
Jacek
Jacek Laskowski | https://medium.com/@jaceklaskowski/
Mastering Apache Spark
==> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
Follow me at https://twitter.com/jaceklaskowski
On Thu, Jan 7, 2016 at 12:11 PM, Ted Yu <yu...@gmail.com> wrote:
> Please take a look at the following for sample on how rowNumber is used:
> https://github.com/apache/spark/pull/9050
>
> BTW 1.4.0 was an old release.
>
> Please consider upgrading.
>
> On Thu, Jan 7, 2016 at 3:04 AM, satish chandra j <js...@gmail.com>
> wrote:
>>
>> HI All,
>> Currently using Spark 1.4.0 version, I have a requirement to add a column
>> having Sequential Numbering to an existing DataFrame
>> I understand Window Function "rowNumber" serves my purpose
>> hence I have below import statements to include the same
>>
>> import org.apache.spark.sql.expressions.Window
>> import org.apache.spark.sql.functions.rowNumber
>>
>> But I am getting an error at the import statement itself such as
>> "object expressions is not a member of package org.apache.spark.sql"
>>
>> "value rowNumber is not a member of object org.apache.spark.sql.functions"
>>
>> Could anybody throw some light if any to fix the issue
>>
>> Regards,
>> Satish Chandra
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Window Functions importing issue in Spark 1.4.0
Posted by Ted Yu <yu...@gmail.com>.
Please take a look at the following for sample on how rowNumber is used:
https://github.com/apache/spark/pull/9050
BTW 1.4.0 was an old release.
Please consider upgrading.
On Thu, Jan 7, 2016 at 3:04 AM, satish chandra j <js...@gmail.com>
wrote:
> HI All,
> Currently using Spark 1.4.0 version, I have a requirement to add a column
> having Sequential Numbering to an existing DataFrame
> I understand Window Function "rowNumber" serves my purpose
> hence I have below import statements to include the same
>
> import org.apache.spark.sql.expressions.Window
> import org.apache.spark.sql.functions.rowNumber
>
> But I am getting an error at the import statement itself such as
> "object expressions is not a member of package org.apache.spark.sql"
>
> "value rowNumber is not a member of object org.apache.spark.sql.functions"
>
> Could anybody throw some light if any to fix the issue
>
> Regards,
> Satish Chandra
>