You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by saurabh3d <sa...@oracle.com> on 2016/09/01 05:16:21 UTC
Window Functions with SQLContext
Hi All,
As per SPARK-11001 <https://issues.apache.org/jira/browse/SPARK-11001> ,
Window functions should be supported by SQLContext. But when i try to run
SQLContext sqlContext = new SQLContext(jsc);
WindowSpec w = Window.partitionBy("assetId").orderBy("assetId");
DataFrame df_2 = df1.withColumn("row_number", row_number().over(w));
df_2.show(false);
it fails with:
Exception in thread "main" org.apache.spark.sql.AnalysisException: Could not
resolve window function 'row_number'. Note that, using window functions
currently requires a HiveContext;
This code runs fine with HiveContext.
Any idea what’s going on? Is this a known issue and is there a workaround
to make Window function work without HiveContext.
Thanks,
Saurabh
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Window-Functions-with-SQLContext-tp27636.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
RE: Window Functions with SQLContext
Posted by Saurabh Dubey <sa...@oracle.com>.
Hi Divya,
Then, how can https://issues.apache.org/jira/browse/SPARK-11001 be resolved.
Thanks,
Saurabh
From: Divya Gehlot [mailto:divya.htconex@gmail.com]
Sent: 01 September 2016 11:33
To: saurabh3d
Cc: user @spark
Subject: Re: Window Functions with SQLContext
Hi Saurabh,
Even I am using Spark 1.6+ version ..and when I didnt create hiveContext it threw the same error .
So have to create HiveContext to access windows function
Thanks,
Divya
On 1 September 2016 at 13:16, saurabh3d <HYPERLINK "mailto:saurabh.s.dubey@oracle.com" \nsaurabh.s.dubey@oracle.com> wrote:
Hi All,
As per SPARK-11001 <https://issues.apache.org/jira/browse/SPARK-11001> ,
Window functions should be supported by SQLContext. But when i try to run
SQLContext sqlContext = new SQLContext(jsc);
WindowSpec w = Window.partitionBy("assetId").orderBy("assetId");
DataFrame df_2 = df1.withColumn("row_number", row_number().over(w));
df_2.show(false);
it fails with:
Exception in thread "main" org.apache.spark.sql.AnalysisException: Could not
resolve window function 'row_number'. Note that, using window functions
currently requires a HiveContext;
This code runs fine with HiveContext.
Any idea what’s going on? Is this a known issue and is there a workaround
to make Window function work without HiveContext.
Thanks,
Saurabh
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Window-Functions-with-SQLContext-tp27636.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: HYPERLINK "mailto:user-unsubscribe@spark.apache.org"user-unsubscribe@spark.apache.org
Re: Window Functions with SQLContext
Posted by Divya Gehlot <di...@gmail.com>.
Hi Saurabh,
Even I am using Spark 1.6+ version ..and when I didnt create hiveContext it
threw the same error .
So have to create HiveContext to access windows function
Thanks,
Divya
On 1 September 2016 at 13:16, saurabh3d <sa...@oracle.com> wrote:
> Hi All,
>
> As per SPARK-11001 <https://issues.apache.org/jira/browse/SPARK-11001>
> ,
> Window functions should be supported by SQLContext. But when i try to run
>
> SQLContext sqlContext = new SQLContext(jsc);
> WindowSpec w = Window.partitionBy("assetId").orderBy("assetId");
> DataFrame df_2 = df1.withColumn("row_number", row_number().over(w));
> df_2.show(false);
>
> it fails with:
> Exception in thread "main" org.apache.spark.sql.AnalysisException: Could
> not
> resolve window function 'row_number'. Note that, using window functions
> currently requires a HiveContext;
>
> This code runs fine with HiveContext.
> Any idea what’s going on? Is this a known issue and is there a workaround
> to make Window function work without HiveContext.
>
> Thanks,
> Saurabh
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/Window-Functions-with-SQLContext-tp27636.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
RE: Window Functions with SQLContext
Posted by Saurabh Dubey <sa...@oracle.com>.
Hi Ayan,
Even with SQL query like:
SQLContext sqlContext = new SQLContext(jsc);
DataFrame df_2_sql = sqlContext.sql("select assetId, row_number() over ( partition by " +
"assetId order by assetId) as " +
"serial from df1");
df_2_sql.show(false);
It fails with:
Exception in thread "main" java.lang.RuntimeException: [1.35] failure: ``union'' expected but `(' found
select assetId, row_number() over ( partition by assetId order by assetId) as serial from df1
^
at scala.sys.package$.error(package.scala:27)
at org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:36)
at org.apache.spark.sql.catalyst.DefaultParserDialect.parse(ParserDialect.scala:67)
at org.apache.spark.sql.SQLContext$$anonfun$2.apply(SQLContext.scala:211)
at org.apache.spark.sql.SQLContext$$anonfun$2.apply(SQLContext.scala:211)
Thanks
From: ayan guha [mailto:guha.ayan@gmail.com]
Sent: 01 September 2016 11:12
To: saurabh3d
Cc: user
Subject: Re: Window Functions with SQLContext
I think you can write the SQL query and run it usin sqlContext.
like
select *,row_number() over(partitin by assetid order by assetid) rn from t
On Thu, Sep 1, 2016 at 3:16 PM, saurabh3d <HYPERLINK "mailto:saurabh.s.dubey@oracle.com" \nsaurabh.s.dubey@oracle.com> wrote:
Hi All,
As per SPARK-11001 <https://issues.apache.org/jira/browse/SPARK-11001> ,
Window functions should be supported by SQLContext. But when i try to run
SQLContext sqlContext = new SQLContext(jsc);
WindowSpec w = Window.partitionBy("assetId").orderBy("assetId");
DataFrame df_2 = df1.withColumn("row_number", row_number().over(w));
df_2.show(false);
it fails with:
Exception in thread "main" org.apache.spark.sql.AnalysisException: Could not
resolve window function 'row_number'. Note that, using window functions
currently requires a HiveContext;
This code runs fine with HiveContext.
Any idea what’s going on? Is this a known issue and is there a workaround
to make Window function work without HiveContext.
Thanks,
Saurabh
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Window-Functions-with-SQLContext-tp27636.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: HYPERLINK "mailto:user-unsubscribe@spark.apache.org"user-unsubscribe@spark.apache.org
--
Best Regards,
Ayan Guha
Re: Window Functions with SQLContext
Posted by ayan guha <gu...@gmail.com>.
I think you can write the SQL query and run it usin sqlContext.
like
select *,row_number() over(partitin by assetid order by assetid) rn from t
On Thu, Sep 1, 2016 at 3:16 PM, saurabh3d <sa...@oracle.com>
wrote:
> Hi All,
>
> As per SPARK-11001 <https://issues.apache.org/jira/browse/SPARK-11001>
> ,
> Window functions should be supported by SQLContext. But when i try to run
>
> SQLContext sqlContext = new SQLContext(jsc);
> WindowSpec w = Window.partitionBy("assetId").orderBy("assetId");
> DataFrame df_2 = df1.withColumn("row_number", row_number().over(w));
> df_2.show(false);
>
> it fails with:
> Exception in thread "main" org.apache.spark.sql.AnalysisException: Could
> not
> resolve window function 'row_number'. Note that, using window functions
> currently requires a HiveContext;
>
> This code runs fine with HiveContext.
> Any idea what’s going on? Is this a known issue and is there a workaround
> to make Window function work without HiveContext.
>
> Thanks,
> Saurabh
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/Window-Functions-with-SQLContext-tp27636.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
--
Best Regards,
Ayan Guha
RE: Window Functions with SQLContext
Posted by Saurabh Dubey <sa...@oracle.com>.
Hi Adline,
rowNumber and row_number are same functions:
@scala.deprecated("Use row_number. This will be removed in Spark 2.0.")
def rowNumber() : org.apache.spark.sql.Column = { /* compiled code */ }
def row_number() : org.apache.spark.sql.Column = { /* compiled code */ }
but the issue here is about using SQLContext instead of HiveContext.
Thanks,
Saurabh
-----Original Message-----
From: Adline Dsilva [mailto:adline.dsilva@mimos.my]
Sent: 01 September 2016 11:05
To: saurabh3d; user@spark.apache.org
Subject: RE: Window Functions with SQLContext
Hi,
Use function rowNumber instead of row_number
df1.withColumn("row_number", rowNumber.over(w));
Regards,
Adline
________________________________________
From: saurabh3d [saurabh.s.dubey@oracle.com]
Sent: 01 September 2016 13:16
To: user@spark.apache.org
Subject: Window Functions with SQLContext
Hi All,
As per SPARK-11001 <https://issues.apache.org/jira/browse/SPARK-11001> ,
Window functions should be supported by SQLContext. But when i try to run
SQLContext sqlContext = new SQLContext(jsc); WindowSpec w = Window.partitionBy("assetId").orderBy("assetId");
DataFrame df_2 = df1.withColumn("row_number", row_number().over(w)); df_2.show(false);
it fails with:
Exception in thread "main" org.apache.spark.sql.AnalysisException: Could not resolve window function 'row_number'. Note that, using window functions currently requires a HiveContext;
This code runs fine with HiveContext.
Any idea what's going on? Is this a known issue and is there a workaround to make Window function work without HiveContext.
Thanks,
Saurabh
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Window-Functions-with-SQLContext-tp27636.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
________________________________
DISCLAIMER:
This e-mail (including any attachments) is for the addressee(s) only and may be confidential, especially as regards personal data. If you are not the intended recipient, please note that any dealing, review, distribution, printing, copying or use of this e-mail is strictly prohibited. If you have received this email in error, please notify the sender immediately and delete the original message (including any attachments).
MIMOS Berhad is a research and development institution under the purview of the Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions and other information in this e-mail that do not relate to the official business of MIMOS Berhad and/or its subsidiaries shall be understood as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts responsibility for the same. All liability arising from or in connection with computer viruses and/or corrupted e-mails is excluded to the fullest extent permitted by law.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
RE: Window Functions with SQLContext
Posted by Adline Dsilva <ad...@mimos.my>.
Hi,
Use function rowNumber instead of row_number
df1.withColumn("row_number", rowNumber.over(w));
Regards,
Adline
________________________________________
From: saurabh3d [saurabh.s.dubey@oracle.com]
Sent: 01 September 2016 13:16
To: user@spark.apache.org
Subject: Window Functions with SQLContext
Hi All,
As per SPARK-11001 <https://issues.apache.org/jira/browse/SPARK-11001> ,
Window functions should be supported by SQLContext. But when i try to run
SQLContext sqlContext = new SQLContext(jsc);
WindowSpec w = Window.partitionBy("assetId").orderBy("assetId");
DataFrame df_2 = df1.withColumn("row_number", row_number().over(w));
df_2.show(false);
it fails with:
Exception in thread "main" org.apache.spark.sql.AnalysisException: Could not
resolve window function 'row_number'. Note that, using window functions
currently requires a HiveContext;
This code runs fine with HiveContext.
Any idea what’s going on? Is this a known issue and is there a workaround
to make Window function work without HiveContext.
Thanks,
Saurabh
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Window-Functions-with-SQLContext-tp27636.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
________________________________
DISCLAIMER:
This e-mail (including any attachments) is for the addressee(s) only and may be confidential, especially as regards personal data. If you are not the intended recipient, please note that any dealing, review, distribution, printing, copying or use of this e-mail is strictly prohibited. If you have received this email in error, please notify the sender immediately and delete the original message (including any attachments).
MIMOS Berhad is a research and development institution under the purview of the Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions and other information in this e-mail that do not relate to the official business of MIMOS Berhad and/or its subsidiaries shall be understood as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts responsibility for the same. All liability arising from or in connection with computer viruses and/or corrupted e-mails is excluded to the fullest extent permitted by law.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org