You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Gerhard Fiedler <gf...@algebraixdata.com> on 2016/06/03 15:01:14 UTC
Where is DataFrame.scala in 2.0?
When I look at the sources in Github, I see DataFrame.scala at https://github.com/apache/spark/blob/branch-1.6/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala in the 1.6 branch. But when I change the branch to branch-2.0 or master, I get a 404 error. I also can't find the file in the directory listings, for example https://github.com/apache/spark/tree/branch-2.0/sql/core/src/main/scala/org/apache/spark/sql (for branch-2.0).
It seems that quite a few APIs use the DataFrame class, even in 2.0. Can someone please point me to its location, or otherwise explain why it is not there?
Thanks,
Gerhard
Re: Where is DataFrame.scala in 2.0?
Posted by Michael Malak <mi...@yahoo.com.INVALID>.
It's been reduced to a single line of code.
http://technicaltidbit.blogspot.com/2016/03/dataframedataset-swap-places-in-spark-20.html
From: Gerhard Fiedler <gf...@algebraixdata.com>
To: "dev@spark.apache.org" <de...@spark.apache.org>
Sent: Friday, June 3, 2016 9:01 AM
Subject: Where is DataFrame.scala in 2.0?
<!--#yiv0106397017 _filtered #yiv0106397017 {font-family:"Cambria Math";panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv0106397017 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv0106397017 {font-family:Verdana;panose-1:2 11 6 4 3 5 4 4 2 4;}#yiv0106397017 #yiv0106397017 p.yiv0106397017MsoNormal, #yiv0106397017 li.yiv0106397017MsoNormal, #yiv0106397017 div.yiv0106397017MsoNormal {margin:0cm;margin-bottom:.0001pt;font-size:11.0pt;font-family:"Calibri", sans-serif;}#yiv0106397017 a:link, #yiv0106397017 span.yiv0106397017MsoHyperlink {color:blue;text-decoration:underline;}#yiv0106397017 a:visited, #yiv0106397017 span.yiv0106397017MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv0106397017 span.yiv0106397017EmailStyle17 {font-family:"Verdana", sans-serif;color:#006300;font-weight:normal;font-style:normal;}#yiv0106397017 .yiv0106397017MsoChpDefault {font-family:"Calibri", sans-serif;} _filtered #yiv0106397017 {margin:72.0pt 72.0pt 72.0pt 72.0pt;}#yiv0106397017 div.yiv0106397017WordSection1 {}-->When I look at the sources in Github, I see DataFrame.scala athttps://github.com/apache/spark/blob/branch-1.6/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala in the 1.6 branch. But when I change the branch to branch-2.0 or master, I get a 404 error. I also can’t find the file in the directory listings, for example https://github.com/apache/spark/tree/branch-2.0/sql/core/src/main/scala/org/apache/spark/sql (for branch-2.0). It seems that quite a few APIs use the DataFrame class, even in 2.0. Can someone please point me to its location, or otherwise explain why it is not there? Thanks, Gerhard
RE: Where is DataFrame.scala in 2.0?
Posted by Gerhard Fiedler <gf...@algebraixdata.com>.
Thanks!
From: Herman van Hövell tot Westerflier [mailto:hvanhovell@questtec.nl]
Sent: Fri, Jun 03, 2016 10:05
To: Gerhard Fiedler <gf...@algebraixdata.com>
Cc: dev@spark.apache.org
Subject: Re: Where is DataFrame.scala in 2.0?
Hi Gerhard,
DataFrame and DataSet have been merged in Spark 2.0. A DataFrame is now a DataSet that contains Row objects. We still maintain a type alias for DataFrame: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/package.scala#L45
HTH
Kind regards,
Herman van Hövell tot Westerflier
2016-06-03 17:01 GMT+02:00 Gerhard Fiedler <gf...@algebraixdata.com>>:
When I look at the sources in Github, I see DataFrame.scala at https://github.com/apache/spark/blob/branch-1.6/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala in the 1.6 branch. But when I change the branch to branch-2.0 or master, I get a 404 error. I also can’t find the file in the directory listings, for example https://github.com/apache/spark/tree/branch-2.0/sql/core/src/main/scala/org/apache/spark/sql (for branch-2.0).
It seems that quite a few APIs use the DataFrame class, even in 2.0. Can someone please point me to its location, or otherwise explain why it is not there?
Thanks,
Gerhard
Re: Where is DataFrame.scala in 2.0?
Posted by Herman van Hövell tot Westerflier <hv...@questtec.nl>.
Hi Gerhard,
DataFrame and DataSet have been merged in Spark 2.0. A DataFrame is now a
DataSet that contains Row objects. We still maintain a type alias for
DataFrame:
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/package.scala#L45
HTH
Kind regards,
Herman van Hövell tot Westerflier
2016-06-03 17:01 GMT+02:00 Gerhard Fiedler <gf...@algebraixdata.com>:
> When I look at the sources in Github, I see DataFrame.scala at
> https://github.com/apache/spark/blob/branch-1.6/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
> in the 1.6 branch. But when I change the branch to branch-2.0 or master, I
> get a 404 error. I also can’t find the file in the directory listings, for
> example
> https://github.com/apache/spark/tree/branch-2.0/sql/core/src/main/scala/org/apache/spark/sql
> (for branch-2.0).
>
>
>
> It seems that quite a few APIs use the DataFrame class, even in 2.0. Can
> someone please point me to its location, or otherwise explain why it is not
> there?
>
>
>
> Thanks,
>
> Gerhard
>
>
>