You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ian Hellstrom (JIRA)" <ji...@apache.org> on 2016/09/27 07:59:21 UTC
[jira] [Updated] (SPARK-17681) Empty DataFrame with non-zero rows
after using drop
[ https://issues.apache.org/jira/browse/SPARK-17681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ian Hellstrom updated SPARK-17681:
----------------------------------
Description:
It is possible to have a {{DataFrame}} with no columns to have a non-zero number of rows, even though the contents are empty:
{code}
val df = Seq((1,2)).toDF("a", "b")
df.drop("a").drop("b").count
{code}
I tried checking it for 2.0 but unfortunately the sbt console for Spark does not work on Windows. Anyway, I think the following snippet ought to be a simple copy-paste exercise:
{code}
import org.apache.spark._
import org.apache.spark.sql._
val conf = new SparkConf()
val sc = new SparkContext("local", "demo", conf)
val ss = SparkSession.builder.getOrCreate()
import ss.implicits._
case class Data(a: Int, b: Int)
val rdd = sc.parallelize(List(Data(1,2)))
val ds = ss.createDataset(rdd)
ds.drop("a").drop("b").count
{code}
was:
It is possible to have a {DataFrame} with no columns to have a non-zero number of rows, even though the contents are empty:
{code}
val df = Seq((1,2)).toDF("a", "b")
df.drop("a").drop("b").count
{code}
I tried checking it for 2.0 but unfortunately the sbt console for Spark does not work on Windows. Anyway, I think the following snippet ought to be a simple copy-paste exercise:
{code}
import org.apache.spark._
import org.apache.spark.sql._
val conf = new SparkConf()
val sc = new SparkContext("local", "demo", conf)
val ss = SparkSession.builder.getOrCreate()
import ss.implicits._
case class Data(a: Int, b: Int)
val rdd = sc.parallelize(List(Data(1,2)))
val ds = ss.createDataset(rdd)
ds.drop("a").drop("b").count
{code}
> Empty DataFrame with non-zero rows after using drop
> ---------------------------------------------------
>
> Key: SPARK-17681
> URL: https://issues.apache.org/jira/browse/SPARK-17681
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.1, 1.6.0
> Reporter: Ian Hellstrom
>
> It is possible to have a {{DataFrame}} with no columns to have a non-zero number of rows, even though the contents are empty:
> {code}
> val df = Seq((1,2)).toDF("a", "b")
> df.drop("a").drop("b").count
> {code}
> I tried checking it for 2.0 but unfortunately the sbt console for Spark does not work on Windows. Anyway, I think the following snippet ought to be a simple copy-paste exercise:
> {code}
> import org.apache.spark._
> import org.apache.spark.sql._
> val conf = new SparkConf()
> val sc = new SparkContext("local", "demo", conf)
> val ss = SparkSession.builder.getOrCreate()
> import ss.implicits._
> case class Data(a: Int, b: Int)
> val rdd = sc.parallelize(List(Data(1,2)))
> val ds = ss.createDataset(rdd)
> ds.drop("a").drop("b").count
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org