You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2016/10/08 11:25:21 UTC
[jira] [Closed] (SPARK-9405) approximateCountDiscting does not work
with GroupBy
[ https://issues.apache.org/jira/browse/SPARK-9405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon closed SPARK-9405.
-------------------------------
Resolution: Not A Problem
> approximateCountDiscting does not work with GroupBy
> ---------------------------------------------------
>
> Key: SPARK-9405
> URL: https://issues.apache.org/jira/browse/SPARK-9405
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.1
> Reporter: Yana Kadiyska
>
> {code}
> case class MockCustomer(val customer_id:Int,val host:String)
> val df =sc.parallelize(1 to 10).map(i=>MockCustomer(1234,if (i%2 ==0) "http://foo.com" else "http://bar.com")).toDF
> //this works OK
> df.groupBy($"host").agg(count($"*"),sum($"customer_id")).show
> but this doesnt:
> df.groupBy($"host").agg(approxCountDistinct($"*"),sum($"customer_id")).show
> 15/07/28 10:46:14 INFO BlockManagerInfo: Removed broadcast_55_piece0 on localhost:33727 in memory (size: 4.4 KB, free: 265.3 MB)
> 15/07/28 10:46:14 INFO BlockManagerInfo: Removed broadcast_54_piece0 on localhost:33727 in memory (size: 4.4 KB, free: 265.3 MB)
> org.apache.spark.sql.AnalysisException: cannot resolve 'host' given input columns customer_id, host;
> at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
> at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:63)
> at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:52)
> at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:286)
> at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:286)
> at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:51)
> at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:285)
> at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$transformExpressionUp$1(QueryPlan.scala:108)
> at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$2$$anonfun$apply$2.apply(QueryPlan.scala:123)
> at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$2.apply(QueryPlan.scala:122)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org