You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2018/11/07 06:40:00 UTC
[jira] [Resolved] (SPARK-18466) Missing withFilter method causes
warnings when using for comprehensions
[ https://issues.apache.org/jira/browse/SPARK-18466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun resolved SPARK-18466.
-----------------------------------
Resolution: Won't Fix
Thank you for reporting and contribution, [~reggert1980]. We know that this has been a long pending PR.
Although this issue causes a regression with Scala 2.12 at Spark 2.4, Apache Spark community will not add this feature due to the risk. Please see the discussion on the PR.
{code}
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context available as 'sc' (master = local[*], app id = local-1541571276105).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0
/_/
Using Scala version 2.12.7 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_181)
Type in expressions to have them evaluated.
Type :help for more information.
scala> (for (n <- sc.parallelize(Seq(1,2,3)) if n > 2) yield n).toDebugString
<console>:25: error: value withFilter is not a member of org.apache.spark.rdd.RDD[Int]
(for (n <- sc.parallelize(Seq(1,2,3)) if n > 2) yield n).toDebugString
{code}
> Missing withFilter method causes warnings when using for comprehensions
> -----------------------------------------------------------------------
>
> Key: SPARK-18466
> URL: https://issues.apache.org/jira/browse/SPARK-18466
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2, 1.6.0, 1.6.1, 1.6.2, 1.6.3, 2.0.0, 2.0.1, 2.0.2, 2.3.0, 2.4.0
> Reporter: Richard W. Eggert II
> Priority: Minor
> Labels: easyfix
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The fact that the RDD class has a {{filter}} method but not a {{withFilter}} method results in compiler warnings when using RDDs in {{for}} comprehensions. As of Scala 2.12, falling back to use of {{filter}} is no longer supported, so {{for}} comprehensions that use filters will no longer compile. Semantically, the only difference between {{withFilter}} and {{filter}} is that {{withFilter}} is lazy, and since RDDs are lazy by nature, one can simply be aliased to the other.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org