You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Frens Jan Rumph (JIRA)" <ji...@apache.org> on 2015/11/10 11:33:11 UTC
[jira] [Updated] (SPARK-11630) ClosureCleaner incorrectly warns for
class based closures
[ https://issues.apache.org/jira/browse/SPARK-11630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Frens Jan Rumph updated SPARK-11630:
------------------------------------
Description:
Spark's `ClosureCleaner` utility seems to check whether a function is an anonymous function: [ClosureCleaner.scala on line 49|https://github.com/apache/spark/blob/f85aa06464a10f5d1563302fd76465dded475a12/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala#L49] If not, it warns the user.
However, I'm using some class based functions. Something along the lines of:
{code}
trait FromUnreadRow[T] extends (UnreadRow => T) with Serializable
object ToPlainRow extends FromUnreadRow[PlainRow] {
override def apply(row: UnreadRow): PlainRow = ???
}
{code}
This works just fine. I can't really see that the warning is actually useful in this case. I appreciate checking for common 'mistakes', but in my case a user might be alarmed unnecessarily.
Anything that can be done about this? Anything I can do?
was:
Spark's `ClosureCleaner` utility seems to check whether a function is an anonymous function: [ClosureCleaner.scala on line 49|https://github.com/apache/spark/blob/f85aa06464a10f5d1563302fd76465dded475a12/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala#L49] If not, it warns the user.
However, I'm using some class based functions. Something along the lines of:
{code}
trait FromUnreadRow[T] extends (UnreadRow => T) with Serializable
object ToPlainRow extends FromUnreadRow[PlainRow] {
override def apply(row: UnreadRow): PlainRow = ???
}
{code}
This works just fine. I can't really see that the warning is actually useful in this case. I appreciate checking for common 'mistakes', but in my case a user might be alarmed unnecessarily.
> ClosureCleaner incorrectly warns for class based closures
> ---------------------------------------------------------
>
> Key: SPARK-11630
> URL: https://issues.apache.org/jira/browse/SPARK-11630
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.4.1
> Reporter: Frens Jan Rumph
> Priority: Trivial
>
> Spark's `ClosureCleaner` utility seems to check whether a function is an anonymous function: [ClosureCleaner.scala on line 49|https://github.com/apache/spark/blob/f85aa06464a10f5d1563302fd76465dded475a12/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala#L49] If not, it warns the user.
> However, I'm using some class based functions. Something along the lines of:
> {code}
> trait FromUnreadRow[T] extends (UnreadRow => T) with Serializable
> object ToPlainRow extends FromUnreadRow[PlainRow] {
> override def apply(row: UnreadRow): PlainRow = ???
> }
> {code}
> This works just fine. I can't really see that the warning is actually useful in this case. I appreciate checking for common 'mistakes', but in my case a user might be alarmed unnecessarily.
> Anything that can be done about this? Anything I can do?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org