You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Sean Owen <so...@cloudera.com> on 2017/09/15 10:21:12 UTC
A little Scala 2.12 help
I'm working on updating to Scala 2.12, and, have hit a compile error in
Scala 2.12 that I'm strugging to design a fix to (that doesn't modify the
API significantly). If you "./dev/change-scala-version.sh 2.12" and
compile, you'll see errors like...
[error]
/Users/srowen/Documents/Cloudera/spark/core/src/test/scala/org/apache/spark/FileSuite.scala:100:
could not find implicit value for parameter kcf: () =>
org.apache.spark.WritableConverter[org.apache.hadoop.io.IntWritable]
[error] Error occurred in an application involving default arguments.
[error] val output = sc.sequenceFile[IntWritable, Text](outputDir)
Clearly implicit resolution changed a little bit in 2.12 somehow. I
actually don't recall seeing this error before, so might be somehow related
to 2.12.3, but not sure.
As you can see the implicits that have always existed and been imported and
should apply here don't seem to be found.
If anyone is a Scala expert and could glance at this, you might help save
me a lot of puzzling.
Re: A little Scala 2.12 help
Posted by Matei Zaharia <ma...@gmail.com>.
Yeah, pretty tricky change but I’m glad that it’s possible to fix it.
Matei
> On Sep 19, 2017, at 12:50 PM, Jacek Laskowski <ja...@japila.pl> wrote:
>
> Hi,
>
> Nice catch, Sean! Learnt this today. They did say you could learn a lot with Spark! :)
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://about.me/JacekLaskowski
> Spark Structured Streaming (Apache Spark 2.2+) https://bit.ly/spark-structured-streaming
> Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
> On Tue, Sep 19, 2017 at 4:23 PM, Sean Owen <so...@cloudera.com> wrote:
> I figured this out. It's another effect of a new behavior in 2.12: Eta-expansion of zero-argument method values is deprecated
> Imagine:
>
> def f(): String = "foo"
> def g(fn: () => String) = ???
>
> g(f) works in 2.11 without warning. It generates a warning in 2.12, because it wants you to explicitly make a function from the method reference: g(() => f). It will maybe be an error in 2.13.
>
> But, this affects implicit resolution. Some of the implicits that power SparkContext.sequenceFile() need to change to be vals of type () => WritableConverter[T], not methods that return WritableConverter[T].
>
> I'm working through this and other deprecated items in 2.12 and preparing more 2.11-compatible changes that allow these to work cleanly in 2.12.
>
> On Fri, Sep 15, 2017 at 11:21 AM Sean Owen <so...@cloudera.com> wrote:
> I'm working on updating to Scala 2.12, and, have hit a compile error in Scala 2.12 that I'm strugging to design a fix to (that doesn't modify the API significantly). If you "./dev/change-scala-version.sh 2.12" and compile, you'll see errors like...
>
> [error] /Users/srowen/Documents/Cloudera/spark/core/src/test/scala/org/apache/spark/FileSuite.scala:100: could not find implicit value for parameter kcf: () => org.apache.spark.WritableConverter[org.apache.hadoop.io.IntWritable]
> [error] Error occurred in an application involving default arguments.
> [error] val output = sc.sequenceFile[IntWritable, Text](outputDir)
>
> Clearly implicit resolution changed a little bit in 2.12 somehow. I actually don't recall seeing this error before, so might be somehow related to 2.12.3, but not sure.
>
> As you can see the implicits that have always existed and been imported and should apply here don't seem to be found.
>
> If anyone is a Scala expert and could glance at this, you might help save me a lot of puzzling.
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: A little Scala 2.12 help
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
Nice catch, Sean! Learnt this today. They did say you could learn a lot
with Spark! :)
Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
Spark Structured Streaming (Apache Spark 2.2+)
https://bit.ly/spark-structured-streaming
Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Tue, Sep 19, 2017 at 4:23 PM, Sean Owen <so...@cloudera.com> wrote:
> I figured this out. It's another effect of a new behavior in
> 2.12: Eta-expansion of zero-argument method values is deprecated
> Imagine:
>
> def f(): String = "foo"
> def g(fn: () => String) = ???
>
> g(f) works in 2.11 without warning. It generates a warning in 2.12,
> because it wants you to explicitly make a function from the method
> reference: g(() => f). It will maybe be an error in 2.13.
>
> But, this affects implicit resolution. Some of the implicits that power
> SparkContext.sequenceFile() need to change to be vals of type () =>
> WritableConverter[T], not methods that return WritableConverter[T].
>
> I'm working through this and other deprecated items in 2.12 and preparing
> more 2.11-compatible changes that allow these to work cleanly in 2.12.
>
> On Fri, Sep 15, 2017 at 11:21 AM Sean Owen <so...@cloudera.com> wrote:
>
>> I'm working on updating to Scala 2.12, and, have hit a compile error in
>> Scala 2.12 that I'm strugging to design a fix to (that doesn't modify the
>> API significantly). If you "./dev/change-scala-version.sh 2.12" and
>> compile, you'll see errors like...
>>
>> [error] /Users/srowen/Documents/Cloudera/spark/core/src/test/
>> scala/org/apache/spark/FileSuite.scala:100: could not find implicit
>> value for parameter kcf: () => org.apache.spark.
>> WritableConverter[org.apache.hadoop.io.IntWritable]
>> [error] Error occurred in an application involving default arguments.
>> [error] val output = sc.sequenceFile[IntWritable, Text](outputDir)
>>
>> Clearly implicit resolution changed a little bit in 2.12 somehow. I
>> actually don't recall seeing this error before, so might be somehow related
>> to 2.12.3, but not sure.
>>
>> As you can see the implicits that have always existed and been imported
>> and should apply here don't seem to be found.
>>
>> If anyone is a Scala expert and could glance at this, you might help save
>> me a lot of puzzling.
>>
>
Re: A little Scala 2.12 help
Posted by Sean Owen <so...@cloudera.com>.
I figured this out. It's another effect of a new behavior in
2.12: Eta-expansion of zero-argument method values is deprecated
Imagine:
def f(): String = "foo"
def g(fn: () => String) = ???
g(f) works in 2.11 without warning. It generates a warning in 2.12, because
it wants you to explicitly make a function from the method reference: g(()
=> f). It will maybe be an error in 2.13.
But, this affects implicit resolution. Some of the implicits that power
SparkContext.sequenceFile() need to change to be vals of type () =>
WritableConverter[T], not methods that return WritableConverter[T].
I'm working through this and other deprecated items in 2.12 and preparing
more 2.11-compatible changes that allow these to work cleanly in 2.12.
On Fri, Sep 15, 2017 at 11:21 AM Sean Owen <so...@cloudera.com> wrote:
> I'm working on updating to Scala 2.12, and, have hit a compile error in
> Scala 2.12 that I'm strugging to design a fix to (that doesn't modify the
> API significantly). If you "./dev/change-scala-version.sh 2.12" and
> compile, you'll see errors like...
>
> [error]
> /Users/srowen/Documents/Cloudera/spark/core/src/test/scala/org/apache/spark/FileSuite.scala:100:
> could not find implicit value for parameter kcf: () =>
> org.apache.spark.WritableConverter[org.apache.hadoop.io.IntWritable]
> [error] Error occurred in an application involving default arguments.
> [error] val output = sc.sequenceFile[IntWritable, Text](outputDir)
>
> Clearly implicit resolution changed a little bit in 2.12 somehow. I
> actually don't recall seeing this error before, so might be somehow related
> to 2.12.3, but not sure.
>
> As you can see the implicits that have always existed and been imported
> and should apply here don't seem to be found.
>
> If anyone is a Scala expert and could glance at this, you might help save
> me a lot of puzzling.
>