You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Ismael Carnales <ic...@gmail.com> on 2018/07/01 17:42:17 UTC
Feature request: Java-specific transform method in Dataset
Hi,
it would be nice to have an easier way to use the Dataset transform method
from Java than implementing a Function1 from Scala.
I've made a simple implentation here:
https://github.com/void/spark/tree/java-transform
Should I open a JIRA?
Ismael Carnales
Re: Feature request: Java-specific transform method in Dataset
Posted by kant kodali <ka...@gmail.com>.
I am not affiliated with Flink or Spark but I do think some of the thoughts
here
<http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Long-term-goal-of-making-flink-table-Scala-free-td22761.html>
makes sense
On Sun, Jul 1, 2018 at 4:12 PM, Sean Owen <sr...@gmail.com> wrote:
> It's true, that is one of the issues to be solved by the 2.12-compatible
> build, because it otherwise introduces an overload ambiguity for Java 8
> lambdas. But for that reason I think the current transform() method would
> start working with lambdas. That would only help 2.12 builds; maybe that's
> an OK solution?
>
>
> On Sun, Jul 1, 2018, 2:36 PM Reynold Xin <rx...@databricks.com> wrote:
>
>> This wouldn’t be a problem with Scala 2.12 right?
>>
>> On Sun, Jul 1, 2018 at 12:23 PM Sean Owen <sr...@gmail.com> wrote:
>>
>>> I see, transform() doesn't have the same overload that other methods do
>>> in order to support Java 8 lambdas as you'd expect. One option is to
>>> introduce something like MapFunction for transform and introduce an
>>> overload.
>>>
>>> I think transform() isn't used much at all, so maybe why it wasn't
>>> Java-fied. Before Java 8 it wouldn't have made much sense in Java. Now it
>>> might. I think it could be OK to add the overload to match how map works.
>>>
>>> On Sun, Jul 1, 2018 at 1:33 PM Ismael Carnales <ic...@gmail.com>
>>> wrote:
>>>
>>>> No, because Function1 from Scala is not a functional interface.
>>>> You can see a simple example of what I'm trying to accomplish In the
>>>> unit test here:
>>>> https://github.com/void/spark/blob/java-transform/sql/core/
>>>> src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java#L73
>>>>
>>>>
>>>> On Sun, Jul 1, 2018 at 2:48 PM Sean Owen <sr...@gmail.com> wrote:
>>>>
>>>>> Don't Java 8 lambdas let you do this pretty immediately? Can you give
>>>>> an example here of what you want to do and how you are trying to do it?
>>>>>
>>>>> On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales <ic...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> it would be nice to have an easier way to use the Dataset transform
>>>>>> method from Java than implementing a Function1 from Scala.
>>>>>>
>>>>>> I've made a simple implentation here:
>>>>>>
>>>>>> https://github.com/void/spark/tree/java-transform
>>>>>>
>>>>>> Should I open a JIRA?
>>>>>>
>>>>>> Ismael Carnales
>>>>>>
>>>>>
Re: Feature request: Java-specific transform method in Dataset
Posted by Sean Owen <sr...@gmail.com>.
It's true, that is one of the issues to be solved by the 2.12-compatible
build, because it otherwise introduces an overload ambiguity for Java 8
lambdas. But for that reason I think the current transform() method would
start working with lambdas. That would only help 2.12 builds; maybe that's
an OK solution?
On Sun, Jul 1, 2018, 2:36 PM Reynold Xin <rx...@databricks.com> wrote:
> This wouldn’t be a problem with Scala 2.12 right?
>
> On Sun, Jul 1, 2018 at 12:23 PM Sean Owen <sr...@gmail.com> wrote:
>
>> I see, transform() doesn't have the same overload that other methods do
>> in order to support Java 8 lambdas as you'd expect. One option is to
>> introduce something like MapFunction for transform and introduce an
>> overload.
>>
>> I think transform() isn't used much at all, so maybe why it wasn't
>> Java-fied. Before Java 8 it wouldn't have made much sense in Java. Now it
>> might. I think it could be OK to add the overload to match how map works.
>>
>> On Sun, Jul 1, 2018 at 1:33 PM Ismael Carnales <ic...@gmail.com>
>> wrote:
>>
>>> No, because Function1 from Scala is not a functional interface.
>>> You can see a simple example of what I'm trying to accomplish In the
>>> unit test here:
>>>
>>> https://github.com/void/spark/blob/java-transform/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java#L73
>>>
>>>
>>> On Sun, Jul 1, 2018 at 2:48 PM Sean Owen <sr...@gmail.com> wrote:
>>>
>>>> Don't Java 8 lambdas let you do this pretty immediately? Can you give
>>>> an example here of what you want to do and how you are trying to do it?
>>>>
>>>> On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales <ic...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> it would be nice to have an easier way to use the Dataset transform
>>>>> method from Java than implementing a Function1 from Scala.
>>>>>
>>>>> I've made a simple implentation here:
>>>>>
>>>>> https://github.com/void/spark/tree/java-transform
>>>>>
>>>>> Should I open a JIRA?
>>>>>
>>>>> Ismael Carnales
>>>>>
>>>>
Re: Feature request: Java-specific transform method in Dataset
Posted by Reynold Xin <rx...@databricks.com>.
This wouldn’t be a problem with Scala 2.12 right?
On Sun, Jul 1, 2018 at 12:23 PM Sean Owen <sr...@gmail.com> wrote:
> I see, transform() doesn't have the same overload that other methods do in
> order to support Java 8 lambdas as you'd expect. One option is to introduce
> something like MapFunction for transform and introduce an overload.
>
> I think transform() isn't used much at all, so maybe why it wasn't
> Java-fied. Before Java 8 it wouldn't have made much sense in Java. Now it
> might. I think it could be OK to add the overload to match how map works.
>
> On Sun, Jul 1, 2018 at 1:33 PM Ismael Carnales <ic...@gmail.com>
> wrote:
>
>> No, because Function1 from Scala is not a functional interface.
>> You can see a simple example of what I'm trying to accomplish In the unit
>> test here:
>>
>> https://github.com/void/spark/blob/java-transform/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java#L73
>>
>>
>> On Sun, Jul 1, 2018 at 2:48 PM Sean Owen <sr...@gmail.com> wrote:
>>
>>> Don't Java 8 lambdas let you do this pretty immediately? Can you give an
>>> example here of what you want to do and how you are trying to do it?
>>>
>>> On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales <ic...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> it would be nice to have an easier way to use the Dataset transform
>>>> method from Java than implementing a Function1 from Scala.
>>>>
>>>> I've made a simple implentation here:
>>>>
>>>> https://github.com/void/spark/tree/java-transform
>>>>
>>>> Should I open a JIRA?
>>>>
>>>> Ismael Carnales
>>>>
>>>
Re: Feature request: Java-specific transform method in Dataset
Posted by Sean Owen <sr...@gmail.com>.
I see, transform() doesn't have the same overload that other methods do in
order to support Java 8 lambdas as you'd expect. One option is to introduce
something like MapFunction for transform and introduce an overload.
I think transform() isn't used much at all, so maybe why it wasn't
Java-fied. Before Java 8 it wouldn't have made much sense in Java. Now it
might. I think it could be OK to add the overload to match how map works.
On Sun, Jul 1, 2018 at 1:33 PM Ismael Carnales <ic...@gmail.com> wrote:
> No, because Function1 from Scala is not a functional interface.
> You can see a simple example of what I'm trying to accomplish In the unit
> test here:
>
> https://github.com/void/spark/blob/java-transform/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java#L73
>
>
> On Sun, Jul 1, 2018 at 2:48 PM Sean Owen <sr...@gmail.com> wrote:
>
>> Don't Java 8 lambdas let you do this pretty immediately? Can you give an
>> example here of what you want to do and how you are trying to do it?
>>
>> On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales <ic...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> it would be nice to have an easier way to use the Dataset transform
>>> method from Java than implementing a Function1 from Scala.
>>>
>>> I've made a simple implentation here:
>>>
>>> https://github.com/void/spark/tree/java-transform
>>>
>>> Should I open a JIRA?
>>>
>>> Ismael Carnales
>>>
>>
Re: Feature request: Java-specific transform method in Dataset
Posted by Ismael Carnales <ic...@gmail.com>.
No, because Function1 from Scala is not a functional interface.
You can see a simple example of what I'm trying to accomplish In the unit
test here:
https://github.com/void/spark/blob/java-transform/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java#L73
On Sun, Jul 1, 2018 at 2:48 PM Sean Owen <sr...@gmail.com> wrote:
> Don't Java 8 lambdas let you do this pretty immediately? Can you give an
> example here of what you want to do and how you are trying to do it?
>
> On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales <ic...@gmail.com> wrote:
>
>> Hi,
>> it would be nice to have an easier way to use the Dataset transform
>> method from Java than implementing a Function1 from Scala.
>>
>> I've made a simple implentation here:
>>
>> https://github.com/void/spark/tree/java-transform
>>
>> Should I open a JIRA?
>>
>> Ismael Carnales
>>
>
Re: Feature request: Java-specific transform method in Dataset
Posted by Sean Owen <sr...@gmail.com>.
Don't Java 8 lambdas let you do this pretty immediately? Can you give an
example here of what you want to do and how you are trying to do it?
On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales <ic...@gmail.com> wrote:
> Hi,
> it would be nice to have an easier way to use the Dataset transform
> method from Java than implementing a Function1 from Scala.
>
> I've made a simple implentation here:
>
> https://github.com/void/spark/tree/java-transform
>
> Should I open a JIRA?
>
> Ismael Carnales
>