[SPARK-20384][SQL] Support value class in schema of Dataset (third time's a charm)

[Emil Ejbyfeldt <ee...@liveintent.com.INVALID>]

Hi dev,

I am interested getting the support value classes in schemas of Dataset 
merged and I am willing to work on it.

There are two previous PRs created for this JIRA (SPARK-20384) first 
https://github.com/apache/spark/pull/22309 and more recently 
https://github.com/apache/spark/pull/27153 (marked stale ~1year ago). It 
does not seem to me that the PR have been meet with any resistance but 
have the activity has just died out and therefore the changes have not 
been merged.

Before spending more time on this I would like to that there is any 
known problems with supporting this that has caused the previous PRs to 
not be merged?

I think the changes proposed in the later PR is still valid and a good 
approach for adding support. Should I ask to have that PR reopened or 
creating a new one since I am not the original author?

/ Emil



---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Re: [SPARK-20384][SQL] Support value class in schema of Dataset (third time's a charm)

["Fitch, Simeon" <fi...@astraea.io>]

Emil,

We too are interested in this work. Thank you for resurrecting it. I hope
the Spark committers work to incorporate it.

Regards,

Simeon


On Mon, Aug 2, 2021 at 9:51 AM Emil Ejbyfeldt
<ee...@liveintent.com.invalid> wrote:

> Hi dev,
>
> After looking into the details of this and discussing with the other
> authors that previously created PR for SPARK-20384. It resulted in the
> following PR: https://github.com/apache/spark/pull/33205
>
> The core of the change (excluding new test cases) is quite small. It is
> only a change of `24 insertions(+), 9 deletions(-)` in
>
> `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala`
> . The change is backwards compatible and will not break any currently
> working code.
>
> The main driver for us wanting this change is that having value classes
> not work means that we can't incrementally use/move to these spark
> features without changing a bunch of our current modeling.
>
> Also I think that the changes being brought up multiple times before
> shows that there is some general interests from the community of fixing
> this.
>
> I encourage anyone interested in this feature to leave a review/comments
> on that PR.
>
> Also it would be great if some admin would be able to review it. Or any
> tips on steps that should be taken in order to have the PR reviewed
> would be appreciated.
>
> / Emil
>
>
> On 25/05/2021 16:33, Emil Ejbyfeldt wrote:
> > Hi dev,
> >
> > I am interested getting the support value classes in schemas of Dataset
> > merged and I am willing to work on it.
> >
> > There are two previous PRs created for this JIRA (SPARK-20384) first
> > https://github.com/apache/spark/pull/22309 and more recently
> > https://github.com/apache/spark/pull/27153 (marked stale ~1year ago).
> It
> > does not seem to me that the PR have been meet with any resistance but
> > have the activity has just died out and therefore the changes have not
> > been merged.
> >
> > Before spending more time on this I would like to that there is any
> > known problems with supporting this that has caused the previous PRs to
> > not be merged?
> >
> > I think the changes proposed in the later PR is still valid and a good
> > approach for adding support. Should I ask to have that PR reopened or
> > creating a new one since I am not the original author?
> >
> > / Emil
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

-- 
Simeon Fitch
Co-founder & VP of R&D
Astraea, Inc.

Re: [SPARK-20384][SQL] Support value class in schema of Dataset (third time's a charm)

[Emil Ejbyfeldt <ee...@liveintent.com.INVALID>]

Hi dev,

After looking into the details of this and discussing with the other 
authors that previously created PR for SPARK-20384. It resulted in the 
following PR: https://github.com/apache/spark/pull/33205

The core of the change (excluding new test cases) is quite small. It is 
only a change of `24 insertions(+), 9 deletions(-)` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala`
. The change is backwards compatible and will not break any currently 
working code.

The main driver for us wanting this change is that having value classes 
not work means that we can't incrementally use/move to these spark 
features without changing a bunch of our current modeling.

Also I think that the changes being brought up multiple times before 
shows that there is some general interests from the community of fixing 
this.

I encourage anyone interested in this feature to leave a review/comments 
on that PR.

Also it would be great if some admin would be able to review it. Or any 
tips on steps that should be taken in order to have the PR reviewed 
would be appreciated.

/ Emil


On 25/05/2021 16:33, Emil Ejbyfeldt wrote:
> Hi dev,
> 
> I am interested getting the support value classes in schemas of Dataset 
> merged and I am willing to work on it.
> 
> There are two previous PRs created for this JIRA (SPARK-20384) first 
> https://github.com/apache/spark/pull/22309 and more recently 
> https://github.com/apache/spark/pull/27153 (marked stale ~1year ago). It 
> does not seem to me that the PR have been meet with any resistance but 
> have the activity has just died out and therefore the changes have not 
> been merged.
> 
> Before spending more time on this I would like to that there is any 
> known problems with supporting this that has caused the previous PRs to 
> not be merged?
> 
> I think the changes proposed in the later PR is still valid and a good 
> approach for adding support. Should I ask to have that PR reopened or 
> creating a new one since I am not the original author?
> 
> / Emil
> 
> 

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org