Re: Bring back support for spark 2.3?

[Pratyaksh Sharma <pr...@gmail.com>]

Hi Vinoth,

It was a while ago but I guess the error was related to parquet-avro
mismatch only. Let me try reproducing it and will file a lira this time.

On Tue, Feb 25, 2020 at 12:29 PM Vinoth Chandar <vi...@apache.org> wrote:

> Hi Pratyaksh,
>
> Actually we voted on the mailing list here, to move over to 2.4..  :) With
> 3.0 on the horizon, it may make sense to actually upgrade?
> Can you file an issue with how exactly 0.5.1 is failing with 2.3? are you
> hitting issues with parquet-avro mismatches?
> I am wondering if we can do a custom build overriding few versions of
> parquet/avro... (spark-avro 2.4.4 should interplay with spark 2.3?)
>
> Thanks
> VInoth
>
> On Sun, Feb 23, 2020 at 2:02 PM Pratyaksh Sharma <pr...@gmail.com>
> wrote:
>
> > Hi,
> >
> > As discussed in last to last week's weekly sync, I want to put forward
> this
> > point on our mailing list also. Since with 0.5.1 release, we have
> upgraded
> > spark to 2.4 in our master branch, we are facing difficulties after
> > rebasing our codebase with master. At our organisation we are using spark
> > 2.3.2 cluster in production and spark 2.3 to 2.4 upgrade has breaking
> > changes.
> >
> > So I wanted to know if any workaround is possible to bring back support
> for
> > spark 2.3? I am sure there are other guys/organisations using Hudi who
> are
> > not yet on spark 2.4.
> >
> > We really do not want to miss out on any of the cool features being
> > developed by the community.
> >
>

Re: Bring back support for spark 2.3?

[Vinoth Chandar <vi...@apache.org>]

Sounds good.. Also the avro versions? If we can make a profile around this,
we could give it a shot..

I reckon most of the world will be moving towards Spark 3, soon though (say
end of 2020)

On Sat, Mar 21, 2020 at 12:19 PM Pratyaksh Sharma <pr...@gmail.com>
wrote:

> Hi Vinoth,
>
> It was a while ago but I guess the error was related to parquet-avro
> mismatch only. Let me try reproducing it and will file a lira this time.
>
> On Tue, Feb 25, 2020 at 12:29 PM Vinoth Chandar <vi...@apache.org> wrote:
>
> > Hi Pratyaksh,
> >
> > Actually we voted on the mailing list here, to move over to 2.4..  :)
> With
> > 3.0 on the horizon, it may make sense to actually upgrade?
> > Can you file an issue with how exactly 0.5.1 is failing with 2.3? are you
> > hitting issues with parquet-avro mismatches?
> > I am wondering if we can do a custom build overriding few versions of
> > parquet/avro... (spark-avro 2.4.4 should interplay with spark 2.3?)
> >
> > Thanks
> > VInoth
> >
> > On Sun, Feb 23, 2020 at 2:02 PM Pratyaksh Sharma <pr...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > As discussed in last to last week's weekly sync, I want to put forward
> > this
> > > point on our mailing list also. Since with 0.5.1 release, we have
> > upgraded
> > > spark to 2.4 in our master branch, we are facing difficulties after
> > > rebasing our codebase with master. At our organisation we are using
> spark
> > > 2.3.2 cluster in production and spark 2.3 to 2.4 upgrade has breaking
> > > changes.
> > >
> > > So I wanted to know if any workaround is possible to bring back support
> > for
> > > spark 2.3? I am sure there are other guys/organisations using Hudi who
> > are
> > > not yet on spark 2.4.
> > >
> > > We really do not want to miss out on any of the cool features being
> > > developed by the community.
> > >
> >
>