You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by Shiyan Xu <xu...@gmail.com> on 2023/05/26 03:00:13 UTC

[Action required] Default Spark profile changed to 3.2

Hi all,

We recently landed a change
<https://github.com/apache/hudi/commit/516c3d59404934e6a142ea1c9d97002c065f8a4f>
in master switching the default Spark profile from 2.4 to 3.2. If your
local Hudi repo is configured to use Spark 2.4, you may need to re-import
the IDEA project (this may involve clearing `.idea/` folder and *.iml files
in each maven module)

I'd also like to acknowledge contributions from Forward Xu, Rahil, Zhang
Yue, and Danny, who have previously worked on or helped tackle this
migration, as it involved a lot of dependency issue wrangling and test
fixes.

Cheers,

-- 
Best,
Shiyan

Re: [Action required] Default Spark profile changed to 3.2

Posted by Y Ethan Guo <yi...@apache.org>.
There is a hotfix (https://github.com/apache/hudi/pull/8822) merged
recently to fix the default build.  SPARK_HOME does not matter and the
latest master builds on my end with Spark 3.3 as SPARK_HOME.

On Fri, Jun 2, 2023 at 1:28 AM Vinoth Chandar <vi...@apache.org> wrote:

> Hi,
>
> Just tried doing a mvn clean install -DskipTests, and the build failed. My
> local SPARK_HOME is pointing to spark 3.3 installation.
> Does that all matter now? Quite possible this is an issue with my setup,
> just flagging.
>
> Thanks
> Vinoth
>
> On Fri, May 26, 2023 at 8:30 AM Shiyan Xu <xu...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > We recently landed a change
> > <
> >
> https://github.com/apache/hudi/commit/516c3d59404934e6a142ea1c9d97002c065f8a4f
> > >
> > in master switching the default Spark profile from 2.4 to 3.2. If your
> > local Hudi repo is configured to use Spark 2.4, you may need to re-import
> > the IDEA project (this may involve clearing `.idea/` folder and *.iml
> files
> > in each maven module)
> >
> > I'd also like to acknowledge contributions from Forward Xu, Rahil, Zhang
> > Yue, and Danny, who have previously worked on or helped tackle this
> > migration, as it involved a lot of dependency issue wrangling and test
> > fixes.
> >
> > Cheers,
> >
> > --
> > Best,
> > Shiyan
> >
>

Re: [Action required] Default Spark profile changed to 3.2

Posted by Vinoth Chandar <vi...@apache.org>.
Hi,

Just tried doing a mvn clean install -DskipTests, and the build failed. My
local SPARK_HOME is pointing to spark 3.3 installation.
Does that all matter now? Quite possible this is an issue with my setup,
just flagging.

Thanks
Vinoth

On Fri, May 26, 2023 at 8:30 AM Shiyan Xu <xu...@gmail.com>
wrote:

> Hi all,
>
> We recently landed a change
> <
> https://github.com/apache/hudi/commit/516c3d59404934e6a142ea1c9d97002c065f8a4f
> >
> in master switching the default Spark profile from 2.4 to 3.2. If your
> local Hudi repo is configured to use Spark 2.4, you may need to re-import
> the IDEA project (this may involve clearing `.idea/` folder and *.iml files
> in each maven module)
>
> I'd also like to acknowledge contributions from Forward Xu, Rahil, Zhang
> Yue, and Danny, who have previously worked on or helped tackle this
> migration, as it involved a lot of dependency issue wrangling and test
> fixes.
>
> Cheers,
>
> --
> Best,
> Shiyan
>