You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "Yohahaha (via GitHub)" <gi...@apache.org> on 2024/10/22 08:56:52 UTC
[I] Iceberg module may breaks with Spark35 [incubator-gluten]
Yohahaha opened a new issue, #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637
### Backend
VL (Velox)
### Bug description
Gluten has upgrade Spark version to v3.5.3 https://github.com/apache/incubator-gluten/pull/7537, but iceberg may has compatibility issue with Spark 3.5.3.
https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2428355077
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg and hudi module may breaks with Spark35 [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2469741405
We were not aware of such issue before. Maybe, it's time to change our previous convention for upgrading Spark minor release. I have created one discussion: https://github.com/apache/incubator-gluten/discussions/7874.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg and hudi module may breaks with Spark35 [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha closed issue #7637: [CORE] Iceberg and hudi module may breaks with Spark35
URL: https://github.com/apache/incubator-gluten/issues/7637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg module may breaks with Spark35 [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2461204251
@Yohahaha Similar to https://github.com/apache/incubator-gluten/issues/7571, I think we should keep iceberg + Spark35 support in Gluten's next minor release (1.3.0). Any thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg and hudi module may breaks with Spark35 [incubator-gluten]
Posted by "gaoyangxiaozhu (via GitHub)" <gi...@apache.org>.
gaoyangxiaozhu commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2472277847
> but I'm confused why GHA still passed with iceberg tests, I only remove hudi profile when upgrading spark 353.
+1 why CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg module may breaks with Spark35 [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2445761026
Seems iceberg skips spark-3.5.3, see https://github.com/apache/iceberg/pull/11160. cc @zhouyuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg and hudi module may breaks with Spark35 [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2461378273
@Yohahaha Gluten tends to support latest spark release but since iceberg doesn't support spark3.5.3. Iceberg support is more important in Gluten now. Can we skip 3.5.3? once 3.5.4 fix the iceberg issue, we can add spark3.5.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg and hudi module may breaks with Spark35 [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2461343216
> Gluten's strategy is keep latest Spark release version, do we want change it?
Gluten's strategy is supporting one version instead of latest release version due to lack of resource. Perhaps starting from version 3.5, we can support a stable version 3.5.2, and a latest version 3.5.3. Of course, this needs further disscussion.
Back to https://github.com/apache/incubator-gluten/pull/7537, we don't change any meaningful codes, we can still using it in spark 3.5.3 even it was reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg and hudi module may breaks with Spark35 [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2472294680
that's interesting, iceberg tests pass actually...
anyway, let's continue revert to spark 352, if no questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg and hudi module may breaks with Spark35 [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2461598922
> @Yohahaha Gluten tends to support latest spark release but since iceberg doesn't support spark3.5.3. Iceberg support is more important in Gluten now. Can we skip 3.5.3? once 3.5.4 fix the iceberg issue, we can add spark3.5.4.
after internal discussion, let's skip spark 3.5.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg and hudi module may breaks with Spark35 [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2472268716
but I'm confused why GHA still passed with iceberg tests, I only remove hudi profile when upgrading spark 353.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg and hudi module may breaks with Spark35 [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2475831068
> > that's interesting, iceberg tests passed actually...
> > anyway, let's continue revert to spark 352, if no questions.
>
> @PHILO-HE any insight?
@FelixYBW, the existing iceberg tests in Gluten may not have enough coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg module may breaks with Spark35 [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2461220246
> @Yohahaha Similar to #7571, I think we should keep iceberg + Spark35 support in Gluten's next minor release (1.3.0). Any thoughts?
iceberg decides skip spark 3.5.3, do we have better options?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg and hudi module may breaks with Spark35 [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2461317998
@baibaichen @zhztheplayer any reason we must checkout release 1.3.0 from current main branch? why not checkout from the commit before 7537?
or we can checkout release 1.3.0 and revert 7537.
The important things is Spark latest release version on branch 3.5 is v3.5.3, Gluten's strategy is keep latest Spark release version, do we want change it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg and hudi module may breaks with Spark35 [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2470006686
I will revert spark version to 3.5.2 if no one takes it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg module may breaks with Spark35 [incubator-gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2461234709
https://github.com/apache/spark/pull/48257 is the related spark pr that fix the extension problem for iceberg, and it will be released in spark 3.5.4. And iceberg would skip 3.5.3 and keep up with 3.5.4 after spark 3.5.4 is released.
I'm afraid they won't be released before we release Gluten 1.3.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg and hudi module may breaks with Spark35 [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2461280425
> @Yohahaha Similar to #7571, I think we should keep iceberg + Spark35 support in Gluten's next minor release (1.3.0). Any thoughts?
>
> cc @baibaichen
@zhztheplayer @zhouyuan @zzcclp
It depends on which is impormant, supporting hudi and icerberg in spark 3.5.3 or supporting spark 3.5.3.
I guess spark 3.5.3 is not used in most cases, is there any reason that we must support spark 3.5.3? if not, I think revert to support 3.5.2 would be better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg and hudi module may breaks with Spark35 [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2461310102
It seems https://github.com/apache/incubator-gluten/pull/7537 didn't involve any code changes, it's probably easy to maintain as an individual patch.
@Yohahaha Would you exclude the possibility maintaining #7537 internally in your Gluten fork? Until iceberg / hudi bump their Spark version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CORE] Iceberg and hudi module may breaks with Spark35 [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #7637:
URL: https://github.com/apache/incubator-gluten/issues/7637#issuecomment-2474969038
>
> that's interesting, iceberg tests passed actually...
>
> anyway, let's continue revert to spark 352, if no questions.
@PHILO-HE any insight?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org