You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/08/31 08:29:38 UTC
[GitHub] [incubator-uniffle] leixm opened a new issue, #194: [Feature] Support spark 3.2
leixm opened a new issue, #194:
URL: https://github.com/apache/incubator-uniffle/issues/194
After SPARK-36892, MapOutputTrackerWorker#getMapSizesByExecutorId has been modified to include method names and return parameter types, Java.lang.NoSuchMethodException will be thrown when the RssShuffleManager#getExpectedTasksByExecutorId method is called, The version of our production environment is 3.2.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #194: [Feature] Support spark 3.2
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1232636735
Maybe we can support spark 3.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi closed issue #194: [Feature] Support spark 3.2.0
Posted by GitBox <gi...@apache.org>.
jerqi closed issue #194: [Feature] Support spark 3.2.0
URL: https://github.com/apache/incubator-uniffle/issues/194
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] smallzhongfeng commented on issue #194: [Feature] Support spark 3.2
Posted by GitBox <gi...@apache.org>.
smallzhongfeng commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1232976034
I think it may not be standard to build code by reflection. It may be more appropriate to create different modules and match different versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #194: [Feature] Support spark 3.2
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1233633501
We already supported Spark 3.2.1. You can see the Spark 3.2 profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #194: [Feature] Support spark 3.2.0
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1233854594
It seems not friendly for us to use extra patch. I think we can use reflection in our code to support 3.2.0 and add a profile.Spark 3.2.0 may be a special case, it shouldn't bring too much burden. If we need to maintain every minor version, it really bring too much burden.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #194: [Feature] Support spark 3.2.0
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1233752764
The main purpose of this issue is to confirm the following two points
1. Uniffle does not support spark3.2.0
2. Should we support spark3.2.0?
Point 1 has been confirmed, I believe that supporting spark 3.2.0 is beneficial to the uniffle community. Some users' production environment is version 3.2.0, but they want to use uniffle to serve their production environment. We can also give these users a good experience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #194: [Feature] Support spark 3.2.0
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1236655769
> > I think we can use reflection in our code to support 3.2.0 and add a profile
>
> We can support spark3.2.0 through reflection, but I don't quite understand why profile is needed. Profile cannot control the code logic, it can only be used to control the version of dependencies.
We need run the test pipeline with the profile to verify that the Spark version is ok and guarantee the new changes won't break the behavior of the version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #194: [Feature] Support spark 3.2
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1233636275
> I think it may not be standard to build code by reflection. It may be more appropriate to create different modules and match different spark versions.
Just avoid too repeated code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #194: [Feature] Support spark 3.2.0
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1233806967
If we want to support Spark 3.2.0, should we add a spark3.2.0 profile? The interface is only removed in the Spark 3.2.0 temporarily wrongly. Is it worth to support Spark 3.2.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #194: [Feature] Support spark 3.2.0
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1233826106
What you said is correct, maybe we can provide a patch file for users who need to support spark3.2.0 to apply it themselves, which will neither cause code redundancy nor confuse users. Taking a step back, if we don't support it, we can explain in the documentation that spark3.2.0 is not supported to avoid confusion for users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #194: [Feature] Support spark 3.2.0
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1236663762
Maybe I can raise a new pr to add reflection code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #194: [Feature] Support spark 3.2
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1232630511
@jerqi What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #194: [Feature] Support spark 3.2.0
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1233701653
> > We already supported Spark 3.2.1. You can see the Spark 3.2 profile.
>
> Spark 3.2.1 version re-added the getMapSizesByExecutorId method, but currently uniffle does not support spark 3.2.0, there is a simple way to verify, change the spark.version of the spark 3.2 profile from 3.2.1 to 3.2.0, and then run spark3 integration test can be reproduced
Why not upgrade your spark client version? It's easy to upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #194: [Feature] Support spark 3.2.0
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1233743964
> > > We already supported Spark 3.2.1. You can see the Spark 3.2 profile.
> >
> >
> > Spark 3.2.1 version re-added the getMapSizesByExecutorId method, but currently uniffle does not support spark 3.2.0, there is a simple way to verify, change the spark.version of the spark 3.2 profile from 3.2.1 to 3.2.0, and then run spark3 integration test can be reproduced
>
> Why not upgrade your spark client version? It's easy to upgrade.
This is not a problem with the spark client. The exception will be thrown in the Driver process. If we use the current uniffle version, we need to upgrade the spark version of the production environment to 3.2.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #194: [Feature] Support spark 3.2.0
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1236642507
> I think we can use reflection in our code to support 3.2.0 and add a profile
We can support spark3.2.0 through reflection, but I don't quite understand why profile is needed. Profile cannot control the code logic, it can only be used to control the version of dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #194: [Feature] Support spark 3.2.0
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1236667648
> Maybe I can raise a new pr to add reflection code?
OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #194: [Feature] Support spark 3.2.0
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1233700267
> We already supported Spark 3.2.1. You can see the Spark 3.2 profile.
Spark 3.2.1 version re-added the getMapSizesByExecutorId method, but currently uniffle does not support spark 3.2.0, there is a simple way to verify, change the spark.version of the spark 3.2 profile from 3.2.1 to 3.2.0, and then run spark3 integration test can be reproduced
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #194: [Feature] Support spark 3.2.0
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #194:
URL: https://github.com/apache/incubator-uniffle/issues/194#issuecomment-1233744525
We prefer to support spark3.2.0 in the uniffle client, so the cost will be lower.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org