You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/05/10 18:05:14 UTC
[I] Will Comet support closed-source forks of Apache Spark (e.g. CSP versions)? [datafusion-comet]
andygrove opened a new issue, #414:
URL: https://github.com/apache/datafusion-comet/issues/414
### What is the problem the feature request solves?
We have our first PR up that works around an issue with Comet working with AWS Spark (https://github.com/apache/datafusion-comet/pull/412).
I think we need to carefully consider our stance on supporting closed-source forks of Spark from the cloud service providers.
Supporting closed-source Spark versions is challenging for many reasons:
- There are often custom operators and expressions that Comet will not be aware of, and, therefore will not be able to map to native versions
- Software updates can be pushed out at any time, potentially breaking compatibility with Comet
- It is difficult to debug issues because source is not available (and reverse engineering is prohibited)
- We would ideally need CI to test against all supported CSP versions
- It increases the maintenance burden for all contributors, and not all contributors will have access to the CSP versions for testing
If the community desires to maintain Comet versions that can work with CSP Spark versions, then I think we would need to find an approach that allows those contributors to extend the "core" Comet project and add CSP support without adding maintenance burden for the core project.
One idea, for example, would be to keep the core `datafusion-comet` project compatible with OSS Apache Spark, and then have specific downstream repositories such as `datafusion-comet-aws` that extend the project to support a specific CSP.
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Will Comet support closed-source forks of Apache Spark (e.g. CSP versions)? [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #414:
URL: https://github.com/apache/datafusion-comet/issues/414#issuecomment-2108818634
I agree that we cannot support (i.e. guarantee compatibility) with proprietary forks, but I guess it is OK to accept PRs like #412 since it doesn't break anything and can only increase adoption.
If the volume of such PRs becomes too large we can consider a `contrib` directory or even another repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Will Comet support closed-source forks of Apache Spark (e.g. CSP versions)? [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #414:
URL: https://github.com/apache/datafusion-comet/issues/414#issuecomment-2105743676
I plan on creating a PR to update our documentation to make it clear that we only support Apache Spark and not other Spark implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Will Comet support closed-source forks of Apache Spark (e.g. CSP versions)? [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #414: Will Comet support closed-source forks of Apache Spark (e.g. CSP versions)?
URL: https://github.com/apache/datafusion-comet/issues/414
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Will Comet support closed-source forks of Apache Spark (e.g. CSP versions)? [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #414:
URL: https://github.com/apache/datafusion-comet/issues/414#issuecomment-2775835470
The documentation does now state that we only support open-source Apache Spark, so I will close this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Will Comet support closed-source forks of Apache Spark (e.g. CSP versions)? [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #414:
URL: https://github.com/apache/datafusion-comet/issues/414#issuecomment-2105063433
Thanks @andygrove for creating this.
I think we don't claim that Comet supports for closed source forks of Spark right now. It would be impossible to make such claims as we don't have such resources to make sure it happens.
For #412, I think although it is proposed to support AWS Spark, but the patch actually can be seen as a prevention to take unexpected constructors which have different parameters. I think it makes sense to do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Will Comet support closed-source forks of Apache Spark (e.g. CSP versions)? [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #414:
URL: https://github.com/apache/datafusion-comet/issues/414#issuecomment-2105071185
There is also a reported compatibility issue with Databricks Spark: #190
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org