You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@graphar.apache.org by "yecol (via GitHub)" <gi...@apache.org> on 2024/08/14 08:33:20 UTC
[I] weird source code structure (maybe bug?) [incubator-graphar]
yecol opened a new issue, #593:
URL: https://github.com/apache/incubator-graphar/issues/593
### Describe the bug, including details regarding any error messages, version, and platform.
<img width="1234" alt="Screenshot 2024-08-14 at 4 29 59 PM" src="https://github.com/user-attachments/assets/487de71e-4fb6-4e8b-8aee-ed22e6fde5d0">
Just check out the latest code and find something weird.
As shown in the above figure, it seems some code is under dir `datasources-34` and `datasources-35`?
- should they be in the `maven-projects/spark/src/main` if they are library code?
- or should be in `maven-projects/spark/src/test` if they are test specific code?
### Component(s)
Spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] weird source code structure (maybe bug?) [incubator-graphar]
Posted by "yecol (via GitHub)" <gi...@apache.org>.
yecol closed issue #593: weird source code structure (maybe bug?)
URL: https://github.com/apache/incubator-graphar/issues/593
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] weird source code structure (maybe bug?) [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #593:
URL: https://github.com/apache/incubator-graphar/issues/593#issuecomment-2288273648
@yecol Our target is to support multiple versions of Apache Spark. Unfortunately, the `DataSource` API of Apache Spark is a Developer API and changing dramatically from one version of spark to another. And sometimes changes are so big, that reflection is not enough.
We [made](https://github.com/apache/incubator-graphar/issues/320) a decision to separate datasource implementation into a maven subpackage.
And we [have](https://github.com/apache/incubator-graphar/blob/main/maven-projects/spark/pom.xml#L51C1-L99C16) the following maven profiles:
```xml
<profiles>
<profile>
<id>datasources-32</id>
<properties>
<sbt.project.name>graphar</sbt.project.name>
<spark.version>3.2.4</spark.version>
</properties>
<modules>
<module>graphar</module>
<module>datasources-32</module>
</modules>
</profile>
<profile>
<id>datasources-33</id>
<properties>
<sbt.project.name>graphar</sbt.project.name>
<spark.version>3.3.4</spark.version>
</properties>
<modules>
<module>graphar</module>
<module>datasources-33</module>
</modules>
</profile>
<profile>
<id>datasources-34</id>
<properties>
<sbt.project.name>graphar</sbt.project.name>
<spark.version>3.4.3</spark.version>
</properties>
<modules>
<module>graphar</module>
<module>datasources-34</module>
</modules>
</profile>
<profile>
<id>datasources-35</id>
<properties>
<sbt.project.name>graphar</sbt.project.name>
<spark.version>3.5.1</spark.version>
</properties>
<modules>
<module>graphar</module>
<module>datasources-35</module>
</modules>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
</profile>
</profiles>
```
so, using that approach we are able to build GraphAr Spark for a different version of spark itself.
At the moment, that approach is used in our CI when we are running tests for all the supported Maven profiles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] weird source code structure (maybe bug?) [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #593:
URL: https://github.com/apache/incubator-graphar/issues/593#issuecomment-2288286760
An alternative way to provide the support is to use tags/branches. But for me it is better to have Meven sub-projects. At the random moment of time about 4-5 versions of spark are maintained, so I don't think that amount of duplicated code will grow infinitely: spark-3.2 is EoL soon, for example, so we can drop it, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] weird source code structure (maybe bug?) [incubator-graphar]
Posted by "yecol (via GitHub)" <gi...@apache.org>.
yecol commented on issue #593:
URL: https://github.com/apache/incubator-graphar/issues/593#issuecomment-2288352214
I see. It makes sense!
I didn't aware the diverged datasource versions of Spark. Thanks for your kindly and detailed response!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org