You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@graphar.apache.org by "SemyonSinchenko (via GitHub)" <gi...@apache.org> on 2024/05/23 08:42:39 UTC
[I] [Spark] Move datasources under org.apache.spark package [incubator-graphar]
SemyonSinchenko opened a new issue, #493:
URL: https://github.com/apache/incubator-graphar/issues/493
### Describe the enhancement requested
Currently datasources implementation are under `org.apache.graphar` space. I suggest to move these classes to `org.apache.spark` package. It allows us, for example, to use spark internals. For example, it resolve the problem in #488 when `JSONOptions` is `private[sql]`.
### Component(s)
Spark, PySpark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [Spark] Move datasources under org.apache.spark package [incubator-graphar]
Posted by "acezen (via GitHub)" <gi...@apache.org>.
acezen closed issue #493: [Spark] Move datasources under org.apache.spark package
URL: https://github.com/apache/incubator-graphar/issues/493
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [Spark] Move datasources under org.apache.spark package [incubator-graphar]
Posted by "acezen (via GitHub)" <gi...@apache.org>.
acezen commented on issue #493:
URL: https://github.com/apache/incubator-graphar/issues/493#issuecomment-2126563955
There is a problem if we move to org.apache.spark is that we can not release the datasource maven package to the org.apache.spark in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [Spark] Move datasources under org.apache.spark package [incubator-graphar]
Posted by "acezen (via GitHub)" <gi...@apache.org>.
acezen commented on issue #493:
URL: https://github.com/apache/incubator-graphar/issues/493#issuecomment-2128817133
> > > > The graphar package relys on datasource, I don't know how can we release graphar without release the datasource package?
> > >
> > >
> > > It will be a compile time dependency. For example, like in delta-spark: https://github.com/delta-io/delta/tree/master/spark/src/main/scala/org/apache/spark/sql
> >
> >
> > It seems that as a compile time dependency, we need to put datasource back to maven-projects/graphar? Or do you know is there a way that datasource still be separated from graphar and can be a compile time dependency too? That would be perfect for this issue.
>
> I can experiment with it. It seems to me that yes, it should be possible.
Great, thanks Sem, Can I assign this issue to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [Spark] Move datasources under org.apache.spark package [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #493:
URL: https://github.com/apache/incubator-graphar/issues/493#issuecomment-2126567158
> There is a problem if we move to org.apache.spark is that we can not release the datasource maven packages to the org.apache.spark in the future.
But do we need it? We may release graphar itself and we may release PyPI package that already include JARs from datasources. Should datasources be released separately?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [Spark] Move datasources under org.apache.spark package [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #493:
URL: https://github.com/apache/incubator-graphar/issues/493#issuecomment-2128778602
> > > The graphar package relys on datasource, I don't know how can we release graphar without release the datasource package?
> >
> > It will be a compile time dependency. For example, like in delta-spark: https://github.com/delta-io/delta/tree/master/spark/src/main/scala/org/apache/spark/sql
>
> It seems that as a compile time dependency, we need to put datasource back to maven-projects/graphar? Or do you know is there a way that datasource still be separated from graphar and can be a compile time dependency too? That would be perfect for this issue.
I can experiment with it. It seems to me that yes, it should be possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [Spark] Move datasources under org.apache.spark package [incubator-graphar]
Posted by "acezen (via GitHub)" <gi...@apache.org>.
acezen commented on issue #493:
URL: https://github.com/apache/incubator-graphar/issues/493#issuecomment-2126578853
> > There is a problem if we move to org.apache.spark is that we can not release the datasource maven packages to the org.apache.spark in the future.
>
> But do we need it? We may release graphar itself and we may release PyPI package that already include JARs from datasources. Should datasources be released separately?
The graphar package relys on datasource, I don't know how can we release graphar without release the datasource package?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [Spark] Move datasources under org.apache.spark package [incubator-graphar]
Posted by "acezen (via GitHub)" <gi...@apache.org>.
acezen commented on issue #493:
URL: https://github.com/apache/incubator-graphar/issues/493#issuecomment-2128314523
> > The graphar package relys on datasource, I don't know how can we release graphar without release the datasource package?
>
> It will be a compile time dependency. For example, like in delta-spark: https://github.com/delta-io/delta/tree/master/spark/src/main/scala/org/apache/spark/sql
It seems that as a compile time dependency, we need to put datasource back to graphar? Or do you know is there a way that datasource still be separated from graphar and can be a compile time dependency? That would be perfect for this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [Spark] Move datasources under org.apache.spark package [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #493:
URL: https://github.com/apache/incubator-graphar/issues/493#issuecomment-2126937259
> The graphar package relys on datasource, I don't know how can we release graphar without release the datasource package?
It wild be a compile time dependency. For example, like in delta-spark: https://github.com/delta-io/delta/tree/master/spark/src/main/scala/org/apache/spark/sql
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [Spark] Move datasources under org.apache.spark package [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #493:
URL: https://github.com/apache/incubator-graphar/issues/493#issuecomment-2126563976
It may also simplify the full support of `v2` datasources. Last time when I tried to switch to `v2` I faced a problem that few important methods are `private[sql]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org