You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@graphar.apache.org by Sem <ss...@apache.org> on 2024/04/11 18:58:56 UTC
A new Pyspark API that will work with botth Spark Classic and Spark Connect
Hello!
The current PySpark implementation has one serious problem: it won't
work with the new Spark Connect because it relies on using `py4j` and
an internal `_jvm` variable.
My suggestion is to rewrite PySpark API from scratch in the following
way:
1. We will have pure Python GraphInfo, EdgeInfo and VertexInfo
2. We will have pure PySpark utils (index generators)
3. We will use spark scala datasources for writing and reading in GAR
format from PySpark
It is a lot of work, but I'm committing to do it and support it in the
future as a PMC of the project. Decoupling PySpark from Scala will also
simplify Scala/Java development. Another good point is that the actual
logic in PySpark will be mostly in Python code that simplifies reading
of source code and debugging for everyone who wants to work with a
library.
Couple of additional dependencies will be introduced:
1. Pydantic for working with YAML-models of Info objects (MIT License,
pure python)
2. PyYaml for the same reason (MIT License, pure python/cython)
Overall it should be good for the project, bceause it will simplify
testing of both part (spark and pyspark).
I see GraphAr PySpark mostly not in production ETL-jobs but in
interactive development and ad-hoc analytics on graph data. And
typically such an analytics happens on Databricks Notebooks (does not
provide an access to `_jvm` in shared clsuters) or in other tools (like
VSCode spark-connect) relies on Spark Connect. So, for that case
support of Spark Connect may be more important than for spark scala
part that should be used for jobs not interactive development.
Thoughts?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@graphar.apache.org
For additional commands, e-mail: dev-help@graphar.apache.org
Re: A new Pyspark API that will work with botth Spark Classic and Spark Connect
Posted by Weibin Zeng <we...@apache.org>.
+1, agree. Providing a pure and easy-to-use PySpark Library and decoupling Spark would really help developers/users to get start with GraphAr project and apply more application scenarios. Like spark-java-info-refacting branch, maybe we can new a feature for starting new-graphar-pyspark. We can merge to `main` when the new-graphar-pyspark is workable.
On 2024/04/11 18:58:56 Sem wrote:
> Hello!
>
> The current PySpark implementation has one serious problem: it won't
> work with the new Spark Connect because it relies on using `py4j` and
> an internal `_jvm` variable.
>
> My suggestion is to rewrite PySpark API from scratch in the following
> way:
>
> 1. We will have pure Python GraphInfo, EdgeInfo and VertexInfo
> 2. We will have pure PySpark utils (index generators)
> 3. We will use spark scala datasources for writing and reading in GAR
> format from PySpark
>
> It is a lot of work, but I'm committing to do it and support it in the
> future as a PMC of the project. Decoupling PySpark from Scala will also
> simplify Scala/Java development. Another good point is that the actual
> logic in PySpark will be mostly in Python code that simplifies reading
> of source code and debugging for everyone who wants to work with a
> library.
>
> Couple of additional dependencies will be introduced:
> 1. Pydantic for working with YAML-models of Info objects (MIT License,
> pure python)
> 2. PyYaml for the same reason (MIT License, pure python/cython)
>
> Overall it should be good for the project, bceause it will simplify
> testing of both part (spark and pyspark).
>
> I see GraphAr PySpark mostly not in production ETL-jobs but in
> interactive development and ad-hoc analytics on graph data. And
> typically such an analytics happens on Databricks Notebooks (does not
> provide an access to `_jvm` in shared clsuters) or in other tools (like
> VSCode spark-connect) relies on Spark Connect. So, for that case
> support of Spark Connect may be more important than for spark scala
> part that should be used for jobs not interactive development.
>
> Thoughts?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@graphar.apache.org
> For additional commands, e-mail: dev-help@graphar.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@graphar.apache.org
For additional commands, e-mail: dev-help@graphar.apache.org
回复：A new Pyspark API that will work with botth Spark Classic and Spark Connect
Posted by "李雪(有理)" <yo...@alibaba-inc.com.INVALID>.
I agree with the proposed revamp of the PySpark API. It's a good move, and to ensure smooth progress, maybe we could break it down into smaller, manageable steps in more detail.
------------------------------------------------------------------
发件人：Sem <ss...@apache.org>
发送时间：2024年4月12日(星期五) 17:05
收件人：dev<de...@graphar.apache.org>
主 题：Re: A new Pyspark API that will work with botth Spark Classic and Spark Connect
Exactly!
It is the topic I wanted to discuss on the latest community meeting!
On Fri, 2024-04-12 at 08:23 +0000, Weibin Zeng wrote:
> Perhaps this is not directly relevant, but is the proposal you're
> mentioning the topic that was intended for discussion at the last
> community meeting?
>
> On 2024/04/11 18:58:56 Sem wrote:
> > Hello!
> >
> > The current PySpark implementation has one serious problem: it
> > won't
> > work with the new Spark Connect because it relies on using `py4j`
> > and
> > an internal `_jvm` variable.
> >
> > My suggestion is to rewrite PySpark API from scratch in the
> > following
> > way:
> >
> > 1. We will have pure Python GraphInfo, EdgeInfo and VertexInfo
> > 2. We will have pure PySpark utils (index generators)
> > 3. We will use spark scala datasources for writing and reading in
> > GAR
> > format from PySpark
> >
> > It is a lot of work, but I'm committing to do it and support it in
> > the
> > future as a PMC of the project. Decoupling PySpark from Scala will
> > also
> > simplify Scala/Java development. Another good point is that the
> > actual
> > logic in PySpark will be mostly in Python code that simplifies
> > reading
> > of source code and debugging for everyone who wants to work with a
> > library.
> >
> > Couple of additional dependencies will be introduced:
> > 1. Pydantic for working with YAML-models of Info objects (MIT
> > License,
> > pure python)
> > 2. PyYaml for the same reason (MIT License, pure python/cython)
> >
> > Overall it should be good for the project, bceause it will simplify
> > testing of both part (spark and pyspark).
> >
> > I see GraphAr PySpark mostly not in production ETL-jobs but in
> > interactive development and ad-hoc analytics on graph data. And
> > typically such an analytics happens on Databricks Notebooks (does
> > not
> > provide an access to `_jvm` in shared clsuters) or in other tools
> > (like
> > VSCode spark-connect) relies on Spark Connect. So, for that case
> > support of Spark Connect may be more important than for spark scala
> > part that should be used for jobs not interactive development.
> >
> > Thoughts?
> >
> > -------------------------------------------------------------------
> > --
> > To unsubscribe, e-mail: dev-unsubscribe@graphar.apache.org
> > For additional commands, e-mail: dev-help@graphar.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@graphar.apache.org
> For additional commands, e-mail: dev-help@graphar.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@graphar.apache.org
For additional commands, e-mail: dev-help@graphar.apache.org
Re: A new Pyspark API that will work with botth Spark Classic and Spark Connect
Posted by Sem <ss...@apache.org>.
Exactly!
It is the topic I wanted to discuss on the latest community meeting!
On Fri, 2024-04-12 at 08:23 +0000, Weibin Zeng wrote:
> Perhaps this is not directly relevant, but is the proposal you're
> mentioning the topic that was intended for discussion at the last
> community meeting?
>
> On 2024/04/11 18:58:56 Sem wrote:
> > Hello!
> >
> > The current PySpark implementation has one serious problem: it
> > won't
> > work with the new Spark Connect because it relies on using `py4j`
> > and
> > an internal `_jvm` variable.
> >
> > My suggestion is to rewrite PySpark API from scratch in the
> > following
> > way:
> >
> > 1. We will have pure Python GraphInfo, EdgeInfo and VertexInfo
> > 2. We will have pure PySpark utils (index generators)
> > 3. We will use spark scala datasources for writing and reading in
> > GAR
> > format from PySpark
> >
> > It is a lot of work, but I'm committing to do it and support it in
> > the
> > future as a PMC of the project. Decoupling PySpark from Scala will
> > also
> > simplify Scala/Java development. Another good point is that the
> > actual
> > logic in PySpark will be mostly in Python code that simplifies
> > reading
> > of source code and debugging for everyone who wants to work with a
> > library.
> >
> > Couple of additional dependencies will be introduced:
> > 1. Pydantic for working with YAML-models of Info objects (MIT
> > License,
> > pure python)
> > 2. PyYaml for the same reason (MIT License, pure python/cython)
> >
> > Overall it should be good for the project, bceause it will simplify
> > testing of both part (spark and pyspark).
> >
> > I see GraphAr PySpark mostly not in production ETL-jobs but in
> > interactive development and ad-hoc analytics on graph data. And
> > typically such an analytics happens on Databricks Notebooks (does
> > not
> > provide an access to `_jvm` in shared clsuters) or in other tools
> > (like
> > VSCode spark-connect) relies on Spark Connect. So, for that case
> > support of Spark Connect may be more important than for spark scala
> > part that should be used for jobs not interactive development.
> >
> > Thoughts?
> >
> > -------------------------------------------------------------------
> > --
> > To unsubscribe, e-mail: dev-unsubscribe@graphar.apache.org
> > For additional commands, e-mail: dev-help@graphar.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@graphar.apache.org
> For additional commands, e-mail: dev-help@graphar.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@graphar.apache.org
For additional commands, e-mail: dev-help@graphar.apache.org
Re: A new Pyspark API that will work with botth Spark Classic and Spark Connect
Posted by Weibin Zeng <we...@apache.org>.
Perhaps this is not directly relevant, but is the proposal you're mentioning the topic that was intended for discussion at the last community meeting?
On 2024/04/11 18:58:56 Sem wrote:
> Hello!
>
> The current PySpark implementation has one serious problem: it won't
> work with the new Spark Connect because it relies on using `py4j` and
> an internal `_jvm` variable.
>
> My suggestion is to rewrite PySpark API from scratch in the following
> way:
>
> 1. We will have pure Python GraphInfo, EdgeInfo and VertexInfo
> 2. We will have pure PySpark utils (index generators)
> 3. We will use spark scala datasources for writing and reading in GAR
> format from PySpark
>
> It is a lot of work, but I'm committing to do it and support it in the
> future as a PMC of the project. Decoupling PySpark from Scala will also
> simplify Scala/Java development. Another good point is that the actual
> logic in PySpark will be mostly in Python code that simplifies reading
> of source code and debugging for everyone who wants to work with a
> library.
>
> Couple of additional dependencies will be introduced:
> 1. Pydantic for working with YAML-models of Info objects (MIT License,
> pure python)
> 2. PyYaml for the same reason (MIT License, pure python/cython)
>
> Overall it should be good for the project, bceause it will simplify
> testing of both part (spark and pyspark).
>
> I see GraphAr PySpark mostly not in production ETL-jobs but in
> interactive development and ad-hoc analytics on graph data. And
> typically such an analytics happens on Databricks Notebooks (does not
> provide an access to `_jvm` in shared clsuters) or in other tools (like
> VSCode spark-connect) relies on Spark Connect. So, for that case
> support of Spark Connect may be more important than for spark scala
> part that should be used for jobs not interactive development.
>
> Thoughts?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@graphar.apache.org
> For additional commands, e-mail: dev-help@graphar.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@graphar.apache.org
For additional commands, e-mail: dev-help@graphar.apache.org