You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@graphar.apache.org by "acezen (via GitHub)" <gi...@apache.org> on 2024/04/11 04:02:41 UTC
[PR] [Docs] Reorg the documents to adapt website building [incubator-graphar]
acezen opened a new pull request, #448:
URL: https://github.com/apache/incubator-graphar/pull/448
## Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._
- [ ] I have read the [CONTRIBUTING](https://github.com/apache/incubator-graphar/blob/main/CONTRIBUTING.rst) doc
- [ ] I have signed the CLA
- [ ] Lint and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] I have added necessary documentation (if appropriate)
## Further comments
If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [PR] [Docs] Reorganize the documents to adapt website building [incubator-graphar]
Posted by "acezen (via GitHub)" <gi...@apache.org>.
acezen commented on code in PR #448:
URL: https://github.com/apache/incubator-graphar/pull/448#discussion_r1560609249
##########
docs/libraries/pyspark/how-to.md:
##########
@@ -0,0 +1,212 @@
+---
+id: how-to
+title: How to use GraphAr PySpark package
+sidebar_position: 1
+---
+
+
+## GraphAr PySpark
+
+``graphar_pyspark`` is implemented as bindings to GraphAr spark scala
+library. You should have ``graphar-0.1.0-SNAPSHOT.jar`` in your
+Apache Spark JVM classpath. Otherwise you will get an exception. To
+add it spceify ``config("spark.jars", "path-to-graphar-jar")`` when
+you create a SparkSession:
+
+```python
+from pyspark.sql import SparkSession
+
+spark = (
+ SparkSession
+ .builder
+ .master("local[1]")
+ .appName("graphar-local-tests")
+ .config("spark.jars", "../../spark/target/graphar-0.1.0-SNAPSHOT.jar")
+ .config("spark.log.level", "INFO")
+ .getOrCreate()
+)
+```
+
+
+ .. rubric:: GraphAr PySpark initialize
+ :name: graphar-pyspark-initialize
+
+## GraphAr PySpark initialize
+
+PySpark bindings are heavily relying on JVM-calls via ``py4j``. To
+initiate all the neccessary things for it just call
+``graphar_pyspark.initialize()``:
+
+```python
+from graphar_pyspark import initialize
+
+initialize(spark)
+```
+
+## GraphAr objects
+
+Now you can import, create and modify all the classes you can work
+call from [scala API of GraphAr](https://graphar.apache.org/docs/libraries/spark).
+For simplify using of graphar from python constants, like GAR-types,
+supported file-types, etc. are placed in ``graphar_pyspark.enums``.
+
+```python
+from graphar_pyspark.info import Property, PropertyGroup, AdjList, AdjListType, VertexInfo, EdgeInfo, GraphInfo
+from graphar_pyspark.enums import GarType, FileType
+```
+
+Main objects of GraphAr are the following:
+
+- GraphInfo
+- VertexInfo
+- EdgeInfo
+
+You can check [Scala library documentation](https://graphar.apache.org/docs/spark#information-classes)
Review Comment:
yes, we can replace with relative path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [PR] [Docs] Reorganize the documents to adapt website building [incubator-graphar]
Posted by "acezen (via GitHub)" <gi...@apache.org>.
acezen commented on code in PR #448:
URL: https://github.com/apache/incubator-graphar/pull/448#discussion_r1560609946
##########
docs/libraries/pyspark/how-to.md:
##########
@@ -0,0 +1,212 @@
+---
+id: how-to
+title: How to use GraphAr PySpark package
+sidebar_position: 1
+---
+
+
+## GraphAr PySpark
+
+``graphar_pyspark`` is implemented as bindings to GraphAr spark scala
+library. You should have ``graphar-0.1.0-SNAPSHOT.jar`` in your
+Apache Spark JVM classpath. Otherwise you will get an exception. To
+add it spceify ``config("spark.jars", "path-to-graphar-jar")`` when
Review Comment:
Thanks for pointing the typo, maybe we can add a CI to check typos, I will try to add in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [PR] [Docs] Reorganize the documents to adapt website building [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on code in PR #448:
URL: https://github.com/apache/incubator-graphar/pull/448#discussion_r1560586502
##########
docs/libraries/pyspark/how-to.md:
##########
@@ -0,0 +1,212 @@
+---
+id: how-to
+title: How to use GraphAr PySpark package
+sidebar_position: 1
+---
+
+
+## GraphAr PySpark
+
+``graphar_pyspark`` is implemented as bindings to GraphAr spark scala
+library. You should have ``graphar-0.1.0-SNAPSHOT.jar`` in your
+Apache Spark JVM classpath. Otherwise you will get an exception. To
+add it spceify ``config("spark.jars", "path-to-graphar-jar")`` when
+you create a SparkSession:
+
+```python
+from pyspark.sql import SparkSession
+
+spark = (
+ SparkSession
+ .builder
+ .master("local[1]")
+ .appName("graphar-local-tests")
+ .config("spark.jars", "../../spark/target/graphar-0.1.0-SNAPSHOT.jar")
+ .config("spark.log.level", "INFO")
+ .getOrCreate()
+)
+```
+
+
+ .. rubric:: GraphAr PySpark initialize
+ :name: graphar-pyspark-initialize
+
+## GraphAr PySpark initialize
+
+PySpark bindings are heavily relying on JVM-calls via ``py4j``. To
+initiate all the neccessary things for it just call
+``graphar_pyspark.initialize()``:
+
+```python
+from graphar_pyspark import initialize
+
+initialize(spark)
+```
+
+## GraphAr objects
+
+Now you can import, create and modify all the classes you can work
Review Comment:
```suggestion
Now you can import, create and modify all the classes you can
```
##########
docs/libraries/pyspark/how-to.md:
##########
@@ -0,0 +1,212 @@
+---
+id: how-to
+title: How to use GraphAr PySpark package
+sidebar_position: 1
+---
+
+
+## GraphAr PySpark
+
+``graphar_pyspark`` is implemented as bindings to GraphAr spark scala
+library. You should have ``graphar-0.1.0-SNAPSHOT.jar`` in your
+Apache Spark JVM classpath. Otherwise you will get an exception. To
+add it spceify ``config("spark.jars", "path-to-graphar-jar")`` when
+you create a SparkSession:
+
+```python
+from pyspark.sql import SparkSession
+
+spark = (
+ SparkSession
+ .builder
+ .master("local[1]")
+ .appName("graphar-local-tests")
+ .config("spark.jars", "../../spark/target/graphar-0.1.0-SNAPSHOT.jar")
+ .config("spark.log.level", "INFO")
+ .getOrCreate()
+)
+```
+
+
+ .. rubric:: GraphAr PySpark initialize
+ :name: graphar-pyspark-initialize
+
+## GraphAr PySpark initialize
+
+PySpark bindings are heavily relying on JVM-calls via ``py4j``. To
+initiate all the neccessary things for it just call
+``graphar_pyspark.initialize()``:
+
+```python
+from graphar_pyspark import initialize
+
+initialize(spark)
+```
+
+## GraphAr objects
+
+Now you can import, create and modify all the classes you can work
+call from [scala API of GraphAr](https://graphar.apache.org/docs/libraries/spark).
+For simplify using of graphar from python constants, like GAR-types,
+supported file-types, etc. are placed in ``graphar_pyspark.enums``.
+
+```python
+from graphar_pyspark.info import Property, PropertyGroup, AdjList, AdjListType, VertexInfo, EdgeInfo, GraphInfo
+from graphar_pyspark.enums import GarType, FileType
+```
+
+Main objects of GraphAr are the following:
+
+- GraphInfo
+- VertexInfo
+- EdgeInfo
+
+You can check [Scala library documentation](https://graphar.apache.org/docs/spark#information-classes)
Review Comment:
Is it possible to use relative paths instead? It would be better than full links. Is it working in apache-like websites?
##########
docs/libraries/pyspark/how-to.md:
##########
@@ -0,0 +1,212 @@
+---
+id: how-to
+title: How to use GraphAr PySpark package
+sidebar_position: 1
+---
+
+
+## GraphAr PySpark
+
+``graphar_pyspark`` is implemented as bindings to GraphAr spark scala
+library. You should have ``graphar-0.1.0-SNAPSHOT.jar`` in your
+Apache Spark JVM classpath. Otherwise you will get an exception. To
+add it spceify ``config("spark.jars", "path-to-graphar-jar")`` when
+you create a SparkSession:
+
+```python
+from pyspark.sql import SparkSession
+
+spark = (
+ SparkSession
+ .builder
+ .master("local[1]")
+ .appName("graphar-local-tests")
+ .config("spark.jars", "../../spark/target/graphar-0.1.0-SNAPSHOT.jar")
Review Comment:
```suggestion
.config("spark.jars", "../../spark/graphar/target/graphar-0.1.0-SNAPSHOT.jar")
```
Because of submodules
##########
docs/libraries/pyspark/how-to.md:
##########
@@ -0,0 +1,212 @@
+---
+id: how-to
+title: How to use GraphAr PySpark package
+sidebar_position: 1
+---
+
+
+## GraphAr PySpark
+
+``graphar_pyspark`` is implemented as bindings to GraphAr spark scala
+library. You should have ``graphar-0.1.0-SNAPSHOT.jar`` in your
+Apache Spark JVM classpath. Otherwise you will get an exception. To
+add it spceify ``config("spark.jars", "path-to-graphar-jar")`` when
+you create a SparkSession:
+
+```python
+from pyspark.sql import SparkSession
+
+spark = (
+ SparkSession
+ .builder
+ .master("local[1]")
+ .appName("graphar-local-tests")
+ .config("spark.jars", "../../spark/target/graphar-0.1.0-SNAPSHOT.jar")
+ .config("spark.log.level", "INFO")
+ .getOrCreate()
+)
+```
+
+
+ .. rubric:: GraphAr PySpark initialize
+ :name: graphar-pyspark-initialize
+
+## GraphAr PySpark initialize
+
+PySpark bindings are heavily relying on JVM-calls via ``py4j``. To
+initiate all the neccessary things for it just call
+``graphar_pyspark.initialize()``:
+
+```python
+from graphar_pyspark import initialize
+
+initialize(spark)
+```
+
+## GraphAr objects
+
+Now you can import, create and modify all the classes you can work
+call from [scala API of GraphAr](https://graphar.apache.org/docs/libraries/spark).
+For simplify using of graphar from python constants, like GAR-types,
+supported file-types, etc. are placed in ``graphar_pyspark.enums``.
+
+```python
+from graphar_pyspark.info import Property, PropertyGroup, AdjList, AdjListType, VertexInfo, EdgeInfo, GraphInfo
+from graphar_pyspark.enums import GarType, FileType
+```
+
+Main objects of GraphAr are the following:
+
+- GraphInfo
+- VertexInfo
+- EdgeInfo
+
+You can check [Scala library documentation](https://graphar.apache.org/docs/spark#information-classes)
+for the more detailed information.
+
+
+## Creating objects in graphar_pyspark
+
+GraphAr PySpark package provide two main ways how to initiate
+objects, like ``GraphInfo``:
+
+#. ``from_python(**args)`` when you create an object based on
+ python-arguments
+#. ``from_scala(jvm_ref)`` when you create an object from the
+ corresponded JVM-object (``py4j.java_gateway.JavaObject``)
+
+
+```python
+help(Property.from_python)
+
+Help on method from_python in module graphar_pyspark.info:
+
+from_python(name: 'str', data_type: 'GarType', is_primary: 'bool') -> 'PropertyType' method of builtins.type instance
+ Create an instance of the Class from Python arguments.
+
+ :param name: property name
+ :param data_type: property data type
+ :param is_primary: flag that property is primary
+ :returns: instance of Python Class.
+```
+
+```python
+python_property = Property.from_python(name="my_property", data_type=GarType.INT64, is_primary=False)
+print(type(python_property))
+
+<class 'graphar_pyspark.info.Property'>
+```
+
+You can always get a reference to the corresponding JVM object. For
+example, you want to use it in your own code and need a direct link
Review Comment:
```suggestion
example, if you want to use it in your own code and need a direct link
```
##########
docs/libraries/spark/spark.md:
##########
@@ -0,0 +1,235 @@
+---
+id: spark
+title: Spark Library
+sidebar_position: 3
+---
+
+## Overview
+
+The GraphAr Spark library is provided for generating, loading and transforming GAR files with Apache Spark easy. It consists of several components:
+
+- **Information Classes**: As same with in the C++ library, the information classes are implemented as a part of the Spark library for constructing and accessing the meta information about the graphs, vertices and edges in GraphAr.
+- **IndexGenerator**: The IndexGenerator helps to generate the indices for vertex/edge DataFrames. In most cases, IndexGenerator is first utilized to generate the indices for a DataFrame (e.g., from primary keys), and then this DataFrame can be written into GAR files through the writer.
+- **Writer**: The GraphAr Spark writer provides a set of interfaces that can be used to write Spark DataFrames into GAR files. Every time it takes a DataFrame as the logical table for a type of vertices or edges, assembles the data in specified format (e.g., reorganize the edges in the CSR way) and then dumps it to standard GAR files (CSV, ORC or Parquet files) under the specific directory path.
+- **Reader**: The GraphAr Spark reader provides a set of interfaces that can be used to read GAR files. It reads a collection of vertices or edges at a time and assembles the result into the Spark DataFrame. Similar with the reader in the C++ library, it supports the users to specify the data they need, e.g., reading a single property group instead of all properties.
+
+## Use Cases
+
+The GraphAr Spark library can be used in a range of scenarios:
+
+- Taking GAR as a data source to execute SQL queries or do graph processing (e.g., using GraphX).
+- Transforming data between GAR and other data sources (e.g., Hive, Neo4j, NebulaGraph, ...).
+- Transforming GAR data between different file types (e.g., from ORC to Parquet).
+- Transforming GAR data between different adjList types (e.g., from COO to CSR).
+- Modifying existing GAR data (e.g., adding new vertices/edges).
+
+For more information on its usage, please refer to the `Examples <examples/spark.html>`_.
+
+
+## Get GraphAr Spark Library
+
+### Building from source
+
+Make the graphar-spark-library directory as the current working directory:
+
+```bash
+cd GraphAr/spark/
+```
+
+Compile package:
+
+```bash
+mvn clean package -DskipTests
+```
+
+GraphAr supports two Apache Spark versions for now and uses Maven Profiles to work with it. The command above built GraphAr with Spark 3.2.2 by default. To built GraphAr with Spark 3.3.4 use the following command:
+
+```bash
+mvn clean package -DskipTests -P datasources-33
+```
+
+After compilation, a similar file *graphar-x.x.x-SNAPSHOT-shaded.jar* is generated in the directory *spark/graphar/target/*.
+
+Please refer to the `building steps <https://github.com/apache/incubator-graphar/tree/main/spark>`_ for more details.
Review Comment:
```suggestion
Please refer to the [building steps](https://github.com/apache/incubator-graphar/tree/main/spark) for more details.
```
##########
docs/libraries/pyspark/how-to.md:
##########
@@ -0,0 +1,212 @@
+---
+id: how-to
+title: How to use GraphAr PySpark package
+sidebar_position: 1
+---
+
+
+## GraphAr PySpark
+
+``graphar_pyspark`` is implemented as bindings to GraphAr spark scala
+library. You should have ``graphar-0.1.0-SNAPSHOT.jar`` in your
+Apache Spark JVM classpath. Otherwise you will get an exception. To
+add it spceify ``config("spark.jars", "path-to-graphar-jar")`` when
+you create a SparkSession:
+
+```python
+from pyspark.sql import SparkSession
+
+spark = (
+ SparkSession
+ .builder
+ .master("local[1]")
+ .appName("graphar-local-tests")
+ .config("spark.jars", "../../spark/target/graphar-0.1.0-SNAPSHOT.jar")
+ .config("spark.log.level", "INFO")
+ .getOrCreate()
+)
+```
+
+
+ .. rubric:: GraphAr PySpark initialize
+ :name: graphar-pyspark-initialize
+
+## GraphAr PySpark initialize
+
+PySpark bindings are heavily relying on JVM-calls via ``py4j``. To
+initiate all the neccessary things for it just call
+``graphar_pyspark.initialize()``:
+
+```python
+from graphar_pyspark import initialize
+
+initialize(spark)
+```
+
+## GraphAr objects
+
+Now you can import, create and modify all the classes you can work
+call from [scala API of GraphAr](https://graphar.apache.org/docs/libraries/spark).
+For simplify using of graphar from python constants, like GAR-types,
+supported file-types, etc. are placed in ``graphar_pyspark.enums``.
+
+```python
+from graphar_pyspark.info import Property, PropertyGroup, AdjList, AdjListType, VertexInfo, EdgeInfo, GraphInfo
+from graphar_pyspark.enums import GarType, FileType
+```
+
+Main objects of GraphAr are the following:
+
+- GraphInfo
+- VertexInfo
+- EdgeInfo
+
+You can check [Scala library documentation](https://graphar.apache.org/docs/spark#information-classes)
+for the more detailed information.
+
+
+## Creating objects in graphar_pyspark
+
+GraphAr PySpark package provide two main ways how to initiate
+objects, like ``GraphInfo``:
+
+#. ``from_python(**args)`` when you create an object based on
+ python-arguments
+#. ``from_scala(jvm_ref)`` when you create an object from the
+ corresponded JVM-object (``py4j.java_gateway.JavaObject``)
+
+
+```python
+help(Property.from_python)
+
+Help on method from_python in module graphar_pyspark.info:
+
+from_python(name: 'str', data_type: 'GarType', is_primary: 'bool') -> 'PropertyType' method of builtins.type instance
+ Create an instance of the Class from Python arguments.
+
+ :param name: property name
+ :param data_type: property data type
+ :param is_primary: flag that property is primary
+ :returns: instance of Python Class.
+```
+
+```python
+python_property = Property.from_python(name="my_property", data_type=GarType.INT64, is_primary=False)
+print(type(python_property))
+
+<class 'graphar_pyspark.info.Property'>
+```
+
+You can always get a reference to the corresponding JVM object. For
+example, you want to use it in your own code and need a direct link
+to the underlaying instance of Scala Class, you can just call
+``to_scala()`` method:
+
+```python
+scala_obj = python_property.to_scala()
+print(type(scala_obj))
+
+<class 'py4j.java_gateway.JavaObject'>
+```
+
+As we already mentioned, you can initialize an instance of the Python
+class from the JVM object:
+
+```python
+help(Property.from_scala)
+
+Help on method from_scala in module graphar_pyspark.info:
+
+ from_scala(jvm_obj: 'JavaObject') -> 'PropertyType' method of builtins.type instance
+ Create an instance of the Class from the corresponding JVM object.
+
+ :param jvm_obj: scala object in JVM.
+ :returns: instance of Python Class.
+```
+
+```python
+python_property = Property.from_scala(scala_obj)
+```
+
+Each public property and method of the Scala API is provided in
+python, but in a pythonic-naming convention. For example, in Scala,
+``Property`` has the following fields:
+
+- name
+- data_type
+- is_primary
+
+For each of such a field in Scala API there is a getter and setter
+methods. You can call them from the Python too:
+
+```python
+python_property.get_name()
+
+'my_property'
+```
+
+You can also modify fields, but be careful: when you modify field of
+instance of the Python class, you modify the underlaying Scala Object
+in the same moment!
Review Comment:
```suggestion
at the same moment!
```
##########
docs/libraries/spark/spark.md:
##########
@@ -0,0 +1,235 @@
+---
+id: spark
+title: Spark Library
+sidebar_position: 3
+---
+
+## Overview
+
+The GraphAr Spark library is provided for generating, loading and transforming GAR files with Apache Spark easy. It consists of several components:
+
+- **Information Classes**: As same with in the C++ library, the information classes are implemented as a part of the Spark library for constructing and accessing the meta information about the graphs, vertices and edges in GraphAr.
+- **IndexGenerator**: The IndexGenerator helps to generate the indices for vertex/edge DataFrames. In most cases, IndexGenerator is first utilized to generate the indices for a DataFrame (e.g., from primary keys), and then this DataFrame can be written into GAR files through the writer.
+- **Writer**: The GraphAr Spark writer provides a set of interfaces that can be used to write Spark DataFrames into GAR files. Every time it takes a DataFrame as the logical table for a type of vertices or edges, assembles the data in specified format (e.g., reorganize the edges in the CSR way) and then dumps it to standard GAR files (CSV, ORC or Parquet files) under the specific directory path.
+- **Reader**: The GraphAr Spark reader provides a set of interfaces that can be used to read GAR files. It reads a collection of vertices or edges at a time and assembles the result into the Spark DataFrame. Similar with the reader in the C++ library, it supports the users to specify the data they need, e.g., reading a single property group instead of all properties.
+
+## Use Cases
+
+The GraphAr Spark library can be used in a range of scenarios:
+
+- Taking GAR as a data source to execute SQL queries or do graph processing (e.g., using GraphX).
+- Transforming data between GAR and other data sources (e.g., Hive, Neo4j, NebulaGraph, ...).
+- Transforming GAR data between different file types (e.g., from ORC to Parquet).
+- Transforming GAR data between different adjList types (e.g., from COO to CSR).
+- Modifying existing GAR data (e.g., adding new vertices/edges).
+
+For more information on its usage, please refer to the `Examples <examples/spark.html>`_.
Review Comment:
```suggestion
For more information on its usage, please refer to the [Examples](examples/spark.md).
```
##########
docs/libraries/pyspark/pyspark.md:
##########
@@ -0,0 +1,107 @@
+---
+id: pyspark
+title: PySpark Library
+sidebar_position: 4
+---
+
+
+> **note:** The current policy of GraphAr project is that for Apache Spark
+> the main API is Scala Spark API. PySpark API follows scala Spark API.
+> Please refer to [GraphAr Spark library](../spark/spark.md)
+> for more detailed information about how to use GraphAr with Apache
+> Spark.
+
+## Overview
+
+The GraphAr PySpark library is provided for generating, loading and
+transforming GAR files with PySpark.
+
+- **Information Classes**: As same with in the C++ library, the
+ information classes are implemented as a part of the PySpark library
+ for constructing and accessing the meta information about the graphs,
+ vertices and edges in GraphAr.
+- **IndexGenerator**: The IndexGenerator helps to generate the indices
+ for vertex/edge DataFrames. In most cases, IndexGenerator is first
+ utilized to generate the indices for a DataFrame (e.g., from primary
+ keys), and then this DataFrame can be written into GAR files through
+ the writer.
+- **Writer**: The GraphAr PySpark writer provides a set of interfaces
+ that can be used to write Spark DataFrames into GAR files. Every time
+ it takes a DataFrame as the logical table for a type of vertices or
+ edges, assembles the data in specified format (e.g., reorganize the
+ edges in the CSR way) and then dumps it to standard GAR files (CSV,
+ ORC or Parquet files) under the specific directory path.
+- **Reader**: The GraphAr PySpark reader provides a set of interfaces
+ that can be used to read GAR files. It reads a collection of vertices
+ or edges at a time and assembles the result into the Spark DataFrame.
+ Similar with the reader in the C++ library, it supports the users to
+ specify the data they need, e.g., reading a single property group
+ instead of all properties.
+
+## Use Cases
+
+The GraphAr Spark library can be used in a range of scenarios:
+
+- Taking GAR as a data source to execute SQL queries or do graph
+ processing (e.g., using GraphX).
+- Transforming data between GAR and other data sources (e.g., Hive,
+ Neo4j, NebulaGraph, …).
+- Transforming GAR data between different file types (e.g., from ORC to
+ Parquet).
+- Transforming GAR data between different adjList types (e.g., from COO
+ to CSR).
+- Modifying existing GAR data (e.g., adding new vertices/edges).
+
+## Get GraphAr Spark Library
+
+### Building from source
+
+GraphAr PySpark uses poetry as a build system. Please refer to
+[Poetry documentation](https://python-poetry.org/docs/#installation)
+to find the manual how to install this tool. Currently GraphAr PySpark
+is build with Python 3.9 and PySpark 3.2
+
+
+Make the graphar-pyspark-library directory as the current working
+directory:
+
+```bash
+cd GraphAr/pyspark
+```
+
+Build package:
+
+
+```bash
+poetry build
+```
+
+After compilation, a similar file *graphar_pyspark-0.0.1.tar.gz* is
+generated in the directory *pyspark/dist/*.
+
+### Get from PyPI
+
+You cannot install graphar-pyspark from PyPi for now.
+
+
+## How to Use
+
+### Initialization
+
+GraphAr PySpark is not a standalone library but bindings to GraphAr
+Scala. You need to have *graphar-spark-x.x.x.jar* in your *spark-jars*.
+Please refer to `GraphAr scala documentation <../spark/index>`\_ to get
Review Comment:
It looks like RST syntax, not MD.
```suggestion
Please refer to [GraphAr scala documentation](../spark/spark.md) to get
```
##########
docs/libraries/pyspark/how-to.md:
##########
@@ -0,0 +1,212 @@
+---
+id: how-to
+title: How to use GraphAr PySpark package
+sidebar_position: 1
+---
+
+
+## GraphAr PySpark
+
+``graphar_pyspark`` is implemented as bindings to GraphAr spark scala
+library. You should have ``graphar-0.1.0-SNAPSHOT.jar`` in your
+Apache Spark JVM classpath. Otherwise you will get an exception. To
+add it spceify ``config("spark.jars", "path-to-graphar-jar")`` when
Review Comment:
```suggestion
add it specify ``config("spark.jars", "path-to-graphar-jar")`` when
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [PR] [Docs] Reorganize the documents to adapt website building [incubator-graphar]
Posted by "acezen (via GitHub)" <gi...@apache.org>.
acezen commented on PR #448:
URL: https://github.com/apache/incubator-graphar/pull/448#issuecomment-2049064241
cc/ @lixueclaire @Thespica @SemyonSinchenko This pull request make a lot of change to the document, may need your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [PR] [Docs] Reorganize the documents to adapt website building [incubator-graphar]
Posted by "Thespica (via GitHub)" <gi...@apache.org>.
Thespica commented on PR #448:
URL: https://github.com/apache/incubator-graphar/pull/448#issuecomment-2049578642
Hi, @acezen the changes itself is LGTM.
But when I try to build locally and take a look, I find it a little bit bothering. Can you write some letters that teach how to build documents locally in docs/README.md or docs for developers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [PR] [Docs] Reorganize the documents to adapt website building [incubator-graphar]
Posted by "acezen (via GitHub)" <gi...@apache.org>.
acezen commented on PR #448:
URL: https://github.com/apache/incubator-graphar/pull/448#issuecomment-2049653020
> Hi, @acezen the change itself is LGTM.
>
> But when I try to build locally and take a look, I find it a little bit bothering. Can you write some letters that teach how to build documents locally in docs/README.md or docs for developers?
hi,@Thespica, I have put the instruction how to build docs in apache-incubator-website README(they share the same process) and would merge after this PR. The docs/README would have a link to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [PR] [Docs] Reorganize the documents to adapt website building [incubator-graphar]
Posted by "acezen (via GitHub)" <gi...@apache.org>.
acezen commented on PR #448:
URL: https://github.com/apache/incubator-graphar/pull/448#issuecomment-2049405354
The path has been replaced with relative path, and I add a TODO for add markdown lint and typo check CI, please review again, thanks~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [PR] [Docs] Reorganize the documents to adapt website building [incubator-graphar]
Posted by "acezen (via GitHub)" <gi...@apache.org>.
acezen merged PR #448:
URL: https://github.com/apache/incubator-graphar/pull/448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org