You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@graphar.apache.org by we...@apache.org on 2024/04/11 14:49:00 UTC
(incubator-graphar) branch main updated: [Docs] Reorganize the documents to adapt website building (#448)
This is an automated email from the ASF dual-hosted git repository.
weibin pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-graphar.git
The following commit(s) were added to refs/heads/main by this push:
new 36dbcbf [Docs] Reorganize the documents to adapt website building (#448)
36dbcbf is described below
commit 36dbcbfd619095f3721b91ba36b1a2bfb42d3a15
Author: Weibin Zeng <qi...@alibaba-inc.com>
AuthorDate: Thu Apr 11 22:48:55 2024 +0800
[Docs] Reorganize the documents to adapt website building (#448)
## Proposed changes
Since we have created a repository [incubator-graphar-website](https://github.com/apache/incubator-graphar-website) as our website repo, we should clean the original website page related code from here and only remain the dev documents of the project core(including format, libraries) . So the changes of this Pull Request includes:
- Remove the website page related code.
- Convert the dev documents from RST to MarkDown and reorganize them.
- Refine the Docs CI base on the website building workflow.
Signed-off-by: acezen <qi...@alibaba-inc.com>
---
.github/workflows/docs.yml | 77 ++---
.github/workflows/license.yml | 2 +-
docs/.gitignore | 1 -
docs/.nojekyll | 0
docs/Makefile | 62 ----
docs/README.md | 10 +
docs/_static/css/brands.min.css | 6 -
docs/_static/css/custom.css | 18 -
docs/_static/css/panels.css | 147 --------
docs/_static/css/v4-shims.min.css | 6 -
docs/_static/webfonts/fa-brands-400.ttf | Bin 181852 -> 0 bytes
docs/_static/webfonts/fa-brands-400.woff2 | Bin 105536 -> 0 bytes
docs/_templates/base.html | 7 -
docs/_templates/footer.html | 75 ----
docs/_templates/layout.html | 17 -
docs/conf.py | 119 -------
docs/cpp/examples/bgl.rst | 93 -----
docs/cpp/examples/graphscope.rst | 32 --
docs/cpp/examples/index.rst | 10 -
docs/cpp/examples/out-of-core.rst | 122 -------
docs/cpp/examples/snap-to-graphar.rst | 81 -----
docs/cpp/getting-started.rst | 199 -----------
docs/cpp/index.rst | 9 -
docs/cpp/reference/index.rst | 301 ----------------
docs/developers/contributing.rst | 4 -
docs/developers/java-dev.rst | 108 ------
docs/format/status.rst | 380 ---------------------
docs/index.md | 17 +
docs/index.rst | 39 ---
docs/java/index.rst | 8 -
docs/java/java-lib.rst | 221 ------------
docs/java/reference/index.rst | 4 -
.../license.yml => docs/libraries/_category_.yml | 37 +-
.../libraries/cpp/_category_.yml | 37 +-
.../libraries/cpp/examples/_category_.yml | 36 +-
docs/libraries/cpp/examples/bgl.md | 108 ++++++
docs/libraries/cpp/examples/graphscope.md | 66 ++++
docs/libraries/cpp/examples/out-of-core.md | 180 ++++++++++
docs/libraries/cpp/examples/snap-to-graphar.md | 91 +++++
docs/libraries/cpp/getting-started.md | 278 +++++++++++++++
docs/libraries/java/how_to_develop_java.md | 106 ++++++
docs/libraries/java/java.md | 215 ++++++++++++
docs/libraries/pyspark/how-to.md | 212 ++++++++++++
docs/libraries/pyspark/pyspark.md | 111 ++++++
docs/libraries/spark/examples.md | 211 ++++++++++++
docs/libraries/spark/spark.md | 237 +++++++++++++
docs/overview/{concepts.rst => concepts.md} | 7 +-
docs/overview/{motivation.rst => motivation.md} | 9 +-
docs/overview/{overview.rst => overview.md} | 16 +-
docs/pyspark/api/graphar_pyspark.rst | 69 ----
docs/pyspark/api/modules.rst | 7 -
docs/pyspark/how-to.rst | 318 -----------------
docs/pyspark/index.rst | 9 -
docs/pyspark/pyspark-lib.rst | 81 -----
docs/spark/examples/spark.rst | 211 ------------
docs/spark/index.rst | 9 -
docs/spark/reference/index.rst | 4 -
docs/spark/spark-lib.rst | 247 --------------
.../specification/_category_.yml | 37 +-
.../file-format.rst => specification/format.md} | 132 ++++---
docs/specification/implementation-status.md | 253 ++++++++++++++
61 files changed, 2229 insertions(+), 3280 deletions(-)
diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
index ec29660..4c65b76 100644
--- a/.github/workflows/docs.yml
+++ b/.github/workflows/docs.yml
@@ -24,65 +24,56 @@ on:
pull_request:
branches:
- main
+ paths:
+ - 'docs/**'
+ - 'cpp/**'
+ - 'java/**'
+ - 'spark/**'
+ - 'pyspark/**'
+ - '.github/workflows/docs.yml'
concurrency:
group: ${{ github.repository }}-${{ github.event.number || github.head_ref || github.sha }}-${{ github.workflow }}
cancel-in-progress: true
jobs:
- build:
+ build-website:
runs-on: ubuntu-latest
steps:
- - uses: actions/setup-node@v3
- with:
- node-version: 16
-
- - name: Checkout Code
- uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
repository: ${{ github.event.pull_request.head.repo.full_name }}
ref: ${{ github.event.pull_request.head.ref }}
submodules: true
fetch-depth: 0
+
+ - uses: pnpm/action-setup@v2
+ with:
+ version: 8
- - name: Install Python
- uses: actions/setup-python@v4
+ - uses: actions/setup-node@v4
with:
- python-version: 3.9
+ node-version: '18'
- - name: Install dependencies
- run: |
- sudo apt-get update -y
- sudo apt-get install -y ca-certificates doxygen
- curl -sSL https://install.python-poetry.org | python3 -
- # install poetry
- yes | sudo python3 -m pip install poetry --quiet
- poetry env use python3
- # install dependencies with poetry
- poetry install --with=docs,spark
- working-directory: ./pyspark
+ - name: Checkout Website
+ uses: actions/checkout@v4
+ with:
+ repository: apache/incubator-graphar-website
+ path: website
+ ref: main
- - name: Generate Doc
+ - name: Sync docs to website
+ working-directory: ${{ github.workspace }}
run: |
- export JAVA_HOME=${JAVA_HOME_11_X64}
- make html
- working-directory: ./docs
+ # Note: DO NOT omit the trailing slash in the source directory
+ rsync -av docs/ website/docs
+
+ - name: Install Dependencies
+ working-directory: website
+ run: pnpm install --frozen-lockfile
- - name: Commit Doc
- # TODO: open this when apache infrastructure is ready
- # if: ${{ github.ref == 'refs/heads/main' && github.event_name == 'push' && github.repository == 'apache/incubator-graphar' }}
- if: false
- run: |
- git config user.email github-actions@github.com
- git config user.name github-actions
- git branch -D gh-pages || true
- git checkout --orphan gh-pages
- shopt -s extglob
- rm -rf !(docs)
- rm -rf .github .clang-format .gitattributes .gitmodules .gitignore
- mv docs/_build/html/* ./
- mv docs/_build/html/.nojekyll ./
- rm -rf docs
- git add -A
- git commit -m "Generate latest docs on CI, from commit ${{ github.sha }}."
- git push -f origin gh-pages
+ - name: Build
+ working-directory: website
+ run: pnpm build
+
+# TODO: enable markdownlint & typos
diff --git a/.github/workflows/license.yml b/.github/workflows/license.yml
index ab2dea7..deb2719 100644
--- a/.github/workflows/license.yml
+++ b/.github/workflows/license.yml
@@ -30,7 +30,7 @@ concurrency:
cancel-in-progress: true
jobs:
- build:
+ check:
runs-on: ubuntu-latest
steps:
diff --git a/docs/.gitignore b/docs/.gitignore
deleted file mode 100644
index e35d885..0000000
--- a/docs/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-_build
diff --git a/docs/.nojekyll b/docs/.nojekyll
deleted file mode 100644
index e69de29..0000000
diff --git a/docs/Makefile b/docs/Makefile
deleted file mode 100644
index 8fc7587..0000000
--- a/docs/Makefile
+++ /dev/null
@@ -1,62 +0,0 @@
-# Minimal makefile for Sphinx documentation
-#
-
-# You can set these variables from the command line, and also
-# from the environment for the first two.
-# Do not fail the build if there are warnings
-SPHINXOPTS = -j8
-SPHINXBUILD ?= sphinx.cmd.build
-SOURCEDIR = .
-BUILDDIR = _build
-
-# Internal variables.
-ALLSPHINXOPTS = -b html -d ../docs/$(BUILDDIR)/doctrees $(SPHINXOPTS)
-DOXYGEN = doxygen
-ROOTDIR = ..
-
-# Put it first so that "make" without argument is like "make help".
-help:
- @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
-
-.PHONY: help Makefile doxygen
-
-# Catch-all target: route all unknown targets to Sphinx using the new
-# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
-%: Makefile
- @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
-
-.PHONY: clean
-clean:
- rm -rf $(BUILDDIR)/*
-
-.PHONY: cpp-apidoc
-cpp-apidoc:
- cd $(ROOTDIR)/cpp/apidoc && \
- $(DOXYGEN) Doxyfile
-
-.PHONY: spark-apidoc
-spark-apidoc:
- cd $(ROOTDIR)/spark && \
- mvn --no-transfer-progress clean install -DskipTests -Dspotless.check.skip=true && \
- mvn --no-transfer-progress scala:doc
-
-.PHONY: pyspark-apidoc
-pyspark-apidoc:
- cd ../pyspark && poetry run sphinx-apidoc -o ../docs/pyspark/api graphar_pyspark/
-
-.PHONY: html
-html: cpp-apidoc spark-apidoc pyspark-apidoc
- echo "Generate docs..."
- cd $(ROOTDIR)/pyspark && poetry run python -m $(SPHINXBUILD) $(ALLSPHINXOPTS) $(ROOTDIR)/docs $(ROOTDIR)/docs/$(BUILDDIR)/html
- rm -fr $(BUILDDIR)/html/spark/reference
- cp -fr $(ROOTDIR)/spark/graphar/target/site/scaladocs $(BUILDDIR)/html/spark/reference/
- cd $(ROOTDIR)/java && \
- mvn --no-transfer-progress -P javadoc javadoc:aggregate \
- -Dmaven.antrun.skip=true \
- -DskipTests \
- -Djavadoc.output.directory=$(ROOTDIR)/docs/$(BUILDDIR)/html/java/ \
- -Djavadoc.output.destDir=reference \
- --quiet
- @echo
- @echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
-
diff --git a/docs/README.md b/docs/README.md
new file mode 100644
index 0000000..b397676
--- /dev/null
+++ b/docs/README.md
@@ -0,0 +1,10 @@
+# Apache GraphAr (incubating) Documentation
+
+This directory contains the source files for building the main
+project documentation for Apache GraphAr (incubating). This include
+the GraphAr Format specification and libraries.
+
+Instructions for building the documentation site are found in the
+[website README][1].
+
+[1]: https://github.com/apache/incubator-graphar-website/blob/main/README.md
diff --git a/docs/_static/css/brands.min.css b/docs/_static/css/brands.min.css
deleted file mode 100644
index 714509e..0000000
--- a/docs/_static/css/brands.min.css
+++ /dev/null
@@ -1,6 +0,0 @@
-/*!
- * Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com
- * License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License)
- * Copyright 2022 Fonticons, Inc.
- */
-:host,:root{--fa-font-brands:normal 400 1em/1 "Font Awesome 6 Brands"}@font-face{font-family:"Font Awesome 6 Brands";font-style:normal;font-weight:400;font-display:block;src:url(../webfonts/fa-brands-400.woff2) format("woff2"),url(../webfonts/fa-brands-400.ttf) format("truetype")}.fa-brands,.fab{font-family:"Font Awesome 6 Brands";font-weight:400}.fa-42-group:before,.fa-innosoft:before{content:"\e080"}.fa-500px:before{content:"\f26e"}.fa-accessible-icon:before{content:"\f368"}.fa-accusof [...]
\ No newline at end of file
diff --git a/docs/_static/css/custom.css b/docs/_static/css/custom.css
deleted file mode 100644
index 398234f..0000000
--- a/docs/_static/css/custom.css
+++ /dev/null
@@ -1,18 +0,0 @@
-.fa.fa-2x {
- font-size: 36px;
-}
-
-/* Decrease the padding of button block in panels */
-div.card-body.card-body-less-padding {
- padding: 0.25em;
-}
-
-/* Disable theme toggle */
-div.theme-toggle-container.theme-toggle-content {
- display: none;
-}
-
-table.docutils {
- width: 100%;
- max-width: none;
-}
diff --git a/docs/_static/css/panels.css b/docs/_static/css/panels.css
deleted file mode 100644
index 5cf775f..0000000
--- a/docs/_static/css/panels.css
+++ /dev/null
@@ -1,147 +0,0 @@
-/* Referred and derived from https://github.com/flyteorg/furo/blob/main/src/furo/assets/styles/flyte.css */
-
-h1,
-h2,
-h3,
-h4,
-h5,
-h6 {
- font-weight: bold;
-}
-
-.caption-text {
- font-size: 15px;
- /* color: #696969; */
- color: #333333;
-}
-
-div.sphinx-bs .card {
- flex-direction: row;
-}
-
-/* sphinx-panels custom styles */
-div.sphinx-bs .card-header {
- border-bottom: none;
- background-color: var(--color-background-primary);
- display: flex;
- align-items: center;
- justify-content: left;
- width: 28%;
- float: left;
-}
-
-.sphinx-bs .card-header:first-child {
- border-radius: calc(0.25rem - 1px) 0 0 calc(0.25rem - 1px);
-}
-
-div.sphinx-bs .card-header .sphinx-bs.btn,
-div.sphinx-bs .card-body .sphinx-bs.btn,
-div.sphinx-bs .card-header p.card-text {
- font-size: 1rem;
- text-decoration: none;
- word-spacing: 2.5px;
- color: var(--color-sidebar-link-text);
-}
-
-div.sphinx-bs .card-header p.card-text a {
- text-align: left;
-}
-
-.sphinx-bs.btn:focus {
- box-shadow: none;
-}
-
-div.sphinx-bs .card-body {
- width: 72%;
- float: left;
-}
-
-.sphinx-bs .card-body .fa {
- color: var(--color-sidebar-link-text);
-}
-
-.sphinx-bs .card-body:hover .fa {
- color: var(--color-link--hover);
-}
-
-.sphinx-bs .card-body .fa {
- font-size: 2rem;
-}
-
-div.sphinx-bs .card:hover {
- box-shadow: none !important;
- border-color: #cca9ff;
-}
-
-div.sphinx-bs .card:hover .card-header {
- background-color: #f2e9ff;
- color: #fff;
-}
-
-body[data-theme="dark"] div.sphinx-bs .card:hover {
- border-color: #2a144a;
-}
-
-body[data-theme="dark"] div.sphinx-bs .card:hover .card-header {
- background-color: #2a144a;
- color: #fff;
-}
-
-/* make sure hover style is consistent if user prefers dark theme at OS level */
-@media (prefers-color-scheme: dark) {
- body:not([data-theme="light"]) div.sphinx-bs .card:hover {
- border-color: #2a144a;
- }
- body:not([data-theme="light"]) div.sphinx-bs .card:hover .card-header {
- background-color: #2a144a;
- color: #fff;
- }
-}
-
-div.sphinx-bs .card:hover .sphinx-bs.btn {
- color: var(--color-link);
-}
-
-div.sphinx-bs .card:hover .card-body .sphinx-bs.btn {
- color: var(--color-link--hover);
-}
-
-.getting-started-panels div.sphinx-bs .sphinx-bs.btn:hover {
- border-color: var(--color-link);
- background-color: #9d68e4;
- color: #ffffff;
-}
-
-div.sphinx-bs .card {
- background-color: var(--color-background-secondary);
- border: 1px solid var(--color-background-border);
-}
-
-.center-card-content p {
- margin: auto !important;
-}
-
-.sphinx-tabs {
- padding-top: 10px;
-}
-
-.sphinx-tabs-tab {
- color: var(--color-link);
-}
-
-/* sphinx tabs */
-.sphinx-tabs-tab[aria-selected="true"] {
- background-color: var(--color-background-secondary);
- border: 1px solid var(--color-background-border);
- border-bottom: 1px solid var(--color-background-secondary);
-}
-
-.sphinx-tabs-panel {
- border: 1px solid var(--color-background-border);
- background: var(--color-background-secondary);
- border-top: 0;
-}
-
-[role="tablist"] {
- border-bottom: 1px solid var(--color-background-border);
-}
diff --git a/docs/_static/css/v4-shims.min.css b/docs/_static/css/v4-shims.min.css
deleted file mode 100644
index f742adc..0000000
--- a/docs/_static/css/v4-shims.min.css
+++ /dev/null
@@ -1,6 +0,0 @@
-/*!
- * Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com
- * License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License)
- * Copyright 2022 Fonticons, Inc.
- */
-.fa.fa-glass:before{content:"\f000"}.fa.fa-envelope-o{font-family:"Font Awesome 6 Free";font-weight:400}.fa.fa-envelope-o:before{content:"\f0e0"}.fa.fa-star-o{font-family:"Font Awesome 6 Free";font-weight:400}.fa.fa-star-o:before{content:"\f005"}.fa.fa-close:before,.fa.fa-remove:before{content:"\f00d"}.fa.fa-gear:before{content:"\f013"}.fa.fa-trash-o{font-family:"Font Awesome 6 Free";font-weight:400}.fa.fa-trash-o:before{content:"\f2ed"}.fa.fa-home:before{content:"\f015"}.fa.fa-file-o{fo [...]
\ No newline at end of file
diff --git a/docs/_static/webfonts/fa-brands-400.ttf b/docs/_static/webfonts/fa-brands-400.ttf
deleted file mode 100644
index 430a02e..0000000
Binary files a/docs/_static/webfonts/fa-brands-400.ttf and /dev/null differ
diff --git a/docs/_static/webfonts/fa-brands-400.woff2 b/docs/_static/webfonts/fa-brands-400.woff2
deleted file mode 100644
index 4d904aa..0000000
Binary files a/docs/_static/webfonts/fa-brands-400.woff2 and /dev/null differ
diff --git a/docs/_templates/base.html b/docs/_templates/base.html
deleted file mode 100644
index 90bcbac..0000000
--- a/docs/_templates/base.html
+++ /dev/null
@@ -1,7 +0,0 @@
-{% extends "!base.html" %}
-
-{% block body %}
- <script>
- document.body.dataset.theme = "light";
- </script>
-{% endblock %}
diff --git a/docs/_templates/footer.html b/docs/_templates/footer.html
deleted file mode 100644
index 0b6d7d8..0000000
--- a/docs/_templates/footer.html
+++ /dev/null
@@ -1,75 +0,0 @@
-<footer>
- {%- if (theme_prev_next_buttons_location == 'bottom' or theme_prev_next_buttons_location == 'both') and (next or prev) %}
- <div class="rst-footer-buttons" role="navigation" aria-label="{{ _('Footer') }}">
- {%- if prev %}
- <a href="{{ prev.link|e }}" class="btn btn-neutral float-left" title="{{ prev.title|striptags|e }}" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> {{ _('Previous') }}</a>
- {%- endif %}
- {%- if next %}
- <a href="{{ next.link|e }}" class="btn btn-neutral float-right" title="{{ next.title|striptags|e }}" accesskey="n" rel="next">{{ _('Next') }} <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
- {%- endif %}
- </div>
- {%- endif %}
-
- <hr/>
-
- <div role="contentinfo">
-
- {%- block contentinfo %}
-
- <p>
- Vineyard is a <a href="https://www.cncf.io/sandbox-projects/">CNCF sandbox project</a>.
- </p>
- <img src="https://v6d.io/_static/cncf-color.svg" alt="CLOUD NATIVE COMPUTING FOUNDATION" width="40%"/>
- <p>
- The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation,
- please see our <a href="https://www.linuxfoundation.org/trademark-usage/">Trademark Usage page</a>.
- </p>
-
- <hr/>
-
- {%- if show_copyright %}
- {%- if hasdoc('copyright') %}
- {%- trans path=pathto('copyright'), copyright=copyright|e %}© <a href="{{ path }}">Copyright</a> {{ copyright }}.{% endtrans %}
- {%- else %}
- {%- trans copyright=copyright|e %}© Copyright {{ copyright }}.{% endtrans %}
- {%- endif %}
- {%- endif %}
-
- <p>
-
- {%- if build_id and build_url %}
- <span class="build">
- {#- Translators: Build is a noun, not a verb -#}
- {%- trans %}Build{% endtrans -%}
- <a href="{{ build_url }}">{{ build_id }}</a>.
- </span>
- {%- elif commit %}
- <span class="commit">
- {#- Translators: the phrase "revision" comes from Git, referring to a commit #}
- {%- trans %}Revision{% endtrans %} <code>{{ commit }}</code>.
- </span>
- {%- endif %}
- {%- if last_updated %}
- <span class="lastupdated">
- {%- trans last_updated=last_updated|e %}Last updated on {{ last_updated }}.{% endtrans %}
- </span>
- {%- endif -%}
-
- </p>
- {%- endblock %}
- </div>
-
- {% if show_sphinx %}
- {%- set sphinx_web = '<a href="https://www.sphinx-doc.org/">Sphinx</a>' %}
- {%- set readthedocs_web = '<a href="https://readthedocs.org">Read the Docs</a>' %}
- {#- Translators: the variable "sphinx_web" is a link to the Sphinx project documentation with the text "Sphinx" #}
- {%- trans sphinx_web=sphinx_web, readthedocs_web=readthedocs_web %}Built with {{ sphinx_web }} using a{% endtrans %}
- {#- Translators: "theme" refers to a theme for Sphinx, which alters the appearance of the generated documenation #}
- <a href="https://github.com/readthedocs/sphinx_rtd_theme">{% trans %}theme{% endtrans %}</a>
- {#- Translators: this is always used as "provided by Read the Docs", and should not imply Read the Docs is an author of the generated documentation. #}
- {%- trans %}provided by {{ readthedocs_web }}{% endtrans %}.
- {% endif %}
-
- {%- block extrafooter %} {% endblock %}
-
- </footer>
diff --git a/docs/_templates/layout.html b/docs/_templates/layout.html
deleted file mode 100644
index 889bb9a..0000000
--- a/docs/_templates/layout.html
+++ /dev/null
@@ -1,17 +0,0 @@
-{% extends "!layout.html" %}
-
-{%- block extrahead %}
-
-{{ super() }}
-
-<!-- Global site tag (gtag.js) - Google Analytics -->
-<script async src="https://www.googletagmanager.com/gtag/js?id=G-6CYXWKLJB8"></script>
-<script>
- window.dataLayer = window.dataLayer || [];
- function gtag(){dataLayer.push(arguments);}
- gtag('js', new Date());
-
- gtag('config', 'G-6CYXWKLJB8');
-</script>
-
-{% endblock %}
diff --git a/docs/conf.py b/docs/conf.py
deleted file mode 100644
index be7688a..0000000
--- a/docs/conf.py
+++ /dev/null
@@ -1,119 +0,0 @@
-# Configuration file for the Sphinx documentation builder.
-#
-# This file only contains a selection of the most common options. For a full
-# list see the documentation:
-# https://www.sphinx-doc.org/en/master/usage/configuration.html
-
-# -- Path setup --------------------------------------------------------------
-
-# If extensions (or modules to document with autodoc) are in another directory,
-# add these directories to sys.path here. If the directory is relative to the
-# documentation root, use os.path.abspath to make it absolute, like shown here.
-#
-import os
-# sys.path.insert(0, os.path.abspath('.'))
-
-
-# -- Project information -----------------------------------------------------
-
-project = 'GraphAr'
-copyright = '2022, The GraphAr Authors'
-author = 'The GraphAr Authors'
-
-# The full version, including alpha/beta/rc tags
-release = '0.1-alpha'
-
-
-# -- General configuration ---------------------------------------------------
-
-# Add any Sphinx extension module names here, as strings. They can be
-# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
-# ones.
-extensions = [
- 'breathe',
- 'nbsphinx',
- 'sphinx.ext.autodoc',
- 'sphinx.ext.autosummary',
- 'sphinx.ext.mathjax',
- 'sphinx.ext.napoleon',
- 'sphinx.ext.viewcode',
- "sphinx_copybutton",
- 'sphinx_panels',
- 'sphinxemoji.sphinxemoji',
- "sphinxext.opengraph",
-]
-
-# breathe
-breathe_projects = {
- 'GraphAr': os.path.abspath('../cpp/apidoc/xml'),
-}
-breathe_default_project = 'GraphAr'
-breathe_debug_trace_directives = True
-breathe_debug_trace_doxygen_ids = True
-breathe_debug_trace_qualification = True
-
-# jupyter notebooks
-nbsphinx_execute = 'never'
-
-source_suffix = ['.rst', '.md']
-
-# Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
-
-# The root document.
-root_doc = 'index'
-
-# List of patterns, relative to source directory, that match files and
-# directories to ignore when looking for source files.
-# This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
-
-
-# -- Options for HTML output -------------------------------------------------
-
-# The theme to use for HTML and HTML Help pages. See the documentation for
-# a list of builtin themes.
-#
-html_theme = "furo"
-
-html_theme_options = {
- "sidebar_hide_name": False, # we use the logo
- "navigation_with_keys": True,
- "source_repository": "https://github.com/apache/incubator-graphar/",
- "source_branch": "main",
- "source_directory": "docs/",
- "footer_icons": [
- {
- "name": "GitHub",
- "url": "https://github.com/apache/incubator-graphar",
- "html": "",
- "class": "fa fa-solid fa-github fa-2x",
- },
- ],
-}
-
-# Add any paths that contain custom static files (such as style sheets) here,
-# relative to this directory. They are copied after the builtin static files,
-# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = [
- 'images/',
- "_static/",
-]
-
-html_css_files = [
- "css/brands.min.css", # font-awesome
- "css/v4-shims.min.css", # font-awesome
- "css/custom.css",
- "css/panels.css",
-]
-
-html_extra_path = [
- './.nojekyll',
-]
-
-html_title = "GraphAr"
-html_favicon = "images/graphar.ico"
-
-html_show_copyright= True
-html_show_sphinx = False
-html_last_updated = True
diff --git a/docs/cpp/examples/bgl.rst b/docs/cpp/examples/bgl.rst
deleted file mode 100644
index 8c0cfe5..0000000
--- a/docs/cpp/examples/bgl.rst
+++ /dev/null
@@ -1,93 +0,0 @@
-Co-Work with BGL
-============================
-
-The `Boost Graph Library (BGL) <https://cs.brown.edu/~jwicks/boost/libs/graph/doc/>`_ is the first C++ library to apply the principles of generic programming to the construction of the advanced data structures and algorithms used in graph computations. The BGL graph interface and graph components are generic in the same sense as the Standard Template Library (STL). And it provides some built-in algorithms which cover a core set of algorithm patterns and a larger set of graph algorithms.
-
-We take calculating CC as an example, to demonstrate how BGL works with GraphAr. A weakly connected component is a maximal subgraph of a graph such that for every pair of vertices in it, there is an undirected path connecting them. And the CC algorithm is to identify all such components in a graph. Learn more about `the CC algorithm <https://en.wikipedia.org/wiki/Connected_component>`_.
-
-The source code of CC based on BGL can be found at `bgl_example.cc`_. In this program, the graph information file is first read to get the metadata:
-
-.. code:: C++
-
- std::string path = ... // the path of the graph information file
- auto graph_info = graphar::GraphInfo::Load(path).value();
-
-And then, the vertex collection and the edge collection are established as the handles to access the graph data:
-
-.. code:: C++
-
- auto maybe_vertices = graphar::VerticesCollection::Make(graph_info, "person");
- auto vertices = maybe_vertices.value();
- auto maybe_edges = graphar::EdgesCollection::Make(graph_info, "person", "knows", "person", graphar::AdjListType::ordered_by_source);
- auto edges = maybe_edges.value();
-
-Next, we construct the in-memory graph data structure for BGL by traversing the vertices and edges via GraphAr's high-level reading interface (the vertex iterator and the edge iterator):
-
-.. code:: C++
-
- // define the Graph type in BGL
- typedef boost::adjacency_list<boost::vecS, // use vector to store edges
- boost::vecS, // use vector to store vertices
- boost::undirectedS, // undirected
- boost::property<boost::vertex_name_t, int64_t>, // vertex property
- boost::no_property> Graph; // no edge property
- // descriptors for vertex in BGL
- typedef typename boost::graph_traits<Graph>::vertex_descriptor Vertex;
-
- // declare a graph object with (num_vertices) vertices and an edge iterator
- std::vector<std::pair<graphar::IdType, graphar::IdType>> edges_array;
- auto it_begin = edges->begin(), it_end = edges->end();
- for (auto it = it_begin; it != it_end; ++it)
- edges_array.push_back(std::make_pair(it.source(), it.destination()));
- Graph g(edges_array.begin(), edges_array.end(), num_vertices);
-
- // define the internal vertex property "id"
- boost::property_map<Graph, boost::vertex_name_t>::type id = get(boost::vertex_name_t(), g);
- auto v_it_begin = vertices->begin(), v_it_end = vertices->end();
- for (auto it = v_it_begin; it != v_it_end; ++it) {
- auto vertex = *it;
- boost::put(id, vertex.id(), vertex.property<int64_t>("id").value());
- }
-
-After that, an internal CC algorithm provided by BGL is called:
-
-.. code:: C++
-
- // define the external vertex property "component"
- std::vector<int> component(num_vertices);
- // call algorithm: cc
- int cc_num = boost::connected_components(g, &component[0]);
- std::cout << "Total number of components: " << cc_num << std::endl;
-
-Finally, we could use a **VerticesBuilder** of GraphAr to write the results to new generated GAR files:
-
-.. code:: C++
-
- // construct a new property group
- graphar::Property cc = {"cc", graphar::int32(), false};
- std::vector<graphar::Property> property_vector = {cc};
- auto group = graphar::CreatePropertyGroup(property_vector, graphar::FileType::PARQUET);
-
- // construct the new vertex info
- std::string vertex_label = "cc_result", vertex_prefix = "result/";
- int chunk_size = 100;
- auto new_info = graphar::CreateVertexInfo(vertex_label, chunk_size, {group}, vertex_prefix);
-
- // access the vertices via the index map and vertex iterator of BGL
- typedef boost::property_map<Graph, boost::vertex_index_t>::type IndexMap;
- IndexMap index = boost::get(boost::vertex_index, g);
- typedef boost::graph_traits<Graph>::vertex_iterator vertex_iter;
- std::pair<vertex_iter, vertex_iter> vp;
-
- // dump the results through the VerticesBuilder
- graphar::builder::VerticesBuilder builder(new_info, "/tmp/");
- for (vp = boost::vertices(g); vp.first!= vp.second; ++vp.first) {
- Vertex v = *vp.first;
- graphar::builder::Vertex vertex(index[v]);
- vertex.AddProperty(cc.name, component[index[v]]);
- builder.AddVertex(vertex);
- }
- builder.Dump();
-
-
-.. _bgl_example.cc: https://github.com/apache/incubator-graphar/blob/main/cpp/examples/bgl_example.cc
diff --git a/docs/cpp/examples/graphscope.rst b/docs/cpp/examples/graphscope.rst
deleted file mode 100644
index 43f195c..0000000
--- a/docs/cpp/examples/graphscope.rst
+++ /dev/null
@@ -1,32 +0,0 @@
-Integrate into GraphScope
-============================
-
-`GraphScope <https://graphscope.io/>`_ is a unified distributed graph computing platform that provides a one-stop environment for performing diverse graph operations on a cluster through a user-friendly Python interface. As an important application case of GraphAr, we have integrated it into GraphScope.
-
-GraphScope works on a graph G fragmented via a partition strategy picked by the user and each worker maintains a fragment of G. Given a query, it posts the same query to all the workers and computes following the BSP (Bulk Synchronous Parallel) model. More specifically, each worker first executes processing against its local fragment, to compute partial answers in parallel. And then each worker may exchange partial results with other processors via synchronous message passing.
-
-To integrate GraphAr into GraphScope, we implemented *ArrowFragmentBuilder* and *ArrowFragmentWriter*. *ArrowFragmentBuilder* establishes the fragments for workers of GraphScope through reading GAR files in parallel. Conversely, *ArrowFragmentWriter* can take the GraphScope fragments and save them as GAR files. If you're interested in knowing more about the implementation, please refer to the `source code <https://github.com/v6d-io/v6d/commit/0eda2067e45fbb4ac46892398af0edc84fe1c27b>`_.
-
-
-Performance Report
-------------------------
-
-Parameter settings
-``````````````````
-The time performance of *ArrowFragmentBuilder* and *ArrowFragmentWriter* in GraphScope is heavily dependent on the partitioning of the graph into GAR files, that is, the *vertex chunk size* and *edge chunk size*, which are specified in the vertex information file and in the edge information file, respectively. See `GraphAr File Format <../user-guide/file-format.html>`_ to understand the chunk size definitions in GAR.
-
-Generally speaking, fewer chunks are created if the file size is large. On small graphs, this can be disadvantageous as it reduces the degree of parallelism, prolonging disk I/O time. On the other hand, having too many small files increases the overhead associated with the file system and the file parser.
-
-We have conducted micro benchmarks to compare the time performance for reading/writing GAR files by *ArrowFragmentBuilder*/*ArrowFragmentWriter*, across different *vertex chunk size* and *edge chunk size* configurations. The settings we recommend for *vertex chunk size* and *edge chunk size* are **2^18** and **2^22**, respectively, which lead to efficient performance in most cases. These settings can be used as the reference values when integrating GraphAr into other systems besides GraphScope.
-
-Time performance results
-````````````````````````
-Here we report the performance results of *ArrowFragmentBuilder*, and compare it with loading the same graph through the default loading strategy of GraphScope (through reading the csv files in parallel) . The execution time reported below includes loading the graph data from the disk into memory, as well as building GraphScope fragments from such data. The experiments are conducted on a cluster of 4 AliCloud ecs.r6.6xlarge instances (24vCPU, 192GB memory), and using `com-friendster <htt [...]
-
-+----------------+---------+-----------------+-----------------+
-| Dataset | Workers | Default Loading | GraphAr Loading |
-+================+=========+=================+=================+
-| com-friendster | 4 | 282s | 54s |
-+----------------+---------+-----------------+-----------------+
-| ldbc-snb-30 | 4 | 196s | 40s |
-+----------------+---------+-----------------+-----------------+
diff --git a/docs/cpp/examples/index.rst b/docs/cpp/examples/index.rst
deleted file mode 100644
index 0d9abdd..0000000
--- a/docs/cpp/examples/index.rst
+++ /dev/null
@@ -1,10 +0,0 @@
-Examples
----------
-
-.. toctree::
- :maxdepth: 2
-
- bgl
- graphscope
- out-of-core
- snap-to-graphar
diff --git a/docs/cpp/examples/out-of-core.rst b/docs/cpp/examples/out-of-core.rst
deleted file mode 100644
index 066afc2..0000000
--- a/docs/cpp/examples/out-of-core.rst
+++ /dev/null
@@ -1,122 +0,0 @@
-Out-of-core Graph Algorithms
-============================
-
-An important application case of GraphAr is to serve out-of-core graph processing scenarios. With the graph data saved as GAR files in the disk, GraphAr provides a set of reading interfaces to allow to load part of graph data into memory when needed, to conduct analytics. While it is more convenient and efficient to store the entirety of the graph in memory (as is done in BGL), out-of-core graph processing makes it possible to complete analytics on the large-scale graphs using limited me [...]
-
-The are some out-of-core graph analytic algorithms that have been implemented based on GraphAr, include:
-
-- PageRank (PR)
-- Connected Components (CC)
-- Breadth First Search (BFS)
-
-These algorithms represent for different compute patterns and are usually building blocks for constructing other graph algorithms.
-
-PageRank
-------------------------
-
-`PageRank (PR) <https://en.wikipedia.org/wiki/PageRank>`_ is an algorithm used by Google Search to rank web pages in their search engine results. The source code of PageRank based on GraphAr located at `pagerank_example.cc`_, and the explanations can be found in the `Getting Started <../user-guide/getting-started.html#a-pagerank-example>`_ page.
-
-Connected Components
-------------------------
-
-A weakly connected component is a maximal subgraph of a graph such that for every pair of vertices in it, there is an undirected path connecting them. And `Connected Components (CC) <https://en.wikipedia.org/wiki/Connected_component>`_ is an algorithm to identify all weakly connected components in a graph. `CC based on BGL <bgl.html>`_ is provided in GraphAr, also, we implement out-of-core algorithms for this application.
-
-A typical method for calculating CC is label propagation. In this algorithm, each vertex is attached with a property which represents its component label, being its own vertex id initially. In the subsequent supersteps (i.e., iterations), a vertex will update its label if it receives a smaller id and then it propagates this id to all its neighbors.
-
-This algorithm can be implemented based on streaming the edges via GraphAr's reading interface. That is to say, the edges are accessed and processed chunk by chunk, instead of being loaded into memory at once (as in the BGL example).
-
-.. code:: C++
-
- // construct the edge collection in GraphAr
- auto edges = ...
- auto it_begin = edges->begin(), it_end = edges->end();
-
- // initialize for all vertices
- std::vector<graphar::IdType> component(num_vertices);
- for (graphar::IdType i = 0; i < num_vertices; i++)
- component[i] = i;
-
- // stream all edges for each iteration
- for (int iter = 0; ; iter++) {
- bool flag = false;
- for (auto it = it_begin; it != it_end; ++it) {
- graphar::IdType src = it.source(), dst = it.destination();
- // update
- if (component[src] < component[dst]) {
- component[dst] = component[src];
- flag = true;
- } else if (component[src] > component[dst]) {
- component[src] = component[dst];
- flag = true;
- }
- }
- // check if it should terminate
- if (!flag) break;
- }
-
-The file `cc_stream_example.cc`_ located inside the source tree contains the complete implementation for this algorithm. Also, we can only process active vertices (the vertices which are updated in the last iteration) and the corresponding edges for each iteration, since an inactive vertex does not need to update its neighbors. Please refer to `cc_push_example.cc`_ for the complete code.
-
-.. tip::
-
- In this example, two kinds of edges are used. The **ordered_by_source** edges are used to access all outgoing edges of an active vertex, and **ordered_by_dest** edges are used to access the incoming edges. In this way, all the neighbors of an active vertex can be accessed and processed.
-
- Although GraphAr supports to get the outgoing (incoming) edges of a single vertex for all adjList types, it is most efficient when the type is **ordered_by_source** (**ordered_by_dest**) since it can avoid to read redundant data.
-
-Breadth First Search
-------------------------
-
-`Breadth First Search (BFS) <https://en.wikipedia.org/wiki/Breadth-first_search>`_ is a traversing algorithm that starts from a selected vertex (the root) and traverse the graph layerwise thus exploring the neighbor vertices (vertices which are directly connected to the root), and then it moves towards the next-level neighbor vertices.
-
-An out-of-core BFS algorithm could be implemented based on streaming the graph data via GraphAr. For each vertex, a property named *distance* is created and initialized to represent the distance from the root to this vertex. As with the standard BFS algorithms in other graph processing systems, for the iteration/superstep *i* (starting from 0), the active vertices contain all vertices reachable from the root in *i* hops (i.e., *distance[v]= i*). At the beginning of the algorithm, only th [...]
-
-.. code:: C++
-
- // construct the edge collection in GraphAr
- auto edges = ...
- auto it_begin = edges->begin(), it_end = edges->end();
-
- // initialize for all vertices
- graphar::IdType root = 0; // the BFS root
- std::vector<int32_t> distance(num_vertices);
- for (graphar::IdType i = 0; i < num_vertices; i++)
- distance[i] = (i == root ? 0 : -1);
-
- // stream all edges for each iteration
- for (int iter = 0; ; iter++) {
- graphar::IdType count = 0;
- for (auto it = it_begin; it != it_end; ++it) {
- graphar::IdType src = it.source(), dst = it.destination();
- // update
- if (distance[src] == iter && distance[dst] == -1) {
- distance[dst] = distance[src] + 1;
- count++;
- }
- }
- // check if it should terminate
- if (count == 0) break;
- }
-
-The above algorithm is implemented based on streaming all edges for each iteration, the source code can be found at `bfs_stream_example.cc`_.
-
-Meanwhile, BFS could be implemented in a **push**-style which only traverses the edges that from active vertices for each iteration, which is typically more efficient on real-world graphs. This implementation can be found at `bfs_push_example.cc`_. Similarly, we provide a BFS implementation in a **pull**-style which only traverses the edges that lead to non-visited vertices (i.e., the vertices that have not been traversed), as shown in `bfs_pull_example.cc`_.
-
-.. tip::
-
- In common cases of graph processing, the **push**-style is more efficient when the set of active vertices is very sparse, while the **pull**-style fits when it is dense.
-
-In some cases, it is required to record the path of BFS, that is, to maintain each vertex's predecessor (also called *father*) in the traversing tree rather than only recording the distance. The implementation of BFS with recording fathers can be found at `bfs_father_example.cc`_.
-
-
-.. _pagerank_example.cc: https://github.com/apache/incubator-graphar/blob/main/cpp/examples/pagerank_example.cc
-
-.. _cc_stream_example.cc: https://github.com/apache/incubator-graphar/blob/main/cpp/examples/cc_stream_example.cc
-
-.. _cc_push_example.cc: https://github.com/apache/incubator-graphar/blob/main/cpp/examples/cc_push_example.cc
-
-.. _bfs_stream_example.cc: https://github.com/apache/incubator-graphar/blob/main/cpp/examples/bfs_stream_example.cc
-
-.. _bfs_push_example.cc: https://github.com/apache/incubator-graphar/blob/main/cpp/examples/bfs_push_example.cc
-
-.. _bfs_pull_example.cc: https://github.com/apache/incubator-graphar/blob/main/cpp/examples/bfs_pull_example.cc
-
-.. _bfs_father_example.cc: https://github.com/apache/incubator-graphar/blob/main/cpp/examples/bfs_father_example.cc
diff --git a/docs/cpp/examples/snap-to-graphar.rst b/docs/cpp/examples/snap-to-graphar.rst
deleted file mode 100644
index 852bc4d..0000000
--- a/docs/cpp/examples/snap-to-graphar.rst
+++ /dev/null
@@ -1,81 +0,0 @@
-Convert SNAP Datasets to GraphAr Format
-=======================================
-
-`SNAP <https://snap.stanford.edu/data/>`_ (Stanford Network Analysis Project) is a general-purpose network analysis and graph mining library. It provides a variety of datasets for research and development. In this section, we will show how to convert the SNAP datasets to GraphAr format, showcasing the process with the `ego-Facebook <https://snap.stanford.edu/data/ego-Facebook.html>`_ graph as a case study. The conversion leverages GraphInfo constructors and the high-level writer function [...]
-
-
-Prepare the SNAP Dataset
-------------------------
-
-Before converting, download the ego-Facebook dataset from the SNAP website. The dataset is a text file with each line representing an edge in the graph.
-
-.. code:: bash
-
- cd /path/to/your/dataset
- wget https://snap.stanford.edu/data/facebook_combined.txt.gz
- gunzip facebook_combined.txt.gz
-
-
-Convert the SNAP Dataset to GraphAr Format
-------------------------------------------
-
-The initial phase involves constructing VertexInfo, EdgeInfo, and GraphInfo objects, which are subsequently serialized into YAML files.
-For instance, the code snippet below illustrates the creation and storage of the vertex information file.
-
-.. code:: C++
-
- auto version = graphar::InfoVersion::Parse("gar/v1").value();
-
- // meta info
- std::string vertex_label = "node", vertex_prefix = "vertex/node/";
-
- // create vertex info
- auto vertex_info = graphar::CreateVertexInfo(
- vertex_label, VERTEX_CHUNK_SIZE, {}, vertex_prefix, version);
-
- // save & dump vertex info
- ASSERT(!vertex_info->Dump().has_error());
- ASSERT(vertex_info->Save(save_path + "node.vertex.yml").ok());
-
- // create and save edge info file ...
- auto edge_info = ...
- ASSERT(!edge_info->Dump().has_error());
- ASSERT(edge_info->Save(save_path + "node_links_node.edge.yml").ok());
-
- // create and save graph info file ...
- auto graph_info = ...
- ASSERT(!graph_info->Dump().has_error());
- ASSERT(graph_info->Save(save_path + graph_name + ".graph.yml").ok());
-
-
-Subsequently, we employ the high-level vertex and edge builders provided by the GraphAr C++ library to generate payload data files with vertex and edge data.
-The code snippet that follows demonstrates the generation and preservation of the edge data file.
-
-.. code:: C++
-
- // construct edges builder
- auto e_builder = graphar::builder::EdgesBuilder::Make(
- edge_info, save_path, ADJLIST_TYPE, VERTEX_COUNT)
- .value();
- // read edge data from file
- std::ifstream file(DATA_PATH);
- std::string line;
- while (std::getline(file, line)) {
- std::istringstream iss(line);
- // skip comments
- if (line[0] == '#') {
- continue;
- }
- int src, dst;
- if (!(iss >> src >> dst)) {
- break;
- }
- graphar::builder::Edge e(src, dst);
- ASSERT(e_builder->AddEdge(e).ok());
- }
-
- // dump & clear
- ASSERT(e_builder->Dump().ok());
- e_builder->Clear();
-
-For comprehensive insights into this example, please consult the accompanying `source code <https://github.com/apache/incubator-graphar/tree/main/docs/cpp/examples/snap_dataset_to_graphar.cc>`_ .
diff --git a/docs/cpp/getting-started.rst b/docs/cpp/getting-started.rst
deleted file mode 100644
index 6c0a341..0000000
--- a/docs/cpp/getting-started.rst
+++ /dev/null
@@ -1,199 +0,0 @@
-Getting Started
-============================
-
-This article is a quick guide that explains how to work with GraphAr C++. To begin with, please refer to the `Building Steps`_ to install GraphAr.
-
-
-GAR Information Files
-------------------------
-
-GAR uses a group of Yaml files to save the meta information for a graph.
-
-Graph information
-`````````````````
-The graph information file defines the most basic information of a graph includes its name, the root directory path of the data files, the vertex information and edge information files it contains, and the version of GraphAr. For example, the file "ldbc_sample.graph.yml" defines an example graph named "ldbc_sample", which includes one type of vertices ("person") and one type of edges ("person knows person").
-
-.. code:: Yaml
-
- name: ldbc_sample
- prefix: ./
- vertices:
- - person.vertex.yml
- edges:
- - person_knows_person.edge.yml
- version: gar/v1
-
-Vertex information
-``````````````````
-Each vertex information file defines a single group of vertices with the same vertex label, e.g., "person" in this case. The vertex chunk size, the relative path for vertex data files and the version of GraphAr are specified. These vertices could have some properties, which are divided into property groups. Each property group has its own file type (CSV, ORC or Parquet) and the prefix of the relative path for its data files, it also lists all properties in this group, with every property [...]
-
-The file `person.vertex.yml`_ located inside the test data contains an example of the vertex information file. In this example, the "person" vertices have two property groups. The first group contains only one property (named "id") and the second group contains three properties ("firstName", "lastName" and "gender").
-
-Edge information
-````````````````
-Each edge information file defines a single type of edges with specific labels for the source vertex, destination vertex and the edge, e.g., "person_knows_person" in this case. It defines the meta information such as the edge chunk size, the source vertex chunk size, the destination vertex chunk size, if the edges are directed or not, the relative file path for edge data files, the adjLists and the version of GraphAr. The file `person_knows_person.edge.yml`_ located inside the test data [...]
-
-In GAR format, separate data files are used to store the structure (called adjList) and the properties for edges. The adjList type can be either of **unordered_by_source**, **unordered_by_dest**, **ordered_by_source** or **ordered_by_dest** (see `Edges in GraphAr <file-format.html#edges-in-graphar>`_ for more). For a specific type of adjList, the meta information includes its file path prefix, the file type, as well as all the property groups attached.
-
-.. note::
-
- It is allowed to store different types of adjLists for a group of edges at the same time.
-
-
-
-GAR Data Files
-------------------------
-
-Property data
-`````````````
-The vertex properties are stored in vertex property chunks with the chunk size specified by the vertex information file. Different property groups correspond to individual groups of data files.
-In our example, the property group ("first name", "last name", "gender") for vertex chunk 0 of "person" vertices are stored in `./vertex/person/firstName_lastName_gender/chunk0`_.
-
-In practice of graph processing, it is common to only query a subset of columns of the properties. Thus, the column-oriented formats like Apache ORC and Apache Parquet are more efficient, which eliminate the need to read columns that are not relevant. We also provide data files in ORC and Parquet for the example graph in the `test data`_.
-
-Similar with vertices, the edge properties are stored in edge property chunks. For each vertex chunk, its associated edges (if the edge type is **ordered_by_source** or **unordered_by_source**, the associated edges are those in which the source vertex is in that chunk; otherwise, if the edge type is **ordered_by_dest** or **unordered_by_dest**, the associated edges are those in which the destination is in that chunk) are maintained in some edge chunks, with the size of each chunk not exc [...]
-
-For instance, the file `./edge/person_knows_person/ordered_by_source/creationDate/part0/chunk0`_ stores the property group "creationDate" of "person_knows_person" edges for the first edge chunk of the first vertex chunk, and the adjList type of the edges is **ordered_by_source**.
-
-AdjList data
-````````````
-The adjList in GAR describes the topology structure, i.e., the internal id of the source vertex and the destination vertex for each of a group of edges. As explained in `Edges in GraphAr <file-format.html#edges-in-graphar>`_, the edges are separated into edge chunks, and each edge chunk has its own adjList table and 0 or more property tables.
-
-For example, the file `./edge/person_knows_person/ordered_by_source/adj_list/part0/chunk0`_ saves the adjList of "person_knows_person" edges for the first edge chunk of the first vertex chunk, and the adjList type of the edges is "ordered_by_source". This adjList table consists of only two columns: one for the source and one for the destination; it can be saved in CSV, ORC, or Parquet files.
-
-.. note::
-
- If the edges are ordered, there may also be offset chunks to construct the index for accessing edges of a single vertex. These chunks will store the start offset of each vertex's edges, see `./edge/person_knows_person/ordered_by_source/offset/chunk0`_ as an example.
-
-
-How to Use GAR
-------------------------
-
-Construct information
-`````````````````````
-It is convenient to construct the GAR metadata and dump it to generate information files. We provide an `example program`_ located in the source code which shows how to construct and dump the files for graph information, vertex information and edge information.
-
-Also, the metadata of a graph can be constructed easily through reading the already existed information files, as the following code illustrates:
-
-.. code:: C++
-
- // construct graph information from file
- std::string path = ... // the path of the graph information file (e.g., ldbc_sample.graph.yml)
- auto graph_info = graphar::GraphInfo::Load(path).value();
-
- // get vertex information
- auto vertex_info = graph_info->GetVertexInfo("person");
- if (vertex_info != nullptr) {
- // use vertex_info ...
- }
-
- // get edge information
- auto edge_info = graph_info->GetEdgeInfo("person", "knows", "person");
- if (edge_info != nullptr) {
- // use edge_info ...
- }
-
-
-Read GAR files
-``````````````
-GAR supports the flexible reading of graph data, e.g., allowing to read data of a single vertex, a vertex chunk, or all vertices with a specific label. In addition, necessary property groups can be selected to read and avoid reading all properties from the files. Furthermore, GAR provides convenient and flexible access to adjList, offset and property chunks for edges.
-
-As a simple case, the following example shows how to read all vertices with label "person" of the graph defined by "graph_info" and output the values of "id" and "firstName" for each vertex.
-
-.. code:: C++
-
- graph_info = ...
- auto vertices = graphar::VerticesCollection::Make(graph_info, "person").value();
-
- for (auto it = vertices->begin(); it != vertices->end(); ++it) {
- // get a vertex and access its data
- auto vertex = *it;
- if (vertex.IsValid("id") && vertex.IsValid("firstName"))
- std::cout << "id=" << vertex.property<int64_t>("id").value() << ", firstName=" << vertex.property<std::string>("firstName").value() << std::endl;
- }
-
-The next example reads all edges with label "person_knows_person" from the above graph and outputs the end vertices for each edge.
-
-.. code:: C++
-
- graph_info = ...
- auto expect = graphar::EdgesCollection::Make(graph_info, "person", "konws", "person", graphar::AdjListType::ordered_by_source);
- auto edges = expect.value();
-
- for (auto it = edges->begin(); it != edges->end(); ++it) {
- // get an edge and access its data
- auto edge = *it;
- std::cout << "src=" << edge.source() << ", dst=" << edge.destination() << std::endl;
- }
-
-See also `C++ Reader API Reference <reference/index.html#readers>`_.
-
-Write GAR files
-```````````````
-As same with the readers, the GAR writers provide different-level methods to output the graph data in memory into GAR files.
-
-As the simplest cases, the fist example below adds vertices to **VerticesBuilder**, and then dumps the data to files; the second example constructs a collection of edges and then dumps them.
-
-.. code:: C++
-
- vertex_info = ...
- prefix = ...
- graphar::builder::VerticesBuilder builder(vertex_info, prefix);
-
- // add a vertex
- graphar::builder::Vertex v;
- v.AddProperty("id", 933);
- v.AddProperty("firstName", "Alice");
- builder.AddVertex(v);
- // add other vertices
- // ...
-
- // write to GAR files
- builder.Dump();
-
-.. code:: C++
-
- edge_info = ...
- prefix = ...
- vertices_num = ...
- graphar::builder::EdgesBuilder builder(edge_info, prefix, graphar::AdjListType::ordered_by_source, vertices_num);
-
- // add an edge (0 -> 3)
- graphar::builder::Edge e(0, 3);
- e.AddProperty("creationDate", "2011-07-20T20:02:04.233+0000");
- builder.AddEdge(e);
- // add other edges
- // ...
-
- // write to GAR files
- builder.Dump();
-
-See also `C++ Writer API Reference <reference/index.html#writer-and-builder>`_.
-
-A PageRank Example
-``````````````````
-Here we will go through an example of out-of-core graph analytic algorithms based on GAR which calculates the PageRank. Please look `here <https://en.wikipedia.org/wiki/PageRank>`_ if you want a detailed explanation of the PageRank algorithm. And the source code can be found at `pagerank_example.cc`_.
-
-This program first reads in the graph information file to obtain the metadata; then, it constructs the vertex and edge collections to enable access to the graph. After that, an implementation of the PageRank algorithm is provided, with data for the vertices stored in memory, and the edges streamed through disk I/O. Finally, the vertex information with type "person" is extended to include a new property named "pagerank" (a new vertex information file named *person-new-pagerank.vertex.yml* [...]
-
-Please refer to `more examples <examples/out-of-core.html>`_ to learn about the other available case studies utilizing GraphAr.
-
-.. _Building Steps: https://github.com/apache/incubator-graphar/blob/main/README.rst#building-libraries
-
-.. _person.vertex.yml: https://github.com/GraphScope/gar-test/blob/main/ldbc_sample/csv/person.vertex.yml
-
-.. _person_knows_person.edge.yml: https://github.com/GraphScope/gar-test/blob/main/ldbc_sample/csv/person_knows_person.edge.yml
-
-.. _./vertex/person/firstName_lastName_gender/chunk0: https://github.com/GraphScope/gar-test/blob/main/ldbc_sample/csv/vertex/person/firstName_lastName_gender/chunk0
-
-.. _test data: https://github.com/GraphScope/gar-test/blob/main/ldbc_sample/
-
-.. _./edge/person_knows_person/ordered_by_source/creationDate/part0/chunk0: https://github.com/GraphScope/gar-test/blob/main/ldbc_sample/csv/edge/person_knows_person/ordered_by_source/creationDate/part0/chunk0
-
-.. _./edge/person_knows_person/ordered_by_source/adj_list/part0/chunk0: https://github.com/GraphScope/gar-test/blob/main/ldbc_sample/csv/edge/person_knows_person/ordered_by_source/adj_list/part0/chunk0
-
-.. _./edge/person_knows_person/ordered_by_source/offset/chunk0: https://github.com/GraphScope/gar-test/blob/main/ldbc_sample/csv/edge/person_knows_person/ordered_by_source/offset/chunk0
-
-.. _example program: https://github.com/apache/incubator-graphar/blob/main/cpp/examples/construct_info_example.cc
-
-.. _pagerank_example.cc: https://github.com/apache/incubator-graphar/blob/main/cpp/examples/pagerank_example.cc
diff --git a/docs/cpp/index.rst b/docs/cpp/index.rst
deleted file mode 100644
index 34b6b6b..0000000
--- a/docs/cpp/index.rst
+++ /dev/null
@@ -1,9 +0,0 @@
-C++ Library
-============================
-
-.. toctree::
- :maxdepth: 2
-
- Getting Started <getting-started>
- Examples <examples/index>
- Reference <reference/index>
diff --git a/docs/cpp/reference/index.rst b/docs/cpp/reference/index.rst
deleted file mode 100644
index 44b62a4..0000000
--- a/docs/cpp/reference/index.rst
+++ /dev/null
@@ -1,301 +0,0 @@
-API Reference
-==================
-
-.. _cpp-api:
-
-.. default-domain:: cpp
-
-Graph Info
------------
-
-.. doxygenclass:: graphar::Property
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::PropertyGroup
- :members:
- :undoc-members:
-
-.. doxygenfunction:: graphar::CreatePropertyGroup
-
-.. doxygenclass:: graphar::AdjacentList
- :members:
- :undoc-members:
-
-.. doxygenfunction:: graphar::CreateAdjacentList
-
-.. doxygenclass:: graphar::VertexInfo
- :members:
- :undoc-members:
-
-.. doxygenfunction:: graphar::CreateVertexInfo
-
-.. doxygenclass:: graphar::EdgeInfo
- :members:
- :undoc-members:
-
-.. doxygenfunction:: graphar::CreateEdgeInfo
-
-.. doxygenclass:: graphar::GraphInfo
- :members:
- :undoc-members:
-
-.. doxygenfunction:: graphar::CreateGraphInfo
-
-Readers
----------------------
-
-Chunk Info Reader
-~~~~~~~~~~~~~~~~~
-
-.. doxygenclass:: graphar::VertexPropertyChunkInfoReader
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::AdjListChunkInfoReader
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::AdjListPropertyChunkInfoReader
- :members:
- :undoc-members:
-
-Arrow Chunk Reader
-~~~~~~~~~~~~~~~~~~
-
-.. doxygenclass:: graphar::VertexPropertyArrowChunkReader
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::AdjListArrowChunkReader
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::AdjListOffsetArrowChunkReader
- :members:
- :undoc-members:
-
-Vertices Collection
-~~~~~~~~~~~~~~~~~~~
-
-.. doxygenclass:: graphar::Vertex
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::VertexIter
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::VerticesCollection
- :members:
- :undoc-members:
-
-Edges Collection
-~~~~~~~~~~~~~~~~~~
-
-.. doxygenclass:: graphar::Edge
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::EdgeIter
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::EdgesCollection
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::OBSEdgeCollection
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::OBDEdgesCollection
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::UBSEdgesCollection
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::UBDEdgesCollection
- :members:
- :undoc-members:
-
-Writer and Builder
----------------------
-
-Chunk Writer
-~~~~~~~~~~~~~~~~~
-
-.. doxygenclass:: graphar::VertexPropertyWriter
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::EdgeChunkWriter
- :members:
- :undoc-members:
-
-Builder
-~~~~~~~~~~~~~~~~~~~
-
-.. doxygenclass:: graphar::builder::Vertex
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::builder::Edge
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::builder::VerticesBuilder
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::builder::EdgesBuilder
- :members:
- :undoc-members:
-
-
-Types
---------
-
-Id Type
-~~~~~~~~~~~~~~~~~~~
-
-.. doxygentypedef:: graphar::IdType
-
-Data Type
-~~~~~~~~~~~~~~~~~~~
-
-.. doxygenclass:: graphar::DataType
- :members:
- :undoc-members:
-
-.. doxygenfunction:: graphar::boolean
-.. doxygenfunction:: graphar::int32
-.. doxygenfunction:: graphar::int64
-.. doxygenfunction:: graphar::float32
-.. doxygenfunction:: graphar::float64
-.. doxygenfunction:: graphar::string
-.. doxygenfunction:: graphar::list
-
-File Type
-~~~~~~~~~~~~~~~~~~~
-.. doxygenenum:: graphar::FileType
-
-Adj List Type
-~~~~~~~~~~~~~~~~~~~
-.. doxygenenum:: graphar::AdjListType
-
-Validate Level
-~~~~~~~~~~~~~~~~~~~
-.. doxygenenum:: graphar::ValidateLevel
-
-
-Utilities
----------
-
-Result and Status
-~~~~~~~~~~~~~~~~~~~
-
-.. doxygentypedef:: graphar::Result
-
-.. doxygenclass:: graphar::Status
- :members:
- :undoc-members:
-
-FileSystem
-~~~~~~~~~~~~~~~~~~~
-
-.. doxygenclass:: graphar::FileSystem
- :members:
- :undoc-members:
-
-.. doxygenfunction:: graphar::FileSystemFromUriOrPath
-
-Yaml Parser
-~~~~~~~~~~~~~~~~~~~
-
-.. doxygenclass:: graphar::Yaml
- :members:
- :undoc-members:
-
-Info Version
-~~~~~~~~~~~~~~~~~~~
-
-.. doxygenclass:: graphar::InfoVersion
- :members:
- :undoc-members:
-
-Expression
-~~~~~~~~~~~~~~~~~~~
-
-.. doxygenclass:: graphar::Expression
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::ExpressionProperty
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::ExpressionLiteral
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::ExpressionNot
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::ExpressionUnaryOp
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::ExpressionBinaryOp
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::ExpressionEqual
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::ExpressionNotEqual
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::ExpressionGreaterThan
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::ExpressionGreaterEqual
- :members:
- :undoc-members:
-
-
-.. doxygenclass:: graphar::ExpressionLessThan
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::ExpressionLessEqual
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::ExpressionAnd
- :members:
- :undoc-members:
-
-.. doxygenclass:: graphar::ExpressionOr
- :members:
- :undoc-members:
-
-.. doxygenfunction:: graphar::_Property(const Property&)
-.. doxygenfunction:: graphar::_Property(const std::string&)
-.. doxygenfunction:: graphar::_Literal
-.. doxygenfunction:: graphar::_Not
-.. doxygenfunction:: graphar::_Equal
-.. doxygenfunction:: graphar::_NotEqual
-.. doxygenfunction:: graphar::_GreaterThan
-.. doxygenfunction:: graphar::_GreaterEqual
-.. doxygenfunction:: graphar::_LessThan
-.. doxygenfunction:: graphar::_LessEqual
-.. doxygenfunction:: graphar::_And
-.. doxygenfunction:: graphar::_Or
-
diff --git a/docs/developers/contributing.rst b/docs/developers/contributing.rst
deleted file mode 100644
index 3dc0f5e..0000000
--- a/docs/developers/contributing.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-.. This file is just a placeholder to refer the top-level CONTRIBUTING.rst
- to sphinx doc workspace.
-
-.. include:: ../../CONTRIBUTING.rst
diff --git a/docs/developers/java-dev.rst b/docs/developers/java-dev.rst
deleted file mode 100644
index 62cbd26..0000000
--- a/docs/developers/java-dev.rst
+++ /dev/null
@@ -1,108 +0,0 @@
-Java Development
-================
-
-Introduction
-------------
-
-GraphAr Java library based on GraphAr C++ library and an efficient FFI
-for Java and C++ called
-`FastFFI <https://github.com/alibaba/fastFFI>`__.
-
-Source Code Level
-~~~~~~~~~~~~~~~~~
-
-- Interface
-
-- Class
-
-- JNI code
-
-- GraphAr C++ library
-
-If you want to use classes or functions of GraphAr C++ library through JAVA SDK, you only need to write interfaces with
-annotations. After the interfaces are ready, the java code for the interfaces and the the C++ code which include JNI
-code for native methods will be automatically generated by FastFFI.For
-annotation's usage, please refer to
-`FastFFI <https://github.com/alibaba/fastFFI>`__.
-
-
-
-Runtime Level
-~~~~~~~~~~~~~
-
-Interfaces and classes will be compiled to bytecode. Usually, JNI code will be compiled to bitcode as a part of
-dynamic library which can be called by native methods directly.
-If llvm4jni is enable, suitable method in JNI will be transferred to bytecode.
-
-For decoupling the implementation of C++ and Java, we use a bridge dynamic library called gar-jni to connect them, it
-will integrate all C++ dependencies(e.g. JNI code, GraphAr C++ library and arrow C++)
-and can be called by native methods in Java directly.
-Most JNI code is generated by FastFFI, but some JNI code is written by ourselves, such as JNI code for
-transferring VectorSchemaRoot into arrow::Table.
-
-To build the bridge dynamic library, here is main part of our CMakeLists.txt:
-
-.. code-block:: cmake
-
- # set auto-generated JNI code and handwriting JNI code as source files
- file(GLOB SOURCES "${CMAKE_CURRENT_SOURCE_DIR}/target/generated-sources/annotations/*.cc" "${CMAKE_CURRENT_SOURCE_DIR}/target/generated-test-sources/test-annotations/*.cc"
- "${CMAKE_CURRENT_SOURCE_DIR}/src/main/cpp/ffi/*.cc")
- # remove auto-generated JNI code for specific method cause we have handwriting JNI code for it
- list(REMOVE_ITEM SOURCES "${CMAKE_CURRENT_SOURCE_DIR}/target/generated-sources/annotations/jni_com_alibaba_graphar_arrow_ArrowTable_Static_cxx_0x58c7409.cc")
-
- set(LIBNAME "gar-jni")
-
- # find JNI related libraries
- find_package(JNI REQUIRED)
- include_directories(SYSTEM ${JAVA_INCLUDE_PATH})
- include_directories(SYSTEM ${JAVA_INCLUDE_PATH2})
-
- # some JNI code depends on arrow
- find_package(Arrow REQUIRED)
- # build graphar-cpp in specific version
- include(graphar-cpp)
- build_graphar_cpp()
-
- # build the bridge JNI library
- add_library(${LIBNAME} SHARED ${SOURCES})
- # include graphar-cpp headers
- target_include_directories(${LIBNAME} SYSTEM BEFORE PRIVATE ${GAR_INCLUDE_DIR})
- # link graphar-cpp and arrow
- target_link_libraries(${LIBNAME} ${CMAKE_JNI_LINKER_FLAGS} gar_shared)
- target_link_libraries(${LIBNAME} ${CMAKE_JNI_LINKER_FLAGS} Arrow::arrow_static)
-
-More about usage of CMake, please refer to `CMake's official website <https://cmake.org/>`__.
-
-Building GraphAr Java
----------------------
-
-Please refer to `GraphAr Java Library user guide <../user-guide/java-lib.html>`__.
-
-How To Test
------------
-
-.. code-block:: bash
-
- $ export GAR_TEST_DATA=$PWD/../testing/
- $ mvn clean test
-
-This will build GraphAr C++ library internally for Java. If you already installed GraphAr C++ library in your system,
-you can append this option to skip: ``-DbuildGarCPP=OFF``.
-
-Code Style
-----------
-
-We follow `AOSP Java code
-style <https://source.android.com/docs/setup/contribute/code-style>`__. To ensure
-CI for checking code style will pass, please ensure check below is
-success:
-
-.. code-block:: bash
-
- $ mvn spotless:check
-
-If there are violations, running command below to automatically format:
-
-.. code-block:: bash
-
- $ mvn spotless:apply
diff --git a/docs/format/status.rst b/docs/format/status.rst
deleted file mode 100644
index 19a0078..0000000
--- a/docs/format/status.rst
+++ /dev/null
@@ -1,380 +0,0 @@
-=====================
-Implementation Status
-=====================
-
-The following tables summarize the features available in the various official GraphAr libraries.
-All libraries currently follow version 1.0.0 of the GraphAr format.
-
-
-Data Types
-----------
-
-+-------------------+-------+-------+-------+------------+
-| Data type | C++ | Java | Scala | Python |
-| (primitive) | | | | |
-+===================+=======+=======+=======+============+
-| Boolean | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-| Int32 | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-| Int64 | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-| Float | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-| Double | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-| String | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-| Date | ✓ | | | |
-+-------------------+-------+-------+-------+------------+
-| Timestamp | ✓ | | | |
-+-------------------+-------+-------+-------+------------+
-| Time | | | | |
-+-------------------+-------+-------+-------+------------+
-
-+-------------------+-------+-------+-------+------------+
-| Data type | C++ | Java | Scala | Python |
-| (nested) | | | | |
-+===================+=======+=======+=======+============+
-| List (*) | ✓ | | | |
-+-------------------+-------+-------+-------+------------+
-
-Notes:
-
-* \(\*) The data type of List is not supported by the CSV payload file format.
-
-
-Payload Data File Formats
--------------------------
-
-+-----------------------------+---------+---------+-------+------------+
-| Format | C++ | Java | Scala | Python |
-| | | | | |
-+=============================+=========+=========+=======+============+
-| CSV | R/W | R (1) | R/W | R/W (2) |
-+-----------------------------+---------+---------+-------+------------+
-| ORC | R/W | R (1) | R/W | R/W (2) |
-+-----------------------------+---------+---------+-------+------------+
-| Parquet | R/W | R (1) | R/W | R/W (2) |
-+-----------------------------+---------+---------+-------+------------+
-| Avro | | | | |
-+-----------------------------+---------+---------+-------+------------+
-| HDF5 | | | | |
-+-----------------------------+---------+---------+-------+------------+
-| JSON | | | | |
-+-----------------------------+---------+---------+-------+------------+
-
-Notes:
-
-* *R* = Read supported
-
-* *W* = Write supported
-
-Supported compression methods for the file formats:
-
-+-----------------------------+---------+---------+-------+------------+
-| Compression | C++ | Java | Scala | Python |
-+=============================+=========+=========+=======+============+
-| ZSTD (*) | ✓ | ✓ | ✓ | ✓ |
-+-----------------------------+---------+---------+-------+------------+
-
-Notes:
-
-* \(\*) Compression is not supported by the CSV payload file format.
-
-
-Property
---------
-+-------------------+-------+-------+-------+------------+
-| Property feature | C++ | Java | Scala | Python |
-+===================+=======+=======+=======+============+
-| primary key | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-| nullable | ✓ | | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-
-
-Supported operations in Property:
-
-+-------------------+-------+-------+-------+------------+
-| Property operation| C++ | Java | Scala | Python |
-+===================+=======+=======+=======+============+
-| create | ✓ | ✓ (1) | ✓ | ✓ (2) |
-+-------------------+-------+-------+-------+------------+
-| get_name | ✓ | ✓ (1) | ✓ | ✓ (2) |
-+-------------------+-------+-------+-------+------------+
-| is_primary_key | ✓ | ✓ (1) | ✓ | ✓ (2) |
-+-------------------+-------+-------+-------+------------+
-| is_nullable | ✓ | | ✓ | ✓ (2) |
-+-------------------+-------+-------+-------+------------+
-
-
-Property Group
---------------
-+-------------------+-------+--------+-------+------------+
-| Property Group | C++ |Java (1)| Scala | Python (2)|
-| (operation) | | | | |
-+===================+=======+========+=======+============+
-| create | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| add property | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| remove property | | | | |
-+-------------------+-------+--------+-------+------------+
-| get properties | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| check property | ✓ | ✓ | | |
-+-------------------+-------+--------+-------+------------+
-| get file type | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| get path prefix | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| check validation | ✓ | | | |
-+-------------------+-------+--------+-------+------------+
-
-
-Adjacency List
---------------
-+-------------------+-------+-------+-------+------------+
-| Adjacency List | C++ | Java | Scala | Python |
-| (type) | | | | |
-+===================+=======+=======+=======+============+
-| CSR | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-| CSC | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-| COO | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-
-Supported operations in Adjacency List:
-
-+-------------------+-------+--------+-------+------------+
-| Adjacency List | C++ |Java (1)| Scala | Python (2)|
-| (operation) | | | | |
-+===================+=======+========+=======+============+
-| create | ✓ | | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| get adjacency type| ✓ | | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| get file type | ✓ | | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| get path prefix | ✓ | | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| check validation | ✓ | | | |
-+-------------------+-------+--------+-------+------------+
-
-
-Vertex
--------
-Vertex features:
-
-+-------------------+-------+-------+-------+------------+
-| Vertex feature | C++ | Java | Scala | Python |
-+===================+=======+=======+=======+============+
-| label | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-| tag | | | | |
-+-------------------+-------+-------+-------+------------+
-| chunk based | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-| property group | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-
-Notes:
-
-* *label* is the vertex label, which is a unique identifier for the vertex.
-* *tag* is the vertex tag, which is tag or category for the vertex.
-
-Supported operations in Vertex Info:
-
-+-------------------+-------+--------+-------+------------+
-| Vertex Info | C++ |Java (1)| Scala | Python (2) |
-| (operation) | | | | |
-+===================+=======+========+=======+============+
-| create | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| add group | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| remove group | | | | |
-+-------------------+-------+--------+-------+------------+
-| get label | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| get chunk size | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| get groups | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| get path prefix | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| check property | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| check validation | ✓ | | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| serialize | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| deserialize | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-
-
-Edge
--------
-Edge features:
-
-+-------------------+-------+-------+-------+------------+
-| Edge feature | C++ | Java | Scala | Python |
-+===================+=======+=======+=======+============+
-| label | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-| chunk based | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-| property group | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-| adjacent list | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-| directed | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+-------+-------+------------+
-
-Supported operations in Edge Info:
-
-+-------------------+-------+--------+-------+------------+
-| Edge Info | C++ |Java (1)| Scala | Python (2) |
-| (operation) | | | | |
-+===================+=======+========+=======+============+
-| create | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| add group | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| remove group | | | | |
-+-------------------+-------+--------+-------+------------+
-| add adj list | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| remove adj list | | | | |
-+-------------------+-------+--------+-------+------------+
-| get label | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| get source label | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| get dest label | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| get chunk size | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| get source chunk | ✓ | ✓ | ✓ | ✓ |
-| size | | | | |
-+-------------------+-------+--------+-------+------------+
-| get dest chunk | ✓ | ✓ | ✓ | ✓ |
-| size | | | | |
-+-------------------+-------+--------+-------+------------+
-| get groups | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| check adj list | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| check property | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| get file type | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| get path prefix | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| is directed | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| check validation | ✓ | | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| serialize | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| deserialize | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-
-Notes:
-
-* *<source label, label, dest label>* is the unique identifier for the edge type.
-
-
-Graph
------
-+-------------------+-------+-------+-------+------------+
-| Graph | C++ | Java | Scala | Python |
-+===================+=======+=======+=======+============+
-| labeled vertex | ✓ | ✓ | ✓ | ✓ |
-| (with property) | | | | |
-+-------------------+-------+-------+-------+------------+
-| labeled edge | ✓ | ✓ | ✓ | ✓ |
-| (with property) | | | | |
-+-------------------+-------+-------+-------+------------+
-| extra info | ✓ | | | |
-+-------------------+-------+-------+-------+------------+
-
-Supported operations in Graph Info:
-
-+-------------------+-------+--------+-------+------------+
-| Graph Info | C++ |Java (1)| Scala | Python (2) |
-| (operation) | | | | |
-+===================+=======+========+=======+============+
-| create | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| add vertex | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| remove vertex | | | | |
-+-------------------+-------+--------+-------+------------+
-| add edge | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| remove edge | | | | |
-+-------------------+-------+--------+-------+------------+
-| get name | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| get vertex | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| get edge | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| add extra info | | | | |
-+-------------------+-------+--------+-------+------------+
-| remove extra info | | | | |
-+-------------------+-------+--------+-------+------------+
-| get extra info | ✓ | | | |
-+-------------------+-------+--------+-------+------------+
-| check validation | ✓ | | | |
-+-------------------+-------+--------+-------+------------+
-| serialize | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-| deserialize | ✓ | ✓ | ✓ | ✓ |
-+-------------------+-------+--------+-------+------------+
-
-
-Notes:
-
-* \(1) Through fastFFI bindings to the GraphAr C++ library.
-
-* \(2) Through py4j bindings to the GraphAr Spark library.
-
-
-Libraries Version Compatibility
--------------------------------
-+-------------------+-------+-------+-------------------+
-| GraphAr C++ | C++ | CMake | Format Version |
-| Version | | | |
-+===================+=======+=======+===================+
-| 0.11.x | 17+ | 2.8+ | 1.0.0 |
-+-------------------+-------+-------+-------------------+
-
-+----------------+----------+--------+------------------+
-| GraphAr Java | Java | Maven | Format Version |
-| Version | | | |
-+================+==========+========+==================+
-| 0.1.0 | 1.8 | 3.6+ | 1.0.0 |
-+----------------+----------+--------+------------------+
-
-+----------------+--------------+---------+---------+---------+----------------+
-| GraphAr Spark | Apache Spark | Scala | Java | Hadoop | Format Version |
-| Version | Version | Version | Version | Version | |
-+================+==============+=========+=========+=========+================+
-| 0.1.0 | 3.2.x-3.3.x | 2.12.x | 1.8, 11 | 3 | 1.0.0 |
-+----------------+--------------+---------+---------+---------+----------------+
-
-+-------------------+---------+-------------+---------+-----------------+
-| GraphAr PySpark | Python | PySpark | Hadoop | Format Version |
-| Version | Version | Version | Version | |
-+===================+=========+=============+=========+=================+
-| 0.1.0 | 3.8+ | 3.2.x | 3 | 1.0.0 |
-+-------------------+---------+-------------+---------+-----------------+
-
-Notes:
-
-* Since the GraphAr PySpark library is bindings to the GraphAr Spark library,
- the PySpark version should be compatible with the Spark version.
diff --git a/docs/index.md b/docs/index.md
new file mode 100644
index 0000000..e033a6f
--- /dev/null
+++ b/docs/index.md
@@ -0,0 +1,17 @@
+---
+id: documentation
+title: Documentation
+sidebar_position: 0
+---
+
+
+Welcome to the documentation for Apache GraphAr. Here, you can find information about the GraphAr File Format, including specification and libraries.
+
+### [Overview](/docs/overview)
+Overview of the Apache GraphAr project.
+
+### [Specification](/docs/category/specification)
+Documentation about the Apache GraphAr file format.
+
+### [Libraries](/docs/category/libraries)
+Documentation about the libraries of Apache GraphAr.
diff --git a/docs/index.rst b/docs/index.rst
deleted file mode 100644
index bb5bb8b..0000000
--- a/docs/index.rst
+++ /dev/null
@@ -1,39 +0,0 @@
-.. GraphAr documentation master file, created by
- sphinx-quickstart on Fri Nov 11 20:47:00 2022.
- You can adapt this file completely to your liking, but it should at least
- contain the root `toctree` directive.
-
-.. toctree::
- :maxdepth: 1
- :caption: Overview
- :hidden:
-
- Overview <overview/overview>
- Motivation <overview/motivation>
- Concepts <overview/concepts>
-
-.. toctree::
- :maxdepth: 1
- :caption: Format Specification
- :hidden:
-
- File Format <format/file-format>
- Implementation Status <format/status>
-
-.. toctree::
- :maxdepth: 2
- :caption: Libraries
- :hidden:
-
- C++ <cpp/index>
- Java <java/index>
- Spark <spark/index>
- PySpark <pyspark/index>
-
-.. toctree::
- :maxdepth: 2
- :caption: Development
- :hidden:
-
- Contributing Overview <developers/contributing>
- Java Development <developers/java-dev>
diff --git a/docs/java/index.rst b/docs/java/index.rst
deleted file mode 100644
index d6d38b7..0000000
--- a/docs/java/index.rst
+++ /dev/null
@@ -1,8 +0,0 @@
-JAVA Library
-============================
-
-.. toctree::
- :maxdepth: 2
-
- Getting started <java-lib>
- Reference <reference/index>
diff --git a/docs/java/java-lib.rst b/docs/java/java-lib.rst
deleted file mode 100644
index 039e558..0000000
--- a/docs/java/java-lib.rst
+++ /dev/null
@@ -1,221 +0,0 @@
-GraphAr Java Library
-====================
-
-Overview
---------
-
-Based on an efficient FFI for Java and C++ called
-`fastFFI <https://github.com/alibaba/fastFFI>`__, the GraphAr Java
-library allows users to write Java for generating, loading and
-transforming GAR files. It consists of several components:
-
-- **Information Classes**: As same with in the C++ library, the
- information classes are implemented to construct and access the meta
- information about the **graphs**, **vertices** and **edges** in
- GraphAr.
-
-- **Writers**: The GraphAr Java writer provides a set of interfaces
- that can be used to write Apache Arrow VectorSchemaRoot into GAR
- files. Every time it takes a VectorSchemaRoot as the logical table
- for a type of vertices or edges, then convert it to ArrowTable, and
- then dumps it to standard GAR files (CSV, ORC or Parquet files) under
- the specific directory path.
-
-- **Readers**: The GraphAr Java reader provides a set of interfaces
- that can be used to read GAR files. It reads a collection of vertices
- or edges at a time and assembles the result into the ArrowTable.
- Similar with the reader in the C++ library, it supports the users to
- specify the data they need, e.g., reading a single property group
- instead of all properties.
-
-Get GraphAr Java Library
-------------------------
-
-Building from source
-~~~~~~~~~~~~~~~~~~~~
-
-Only support installing from source currently, but we will support
-installing from Maven in the future.
-
-Firstly, install llvm-11. ``LLVM11_HOME`` should point to the home of
-LLVM 11. In Ubuntu, it is at ``/usr/lib/llvm-11``. Basically, the build
-procedure the following binary:
-
-- ``$LLVM11_HOME/bin/clang++``
-
-- ``$LLVM11_HOME/bin/ld.lld``
-
-- ``$LLVM11_HOME/lib/cmake/llvm``
-
-Tips:
-
-- Use Ubuntu as example:
-
- .. code-block:: bash
-
- $ sudo apt-get install llvm-11 clang-11 lld-11 libclang-11-dev libz-dev -y
- $ export LLVM11_HOME=/usr/lib/llvm-11
-
-- Or compile from source with this
- `script <https://github.com/alibaba/fastFFI/blob/main/docker/install-llvm11.sh>`__:
-
- .. code-block:: bash
-
- $ export LLVM11_HOME=/usr/lib/llvm-11
- $ export LLVM_VAR=11.0.0
- $ sudo ./install-llvm11.sh
-
-Make the graphar-java-library directory as the current working
-directory:
-
-.. code-block:: bash
-
- $ git clone https://github.com/apache/incubator-graphar.git
- $ cd GraphAr
- $ git submodule update --init
- $ cd java
-
-Compile package:
-
-.. code-block:: bash
-
- $ mvn clean install -DskipTests
-
-This will build GraphAr C++ library internally for Java. If you already installed GraphAr C++ library in your system,
-you can append this option to skip: ``-DbuildGarCPP=OFF``.
-
-Then set GraphAr as a dependency in maven project:
-
-.. code-block:: xml
-
- <dependencies>
- <dependency>
- <groupId>org.apache.graphar</groupId>
- <artifactId>gar-java</artifactId>
- <version>0.1.0</version>
- </dependency>
- </dependencies>
-
-How to use
-----------
-
-Information classes
-~~~~~~~~~~~~~~~~~~~
-
-The Java library for GraphAr provides distinct information classes for
-constructing and accessing meta information about graphs, vertices, and
-edges. These classes act as essential parameters for constructing
-readers and writers, and they can be built either from the existing meta
-files (in the Yaml format) or in-memory from scratch.
-
-To construct information from a Yaml file, please refer to the following
-example code.
-
-.. code-block:: java
-
- // read graph yaml and construct information
- String path = ...; // the path to the yaml file
- Result<GraphInfo> graphInfoResult = GraphInfo.load(path);
- if (!graphInfoResult.hasError()) {
- GraphInfo graphInfo = graphInfoResult.value();
- // use information classes
- StdMap<StdString, VertexInfo> vertexInfos = graphInfo.getVertexInfos();
- StdMap<StdString, EdgeInfo> edgeInfos = graphInfo.getEdgeInfos();
- }
-
-See `test for
-graphinfo <https://github.com/apache/incubator-graphar/tree/main/java/src/test/java/com/apache/incubator-graphar/graphinfo>`__
-for the complete example.
-
-Writers
-~~~~~~~
-
-The GraphAr Java writers wrap C++ interfaces to write arrow::Table into GraphAr
-formatted files in a batch-import fashion. But arrow::Table is not easy
-to build in Java. Instead, the GraphAr Java library provide a static
-method to convert VectorSchemaRoot into arrow::Table. Warning: There are
-some problems concerning this method which lead to memory leaks. We will
-fix it or rewrite writers with Apache arrow Java.
-
-With the VertexWriter, users can specify a particular property group to
-be written into its corresponding chunks, or choose to write all
-property groups. For edge chunks, besides the meta data (edge info), the
-adjList type should also be specified. The adjList/properties can be
-written alone, or alternatively, all adjList, properties, and the offset
-(for CSR and CSC format) chunks can be written simultaneously.
-
-To utilize the GAR Java writer, please refer to the following example
-code.
-
-.. code-block:: java
-
- // common steps to construct VectorSchemaRoot
- String uri = "file:" + ...; // data source
- ScanOptions options = new ScanOptions(/*batchSize*/ 32768);
- StdSharedPtr<ArrowTable> table = null;
- try (BufferAllocator allocator = new RootAllocator();
- DatasetFactory datasetFactory =
- new FileSystemDatasetFactory(
- allocator, NativeMemoryPool.getDefault(), FileFormat.PARQUET, uri);
- Dataset dataset = datasetFactory.finish();
- Scanner scanner = dataset.newScan(options);
- ArrowReader reader = scanner.scanBatches()) {
- while (reader.loadNextBatch()) {
- try (VectorSchemaRoot root = reader.getVectorSchemaRoot()) {
- // convert VectorSchemaRoot to ArrowTable
- table = ArrowTable.fromVectorSchemaRoot(allocator, root, reader);
- }
- }
- } catch (Exception e) {
- e.printStackTrace();
- }
-
- // construct writer object
- String path = ...; // file to be wrote
- StdString edgeMetaFile = StdString.create(path);
- StdSharedPtr<Yaml> edgeMeta = Yaml.loadFile(edgeMetaFile).value();
- EdgeInfo edgeInfo = EdgeInfo.load(edgeMeta).value();
- EdgeChunkWriter writer = EdgeChunkWriter.factory.create(
- edgeInfo, StdString.create("/tmp/"), AdjListType.ordered_by_source);
-
- // write table with writer object
- writer.sortAndWriteAdjListTable(table, 0, 0); // Write adj list of vertex chunk 0 to files
-
-See `test for
-writers <https://github.com/apache/incubator-graphar/tree/main/java/src/test/java/com/apache/incubator-graphar/writers>`__
-for the complete example.
-
-Readers
-~~~~~~~
-
-The GraphAr Java reader provides an extensive set of interfaces to read
-GAR files. It reads a collection of vertices or edges at a time as
-ArrowTable. Similar with the reader in C++ library, it supports the
-users to specify the data they need, e.g., a single property group.
-
-To utilize the GAR Java reader, please refer to the following example
-code.
-
-.. code-block:: java
-
- // construct vertex chunk reader
- GraphInfo graphInfo = ...; // load graph meta info
- StdString label = StdString.create("person");
- StdString propertyName = StdString.create("id");
- if (graphInfo.getVertexInfo(label).hasError()) {
- // throw Exception or do other things
- }
- PropertyGroup group = graphInfo.getVertexPropertyGroup(label, propertyName).value();
- Result<VertexPropertyArrowChunkReader> maybeReader =
- GrapharStaticFunctions.INSTANCE.constructVertexPropertyArrowChunkReader(
- graphInfo, label, group);
- // check reader's status if needed
- VertexPropertyArrowChunkReader reader = maybeReader.value();
- Result<StdSharedPtr<ArrowTable>> result = reader.getChunk();
- // check table's status if needed
- StdSharedPtr<ArrowTable> table = result.value();
- StdPair<Long, Long> range = reader.getRange().value();
-
-See `test for
-readers <https://github.com/apache/incubator-graphar/tree/main/java/src/test/java/com/apache/incubator-graphar/readers>`__
-for the complete example.
diff --git a/docs/java/reference/index.rst b/docs/java/reference/index.rst
deleted file mode 100644
index c14872a..0000000
--- a/docs/java/reference/index.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-API Reference (javadoc)
-==============================
-
-Stub page for the Java reference docs; actual source is located in the java-api/ directory.
\ No newline at end of file
diff --git a/.github/workflows/license.yml b/docs/libraries/_category_.yml
similarity index 52%
copy from .github/workflows/license.yml
copy to docs/libraries/_category_.yml
index ab2dea7..4c5471e 100644
--- a/.github/workflows/license.yml
+++ b/docs/libraries/_category_.yml
@@ -15,34 +15,9 @@
# specific language governing permissions and limitations
# under the License.
-name: license
-
-on:
- push:
- branches:
- - main
- pull_request:
- branches:
- - main
-
-concurrency:
- group: ${{ github.repository }}-${{ github.event.number || github.head_ref || github.sha }}-${{ github.workflow }}
- cancel-in-progress: true
-
-jobs:
- build:
- runs-on: ubuntu-latest
- steps:
-
- - uses: actions/checkout@v4
- with:
- repository: ${{ github.event.pull_request.head.repo.full_name }}
- ref: ${{ github.event.pull_request.head.ref }}
- submodules: false
- fetch-depth: 0
-
- - name: Check License Header
- uses: apache/skywalking-eyes/header@main
- with:
- config: .licenserc.yaml
- mode: check
+position: 3
+label: 'Libraries'
+collapsible: true
+collapsed: false
+link:
+ type: generated-index
diff --git a/.github/workflows/license.yml b/docs/libraries/cpp/_category_.yml
similarity index 52%
copy from .github/workflows/license.yml
copy to docs/libraries/cpp/_category_.yml
index ab2dea7..0f17ffe 100644
--- a/.github/workflows/license.yml
+++ b/docs/libraries/cpp/_category_.yml
@@ -15,34 +15,9 @@
# specific language governing permissions and limitations
# under the License.
-name: license
-
-on:
- push:
- branches:
- - main
- pull_request:
- branches:
- - main
-
-concurrency:
- group: ${{ github.repository }}-${{ github.event.number || github.head_ref || github.sha }}-${{ github.workflow }}
- cancel-in-progress: true
-
-jobs:
- build:
- runs-on: ubuntu-latest
- steps:
-
- - uses: actions/checkout@v4
- with:
- repository: ${{ github.event.pull_request.head.repo.full_name }}
- ref: ${{ github.event.pull_request.head.ref }}
- submodules: false
- fetch-depth: 0
-
- - name: Check License Header
- uses: apache/skywalking-eyes/header@main
- with:
- config: .licenserc.yaml
- mode: check
+position: 1
+label: 'C++ Library'
+collapsible: true
+collapsed: false
+link:
+ type: generated-index
diff --git a/.github/workflows/license.yml b/docs/libraries/cpp/examples/_category_.yml
similarity index 52%
copy from .github/workflows/license.yml
copy to docs/libraries/cpp/examples/_category_.yml
index ab2dea7..4e8e6a0 100644
--- a/.github/workflows/license.yml
+++ b/docs/libraries/cpp/examples/_category_.yml
@@ -15,34 +15,8 @@
# specific language governing permissions and limitations
# under the License.
-name: license
-
-on:
- push:
- branches:
- - main
- pull_request:
- branches:
- - main
-
-concurrency:
- group: ${{ github.repository }}-${{ github.event.number || github.head_ref || github.sha }}-${{ github.workflow }}
- cancel-in-progress: true
-
-jobs:
- build:
- runs-on: ubuntu-latest
- steps:
-
- - uses: actions/checkout@v4
- with:
- repository: ${{ github.event.pull_request.head.repo.full_name }}
- ref: ${{ github.event.pull_request.head.ref }}
- submodules: false
- fetch-depth: 0
-
- - name: Check License Header
- uses: apache/skywalking-eyes/header@main
- with:
- config: .licenserc.yaml
- mode: check
+label: 'Examples'
+collapsible: true
+collapsed: false
+link:
+ type: generated-index
diff --git a/docs/libraries/cpp/examples/bgl.md b/docs/libraries/cpp/examples/bgl.md
new file mode 100644
index 0000000..0c9a8fd
--- /dev/null
+++ b/docs/libraries/cpp/examples/bgl.md
@@ -0,0 +1,108 @@
+# Co-Work with BGL
+
+The [Boost Graph Library
+(BGL)](https://cs.brown.edu/~jwicks/boost/libs/graph/doc/) is the first
+C++ library to apply the principles of generic programming to the
+construction of the advanced data structures and algorithms used in
+graph computations. The BGL graph interface and graph components are
+generic in the same sense as the Standard Template Library (STL). And it
+provides some built-in algorithms which cover a core set of algorithm
+patterns and a larger set of graph algorithms.
+
+We take calculating CC as an example, to demonstrate how BGL works with
+GraphAr. A weakly connected component is a maximal subgraph of a graph
+such that for every pair of vertices in it, there is an undirected path
+connecting them. And the CC algorithm is to identify all such components
+in a graph. Learn more about [the CC
+algorithm](https://en.wikipedia.org/wiki/Connected_component).
+
+The source code of CC based on BGL can be found at
+[bgl_example.cc](https://github.com/apache/incubator-graphar/blob/main/cpp/examples/bgl_example.cc).
+In this program, the graph information file is first read to get the
+metadata:
+
+``` C++
+std::string path = ... // the path of the graph information file
+auto graph_info = graphar::GraphInfo::Load(path).value();
+```
+
+And then, the vertex collection and the edge collection are established
+as the handles to access the graph data:
+
+``` C++
+auto maybe_vertices = graphar::VerticesCollection::Make(graph_info, "person");
+auto vertices = maybe_vertices.value();
+auto maybe_edges = graphar::EdgesCollection::Make(graph_info, "person", "knows", "person", graphar::AdjListType::ordered_by_source);
+auto edges = maybe_edges.value();
+```
+
+Next, we construct the in-memory graph data structure for BGL by
+traversing the vertices and edges via GraphAr's high-level reading
+interface (the vertex iterator and the edge iterator):
+
+``` C++
+// define the Graph type in BGL
+typedef boost::adjacency_list<boost::vecS, // use vector to store edges
+ boost::vecS, // use vector to store vertices
+ boost::undirectedS, // undirected
+ boost::property<boost::vertex_name_t, int64_t>, // vertex property
+ boost::no_property> Graph; // no edge property
+// descriptors for vertex in BGL
+typedef typename boost::graph_traits<Graph>::vertex_descriptor Vertex;
+
+// declare a graph object with (num_vertices) vertices and an edge iterator
+std::vector<std::pair<graphar::IdType, graphar::IdType>> edges_array;
+auto it_begin = edges->begin(), it_end = edges->end();
+for (auto it = it_begin; it != it_end; ++it)
+ edges_array.push_back(std::make_pair(it.source(), it.destination()));
+Graph g(edges_array.begin(), edges_array.end(), num_vertices);
+
+// define the internal vertex property "id"
+boost::property_map<Graph, boost::vertex_name_t>::type id = get(boost::vertex_name_t(), g);
+auto v_it_begin = vertices->begin(), v_it_end = vertices->end();
+for (auto it = v_it_begin; it != v_it_end; ++it) {
+ auto vertex = *it;
+ boost::put(id, vertex.id(), vertex.property<int64_t>("id").value());
+}
+```
+
+After that, an internal CC algorithm provided by BGL is called:
+
+``` C++
+// define the external vertex property "component"
+std::vector<int> component(num_vertices);
+// call algorithm: cc
+int cc_num = boost::connected_components(g, &component[0]);
+std::cout << "Total number of components: " << cc_num << std::endl;
+```
+
+Finally, we could use a **VerticesBuilder** of GraphAr to write the
+results to new generated GAR files:
+
+``` C++
+// construct a new property group
+graphar::Property cc = {"cc", graphar::int32(), false};
+std::vector<graphar::Property> property_vector = {cc};
+auto group = graphar::CreatePropertyGroup(property_vector, graphar::FileType::PARQUET);
+
+// construct the new vertex info
+std::string vertex_label = "cc_result", vertex_prefix = "result/";
+int chunk_size = 100;
+auto new_info = graphar::CreateVertexInfo(vertex_label, chunk_size, {group}, vertex_prefix);
+
+// access the vertices via the index map and vertex iterator of BGL
+typedef boost::property_map<Graph, boost::vertex_index_t>::type IndexMap;
+IndexMap index = boost::get(boost::vertex_index, g);
+typedef boost::graph_traits<Graph>::vertex_iterator vertex_iter;
+std::pair<vertex_iter, vertex_iter> vp;
+
+// dump the results through the VerticesBuilder
+graphar::builder::VerticesBuilder builder(new_info, "/tmp/");
+for (vp = boost::vertices(g); vp.first!= vp.second; ++vp.first) {
+ Vertex v = *vp.first;
+ graphar::builder::Vertex vertex(index[v]);
+ vertex.AddProperty(cc.name, component[index[v]]);
+ builder.AddVertex(vertex);
+}
+builder.Dump();
+```
diff --git a/docs/libraries/cpp/examples/graphscope.md b/docs/libraries/cpp/examples/graphscope.md
new file mode 100644
index 0000000..e1c996b
--- /dev/null
+++ b/docs/libraries/cpp/examples/graphscope.md
@@ -0,0 +1,66 @@
+# Integrate into GraphScope
+
+[GraphScope](https://graphscope.io/) is a unified distributed graph
+computing platform that provides a one-stop environment for performing
+diverse graph operations on a cluster through a user-friendly Python
+interface. As an important application case of GraphAr, we have
+integrated it into GraphScope.
+
+GraphScope works on a graph G fragmented via a partition strategy picked
+by the user and each worker maintains a fragment of G. Given a query, it
+posts the same query to all the workers and computes following the BSP
+(Bulk Synchronous Parallel) model. More specifically, each worker first
+executes processing against its local fragment, to compute partial
+answers in parallel. And then each worker may exchange partial results
+with other processors via synchronous message passing.
+
+To integrate GraphAr into GraphScope, we implemented
+*ArrowFragmentBuilder* and *ArrowFragmentWriter*. *ArrowFragmentBuilder*
+establishes the fragments for workers of GraphScope through reading GAR
+files in parallel. Conversely, *ArrowFragmentWriter* can take the
+GraphScope fragments and save them as GAR files. If you're interested in
+knowing more about the implementation, please refer to the [source
+code](https://github.com/v6d-io/v6d/commit/0eda2067e45fbb4ac46892398af0edc84fe1c27b).
+
+## Performance Report
+
+### Parameter settings
+
+The time performance of *ArrowFragmentBuilder* and *ArrowFragmentWriter*
+in GraphScope is heavily dependent on the partitioning of the graph into
+GAR files, that is, the *vertex chunk size* and *edge chunk size*, which
+are specified in the vertex information file and in the edge information
+file, respectively.
+
+Generally speaking, fewer chunks are created if the file size is large.
+On small graphs, this can be disadvantageous as it reduces the degree of
+parallelism, prolonging disk I/O time. On the other hand, having too
+many small files increases the overhead associated with the file system
+and the file parser.
+
+We have conducted micro benchmarks to compare the time performance for
+reading/writing GAR files by
+*ArrowFragmentBuilder*/*ArrowFragmentWriter*, across different *vertex
+chunk size* and *edge chunk size* configurations. The settings we
+recommend for *vertex chunk size* and *edge chunk size* are **2^18** and
+**2^22**, respectively, which lead to efficient performance in most
+cases. These settings can be used as the reference values when
+integrating GraphAr into other systems besides GraphScope.
+
+### Time performance results
+
+Here we report the performance results of *ArrowFragmentBuilder*, and
+compare it with loading the same graph through the default loading
+strategy of GraphScope (through reading the csv files in parallel) . The
+execution time reported below includes loading the graph data from the
+disk into memory, as well as building GraphScope fragments from such
+data. The experiments are conducted on a cluster of 4 AliCloud
+ecs.r6.6xlarge instances (24vCPU, 192GB memory), and using
+[com-friendster](https://snap.stanford.edu/data/com-Friendster.html) (a
+simple graph) and [ldbc-snb-30](https://ldbcouncil.org/benchmarks/snb/)
+(a multi-labeled property graph) as datasets.
+
+| Dataset | Workers | Default Loading | GraphAr Loading |
+|----------------|---------|-----------------|-----------------|
+| com-friendster | 4 | 282s | 54s |
+| ldbc-snb-30 | 4 | 196s | 40s |
diff --git a/docs/libraries/cpp/examples/out-of-core.md b/docs/libraries/cpp/examples/out-of-core.md
new file mode 100644
index 0000000..5978f0f
--- /dev/null
+++ b/docs/libraries/cpp/examples/out-of-core.md
@@ -0,0 +1,180 @@
+# Out-of-core Graph Algorithms
+
+An important application case of GraphAr is to serve out-of-core graph
+processing scenarios. With the graph data saved as GAR files in the
+disk, GraphAr provides a set of reading interfaces to allow to load part
+of graph data into memory when needed, to conduct analytics. While it is
+more convenient and efficient to store the entirety of the graph in
+memory (as is done in BGL), out-of-core graph processing makes it
+possible to complete analytics on the large-scale graphs using limited
+memory/computing resources.
+
+The are some out-of-core graph analytic algorithms that have been
+implemented based on GraphAr, include:
+
+- PageRank (PR)
+- Connected Components (CC)
+- Breadth First Search (BFS)
+
+These algorithms represent for different compute patterns and are
+usually building blocks for constructing other graph algorithms.
+
+## PageRank
+
+[PageRank (PR)](https://en.wikipedia.org/wiki/PageRank) is an algorithm
+used by Google Search to rank web pages in their search engine results.
+The source code of PageRank based on GraphAr located at
+[pagerank_example.cc](https://github.com/apache/incubator-graphar/blob/main/cpp/examples/pagerank_example.cc),
+and the explanations can be found in the [Getting
+Started](../getting-started#a-pagerank-example) page.
+
+## Connected Components
+
+A weakly connected component is a maximal subgraph of a graph such that
+for every pair of vertices in it, there is an undirected path connecting
+them. And [Connected Components
+(CC)](https://en.wikipedia.org/wiki/Connected_component) is an algorithm
+to identify all weakly connected components in a graph. [CC based on
+BGL](./bgl) is provided in GraphAr, also, we implement out-of-core
+algorithms for this application.
+
+A typical method for calculating CC is label propagation. In this
+algorithm, each vertex is attached with a property which represents its
+component label, being its own vertex id initially. In the subsequent
+supersteps (i.e., iterations), a vertex will update its label if it
+receives a smaller id and then it propagates this id to all its
+neighbors.
+
+This algorithm can be implemented based on streaming the edges via
+GraphAr's reading interface. That is to say, the edges are accessed and
+processed chunk by chunk, instead of being loaded into memory at once
+(as in the BGL example).
+
+```c++
+// construct the edge collection in GraphAr
+auto edges = ...
+auto it_begin = edges->begin(), it_end = edges->end();
+
+// initialize for all vertices
+std::vector<graphar::IdType> component(num_vertices);
+for (graphar::IdType i = 0; i < num_vertices; i++)
+ component[i] = i;
+
+// stream all edges for each iteration
+for (int iter = 0; ; iter++) {
+ bool flag = false;
+ for (auto it = it_begin; it != it_end; ++it) {
+ graphar::IdType src = it.source(), dst = it.destination();
+ // update
+ if (component[src] < component[dst]) {
+ component[dst] = component[src];
+ flag = true;
+ } else if (component[src] > component[dst]) {
+ component[src] = component[dst];
+ flag = true;
+ }
+ }
+ // check if it should terminate
+ if (!flag) break;
+}
+```
+
+The file
+[cc_stream_example.cc](https://github.com/apache/incubator-graphar/blob/main/cpp/examples/cc_stream_example.cc)
+located inside the source tree contains the complete implementation for
+this algorithm. Also, we can only process active vertices (the vertices
+which are updated in the last iteration) and the corresponding edges for
+each iteration, since an inactive vertex does not need to update its
+neighbors. Please refer to
+[cc_push_example.cc](https://github.com/apache/incubator-graphar/blob/main/cpp/examples/cc_push_example.cc)
+for the complete code.
+
+
+:::tip
+
+In this example, two kinds of edges are used. The
+**ordered_by_source** edges are used to access all outgoing edges of
+an active vertex, and **ordered_by_dest** edges are used to access the
+incoming edges. In this way, all the neighbors of an active vertex can
+be accessed and processed.
+
+Although GraphAr supports to get the outgoing (incoming) edges of a
+single vertex for all adjList types, it is most efficient when the
+type is **ordered_by_source** (**ordered_by_dest**) since it can avoid
+to read redundant data.
+
+:::
+
+## Breadth First Search
+
+[Breadth First Search
+(BFS)](https://en.wikipedia.org/wiki/Breadth-first_search) is a
+traversing algorithm that starts from a selected vertex (the root) and
+traverse the graph layer-wise thus exploring the neighbor vertices
+(vertices which are directly connected to the root), and then it moves
+towards the next-level neighbor vertices.
+
+An out-of-core BFS algorithm could be implemented based on streaming the
+graph data via GraphAr. For each vertex, a property named *distance* is
+created and initialized to represent the distance from the root to this
+vertex. As with the standard BFS algorithms in other graph processing
+systems, for the iteration/superstep *i* (starting from 0), the active
+vertices contain all vertices reachable from the root in *i* hops (i.e.,
+*distance\[v\]= i*). At the beginning of the algorithm, only the root
+vertex is active. This algorithm terminates when there are no more
+active vertices.
+
+```c++
+// construct the edge collection in GraphAr
+auto edges = ...
+auto it_begin = edges->begin(), it_end = edges->end();
+
+// initialize for all vertices
+graphar::IdType root = 0; // the BFS root
+std::vector<int32_t> distance(num_vertices);
+for (graphar::IdType i = 0; i < num_vertices; i++)
+ distance[i] = (i == root ? 0 : -1);
+
+// stream all edges for each iteration
+for (int iter = 0; ; iter++) {
+ graphar::IdType count = 0;
+ for (auto it = it_begin; it != it_end; ++it) {
+ graphar::IdType src = it.source(), dst = it.destination();
+ // update
+ if (distance[src] == iter && distance[dst] == -1) {
+ distance[dst] = distance[src] + 1;
+ count++;
+ }
+ }
+ // check if it should terminate
+ if (count == 0) break;
+}
+```
+
+The above algorithm is implemented based on streaming all edges for each
+iteration, the source code can be found at
+[bfs_stream_example.cc](https://github.com/apache/incubator-graphar/blob/main/cpp/examples/bfs_stream_example.cc).
+
+Meanwhile, BFS could be implemented in a **push**-style which only
+traverses the edges that from active vertices for each iteration, which
+is typically more efficient on real-world graphs. This implementation
+can be found at
+[bfs_push_example.cc](https://github.com/apache/incubator-graphar/blob/main/cpp/examples/bfs_push_example.cc).
+Similarly, we provide a BFS implementation in a **pull**-style which
+only traverses the edges that lead to non-visited vertices (i.e., the
+vertices that have not been traversed), as shown in
+[bfs_pull_example.cc](https://github.com/apache/incubator-graphar/blob/main/cpp/examples/bfs_pull_example.cc).
+
+:::tip
+
+In common cases of graph processing, the **push**-style is more
+efficient when the set of active vertices is very sparse, while the
+**pull**-style fits when it is dense.
+
+:::
+
+In some cases, it is required to record the path of BFS, that is, to
+maintain each vertex's predecessor (also called *father*) in the
+traversing tree rather than only recording the distance. The
+implementation of BFS with recording fathers can be found at
+[bfs_father_example.cc](https://github.com/apache/incubator-graphar/blob/main/cpp/examples/bfs_father_example.cc).
diff --git a/docs/libraries/cpp/examples/snap-to-graphar.md b/docs/libraries/cpp/examples/snap-to-graphar.md
new file mode 100644
index 0000000..91ed659
--- /dev/null
+++ b/docs/libraries/cpp/examples/snap-to-graphar.md
@@ -0,0 +1,91 @@
+# Convert SNAP Datasets to GraphAr Format
+
+[SNAP](https://snap.stanford.edu/data/) (Stanford Network Analysis
+Project) is a general-purpose network analysis and graph mining library.
+It provides a variety of datasets for research and development. In this
+section, we will show how to convert the SNAP datasets to GraphAr
+format, showcasing the process with the
+[ego-Facebook](https://snap.stanford.edu/data/ego-Facebook.html) graph
+as a case study. The conversion leverages GraphInfo constructors and the
+high-level writer functions from the C++ library.
+
+## Prepare the SNAP Dataset
+
+Before converting, download the ego-Facebook dataset from the SNAP
+website. The dataset is a text file with each line representing an edge
+in the graph.
+
+``` bash
+cd /path/to/your/dataset
+wget https://snap.stanford.edu/data/facebook_combined.txt.gz
+gunzip facebook_combined.txt.gz
+```
+
+## Convert the SNAP Dataset to GraphAr Format
+
+The initial phase involves constructing VertexInfo, EdgeInfo, and
+GraphInfo objects, which are subsequently serialized into YAML files.
+For instance, the code snippet below illustrates the creation and
+storage of the vertex information file.
+
+``` C++
+auto version = graphar::InfoVersion::Parse("gar/v1").value();
+
+// meta info
+std::string vertex_label = "node", vertex_prefix = "vertex/node/";
+
+// create vertex info
+auto vertex_info = graphar::CreateVertexInfo(
+ vertex_label, VERTEX_CHUNK_SIZE, {}, vertex_prefix, version);
+
+// save & dump vertex info
+ASSERT(!vertex_info->Dump().has_error());
+ASSERT(vertex_info->Save(save_path + "node.vertex.yml").ok());
+
+// create and save edge info file ...
+auto edge_info = ...
+ASSERT(!edge_info->Dump().has_error());
+ASSERT(edge_info->Save(save_path + "node_links_node.edge.yml").ok());
+
+// create and save graph info file ...
+auto graph_info = ...
+ASSERT(!graph_info->Dump().has_error());
+ASSERT(graph_info->Save(save_path + graph_name + ".graph.yml").ok());
+```
+
+Subsequently, we employ the high-level vertex and edge builders provided
+by the GraphAr C++ library to generate payload data files with vertex
+and edge data. The code snippet that follows demonstrates the generation
+and preservation of the edge data file.
+
+``` C++
+// construct edges builder
+auto e_builder = graphar::builder::EdgesBuilder::Make(
+ edge_info, save_path, ADJLIST_TYPE, VERTEX_COUNT)
+ .value();
+// read edge data from file
+std::ifstream file(DATA_PATH);
+std::string line;
+while (std::getline(file, line)) {
+ std::istringstream iss(line);
+ // skip comments
+ if (line[0] == '#') {
+ continue;
+ }
+ int src, dst;
+ if (!(iss >> src >> dst)) {
+ break;
+ }
+ graphar::builder::Edge e(src, dst);
+ ASSERT(e_builder->AddEdge(e).ok());
+}
+
+// dump & clear
+ASSERT(e_builder->Dump().ok());
+e_builder->Clear();
+```
+
+For comprehensive insights into this example, please consult the
+accompanying [source
+code](https://github.com/apache/incubator-graphar/tree/main/docs/cpp/examples/snap_dataset_to_graphar.cc)
+.
diff --git a/docs/libraries/cpp/getting-started.md b/docs/libraries/cpp/getting-started.md
new file mode 100644
index 0000000..f7dc186
--- /dev/null
+++ b/docs/libraries/cpp/getting-started.md
@@ -0,0 +1,278 @@
+---
+id: getting-started
+title: Getting Started
+sidebar_position: 1
+---
+
+
+# Getting Started
+
+This article is a quick guide that explains how to work with GraphAr
+C++. To begin with, please refer to the [Building
+Steps](https://github.com/apache/incubator-graphar/blob/main/README.rst#building-libraries)
+to install GraphAr.
+
+## GAR Information Files
+
+GAR uses a group of Yaml files to save the meta information for a graph.
+
+### Graph information
+
+The graph information file defines the most basic information of a graph
+includes its name, the root directory path of the data files, the vertex
+information and edge information files it contains, and the version of
+GraphAr. For example, the file "ldbc_sample.graph.yml" defines an
+example graph named "ldbc_sample", which includes one type of vertices
+("person") and one type of edges ("person knows person").
+
+``` Yaml
+name: ldbc_sample
+prefix: ./
+vertices:
+ - person.vertex.yml
+edges:
+ - person_knows_person.edge.yml
+version: gar/v1
+```
+
+### Vertex information
+
+Each vertex information file defines a single group of vertices with the
+same vertex label, e.g., "person" in this case. The vertex chunk size,
+the relative path for vertex data files and the version of GraphAr are
+specified. These vertices could have some properties, which are divided
+into property groups. Each property group has its own file type (CSV,
+ORC or Parquet) and the prefix of the relative path for its data files,
+it also lists all properties in this group, with every property contains
+the name, data type and if it is the primary key.
+
+The file
+[person.vertex.yml](https://github.com/apache/incubator-graphar-testing/blob/main/ldbc_sample/csv/person.vertex.yml)
+located inside the test data contains an example of the vertex
+information file. In this example, the "person" vertices have two
+property groups. The first group contains only one property (named "id")
+and the second group contains three properties ("firstName", "lastName"
+and "gender").
+
+### Edge information
+
+Each edge information file defines a single type of edges with specific
+labels for the source vertex, destination vertex and the edge, e.g.,
+"person_knows_person" in this case. It defines the meta information such
+as the edge chunk size, the source vertex chunk size, the destination
+vertex chunk size, if the edges are directed or not, the relative file
+path for edge data files, the adjLists and the version of GraphAr. The
+file
+[person_knows_person.edge.yml](https://github.com/apache/incubator-graphar-testing/blob/main/ldbc_sample/csv/person_knows_person.edge.yml)
+located inside the test data contains an example of the edge information
+file.
+
+In GAR format, separate data files are used to store the structure
+(called adjList) and the properties for edges. The adjList type can be
+either of **unordered_by_source**, **unordered_by_dest**,
+**ordered_by_source** or **ordered_by_dest**. For a specific
+type of adjList, the meta information includes its file path prefix, the
+file type, as well as all the property groups attached.
+
+:::note
+
+It is allowed to store different types of adjLists for a group of
+edges at the same time.
+
+::
+
+## GAR Data Files
+
+### Property data
+
+The vertex properties are stored in vertex property chunks with the
+chunk size specified by the vertex information file. Different property
+groups correspond to individual groups of data files. In our example,
+the property group ("first name", "last name", "gender") for vertex
+chunk 0 of "person" vertices are stored in
+[./vertex/person/firstName_lastName_gender/chunk0](https://github.com/apache/incubator-graphar-testing/blob/main/ldbc_sample/csv/vertex/person/firstName_lastName_gender/chunk0).
+
+In practice of graph processing, it is common to only query a subset of
+columns of the properties. Thus, the column-oriented formats like Apache
+ORC and Apache Parquet are more efficient, which eliminate the need to
+read columns that are not relevant. We also provide data files in ORC
+and Parquet for the example graph in the [test
+data](https://github.com/apache/incubator-graphar-testing/blob/main/ldbc_sample/).
+
+Similar with vertices, the edge properties are stored in edge property
+chunks. For each vertex chunk, its associated edges (if the edge type is
+**ordered_by_source** or **unordered_by_source**, the associated edges
+are those in which the source vertex is in that chunk; otherwise, if the
+edge type is **ordered_by_dest** or **unordered_by_dest**, the
+associated edges are those in which the destination is in that chunk)
+are maintained in some edge chunks, with the size of each chunk not
+exceeding the edge chunk size specified in the edge information file.
+
+For instance, the file
+[./edge/person_knows_person/ordered_by_source/creationDate/part0/chunk0](https://github.com/apache/incubator-graphar-testing/blob/main/ldbc_sample/csv/edge/person_knows_person/ordered_by_source/creationDate/part0/chunk0)
+stores the property group "creationDate" of "person_knows_person" edges
+for the first edge chunk of the first vertex chunk, and the adjList type
+of the edges is **ordered_by_source**.
+
+### AdjList data
+
+The adjList in GAR describes the topology structure, i.e., the internal
+id of the source vertex and the destination vertex for each of a group
+of edges. As explained in [Edges in GraphAr](https://graphar.apache.org/docs/specification/format#edge-chunks-in-graphar), the edges are separated
+into edge chunks, and each edge chunk has its own adjList table and 0 or
+more property tables.
+
+For example, the file
+[./edge/person_knows_person/ordered_by_source/adj_list/part0/chunk0](https://github.com/apache/incubator-graphar-testing/blob/main/ldbc_sample/csv/edge/person_knows_person/ordered_by_source/adj_list/part0/chunk0)
+saves the adjList of "person_knows_person" edges for the first edge
+chunk of the first vertex chunk, and the adjList type of the edges is
+"ordered_by_source". This adjList table consists of only two columns:
+one for the source and one for the destination; it can be saved in CSV,
+ORC, or Parquet files.
+
+:::note
+
+If the edges are ordered, there may also be offset chunks to construct
+the index for accessing edges of a single vertex. These chunks will
+tore the start offset of each vertex's edges, see
+[./edge/person_knows_person/ordered_by_source/offset/chunk0](https://github.com/apache/incubator-graphar-testing/blob/main/ldbc_sample/csv/edge/person_knows_person/ordered_by_source/offset/chunk0)
+as an example.
+
+:::
+
+## How to Use GAR
+
+### Construct information
+
+It is convenient to construct the GAR metadata and dump it to generate
+information files. We provide an [example
+program](https://github.com/apache/incubator-graphar/blob/main/cpp/examples/construct_info_example.cc)
+located in the source code which shows how to construct and dump the
+files for graph information, vertex information and edge information.
+
+Also, the metadata of a graph can be constructed easily through reading
+the already existed information files, as the following code
+illustrates:
+
+``` C++
+// construct graph information from file
+std::string path = ... // the path of the graph information file (e.g., ldbc_sample.graph.yml)
+auto graph_info = graphar::GraphInfo::Load(path).value();
+
+// get vertex information
+auto vertex_info = graph_info->GetVertexInfo("person");
+if (vertex_info != nullptr) {
+ // use vertex_info ...
+}
+
+// get edge information
+auto edge_info = graph_info->GetEdgeInfo("person", "knows", "person");
+if (edge_info != nullptr) {
+ // use edge_info ...
+}
+```
+
+### Read GAR files
+
+GAR supports the flexible reading of graph data, e.g., allowing to read
+data of a single vertex, a vertex chunk, or all vertices with a specific
+label. In addition, necessary property groups can be selected to read
+and avoid reading all properties from the files. Furthermore, GAR
+provides convenient and flexible access to adjList, offset and property
+chunks for edges.
+
+As a simple case, the following example shows how to read all vertices
+with label "person" of the graph defined by "graph_info" and output the
+values of "id" and "firstName" for each vertex.
+
+``` C++
+graph_info = ...
+auto vertices = graphar::VerticesCollection::Make(graph_info, "person").value();
+
+for (auto it = vertices->begin(); it != vertices->end(); ++it) {
+ // get a vertex and access its data
+ auto vertex = *it;
+ if (vertex.IsValid("id") && vertex.IsValid("firstName"))
+ std::cout << "id=" << vertex.property<int64_t>("id").value() << ", firstName=" << vertex.property<std::string>("firstName").value() << std::endl;
+}
+```
+
+The next example reads all edges with label "person_knows_person" from
+the above graph and outputs the end vertices for each edge.
+
+``` C++
+graph_info = ...
+auto expect = graphar::EdgesCollection::Make(graph_info, "person", "konws", "person", graphar::AdjListType::ordered_by_source);
+auto edges = expect.value();
+
+for (auto it = edges->begin(); it != edges->end(); ++it) {
+ // get an edge and access its data
+ auto edge = *it;
+ std::cout << "src=" << edge.source() << ", dst=" << edge.destination() << std::endl;
+}
+```
+
+### Write GAR files
+
+As same with the readers, the GAR writers provide different-level
+methods to output the graph data in memory into GAR files.
+
+As the simplest cases, the fist example below adds vertices to
+**VerticesBuilder**, and then dumps the data to files; the second
+example constructs a collection of edges and then dumps them.
+
+``` C++
+vertex_info = ...
+prefix = ...
+graphar::builder::VerticesBuilder builder(vertex_info, prefix);
+
+// add a vertex
+graphar::builder::Vertex v;
+v.AddProperty("id", 933);
+v.AddProperty("firstName", "Alice");
+builder.AddVertex(v);
+// add other vertices
+// ...
+
+// write to GAR files
+builder.Dump();
+```
+
+``` C++
+edge_info = ...
+prefix = ...
+vertices_num = ...
+graphar::builder::EdgesBuilder builder(edge_info, prefix, graphar::AdjListType::ordered_by_source, vertices_num);
+
+// add an edge (0 -> 3)
+graphar::builder::Edge e(0, 3);
+e.AddProperty("creationDate", "2011-07-20T20:02:04.233+0000");
+builder.AddEdge(e);
+// add other edges
+// ...
+
+// write to GAR files
+builder.Dump();
+```
+
+### A PageRank Example
+
+Here we will go through an example of out-of-core graph analytic
+algorithms based on GAR which calculates the PageRank. Please look
+[here](https://en.wikipedia.org/wiki/PageRank) if you want a detailed
+explanation of the PageRank algorithm. And the source code can be found
+at
+[pagerank_example.cc](https://github.com/apache/incubator-graphar/blob/main/cpp/examples/pagerank_example.cc).
+
+This program first reads in the graph information file to obtain the
+metadata; then, it constructs the vertex and edge collections to enable
+access to the graph. After that, an implementation of the PageRank
+algorithm is provided, with data for the vertices stored in memory, and
+the edges streamed through disk I/O. Finally, the vertex information
+with type "person" is extended to include a new property named
+"pagerank" (a new vertex information file named
+*person-new-pagerank.vertex.yml* is saved) and the **VerticesBuilder**
+is used to write the results to new generated data chunks.
+
+Please refer to [more examples](examples/out-of-core.md) to learn
+about the other available case studies utilizing GraphAr.
diff --git a/docs/libraries/java/how_to_develop_java.md b/docs/libraries/java/how_to_develop_java.md
new file mode 100644
index 0000000..db2146f
--- /dev/null
+++ b/docs/libraries/java/how_to_develop_java.md
@@ -0,0 +1,106 @@
+---
+id: how_to_develop_java
+title: How to Develop Java Library
+sidebar_position: 1
+---
+
+## Introduction
+
+GraphAr Java library based on GraphAr C++ library and an efficient FFI
+for Java and C++ called
+[FastFFI](https://github.com/alibaba/fastFFI).
+
+### Source Code Level
+
+- Interface
+- Class
+- JNI code
+- GraphAr C++ library
+
+If you want to use classes or functions of GraphAr C++ library through
+JAVA SDK, you only need to write interfaces with annotations. After the
+interfaces are ready, the java code for the interfaces and the the C++
+code which include JNI code for native methods will be automatically
+generated by FastFFI.For annotation’s usage, please refer to
+[FastFFI](https://github.com/alibaba/fastFFI).
+
+### Runtime Level
+
+Interfaces and classes will be compiled to bytecode. Usually, JNI code
+will be compiled to bitcode as a part of dynamic library which can be
+called by native methods directly. If llvm4jni is enable, suitable
+method in JNI will be transferred to bytecode.
+
+For decoupling the implementation of C++ and Java, we use a bridge
+dynamic library called gar-jni to connect them, it will integrate all
+C++ dependencies(e.g. JNI code, GraphAr C++ library and arrow C++) and
+can be called by native methods in Java directly. Most JNI code is
+generated by FastFFI, but some JNI code is written by ourselves, such as
+JNI code for transferring VectorSchemaRoot into arrow::Table.
+
+To build the bridge dynamic library, here is main part of our CMakeLists.txt:
+
+```cmake
+# set auto-generated JNI code and handwriting JNI code as source files
+file(GLOB SOURCES "${CMAKE_CURRENT_SOURCE_DIR}/target/generated-sources/annotations/*.cc" "${CMAKE_CURRENT_SOURCE_DIR}/target/generated-test-sources/test-annotations/*.cc"
+ "${CMAKE_CURRENT_SOURCE_DIR}/src/main/cpp/ffi/*.cc")
+# remove auto-generated JNI code for specific method cause we have handwriting JNI code for it
+list(REMOVE_ITEM SOURCES "${CMAKE_CURRENT_SOURCE_DIR}/target/generated-sources/annotations/jni_com_alibaba_graphar_arrow_ArrowTable_Static_cxx_0x58c7409.cc")
+
+set(LIBNAME "gar-jni")
+
+# find JNI related libraries
+find_package(JNI REQUIRED)
+include_directories(SYSTEM ${JAVA_INCLUDE_PATH})
+include_directories(SYSTEM ${JAVA_INCLUDE_PATH2})
+
+# some JNI code depends on arrow
+find_package(Arrow REQUIRED)
+# build graphar-cpp in specific version
+include(graphar-cpp)
+build_graphar_cpp()
+
+# build the bridge JNI library
+add_library(${LIBNAME} SHARED ${SOURCES})
+# include graphar-cpp headers
+target_include_directories(${LIBNAME} SYSTEM BEFORE PRIVATE ${GAR_INCLUDE_DIR})
+# link graphar-cpp and arrow
+target_link_libraries(${LIBNAME} ${CMAKE_JNI_LINKER_FLAGS} gar_shared)
+target_link_libraries(${LIBNAME} ${CMAKE_JNI_LINKER_FLAGS} Arrow::arrow_static)
+```
+
+More about usage of CMake, please refer to
+[CMake's official website](https://cmake.org/).
+
+## Building GraphAr Java
+
+Please refer to
+[GraphAr Java Library user guide](java.md).
+
+## How To Test
+
+```bash
+$ export GAR_TEST_DATA=$PWD/../testing/
+$ mvn clean test
+```
+
+This will build GraphAr C++ library internally for Java. If you already
+installed GraphAr C++ library in your system, you can append this option
+to skip: `-DbuildGarCPP=OFF`.
+
+## Code Style
+
+We follow
+[AOSP Java code style](https://source.android.com/docs/setup/contribute/code-style).
+To ensure CI for checking code style will pass, please ensure check
+below is success:
+
+```bash
+$ mvn spotless:check
+```
+
+If there are violations, running command below to automatically format:
+
+```bash
+$ mvn spotless:apply
+```
diff --git a/docs/libraries/java/java.md b/docs/libraries/java/java.md
new file mode 100644
index 0000000..66967a5
--- /dev/null
+++ b/docs/libraries/java/java.md
@@ -0,0 +1,215 @@
+---
+id: java
+title: Java Library
+sidebar_position: 2
+---
+
+## Overview
+
+Based on an efficient FFI for Java and C++ called
+[fastFFI](https://github.com/alibaba/fastFFI), the GraphAr Java
+library allows users to write Java for generating, loading and
+transforming GAR files. It consists of several components:
+
+- **Information Classes**: As same with in the C++ library, the
+ information classes are implemented to construct and access the meta
+ information about the **graphs**, **vertices** and **edges** in
+ GraphAr.
+
+- **Writers**: The GraphAr Java writer provides a set of interfaces
+ that can be used to write Apache Arrow VectorSchemaRoot into GAR
+ files. Every time it takes a VectorSchemaRoot as the logical table
+ for a type of vertices or edges, then convert it to ArrowTable, and
+ then dumps it to standard GAR files (CSV, ORC or Parquet files) under
+ the specific directory path.
+
+- **Readers**: The GraphAr Java reader provides a set of interfaces
+ that can be used to read GAR files. It reads a collection of vertices
+ or edges at a time and assembles the result into the ArrowTable.
+ Similar with the reader in the C++ library, it supports the users to
+ specify the data they need, e.g., reading a single property group
+ instead of all properties.
+
+## Get GraphAr Java Library
+
+### Building from source
+
+Only support installing from source currently, but we will support
+installing from Maven in the future.
+
+Firstly, install llvm-11. `LLVM11_HOME` should point to the home of
+LLVM 11. In Ubuntu, it is at `/usr/lib/llvm-11`. Basically, the build
+procedure the following binary:
+
+- `$LLVM11_HOME/bin/clang++`
+- `$LLVM11_HOME/bin/ld.lld`
+- `$LLVM11_HOME/lib/cmake/llvm`
+
+Tips:
+
+- Use Ubuntu as example:
+
+```bash
+$ sudo apt-get install llvm-11 clang-11 lld-11 libclang-11-dev libz-dev -y
+$ export LLVM11_HOME=/usr/lib/llvm-11
+```
+
+- Or compile from source with this [script](https://github.com/alibaba/fastFFI/blob/main/docker/install-llvm11.sh):
+
+```bash
+$ export LLVM11_HOME=/usr/lib/llvm-11
+$ export LLVM_VAR=11.0.0
+$ sudo ./install-llvm11.sh
+```
+
+Make the graphar-java-library directory as the current working
+directory:
+
+```bash
+$ git clone https://github.com/apache/incubator-graphar.git
+$ cd GraphAr
+$ git submodule update --init
+$ cd java
+```
+
+Compile package:
+
+```bash
+$ mvn clean install -DskipTests
+```
+
+This will build GraphAr C++ library internally for Java. If you already installed GraphAr C++ library in your system,
+you can append this option to skip: `-DbuildGarCPP=OFF`.
+
+Then set GraphAr as a dependency in maven project:
+
+
+```xml
+<dependencies>
+ <dependency>
+ <groupId>org.apache.graphar</groupId>
+ <artifactId>gar-java</artifactId>
+ <version>0.1.0</version>
+ </dependency>
+</dependencies>
+```
+
+## How to use
+
+### Information classes
+
+The Java library for GraphAr provides distinct information classes for
+constructing and accessing meta information about graphs, vertices, and
+edges. These classes act as essential parameters for constructing
+readers and writers, and they can be built either from the existing meta
+files (in the Yaml format) or in-memory from scratch.
+
+To construct information from a Yaml file, please refer to the following
+example code.
+
+```java
+// read graph yaml and construct information
+String path = ...; // the path to the yaml file
+Result<GraphInfo> graphInfoResult = GraphInfo.load(path);
+if (!graphInfoResult.hasError()) {
+ GraphInfo graphInfo = graphInfoResult.value();
+ // use information classes
+ StdMap<StdString, VertexInfo> vertexInfos = graphInfo.getVertexInfos();
+ StdMap<StdString, EdgeInfo> edgeInfos = graphInfo.getEdgeInfos();
+}
+```
+
+See [test for
+graphinfo](https://github.com/apache/incubator-graphar/tree/main/java/src/test/java/com/apache/incubator-graphar/graphinfo)
+for the complete example.
+
+### Writers
+
+The GraphAr Java writers wrap C++ interfaces to write arrow::Table into GraphAr
+formatted files in a batch-import fashion. But arrow::Table is not easy
+to build in Java. Instead, the GraphAr Java library provide a static
+method to convert VectorSchemaRoot into arrow::Table. Warning: There are
+some problems concerning this method which lead to memory leaks. We will
+fix it or rewrite writers with Apache arrow Java.
+
+With the VertexWriter, users can specify a particular property group to
+be written into its corresponding chunks, or choose to write all
+property groups. For edge chunks, besides the meta data (edge info), the
+adjList type should also be specified. The adjList/properties can be
+written alone, or alternatively, all adjList, properties, and the offset
+(for CSR and CSC format) chunks can be written simultaneously.
+
+To utilize the GAR Java writer, please refer to the following example
+code.
+
+```java
+// common steps to construct VectorSchemaRoot
+String uri = "file:" + ...; // data source
+ScanOptions options = new ScanOptions(/*batchSize*/ 32768);
+StdSharedPtr<ArrowTable> table = null;
+try (BufferAllocator allocator = new RootAllocator();
+ DatasetFactory datasetFactory =
+ new FileSystemDatasetFactory(
+ allocator, NativeMemoryPool.getDefault(), FileFormat.PARQUET, uri);
+ Dataset dataset = datasetFactory.finish();
+ Scanner scanner = dataset.newScan(options);
+ ArrowReader reader = scanner.scanBatches()) {
+ while (reader.loadNextBatch()) {
+ try (VectorSchemaRoot root = reader.getVectorSchemaRoot()) {
+ // convert VectorSchemaRoot to ArrowTable
+ table = ArrowTable.fromVectorSchemaRoot(allocator, root, reader);
+ }
+ }
+} catch (Exception e) {
+ e.printStackTrace();
+}
+
+// construct writer object
+String path = ...; // file to be wrote
+StdString edgeMetaFile = StdString.create(path);
+StdSharedPtr<Yaml> edgeMeta = Yaml.loadFile(edgeMetaFile).value();
+EdgeInfo edgeInfo = EdgeInfo.load(edgeMeta).value();
+EdgeChunkWriter writer = EdgeChunkWriter.factory.create(
+ edgeInfo, StdString.create("/tmp/"), AdjListType.ordered_by_source);
+
+// write table with writer object
+writer.sortAndWriteAdjListTable(table, 0, 0); // Write adj list of vertex chunk 0 to files
+```
+
+See [test for
+writers](https://github.com/apache/incubator-graphar/tree/main/java/src/test/java/com/apache/incubator-graphar/writers)
+for the complete example.
+
+### Readers
+
+The GraphAr Java reader provides an extensive set of interfaces to read
+GAR files. It reads a collection of vertices or edges at a time as
+ArrowTable. Similar with the reader in C++ library, it supports the
+users to specify the data they need, e.g., a single property group.
+
+To utilize the GAR Java reader, please refer to the following example
+code.
+
+```java
+// construct vertex chunk reader
+GraphInfo graphInfo = ...; // load graph meta info
+StdString label = StdString.create("person");
+StdString propertyName = StdString.create("id");
+if (graphInfo.getVertexInfo(label).hasError()) {
+ // throw Exception or do other things
+}
+PropertyGroup group = graphInfo.getVertexPropertyGroup(label, propertyName).value();
+Result<VertexPropertyArrowChunkReader> maybeReader =
+ GrapharStaticFunctions.INSTANCE.constructVertexPropertyArrowChunkReader(
+ graphInfo, label, group);
+// check reader's status if needed
+VertexPropertyArrowChunkReader reader = maybeReader.value();
+Result<StdSharedPtr<ArrowTable>> result = reader.getChunk();
+// check table's status if needed
+StdSharedPtr<ArrowTable> table = result.value();
+StdPair<Long, Long> range = reader.getRange().value();
+```
+
+See [test for
+readers](https://github.com/apache/incubator-graphar/tree/main/java/src/test/java/com/apache/incubator-graphar/readers)
+for the complete example.
\ No newline at end of file
diff --git a/docs/libraries/pyspark/how-to.md b/docs/libraries/pyspark/how-to.md
new file mode 100644
index 0000000..2df0764
--- /dev/null
+++ b/docs/libraries/pyspark/how-to.md
@@ -0,0 +1,212 @@
+---
+id: how-to
+title: How to use GraphAr PySpark package
+sidebar_position: 1
+---
+
+
+## GraphAr PySpark
+
+``graphar_pyspark`` is implemented as bindings to GraphAr spark scala
+library. You should have ``graphar-0.1.0-SNAPSHOT.jar`` in your
+Apache Spark JVM classpath. Otherwise you will get an exception. To
+add it specify ``config("spark.jars", "path-to-graphar-jar")`` when
+you create a SparkSession:
+
+```python
+from pyspark.sql import SparkSession
+
+spark = (
+ SparkSession
+ .builder
+ .master("local[1]")
+ .appName("graphar-local-tests")
+ .config("spark.jars", "../../spark/graphar/target/graphar-0.1.0-SNAPSHOT.jar")
+ .config("spark.log.level", "INFO")
+ .getOrCreate()
+)
+```
+
+
+ .. rubric:: GraphAr PySpark initialize
+ :name: graphar-pyspark-initialize
+
+## GraphAr PySpark initialize
+
+PySpark bindings are heavily relying on JVM-calls via ``py4j``. To
+initiate all the neccessary things for it just call
+``graphar_pyspark.initialize()``:
+
+```python
+from graphar_pyspark import initialize
+
+initialize(spark)
+```
+
+## GraphAr objects
+
+Now you can import, create and modify all the classes you can
+call from [scala API of GraphAr](https://graphar.apache.org/docs/spark/).
+For simplify using of graphar from python constants, like GAR-types,
+supported file-types, etc. are placed in ``graphar_pyspark.enums``.
+
+```python
+from graphar_pyspark.info import Property, PropertyGroup, AdjList, AdjListType, VertexInfo, EdgeInfo, GraphInfo
+from graphar_pyspark.enums import GarType, FileType
+```
+
+Main objects of GraphAr are the following:
+
+- GraphInfo
+- VertexInfo
+- EdgeInfo
+
+You can check [Scala library documentation](../spark/spark.md)
+for the more detailed information.
+
+
+## Creating objects in graphar_pyspark
+
+GraphAr PySpark package provide two main ways how to initiate
+objects, like ``GraphInfo``:
+
+#. ``from_python(**args)`` when you create an object based on
+ python-arguments
+#. ``from_scala(jvm_ref)`` when you create an object from the
+ corresponded JVM-object (``py4j.java_gateway.JavaObject``)
+
+
+```python
+help(Property.from_python)
+
+Help on method from_python in module graphar_pyspark.info:
+
+from_python(name: 'str', data_type: 'GarType', is_primary: 'bool') -> 'PropertyType' method of builtins.type instance
+ Create an instance of the Class from Python arguments.
+
+ :param name: property name
+ :param data_type: property data type
+ :param is_primary: flag that property is primary
+ :returns: instance of Python Class.
+```
+
+```python
+python_property = Property.from_python(name="my_property", data_type=GarType.INT64, is_primary=False)
+print(type(python_property))
+
+<class 'graphar_pyspark.info.Property'>
+```
+
+You can always get a reference to the corresponding JVM object. For
+example, if you want to use it in your own code and need a direct link
+to the underlaying instance of Scala Class, you can just call
+``to_scala()`` method:
+
+```python
+scala_obj = python_property.to_scala()
+print(type(scala_obj))
+
+<class 'py4j.java_gateway.JavaObject'>
+```
+
+As we already mentioned, you can initialize an instance of the Python
+class from the JVM object:
+
+```python
+help(Property.from_scala)
+
+Help on method from_scala in module graphar_pyspark.info:
+
+ from_scala(jvm_obj: 'JavaObject') -> 'PropertyType' method of builtins.type instance
+ Create an instance of the Class from the corresponding JVM object.
+
+ :param jvm_obj: scala object in JVM.
+ :returns: instance of Python Class.
+```
+
+```python
+python_property = Property.from_scala(scala_obj)
+```
+
+Each public property and method of the Scala API is provided in
+python, but in a pythonic-naming convention. For example, in Scala,
+``Property`` has the following fields:
+
+- name
+- data_type
+- is_primary
+
+For each of such a field in Scala API there is a getter and setter
+methods. You can call them from the Python too:
+
+```python
+python_property.get_name()
+
+'my_property'
+```
+
+You can also modify fields, but be careful: when you modify field of
+instance of the Python class, you modify the underlaying Scala Object
+at the same moment!
+
+```python
+new_name = "my_renamed_property"
+python_property.set_name(new_name)
+python_property.get_name()
+
+'my_renamed_property'
+```
+
+## Loading Info objects from YAML
+
+But manual creating of objects is not a primary way of using GraphAr
+PySpark. ``GraphInfo``, ``VertexInfo`` and ``EdgeInfo`` can be also
+initialized by reading from YAML-files:
+
+```python
+modern_graph_v_person = VertexInfo.load_vertex_info("../../testing/modern_graph/person.vertex.yml")
+modern_graph_e_person_person = EdgeInfo.load_edge_info("../../testing/modern_graph/person_knows_person.edge.yml")
+modern_graph = GraphInfo.load_graph_info("../../testing/modern_graph/modern_graph.graph.yml")
+```
+
+After that you can work with such an objects like regular python
+objects:
+
+
+```python
+print(modern_graph_v_person.dump())
+
+"
+chunk_size: 2
+prefix: vertex/person/
+property_groups:
+ - prefix: id/
+ file_type: csv
+ properties:
+ - is_primary: true
+ name: id
+ data_type: int64
+ - prefix: name_age/
+ file_type: csv
+ properties:
+ - is_primary: false
+ name: name
+ data_type: string
+ - is_primary: false
+ name: age
+ data_type: int64
+label: person
+version: gar/v1
+"
+```
+
+```python
+print(modern_graph_v_person.contain_property("id") is True)
+print(modern_graph_v_person.contain_property("bad_id?") is False)
+
+True
+True
+```
+
+Please, refer to Scala API and examples of GraphAr Spark Scala
+library to see detailed and business-case oriented examples!
diff --git a/docs/libraries/pyspark/pyspark.md b/docs/libraries/pyspark/pyspark.md
new file mode 100644
index 0000000..e2c085e
--- /dev/null
+++ b/docs/libraries/pyspark/pyspark.md
@@ -0,0 +1,111 @@
+---
+id: pyspark
+title: PySpark Library
+sidebar_position: 4
+---
+
+
+:::note
+
+The current policy of GraphAr project is that for Apache Spark
+the main API is Scala Spark API. PySpark API follows scala Spark API.
+Please refer to [GraphAr Spark library](../spark/spark.md)
+for more detailed information about how to use GraphAr with Apache
+Spark.
+
+:::
+
+## Overview
+
+The GraphAr PySpark library is provided for generating, loading and
+transforming GAR files with PySpark.
+
+- **Information Classes**: As same with in the C++ library, the
+ information classes are implemented as a part of the PySpark library
+ for constructing and accessing the meta information about the graphs,
+ vertices and edges in GraphAr.
+- **IndexGenerator**: The IndexGenerator helps to generate the indices
+ for vertex/edge DataFrames. In most cases, IndexGenerator is first
+ utilized to generate the indices for a DataFrame (e.g., from primary
+ keys), and then this DataFrame can be written into GAR files through
+ the writer.
+- **Writer**: The GraphAr PySpark writer provides a set of interfaces
+ that can be used to write Spark DataFrames into GAR files. Every time
+ it takes a DataFrame as the logical table for a type of vertices or
+ edges, assembles the data in specified format (e.g., reorganize the
+ edges in the CSR way) and then dumps it to standard GAR files (CSV,
+ ORC or Parquet files) under the specific directory path.
+- **Reader**: The GraphAr PySpark reader provides a set of interfaces
+ that can be used to read GAR files. It reads a collection of vertices
+ or edges at a time and assembles the result into the Spark DataFrame.
+ Similar with the reader in the C++ library, it supports the users to
+ specify the data they need, e.g., reading a single property group
+ instead of all properties.
+
+## Use Cases
+
+The GraphAr Spark library can be used in a range of scenarios:
+
+- Taking GAR as a data source to execute SQL queries or do graph
+ processing (e.g., using GraphX).
+- Transforming data between GAR and other data sources (e.g., Hive,
+ Neo4j, NebulaGraph, …).
+- Transforming GAR data between different file types (e.g., from ORC to
+ Parquet).
+- Transforming GAR data between different adjList types (e.g., from COO
+ to CSR).
+- Modifying existing GAR data (e.g., adding new vertices/edges).
+
+## Get GraphAr Spark Library
+
+### Building from source
+
+GraphAr PySpark uses poetry as a build system. Please refer to
+[Poetry documentation](https://python-poetry.org/docs/#installation)
+to find the manual how to install this tool. Currently GraphAr PySpark
+is build with Python 3.9 and PySpark 3.2
+
+
+Make the graphar-pyspark-library directory as the current working
+directory:
+
+```bash
+cd GraphAr/pyspark
+```
+
+Build package:
+
+
+```bash
+poetry build
+```
+
+After compilation, a similar file *graphar_pyspark-0.0.1.tar.gz* is
+generated in the directory *pyspark/dist/*.
+
+### Get from PyPI
+
+You cannot install graphar-pyspark from PyPi for now.
+
+
+## How to Use
+
+### Initialization
+
+GraphAr PySpark is not a standalone library but bindings to GraphAr
+Scala. You need to have *graphar-spark-x.x.x.jar* in your *spark-jars*.
+Please refer to [GraphAr scala documentation](../spark/spark.md) to get
+this JAR.
+
+
+```python
+// create a SparkSession from pyspark.sql import SparkSession
+
+spark = ( SparkSession .builder. … .conf(“spark-jars”,
+“path-to-graphara-spark-x.x.x.jar-file”) .getOrCreate() )
+
+from graphar_pyspark import initialize initialize(spark)
+```
+
+After initialization you can use the same API like in GraphAr scala
+library.
diff --git a/docs/libraries/spark/examples.md b/docs/libraries/spark/examples.md
new file mode 100644
index 0000000..ced535e
--- /dev/null
+++ b/docs/libraries/spark/examples.md
@@ -0,0 +1,211 @@
+---
+id: examples
+title: Examples
+sidebar_position: 1
+---
+
+
+## Co-Work with Apache Spark
+
+[Apache Spark](https://spark.apache.org/) is a multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters. The GraphAr Spark library is developed to make the integration of GraphAr with Spark easy. This library allows users to efficiently generate, load, and transform GAR files, and to integrate GraphAr with other Spark-compatible systems.
+
+Examples of this co-working integration have been provided as showcases.
+
+
+### Examples
+
+### Transform GAR files
+
+We provide an example in [TestGraphTransformer.scala][test-graph-transformer], which demonstrates
+how to conduct data transformation at the graph level. [TransformExample.scala][transformer-example] is
+another example for graph data conversion between different file types or different
+adjList types, which is implemented at the vertex/edge table level. To do this,
+the original data is first loaded into a Spark DataFrame using the GraphAr Spark Reader.
+Then, the DataFrame is written into generated GAR files through a GraphAr Spark Writer,
+following the meta data defined in a new information file.
+
+
+### Compute with GraphX
+
+Another important use case of GraphAr is to use it as a data source for graph
+computing or analytics; [ComputeExample.scala][compute-example] provides an example of constructing
+a GraphX graph from reading GAR files and executing a connected-components computation.
+Also, executing queries with Spark SQL and running other graph analytic algorithms
+can be implemented in a similar fashion.
+
+
+### Import/Export graphs of Neo4j
+
+[Neo4j](https://neo4j.com/product/neo4j-graph-database) graph database provides
+a [spark connector](https://neo4j.com/docs/spark/current/overview/) for integration
+with Spark. The Neo4j Spark Connector, combined with the GraphAr Spark library,
+enables us to migrate graph data between Neo4j and GraphAr. This is also a key application
+of GraphAr in which it acts as a persistent storage of graph data in the database.
+
+We provide two example programs that demonstrate how GraphAr can be used in conjunction
+with Neo4j. It utilizes one of the built-in Neo4j datasets, the [Movie Graph](https://neo4j.com/developer/example-data/#built-in-examples),
+which is a mini graph application containing actors and directors that are related through the movies they have collaborated on.
+Given some necessary information like the chunk size, the prefix of the file path, and the file type,
+the program can read the graph data from Neo4j and write it into GraphAr files.
+When exporting graph data from Neo4j and writing to GraphAr, please refer to the following code,
+with [Neo4j2GraphAr.scala][neo4j2graphar] providing a complete example.
+
+```scala
+def main(args: Array[String]): Unit = {
+ // connect to the Neo4j instance
+ val spark = SparkSession.builder()
+ .appName("Neo4j to GraphAr for Movie Graph")
+ .config("neo4j.url", "bolt://localhost:7687")
+ .config("neo4j.authentication.type", "basic")
+ .config("neo4j.authentication.basic.username", sys.env.get("Neo4j_USR").get)
+ .config("neo4j.authentication.basic.password", sys.env.get("Neo4j_PWD").get)
+ .config("spark.master", "local")
+ .getOrCreate()
+
+ // initialize a graph writer
+ val writer: GraphWriter = new GraphWriter()
+
+ // put movie graph data into writer
+ readAndPutDataIntoWriter(writer, spark)
+
+ // write in graphar format
+ val outputPath: String = args(0)
+ val vertexChunkSize: Long = args(1).toLong
+ val edgeChunkSize: Long = args(2).toLong
+ val fileType: String = args(3)
+
+ writer.write(outputPath, spark, "MovieGraph", vertexChunkSize, edgeChunkSize, fileType)
+}
+```
+
+The `readAndPutDataIntoWriter` method read the vertex and edge from Neo4j to DataFrame
+(Refer to [Neo4j docs](https://neo4j.com/docs/spark/current/reading/) for more details),
+and then put then into a GraphAr GraphWriter.
+
+```scala
+def readAndPutDataIntoWriter(writer: GraphWriter, spark: SparkSession): Unit = {
+ // read vertices with label "Person" from Neo4j as a DataFrame
+ val person_df = spark.read.format("org.neo4j.spark.DataSource")
+ .option("query", "MATCH (n:Person) RETURN n.name AS name, n.born as born")
+ .load()
+ // put into writer, vertex label is "Person"
+ writer.PutVertexData("Person", person_df)
+
+ // read vertices with label "Movie" from Neo4j as a DataFrame
+ val movie_df = spark.read.format("org.neo4j.spark.DataSource")
+ .option("query", "MATCH (n:Movie) RETURN n.title AS title, n.tagline as tagline")
+ .load()
+ // put into writer, vertex label is "Movie"
+ writer.PutVertexData("Movie", movie_df)
+
+ // read edges with type "Person"->"PRODUCED"->"Movie" from Neo4j as a DataFrame
+ val produced_edge_df = spark.read.format("org.neo4j.spark.DataSource")
+ .option("query", "MATCH (a:Person)-[r:PRODUCED]->(b:Movie) return a.name as src, b.title as dst")
+ .load()
+ // put into writer, source vertex label is "Person", edge label is "PRODUCED"
+ // target vertex label is "Movie"
+ writer.PutEdgeData(("Person", "PRODUCED", "Movie"), produced_edge_df)
+```
+
+Finally, the `write` method writes the graph data in GraphAr format to the specified path.
+
+Additionally, when importing data from GraphAr to create/update instances in Neo4j, please refer to the following code:
+
+```scala
+def main(args: Array[String]): Unit = {
+ // connect to the Neo4j instance
+ val spark = SparkSession.builder()
+ .appName("GraphAr to Neo4j for Movie Graph")
+ .config("neo4j.url", "bolt://localhost:7687")
+ .config("neo4j.authentication.type", "basic")
+ .config("neo4j.authentication.basic.username", sys.env.get("Neo4j_USR").get)
+ .config("neo4j.authentication.basic.password", sys.env.get("Neo4j_PWD").get)
+ .config("spark.master", "local")
+ .getOrCreate()
+
+ // path to the graph information file
+ val graphInfoPath: String = args(0)
+ val graphInfo = GraphInfo.loadGraphInfo(graphInfoPath, spark)
+
+ val graphData = GraphReader.read(graphInfoPath, spark)
+ val vertexData = graphData._1
+ val edgeData = graphData._2
+
+ putVertexDataIntoNeo4j(graphInfo, vertexData, spark)
+ putEdgeDataIntoNeo4j(graphInfo, vertexData, edgeData, spark)
+}
+```
+
+Pass the graph information file path to `loadGraphInfo` method to get the graph information.
+Then, read the graph data from GraphAr files with `GraphReader` as DataFrame pair,
+`_1` for vertices and `_2` for edges.
+
+The `putVertexDataIntoNeo4j` and `putEdgeDataIntoNeo4j` methods creates or updates the vertices DataFrame and edges DataFrame in Neo4j.
+
+```scala
+def putVertexDataIntoNeo4j(graphInfo: GraphInfo,
+ vertexData: Map[String, DataFrame],
+ spark: SparkSession): Unit = {
+ // write each vertex type to Neo4j
+ vertexData.foreach { case (key, df) => {
+ val primaryKey = graphInfo.getVertexInfo(key).getPrimaryKey()
+ // the vertex index column is not needed in Neo4j
+ // write to Neo4j, refer to https://neo4j.com/docs/spark/current/writing/
+ df.drop(GeneralParams.vertexIndexCol).write.format("org.neo4j.spark.DataSource")
+ .mode(SaveMode.Overwrite)
+ .option("labels", ":" + key)
+ .option("node.keys", primaryKey)
+ .save()
+ }}
+}
+
+def putEdgeDataIntoNeo4j(graphInfo: GraphInfo,
+ vertexData: Map[String, DataFrame],
+ edgeData: Map[(String, String, String), Map[String, DataFrame]],
+ spark: SparkSession): Unit = {
+ // write each edge type to Neo4j
+ edgeData.foreach { case (key, value) => {
+ // key is (source vertex label, edge label, target vertex label)
+ val sourceLabel = key._1
+ val edgeLabel = key._2
+ val targetLabel = key._3
+ val sourcePrimaryKey = graphInfo.getVertexInfo(sourceLabel).getPrimaryKey()
+ val targetPrimaryKey = graphInfo.getVertexInfo(targetLabel).getPrimaryKey()
+ val sourceDf = vertexData(sourceLabel)
+ val targetDf = vertexData(targetLabel)
+ // convert the source and target index column to the primary key column
+ val df = Utils.joinEdgesWithVertexPrimaryKey(value.head._2, sourceDf, targetDf, sourcePrimaryKey, targetPrimaryKey) // use the first DataFrame of (adj_list_type_str, DataFrame) map
+
+ val properties = if (edgeLabel == "REVIEWED") "rating,summary" else ""
+
+ // write to Neo4j, refer to https://neo4j.com/docs/spark/current/writing/
+ df.write.format("org.neo4j.spark.DataSource")
+ .mode(SaveMode.Overwrite)
+ .option("relationship", edgeLabel)
+ .option("relationship.save.strategy", "keys")
+ .option("relationship.source.labels", ":" + sourceLabel)
+ .option("relationship.source.save.mode", "match")
+ .option("relationship.source.node.keys", "src:" + sourcePrimaryKey)
+ .option("relationship.target.labels", ":" + targetLabel)
+ .option("relationship.target.save.mode", "match")
+ .option("relationship.target.node.keys", "dst:" + targetPrimaryKey)
+ .option("relationship.properties", properties)
+ .save()
+ }}
+}
+```
+
+Finally, you will see the graph in Neo4j Browser after running the above code.
+
+See [GraphAr2Neo4j.scala][graphar2neo4j] for the complete example.
+
+> **tip:**
+> - The Neo4j Spark Connector offers different save modes and writing options, such as Append(CREATE) or Overwrite(MERGE). Please refer to its [documentation](https://neo4j.com/docs/spark/current/writing/) for more information and take the most appropriate method while using.
+> - The Neo4j Spark Connector supports to use [Spark structured streaming API](https://neo4j.com/docs/spark/current/streaming), which works differently from Spark batching. One can utilize this API to read/write a stream from/to Neo4j, avoiding to maintain all data in the memory.
+
+
+[test-graph-transformer]: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TestGraphTransformer.scala
+[transformer-example]: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TransformExample.scala
+[compute-example]: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/ComputeExample.scala
+[neo4j2graphar]: https://github.com/apache/incubator-graphar/blob/main/spark/src/main/scala/com/apache/incubator-graphar/example/Neo4j2GraphAr.scala
+[graphar2neo4j]: https://github.com/apache/incubator-graphar/blob/main/spark/src/main/scala/com/apache/incubator-graphar/example/GraphAr2Neo4j.scala
\ No newline at end of file
diff --git a/docs/libraries/spark/spark.md b/docs/libraries/spark/spark.md
new file mode 100644
index 0000000..b313584
--- /dev/null
+++ b/docs/libraries/spark/spark.md
@@ -0,0 +1,237 @@
+---
+id: spark
+title: Spark Library
+sidebar_position: 3
+---
+
+## Overview
+
+The GraphAr Spark library is provided for generating, loading and transforming GAR files with Apache Spark easy. It consists of several components:
+
+- **Information Classes**: As same with in the C++ library, the information classes are implemented as a part of the Spark library for constructing and accessing the meta information about the graphs, vertices and edges in GraphAr.
+- **IndexGenerator**: The IndexGenerator helps to generate the indices for vertex/edge DataFrames. In most cases, IndexGenerator is first utilized to generate the indices for a DataFrame (e.g., from primary keys), and then this DataFrame can be written into GAR files through the writer.
+- **Writer**: The GraphAr Spark writer provides a set of interfaces that can be used to write Spark DataFrames into GAR files. Every time it takes a DataFrame as the logical table for a type of vertices or edges, assembles the data in specified format (e.g., reorganize the edges in the CSR way) and then dumps it to standard GAR files (CSV, ORC or Parquet files) under the specific directory path.
+- **Reader**: The GraphAr Spark reader provides a set of interfaces that can be used to read GAR files. It reads a collection of vertices or edges at a time and assembles the result into the Spark DataFrame. Similar with the reader in the C++ library, it supports the users to specify the data they need, e.g., reading a single property group instead of all properties.
+
+## Use Cases
+
+The GraphAr Spark library can be used in a range of scenarios:
+
+- Taking GAR as a data source to execute SQL queries or do graph processing (e.g., using GraphX).
+- Transforming data between GAR and other data sources (e.g., Hive, Neo4j, NebulaGraph, ...).
+- Transforming GAR data between different file types (e.g., from ORC to Parquet).
+- Transforming GAR data between different adjList types (e.g., from COO to CSR).
+- Modifying existing GAR data (e.g., adding new vertices/edges).
+
+For more information on its usage, please refer to the [Examples](examples.md).
+
+
+## Get GraphAr Spark Library
+
+### Building from source
+
+Make the graphar-spark-library directory as the current working directory:
+
+```bash
+cd GraphAr/spark/
+```
+
+Compile package:
+
+```bash
+mvn clean package -DskipTests
+```
+
+GraphAr supports two Apache Spark versions for now and uses Maven Profiles to work with it. The command above built GraphAr with Spark 3.2.2 by default. To built GraphAr with Spark 3.3.4 use the following command:
+
+```bash
+mvn clean package -DskipTests -P datasources-33
+```
+
+After compilation, a similar file *graphar-x.x.x-SNAPSHOT-shaded.jar* is generated in the directory *spark/graphar/target/*.
+
+Please refer to the [building steps](https://github.com/apache/incubator-graphar/tree/main/spark) for more details.
+
+
+## How to Use
+
+### Information classes
+
+The Spark library for GraphAr provides distinct information classes for constructing and accessing meta information about graphs, vertices, and edges. These classes act as essential parameters for constructing readers and writers, and they can be built either from the existing meta files (in the Yaml format) or in-memory from scratch.
+
+To construct information from a Yaml file, please refer to the following example code.
+
+```scala
+// read graph yaml and construct information
+val spark = ... // the Spark session
+val file_path = ... // the path to the yaml file
+val graph_info = GraphInfo.loadGraphInfo(file_path, spark)
+
+// use information classes
+val vertices = graph_info.getVertices
+val edges = graph_info.getEdges
+val version = graph_info.getVersion
+```
+
+See [TestGraphInfo.scala][test-graph-info] for the complete example.
+
+
+### IndexGenerator
+
+The GraphAr file format assigns each vertex with a unique index inside the vertex type (which called internal vertex id) starting from 0 and increasing continuously for each type of vertex (i.e., with the same vertex label). However, the vertex/edge tables in Spark often lack this information, requiring special attention. For example, an edge table typically uses the primary key (e.g., "id", which is a string) to identify its source and destination vertices.
+
+To address this issue, the GraphAr Spark library offers the IndexGenerator which is used to generate indices for vertex/edge DataFrames. For a vertex DataFrame, a mapping from the primary keys to GAR indices can be constructed, or an index column can be generated directly if no primary keys are available. For an edge DataFrame, source and destination columns can be generated from the vertex index mapping (when the end vertices are represented by the primary keys), or they may be generate [...]
+
+> **tip:**
+> In most cases, IndexGenerator is first utilized to generate the indices for a DataFrame, and then this DataFrame can be written into GAR files through the writer.
+
+To utilize IndexGenerator, please refer to the following example code.
+
+```scala
+// generate indices for vertex DataFrame
+val vertex_df = ...
+val vertex_df_with_index = IndexGenerator.generateVertexIndexColumn(vertex_df)
+
+// generate indices for src & dst columns of edge DataFrame
+val edge_df = ...
+val edge_df_with_index = IndexGenerator.generateSrcAndDstIndexUnitedlyForEdges(edge_df, "src", "dst")
+
+// generate indices for src & dst columns of edge DataFrame from vertex primary keys
+val vertex_df = ...
+val edge_df = ...
+val vertex_mapping = IndexGenerator.constructVertexIndexMapping(vertex_df, "id")
+val edge_df_src_index = IndexGenerator.generateSrcIndexForEdgesFromMapping(edge_df, "src", vertex_mapping)
+val edge_df_src_dst_index = IndexGenerator.generateDstIndexForEdgesFromMapping(edge_df_src_index, "dst", vertex_mapping)
+```
+
+See [TestIndexGenerator.scala][test-index-generator] for the complete example.
+
+
+### Writer
+
+The GraphAr Spark writer provides the necessary Spark interfaces to write DataFrames into GraphAr formatted files in a batch-import fashion. With the VertexWriter, users can specify a particular property group to be written into its corresponding chunks, or choose to write all property groups. For edge chunks, besides the meta data (edge info), the adjList type should also be specified. The adjList/properties can be written alone, or alternatively, all adjList, properties, and the offset [...]
+
+To utilize the GAR Spark writer, please refer to the following example code.
+
+```scala
+// generate the vertex index column for vertex DataFrame
+val vertex_df = ...
+val vertex_df_with_index = IndexGenerator.generateVertexIndexColumn(vertex_df)
+// construct the vertex writer
+val vertex_info = ...
+val prefix = ...
+val writer = new VertexWriter(prefix, vertex_info, vertex_df_with_index)
+// write certain property group
+val property_group = vertex_info.getPropertyGroup("id")
+writer.writeVertexProperties(property_group)
+// write all properties
+writer.writeVertexProperties()
+
+// generate vertex index for edge DataFrame
+val edge_df = ...
+val edge_df_with_index = IndexGenerator.generateSrcAndDstIndexUnitedlyForEdges(edge_df, "src", "dst")
+// construct the edge writer
+val edge_info = ...
+val adj_list_type = AdjListType.ordered_by_source
+val writer = new EdgeWriter(prefix, edge_info, adj_list_type, edge_df_with_index)
+// write adjList
+writer.writeAdjList()
+// write certain property group
+val property_group = edge_info.getPropertyGroup("creationDate", adj_list_type)
+writer.writeEdgeProperties(property_group)
+// write all of adjList and properties
+writer.writeEdges()
+```
+
+See [TestWriter.scala][test-writer] for the complete example.
+
+
+### Reader
+
+The GraphAr Spark reader provides an extensive set of interfaces to read GAR files. It reads a collection of vertices or edges at a time and assembles the result into the Spark DataFrame. Similar with the reader in C++ library, it supports the users to specify the data they need, e.g., a single property group.
+
+After content has been read into the Spark DataFrame, users can leverage it to do graph processing, execute SQL queries or perform various transformations (such as adding new vertices/edges, reorganizing the edge order, and changing the file type) and then write it back into GAR files if desired.
+
+To utilize the GAR Spark reader, please refer to the following example code.
+
+```scala
+// construct the vertex reader
+val prefix = ...
+val vertex_info = ...
+val reader = new VertexReader(prefix, vertex_info, spark)
+val property_group = vertex_info.getPropertyGroup("gender")
+// read a single chunk
+val single_chunk_df = reader.readVertexPropertyChunk(property_group, 0)
+// ...
+// read all property chunks
+val vertex_df = reader.readAllVertexPropertyGroups()
+
+//construct the edge reader
+val edge_info = ...
+val adj_list_type = AdjListType.ordered_by_source
+val reader = new EdgeReader(prefix, edge_info, adj_list_type, spark)
+// read a single adjList chunk
+val single_adj_list_df = reader.readAdjListChunk(2, 0)
+// read all adjList chunks for a vertex chunk
+val adj_list_df_chunk_2 = reader.readAdjListForVertexChunk(2)
+// ...
+// read all edge chunks (including adjList and all properties)
+val edge_df = reader.readEdges()
+```
+
+See [TestReader.scala][test-reader] for the complete example.
+
+
+### Graph-level APIs
+
+To improve the usability of the GraphAr Spark library, a set of APIs are provided to allow users to easily perform operations such as reading, writing, and transforming data at the graph level. These APIs are fairly easy to use, while the previous methods of using reader, writer and information classes are more flexibly and can be highly customized.
+
+The Graph Reader is a helper object which enables users to read all the chunk files from GraphAr for a single graph. The only input required is a GraphInfo object or the path to the information yaml file. On successful completion, it returns a set of vertex DataFrames and edge DataFrames, each of which can be accessed by specifying the vertex/edge label. The Graph Writer is used for writing all vertex DataFrames and edge DataFrames of a graph to generate GraphAr chunk files. For more det [...]
+
+The Graph Transformer is a helper object in the GraphAr Spark library, designed to assist with data transformation at the graph level. It takes two GraphInfo objects (or paths of two yaml files) as inputs: one for the source graph, and one for the destination graph. The transformer will then load data from existing GAR files for the source graph, utilizing the GraphAr Spark Reader and the meta data defined in the source GraphInfo. After reorganizing the data according to the destination [...]
+
+```scala
+// transform graphs by yaml paths
+val spark = ... // the Spark session
+val source_path = ... // e.g., /tmp/source.graph.yml
+val dest_path = ... // e.g., /tmp/dest.graph.yml
+GraphTransformer.transform(source_path, dest_path, spark)
+
+// transform graphs by information objects
+val source_info = ...
+val dest_info = ...
+GraphTransformer.transform(source_info, dest_info, spark)
+```
+
+We provide an example in [TestGraphTransformer.scala][test-graph-transformer], which demonstrates how to conduct data transformation from the [source graph](https://github.com/apache/incubator-graphar-testing/blob/main/ldbc_sample/parquet/ldbc_sample.graph.yml) to the [destination graph](https://github.com/apache/incubator-graphar-testing/blob/main/transformer/ldbc_sample.graph.yml).
+
+The Graph Transformer can be used for various purposes, including transforming GAR data between different file types (e.g. from ORC to Parquet), transforming between different adjList types (e.g. from COO to CSR), selecting properties or regrouping them, and setting a new chunk size.
+
+:::note
+
+There are certain limitations while using the Graph Transformer:
+ - The vertices (or edges) of the source and destination graphs are aligned by labels, meaning each vertex/edge label included in the destination graph must have an equivalent in the source graph, in order for the related chunks to be loaded as the data source.
+ - For each group of vertices/edges (i.e., each single label), each property included in the destination graph (defined in the relevant VertexInfo/EdgeInfo) must also be present in the source graph.
+
+ In addition, users can use the GraphAr Spark Reader/Writer to conduct data transformation more flexibly at the vertex/edge table level, as opposed to the graph level. This allows for a more granular approach to transforming data, as `TransformExample.scala`_ shows.
+
+:::
+
+### More examples
+
+For more information on usage, please refer to the examples:
+
+- [ComputeExample.scala][compute-example] includes an example for constructing the GraphX graph from GAR files and executing a connected-components computation.
+- [TransformExample.scala][transform-example] shows an example for graph data conversion between different file types or different adjList types.
+- [Neo4j2GraphAr.scala][neo4j2graphar] and [GraphAr2Neo4j.scala][graphar2neo4j] are examples to conduct data importing/exporting for Neo4j.
+
+
+[test-graph-info]: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TestGraphInfo.scala
+[test-index-generator]: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TestIndexGenerator.scala
+[test-writer]: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TestWriter.scala
+[test-reader]: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TestReader.scala
+[test-graph-transformer]: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TestGraphTransformer.scala
+[compute-example]: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/ComputeExample.scala
+[transform-example]: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TransformExample.scala
+[neo4j2graphar]: https://github.com/apache/incubator-graphar/blob/main/spark/src/main/scala/com/apache/incubator-graphar/example/Neo4j2GraphAr.scala
+[graphar2neo4j]: https://github.com/apache/incubator-graphar/blob/main/spark/src/main/scala/com/apache/incubator-graphar/example/GraphAr2Neo4j.scala
\ No newline at end of file
diff --git a/docs/overview/concepts.rst b/docs/overview/concepts.md
similarity index 97%
rename from docs/overview/concepts.rst
rename to docs/overview/concepts.md
index 4427163..2039e8d 100644
--- a/docs/overview/concepts.rst
+++ b/docs/overview/concepts.md
@@ -1,5 +1,8 @@
-Concepts
-=========
+---
+id: concepts
+title: Concepts
+sidebar_position: 3
+---
Glossary of relevant concepts and terms.
diff --git a/docs/overview/motivation.rst b/docs/overview/motivation.md
similarity index 90%
rename from docs/overview/motivation.rst
rename to docs/overview/motivation.md
index fa48ee7..a262c87 100644
--- a/docs/overview/motivation.rst
+++ b/docs/overview/motivation.md
@@ -1,5 +1,8 @@
-Motivation
-===========
+---
+id: motivation
+title: Motivation
+sidebar_position: 2
+---
Numerous graph systems,
such as Neo4j, Nebula Graph, and Apache HugeGraph, have been developed in recent years.
@@ -8,4 +11,4 @@ The need for a standard data file format for large-scale graph data storage and
Our aim is to fill this gap and contribute to the open-source community by providing a standard data file format for graph data storage and exchange, as well as for out-of-core querying.
This format, which we have named GraphAr, is engineered to be efficient, cross-language compatible, and to support out-of-core processing scenarios, such as those commonly found in data lakes.
-Furthermore, GraphAr's flexible design ensures that it can be easily extended to accommodate a broader array of graph data storage and exchange use cases in the future.
+Furthermore, GraphAr's flexible design ensures that it can be easily extended to accommodate a broader array of graph data storage and exchange use cases in the future.
\ No newline at end of file
diff --git a/docs/overview/overview.rst b/docs/overview/overview.md
similarity index 69%
rename from docs/overview/overview.rst
rename to docs/overview/overview.md
index 449b848..eebe4c2 100644
--- a/docs/overview/overview.rst
+++ b/docs/overview/overview.md
@@ -1,8 +1,10 @@
-Overview
-=========
+---
+id: overview
+title: Overview
+sidebar_position: 1
+---
-.. image:: ../images/overview.png
- :alt: overview
+<img src="../images/overview.png" alt="overview" width="700"/>
GraphAr is a project to standardize the graph data format and provide a set of libraries to generate, access and transform such formatted files.
@@ -10,7 +12,5 @@ It is intended to serve as the standard file format for importing/exporting and
Additionally, it can also serve as the direct data source for graph processing applications.
-
-`Motivation <motivation.html>`_
-
-`Concepts <concepts.html>`_
+### [Motivation](/docs/overview/motivation)
+### [Concepts](/docs/overview/concepts)
diff --git a/docs/pyspark/api/graphar_pyspark.rst b/docs/pyspark/api/graphar_pyspark.rst
deleted file mode 100644
index 7035cb9..0000000
--- a/docs/pyspark/api/graphar_pyspark.rst
+++ /dev/null
@@ -1,69 +0,0 @@
-graphar\_pyspark package
-========================
-
-Submodules
-----------
-
-graphar\_pyspark.enums module
------------------------------
-
-.. automodule:: graphar_pyspark.enums
- :members:
- :undoc-members:
- :show-inheritance:
-
-graphar\_pyspark.errors module
-------------------------------
-
-.. automodule:: graphar_pyspark.errors
- :members:
- :undoc-members:
- :show-inheritance:
-
-graphar\_pyspark.graph module
------------------------------
-
-.. automodule:: graphar_pyspark.graph
- :members:
- :undoc-members:
- :show-inheritance:
-
-graphar\_pyspark.info module
-----------------------------
-
-.. automodule:: graphar_pyspark.info
- :members:
- :undoc-members:
- :show-inheritance:
-
-graphar\_pyspark.reader module
-------------------------------
-
-.. automodule:: graphar_pyspark.reader
- :members:
- :undoc-members:
- :show-inheritance:
-
-graphar\_pyspark.util module
-----------------------------
-
-.. automodule:: graphar_pyspark.util
- :members:
- :undoc-members:
- :show-inheritance:
-
-graphar\_pyspark.writer module
-------------------------------
-
-.. automodule:: graphar_pyspark.writer
- :members:
- :undoc-members:
- :show-inheritance:
-
-Module contents
----------------
-
-.. automodule:: graphar_pyspark
- :members:
- :undoc-members:
- :show-inheritance:
diff --git a/docs/pyspark/api/modules.rst b/docs/pyspark/api/modules.rst
deleted file mode 100644
index b8915d9..0000000
--- a/docs/pyspark/api/modules.rst
+++ /dev/null
@@ -1,7 +0,0 @@
-graphar_pyspark
-===============
-
-.. toctree::
- :maxdepth: 4
-
- graphar_pyspark
diff --git a/docs/pyspark/how-to.rst b/docs/pyspark/how-to.rst
deleted file mode 100644
index 8f16241..0000000
--- a/docs/pyspark/how-to.rst
+++ /dev/null
@@ -1,318 +0,0 @@
-How to use GraphAr PySpark package
-==================================
-
-.. container:: cell markdown
- :name: b23d0681-da6d-4759-9d62-08d9376712ef
-
- .. rubric:: GraphAr PySpark
- :name: graphar-pyspark
-
- ``graphar_pyspark`` is implemented as bindings to GraphAr spark scala
- library. You should have ``graphar-0.1.0-SNAPSHOT.jar`` in your
- Apache Spark JVM classpath. Otherwise you will get an exception. To
- add it spceify ``config("spark.jars", "path-to-graphar-jar")`` when
- you create a SparkSession:
-
-.. container:: cell code
- :name: 40fa9a16-66b7-44d7-8aff-dd84fed0303a
-
- .. code:: python
-
- from pyspark.sql import SparkSession
-
- spark = (
- SparkSession
- .builder
- .master("local[1]")
- .appName("graphar-local-tests")
- .config("spark.jars", "../../spark/target/graphar-0.1.0-SNAPSHOT.jar")
- .config("spark.log.level", "INFO")
- .getOrCreate()
- )
-
-.. container:: cell markdown
- :name: 1e40491b-9395-469c-bc30-ac4378d11265
-
- .. rubric:: GraphAr PySpark initialize
- :name: graphar-pyspark-initialize
-
- PySpark bindings are heavily relying on JVM-calls via ``py4j``. To
- initiate all the neccessary things for it just call
- ``graphar_pyspark.initialize()``:
-
-.. container:: cell code
- :name: a1ff3f35-2a5a-4111-a296-b678b318b4dd
-
- .. code:: python
-
- from graphar_pyspark import initialize
-
- initialize(spark)
-
-.. container:: cell markdown
- :name: 180b35c8-c0aa-4c6c-abc0-ffbf2ea1d833
-
- .. rubric:: GraphAr objects
- :name: graphar-objects
-
- Now you can import, create and modify all the classes you can work
- call from `scala API of
- GraphAr <https://graphar.apache.org/docs/libraries/cpp>`__.
- For simplify using of graphar from python constants, like GAR-types,
- supported file-types, etc. are placed in ``graphar_pyspark.enums``.
-
-.. container:: cell code
- :name: 85e186a4-0c44-450b-ac9d-d8624bb3d1d1
-
- .. code:: python
-
- from graphar_pyspark.info import Property, PropertyGroup, AdjList, AdjListType, VertexInfo, EdgeInfo, GraphInfo
- from graphar_pyspark.enums import GarType, FileType
-
-.. container:: cell markdown
- :name: 4b0aad82-df2d-47f9-9799-89b45fe61519
-
- Main objects of GraphAr are the following:
-
- - GraphInfo
- - VertexInfo
- - EdgeInfo
-
- You can check `Scala library
- documentation <https://graphar.apache.org/GraphAr/spark/spark-lib.html#information-classes>`__
- for the more detailed information.
-
-.. container:: cell markdown
- :name: 71ac9d59-521c-41bf-a951-b5d08768096e
-
- .. rubric:: Creating objects in graphar_pyspark
- :name: creating-objects-in-graphar_pyspark
-
- GraphAr PySpark package provide two main ways how to initiate
- objects, like ``GraphInfo``:
-
- #. ``from_python(**args)`` when you create an object based on
- python-arguments
- #. ``from_scala(jvm_ref)`` when you create an object from the
- corresponded JVM-object (``py4j.java_gateway.JavaObject``)
-
-.. container:: cell code
- :name: 560cec49-bb31-4ae5-86aa-f9b24642c283
-
- .. code:: python
-
- help(Property.from_python)
-
- .. container:: output stream stdout
-
- ::
-
- Help on method from_python in module graphar_pyspark.info:
-
- from_python(name: 'str', data_type: 'GarType', is_primary: 'bool') -> 'PropertyType' method of builtins.type instance
- Create an instance of the Class from Python arguments.
-
- :param name: property name
- :param data_type: property data type
- :param is_primary: flag that property is primary
- :returns: instance of Python Class.
-
-.. container:: cell code
- :name: 809301c2-89f3-4ea3-9afd-9154be317972
-
- .. code:: python
-
- python_property = Property.from_python(name="my_property", data_type=GarType.INT64, is_primary=False)
- print(type(python_property))
-
- .. container:: output stream stdout
-
- ::
-
- <class 'graphar_pyspark.info.Property'>
-
-.. container:: cell markdown
- :name: 45f0f74d-9568-467f-809a-832f80d5afc6
-
- You can always get a reference to the corresponding JVM object. For
- example, you want to use it in your own code and need a direct link
- to the underlaying instance of Scala Class, you can just call
- ``to_scala()`` method:
-
-.. container:: cell code
- :name: 9c29c329-76ad-4908-84b6-06e004963ae5
-
- .. code:: python
-
- scala_obj = python_property.to_scala()
- print(type(scala_obj))
-
- .. container:: output stream stdout
-
- ::
-
- <class 'py4j.java_gateway.JavaObject'>
-
-.. container:: cell markdown
- :name: 0703a9e0-a48a-4380-8ea6-383cc8164650
-
- As we already mentioned, you can initialize an instance of the Python
- class from the JVM object:
-
-.. container:: cell code
- :name: 25d243a3-645c-4777-b54e-9175b0685c6f
-
- .. code:: python
-
- help(Property.from_scala)
-
- .. container:: output stream stdout
-
- ::
-
- Help on method from_scala in module graphar_pyspark.info:
-
- from_scala(jvm_obj: 'JavaObject') -> 'PropertyType' method of builtins.type instance
- Create an instance of the Class from the corresponding JVM object.
-
- :param jvm_obj: scala object in JVM.
- :returns: instance of Python Class.
-
-.. container:: cell code
- :name: fbea761b-a843-4225-a589-c66f98d7799c
-
- .. code:: python
-
- python_property = Property.from_scala(scala_obj)
-
-.. container:: cell markdown
- :name: 7c54a9b6-29f1-4a57-aa14-30679613b128
-
- Each public property and method of the Scala API is provided in
- python, but in a pythonic-naming convention. For example, in Scala,
- ``Property`` has the following fields:
-
- - name
- - data_type
- - is_primary
-
- For each of such a field in Scala API there is a getter and setter
- methods. You can call them from the Python too:
-
-.. container:: cell code
- :name: ec90236a-cc39-42bc-a5d1-2f57db3a3d8b
-
- .. code:: python
-
- python_property.get_name()
-
- .. container:: output execute_result
-
- ::
-
- 'my_property'
-
-.. container:: cell markdown
- :name: 81f26098-7eb8-4df5-9764-1b2710f8198c
-
- You can also modify fields, but be careful: when you modify field of
- instance of the Python class, you modify the underlaying Scala Object
- in the same moment!
-
-.. container:: cell code
- :name: ea88b175-ed5f-4fe3-8753-84916b52c7f2
-
- .. code:: python
-
- new_name = "my_renamed_property"
- python_property.set_name(new_name)
- python_property.get_name()
-
- .. container:: output execute_result
-
- ::
-
- 'my_renamed_property'
-
-.. container:: cell markdown
- :name: 3b013f4b-ffd6-4de0-9587-d7273cb9c90c
-
- .. rubric:: Loading Info objects from YAML
- :name: loading-info-objects-from-yaml
-
- But manual creating of objects is not a primary way of using GraphAr
- PySpark. ``GraphInfo``, ``VertexInfo`` and ``EdgeInfo`` can be also
- initialized by reading from YAML-files:
-
-.. container:: cell code
- :name: 3a6dafea-346f-4905-84c1-6c5eda86bba4
-
- .. code:: python
-
- modern_graph_v_person = VertexInfo.load_vertex_info("../../testing/modern_graph/person.vertex.yml")
- modern_graph_e_person_person = EdgeInfo.load_edge_info("../../testing/modern_graph/person_knows_person.edge.yml")
- modern_graph = GraphInfo.load_graph_info("../../testing/modern_graph/modern_graph.graph.yml")
-
-.. container:: cell markdown
- :name: 23c145ba-c4ee-43f0-a8d5-62ffebf1ebf3
-
- After that you can work with such an objects like regular python
- objects:
-
-.. container:: cell code
- :name: 87e99095-0a5f-4e84-9e0a-35f97e1bf9f5
-
- .. code:: python
-
- print(modern_graph_v_person.dump())
-
- .. container:: output stream stdout
-
- ::
-
- chunk_size: 2
- prefix: vertex/person/
- property_groups:
- - prefix: id/
- file_type: csv
- properties:
- - is_primary: true
- name: id
- data_type: int64
- - prefix: name_age/
- file_type: csv
- properties:
- - is_primary: false
- name: name
- data_type: string
- - is_primary: false
- name: age
- data_type: int64
- label: person
- version: gar/v1
-
-.. container:: cell code
- :name: b9d44691-d07d-43d5-833c-0ede2d9b99d9
-
- .. code:: python
-
- print(modern_graph_v_person.contain_property("id") is True)
- print(modern_graph_v_person.contain_property("bad_id?") is False)
-
- .. container:: output stream stdout
-
- ::
-
- True
- True
-
-.. container:: cell markdown
- :name: cc992785-eabc-44c2-a63a-b5eebc9af996
-
- Please, refer to Scala API and examples of GraphAr Spark Scala
- library to see detailed and business-case oriented examples!
-
-.. container:: cell code
- :name: c8fbdbb6-6d48-47fa-b69e-8022c8d8f5a1
-
- .. code:: python
diff --git a/docs/pyspark/index.rst b/docs/pyspark/index.rst
deleted file mode 100644
index b37253e..0000000
--- a/docs/pyspark/index.rst
+++ /dev/null
@@ -1,9 +0,0 @@
-PySpark Library
-============================
-
-.. toctree::
- :maxdepth: 2
-
- Getting Started <pyspark-lib>
- How to use <how-to>
- Reference <api/modules>
diff --git a/docs/pyspark/pyspark-lib.rst b/docs/pyspark/pyspark-lib.rst
deleted file mode 100644
index f026743..0000000
--- a/docs/pyspark/pyspark-lib.rst
+++ /dev/null
@@ -1,81 +0,0 @@
-GraphAr PySpark Library
-============================
-
-.. note::
- The current policy of GraphAr project is that for Apache Spark the main API is Scala Spark API. PySpark API follows scala Spark API. Please refer to `GraphAr Spark Scala library <../spark/index.html>`_ for more detailed information about how to use GraphAr with Apache Spark.
-
-Overview
------------
-
-The GraphAr PySpark library is provided for generating, loading and transforming GAR files with PySpark.
-
-- **Information Classes**: As same with in the C++ library, the information classes are implemented as a part of the PySpark library for constructing and accessing the meta information about the graphs, vertices and edges in GraphAr.
-- **IndexGenerator**: The IndexGenerator helps to generate the indices for vertex/edge DataFrames. In most cases, IndexGenerator is first utilized to generate the indices for a DataFrame (e.g., from primary keys), and then this DataFrame can be written into GAR files through the writer.
-- **Writer**: The GraphAr PySpark writer provides a set of interfaces that can be used to write Spark DataFrames into GAR files. Every time it takes a DataFrame as the logical table for a type of vertices or edges, assembles the data in specified format (e.g., reorganize the edges in the CSR way) and then dumps it to standard GAR files (CSV, ORC or Parquet files) under the specific directory path.
-- **Reader**: The GraphAr PySpark reader provides a set of interfaces that can be used to read GAR files. It reads a collection of vertices or edges at a time and assembles the result into the Spark DataFrame. Similar with the reader in the C++ library, it supports the users to specify the data they need, e.g., reading a single property group instead of all properties.
-
-Use Cases
-----------
-
-The GraphAr Spark library can be used in a range of scenarios:
-
-- Taking GAR as a data source to execute SQL queries or do graph processing (e.g., using GraphX).
-- Transforming data between GAR and other data sources (e.g., Hive, Neo4j, NebulaGraph, ...).
-- Transforming GAR data between different file types (e.g., from ORC to Parquet).
-- Transforming GAR data between different adjList types (e.g., from COO to CSR).
-- Modifying existing GAR data (e.g., adding new vertices/edges).
-
-
-Get GraphAr Spark Library
-------------------------------
-
-Building from source
-
-GraphAr PySpark uses poetry as a build system. Please refer to `Poetry documentation <https://python-poetry.org/docs/#installation>`_ to find the manual how to install this tool. Currently GraphAr PySpark is build with Python 3.9 and PySpark 3.2
-
-`````````````````````
-
-Make the graphar-pyspark-library directory as the current working directory:
-
-.. code-block:: shell
-
- cd GraphAr/pyspark
-
-Build package:
-
-.. code-block:: shell
-
- poetry build
-
-After compilation, a similar file *graphar_pyspark-0.0.1.tar.gz* is generated in the directory *pyspark/dist/*.
-
-Get from PyPI
-```````````````
-
-You cannot install graphar-pyspark from PyPi for now.
-
-
-How to Use
------------------
-
-Initialization
-```````````````
-GraphAr PySpark is not a standalone library but bindings to GraphAr Scala. You need to have *graphar-spark-x.x.x.jar* in your *spark-jars*. Please refer to `GraphAr scala documentation <../spark/index>`_ to get this JAR.
-
-.. code-block:: python
-
- // create a SparkSession
- from pyspark.sql import SparkSession
-
- spark = (
- SparkSession
- .builder.
- ...
- .conf("spark-jars", "path-to-graphara-spark-x.x.x.jar-file")
- .getOrCreate()
- )
-
- from graphar_pyspark import initialize
- initialize(spark)
-
-After initialization you can use the same API like in GraphAr scala library.
diff --git a/docs/spark/examples/spark.rst b/docs/spark/examples/spark.rst
deleted file mode 100644
index 4def182..0000000
--- a/docs/spark/examples/spark.rst
+++ /dev/null
@@ -1,211 +0,0 @@
-Co-Work with Apache Spark
-============================
-
-`Apache Spark <https://spark.apache.org/>`_ is a multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters. The GraphAr Spark library is developed to make the integration of GraphAr with Spark easy. This library allows users to efficiently generate, load, and transform GAR files, and to integrate GraphAr with other Spark-compatible systems.
-
-Examples of this co-working integration have been provided as showcases.
-
-
-Examples
-------------------------
-
-Transform GAR files
-`````````````````````
-We provide an example in `TestGraphTransformer.scala`_, which demonstrates
-how to conduct data transformation at the graph level. `TransformExample.scala`_ is
-another example for graph data conversion between different file types or different
-adjList types, which is implemented at the vertex/edge table level. To do this,
-the original data is first loaded into a Spark DataFrame using the GraphAr Spark Reader.
-Then, the DataFrame is written into generated GAR files through a GraphAr Spark Writer,
-following the meta data defined in a new information file.
-
-
-Compute with GraphX
-`````````````````````
-Another important use case of GraphAr is to use it as a data source for graph
-computing or analytics; `ComputeExample.scala`_ provides an example of constructing
-a GraphX graph from reading GAR files and executing a connected-components computation.
-Also, executing queries with Spark SQL and running other graph analytic algorithms
-can be implemented in a similar fashion.
-
-
-Import/Export graphs of Neo4j
-```````````````````````````````
-`Neo4j <https://neo4j.com/product/neo4j-graph-database/>`_ graph database provides
-a `spark connector <https://neo4j.com/docs/spark/current/overview/>`_ for integration
-with Spark. The Neo4j Spark Connector, combined with the GraphAr Spark library,
-enables us to migrate graph data between Neo4j and GraphAr. This is also a key application
-of GraphAr in which it acts as a persistent storage of graph data in the database.
-
-We provide two example programs that demonstrate how GraphAr can be used in conjunction
-with Neo4j. It utilizes one of the built-in Neo4j datasets, the `Movie Graph <https://neo4j.com/developer/example-data/#built-in-examples>`_,
-which is a mini graph application containing actors and directors that are related through the movies they have collaborated on.
-Given some necessary information like the chunk size, the prefix of the file path, and the file type,
-the program can read the graph data from Neo4j and write it into GraphAr files.
-When exporting graph data from Neo4j and writing to GraphAr, please refer to the following code,
-with `Neo4j2GraphAr.scala`_ providing a complete example.
-
-.. code:: scala
-
- def main(args: Array[String]): Unit = {
- // connect to the Neo4j instance
- val spark = SparkSession.builder()
- .appName("Neo4j to GraphAr for Movie Graph")
- .config("neo4j.url", "bolt://localhost:7687")
- .config("neo4j.authentication.type", "basic")
- .config("neo4j.authentication.basic.username", sys.env.get("Neo4j_USR").get)
- .config("neo4j.authentication.basic.password", sys.env.get("Neo4j_PWD").get)
- .config("spark.master", "local")
- .getOrCreate()
-
- // initialize a graph writer
- val writer: GraphWriter = new GraphWriter()
-
- // put movie graph data into writer
- readAndPutDataIntoWriter(writer, spark)
-
- // write in graphar format
- val outputPath: String = args(0)
- val vertexChunkSize: Long = args(1).toLong
- val edgeChunkSize: Long = args(2).toLong
- val fileType: String = args(3)
-
- writer.write(outputPath, spark, "MovieGraph", vertexChunkSize, edgeChunkSize, fileType)
- }
-
-The `readAndPutDataIntoWriter` method read the vertex and edge from Neo4j to DataFrame
-(Refer to `Neo4j docs <https://neo4j.com/docs/spark/current/reading/>`_ for more details),
-and then put then into a GraphAr GraphWriter.
-
-.. code:: scala
-
- def readAndPutDataIntoWriter(writer: GraphWriter, spark: SparkSession): Unit = {
- // read vertices with label "Person" from Neo4j as a DataFrame
- val person_df = spark.read.format("org.neo4j.spark.DataSource")
- .option("query", "MATCH (n:Person) RETURN n.name AS name, n.born as born")
- .load()
- // put into writer, vertex label is "Person"
- writer.PutVertexData("Person", person_df)
-
- // read vertices with label "Movie" from Neo4j as a DataFrame
- val movie_df = spark.read.format("org.neo4j.spark.DataSource")
- .option("query", "MATCH (n:Movie) RETURN n.title AS title, n.tagline as tagline")
- .load()
- // put into writer, vertex label is "Movie"
- writer.PutVertexData("Movie", movie_df)
-
- // read edges with type "Person"->"PRODUCED"->"Movie" from Neo4j as a DataFrame
- val produced_edge_df = spark.read.format("org.neo4j.spark.DataSource")
- .option("query", "MATCH (a:Person)-[r:PRODUCED]->(b:Movie) return a.name as src, b.title as dst")
- .load()
- // put into writer, source vertex label is "Person", edge label is "PRODUCED"
- // target vertex label is "Movie"
- writer.PutEdgeData(("Person", "PRODUCED", "Movie"), produced_edge_df)
-
-Finally, the `write` method writes the graph data in GraphAr format to the specified path.
-
-Additionally, when importing data from GraphAr to create/update instances in Neo4j, please refer to the following code:
-
-.. code:: scala
-
- def main(args: Array[String]): Unit = {
- // connect to the Neo4j instance
- val spark = SparkSession.builder()
- .appName("GraphAr to Neo4j for Movie Graph")
- .config("neo4j.url", "bolt://localhost:7687")
- .config("neo4j.authentication.type", "basic")
- .config("neo4j.authentication.basic.username", sys.env.get("Neo4j_USR").get)
- .config("neo4j.authentication.basic.password", sys.env.get("Neo4j_PWD").get)
- .config("spark.master", "local")
- .getOrCreate()
-
- // path to the graph information file
- val graphInfoPath: String = args(0)
- val graphInfo = GraphInfo.loadGraphInfo(graphInfoPath, spark)
-
- val graphData = GraphReader.read(graphInfoPath, spark)
- val vertexData = graphData._1
- val edgeData = graphData._2
-
- putVertexDataIntoNeo4j(graphInfo, vertexData, spark)
- putEdgeDataIntoNeo4j(graphInfo, vertexData, edgeData, spark)
- }
-
-Pass the graph information file path to `loadGraphInfo` method to get the graph information.
-Then, read the graph data from GraphAr files with `GraphReader` as DataFrame pair,
-`_1` for vertices and `_2` for edges.
-
-The `putVertexDataIntoNeo4j` and `putEdgeDataIntoNeo4j` methods creates or updates the vertices DataFrame and edges DataFrame in Neo4j.
-
-.. code:: scala
-
- def putVertexDataIntoNeo4j(graphInfo: GraphInfo,
- vertexData: Map[String, DataFrame],
- spark: SparkSession): Unit = {
- // write each vertex type to Neo4j
- vertexData.foreach { case (key, df) => {
- val primaryKey = graphInfo.getVertexInfo(key).getPrimaryKey()
- // the vertex index column is not needed in Neo4j
- // write to Neo4j, refer to https://neo4j.com/docs/spark/current/writing/
- df.drop(GeneralParams.vertexIndexCol).write.format("org.neo4j.spark.DataSource")
- .mode(SaveMode.Overwrite)
- .option("labels", ":" + key)
- .option("node.keys", primaryKey)
- .save()
- }}
- }
-
- def putEdgeDataIntoNeo4j(graphInfo: GraphInfo,
- vertexData: Map[String, DataFrame],
- edgeData: Map[(String, String, String), Map[String, DataFrame]],
- spark: SparkSession): Unit = {
- // write each edge type to Neo4j
- edgeData.foreach { case (key, value) => {
- // key is (source vertex label, edge label, target vertex label)
- val sourceLabel = key._1
- val edgeLabel = key._2
- val targetLabel = key._3
- val sourcePrimaryKey = graphInfo.getVertexInfo(sourceLabel).getPrimaryKey()
- val targetPrimaryKey = graphInfo.getVertexInfo(targetLabel).getPrimaryKey()
- val sourceDf = vertexData(sourceLabel)
- val targetDf = vertexData(targetLabel)
- // convert the source and target index column to the primary key column
- val df = Utils.joinEdgesWithVertexPrimaryKey(value.head._2, sourceDf, targetDf, sourcePrimaryKey, targetPrimaryKey) // use the first DataFrame of (adj_list_type_str, DataFrame) map
-
- val properties = if (edgeLabel == "REVIEWED") "rating,summary" else ""
-
- // write to Neo4j, refer to https://neo4j.com/docs/spark/current/writing/
- df.write.format("org.neo4j.spark.DataSource")
- .mode(SaveMode.Overwrite)
- .option("relationship", edgeLabel)
- .option("relationship.save.strategy", "keys")
- .option("relationship.source.labels", ":" + sourceLabel)
- .option("relationship.source.save.mode", "match")
- .option("relationship.source.node.keys", "src:" + sourcePrimaryKey)
- .option("relationship.target.labels", ":" + targetLabel)
- .option("relationship.target.save.mode", "match")
- .option("relationship.target.node.keys", "dst:" + targetPrimaryKey)
- .option("relationship.properties", properties)
- .save()
- }}
- }
-
-Finally, you will see the graph in Neo4j Browser after running the above code.
-
-See `GraphAr2Neo4j.scala`_ for the complete example.
-
-.. tip::
-
- - The Neo4j Spark Connector offers different save modes and writing options, such as Append(CREATE) or Overwrite(MERGE). Please refer to its `documentation <https://neo4j.com/docs/spark/current/writing/>`_ for more information and take the most appropriate method while using.
- - The Neo4j Spark Connector supports to use `Spark structured streaming API <https://neo4j.com/docs/spark/current/streaming/>`_, which works differently from Spark batching. One can utilize this API to read/write a stream from/to Neo4j, avoiding to maintain all data in the memory.
-
-
-.. _TestGraphTransformer.scala: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TestGraphTransformer.scala
-
-.. _TransformExample.scala: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TransformExample.scala
-
-.. _ComputeExample.scala: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/ComputeExample.scala
-
-.. _Neo4j2GraphAr.scala: https://github.com/apache/incubator-graphar/blob/main/spark/src/main/scala/com/apache/incubator-graphar/example/Neo4j2GraphAr.scala
-
-.. _GraphAr2Neo4j.scala: https://github.com/apache/incubator-graphar/blob/main/spark/src/main/scala/com/apache/incubator-graphar/example/GraphAr2Neo4j.scala
diff --git a/docs/spark/index.rst b/docs/spark/index.rst
deleted file mode 100644
index ccbb42e..0000000
--- a/docs/spark/index.rst
+++ /dev/null
@@ -1,9 +0,0 @@
-Spark Library
-============================
-
-.. toctree::
- :maxdepth: 2
-
- Getting Started <spark-lib>
- Examples <examples/spark>
- Reference <reference/index>
diff --git a/docs/spark/reference/index.rst b/docs/spark/reference/index.rst
deleted file mode 100644
index 85d82ae..0000000
--- a/docs/spark/reference/index.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-API Reference (scaladoc)
-==============================
-
-Stub page for the Spark reference docs; actual source is located in the spark-api/ directory.
\ No newline at end of file
diff --git a/docs/spark/spark-lib.rst b/docs/spark/spark-lib.rst
deleted file mode 100644
index 79bad54..0000000
--- a/docs/spark/spark-lib.rst
+++ /dev/null
@@ -1,247 +0,0 @@
-GraphAr Spark Library
-============================
-
-Overview
------------
-
-The GraphAr Spark library is provided for generating, loading and transforming GAR files with Apache Spark easy. It consists of several components:
-
-- **Information Classes**: As same with in the C++ library, the information classes are implemented as a part of the Spark library for constructing and accessing the meta information about the graphs, vertices and edges in GraphAr.
-- **IndexGenerator**: The IndexGenerator helps to generate the indices for vertex/edge DataFrames. In most cases, IndexGenerator is first utilized to generate the indices for a DataFrame (e.g., from primary keys), and then this DataFrame can be written into GAR files through the writer.
-- **Writer**: The GraphAr Spark writer provides a set of interfaces that can be used to write Spark DataFrames into GAR files. Every time it takes a DataFrame as the logical table for a type of vertices or edges, assembles the data in specified format (e.g., reorganize the edges in the CSR way) and then dumps it to standard GAR files (CSV, ORC or Parquet files) under the specific directory path.
-- **Reader**: The GraphAr Spark reader provides a set of interfaces that can be used to read GAR files. It reads a collection of vertices or edges at a time and assembles the result into the Spark DataFrame. Similar with the reader in the C++ library, it supports the users to specify the data they need, e.g., reading a single property group instead of all properties.
-
-Use Cases
-----------
-
-The GraphAr Spark library can be used in a range of scenarios:
-
-- Taking GAR as a data source to execute SQL queries or do graph processing (e.g., using GraphX).
-- Transforming data between GAR and other data sources (e.g., Hive, Neo4j, NebulaGraph, ...).
-- Transforming GAR data between different file types (e.g., from ORC to Parquet).
-- Transforming GAR data between different adjList types (e.g., from COO to CSR).
-- Modifying existing GAR data (e.g., adding new vertices/edges).
-
-For more information on its usage, please refer to the `Examples <examples/spark.html>`_.
-
-
-Get GraphAr Spark Library
-------------------------------
-
-Building from source
-`````````````````````
-
-Make the graphar-spark-library directory as the current working directory:
-
-.. code-block:: shell
-
- cd GraphAr/spark/
-
-Compile package:
-
-.. code-block:: shell
-
- mvn clean package -DskipTests
-
-GraphAr supports two Apache Spark versions for now and uses Maven Profiles to work with it. The command above built GrahpAr with Spark 3.2.2 by default. To built GraphAr with Spark 3.3.4 use the following command:
-
-.. code-block:: shell
-
- mvn clean package -DskipTests -P datasources-33
-
-After compilation, a similar file *graphar-x.x.x-SNAPSHOT-shaded.jar* is generated in the directory *spark/graphar/target/*.
-
-Please refer to the `building steps <https://github.com/apache/incubator-graphar/tree/main/spark>`_ for more details.
-
-
-How to Use
------------------
-
-Information classes
-`````````````````````
-The Spark library for GraphAr provides distinct information classes for constructing and accessing meta information about graphs, vertices, and edges. These classes act as essential parameters for constructing readers and writers, and they can be built either from the existing meta files (in the Yaml format) or in-memory from scratch.
-
-To construct information from a Yaml file, please refer to the following example code.
-
-.. code-block:: scala
-
- // read graph yaml and construct information
- val spark = ... // the Spark session
- val file_path = ... // the path to the yaml file
- val graph_info = GraphInfo.loadGraphInfo(file_path, spark)
-
- // use information classes
- val vertices = graph_info.getVertices
- val edges = graph_info.getEdges
- val version = graph_info.getVersion
-
-See `TestGraphInfo.scala`_ for the complete example.
-
-
-IndexGenerator
-``````````````````
-The GraphAr file format assigns each vertex with a unique index inside the vertex type (which called internal vertex id) starting from 0 and increasing continuously for each type of vertex (i.e., with the same vertex label). However, the vertex/edge tables in Spark often lack this information, requiring special attention. For example, an edge table typically uses the primary key (e.g., "id", which is a string) to identify its source and destination vertices.
-
-To address this issue, the GraphAr Spark library offers the IndexGenerator which is used to generate indices for vertex/edge DataFrames. For a vertex DataFrame, a mapping from the primary keys to GAR indices can be constructed, or an index column can be generated directly if no primary keys are available. For an edge DataFrame, source and destination columns can be generated from the vertex index mapping (when the end vertices are represented by the primary keys), or they may be generate [...]
-
-.. tip::
- In most cases, IndexGenerator is first utilized to generate the indices for a DataFrame, and then this DataFrame can be written into GAR files through the writer.
-
-To utilize IndexGenerator, please refer to the following example code.
-
-.. code-block:: scala
-
- // generate indices for vertex DataFrame
- val vertex_df = ...
- val vertex_df_with_index = IndexGenerator.generateVertexIndexColumn(vertex_df)
-
- // generate indices for src & dst columns of edge DataFrame
- val edge_df = ...
- val edge_df_with_index = IndexGenerator.generateSrcAndDstIndexUnitedlyForEdges(edge_df, "src", "dst")
-
- // generate indices for src & dst columns of edge DataFrame from vertex primary keys
- val vertex_df = ...
- val edge_df = ...
- val vertex_mapping = IndexGenerator.constructVertexIndexMapping(vertex_df, "id")
- val edge_df_src_index = IndexGenerator.generateSrcIndexForEdgesFromMapping(edge_df, "src", vertex_mapping)
- val edge_df_src_dst_index = IndexGenerator.generateDstIndexForEdgesFromMapping(edge_df_src_index, "dst", vertex_mapping)
-
-See `TestIndexGenerator.scala`_ for the complete example.
-
-
-Writer
-``````````````````
-The GraphAr Spark writer provides the necessary Spark interfaces to write DataFrames into GraphAr formatted files in a batch-import fashion. With the VertexWriter, users can specify a particular property group to be written into its corresponding chunks, or choose to write all property groups. For edge chunks, besides the meta data (edge info), the adjList type should also be specified. The adjList/properties can be written alone, or alternatively, all adjList, properties, and the offset [...]
-
-To utilize the GAR Spark writer, please refer to the following example code.
-
-.. code-block:: scala
-
- // generate the vertex index column for vertex DataFrame
- val vertex_df = ...
- val vertex_df_with_index = IndexGenerator.generateVertexIndexColumn(vertex_df)
- // construct the vertex writer
- val vertex_info = ...
- val prefix = ...
- val writer = new VertexWriter(prefix, vertex_info, vertex_df_with_index)
- // write certain property group
- val property_group = vertex_info.getPropertyGroup("id")
- writer.writeVertexProperties(property_group)
- // write all properties
- writer.writeVertexProperties()
-
- // generate vertex index for edge DataFrame
- val edge_df = ...
- val edge_df_with_index = IndexGenerator.generateSrcAndDstIndexUnitedlyForEdges(edge_df, "src", "dst")
- // construct the edge writer
- val edge_info = ...
- val adj_list_type = AdjListType.ordered_by_source
- val writer = new EdgeWriter(prefix, edge_info, adj_list_type, edge_df_with_index)
- // write adjList
- writer.writeAdjList()
- // write certain property group
- val property_group = edge_info.getPropertyGroup("creationDate", adj_list_type)
- writer.writeEdgeProperties(property_group)
- // write all of adjList and properties
- writer.writeEdges()
-
-See `TestWriter.scala`_ for the complete example.
-
-
-Reader
-``````````````````
-The GraphAr Spark reader provides an extensive set of interfaces to read GAR files. It reads a collection of vertices or edges at a time and assembles the result into the Spark DataFrame. Similar with the reader in C++ library, it supports the users to specify the data they need, e.g., a single property group.
-
-After content has been read into the Spark DataFrame, users can leverage it to do graph processing, execute SQL queries or perform various transformations (such as adding new vertices/edges, reorganizing the edge order, and changing the file type) and then write it back into GAR files if desired.
-
-To utilize the GAR Spark reader, please refer to the following example code.
-
-.. code-block:: scala
-
- // construct the vertex reader
- val prefix = ...
- val vertex_info = ...
- val reader = new VertexReader(prefix, vertex_info, spark)
- val property_group = vertex_info.getPropertyGroup("gender")
- // read a single chunk
- val single_chunk_df = reader.readVertexPropertyChunk(property_group, 0)
- // ...
- // read all property chunks
- val vertex_df = reader.readAllVertexPropertyGroups()
-
- //construct the edge reader
- val edge_info = ...
- val adj_list_type = AdjListType.ordered_by_source
- val reader = new EdgeReader(prefix, edge_info, adj_list_type, spark)
- // read a single adjList chunk
- val single_adj_list_df = reader.readAdjListChunk(2, 0)
- // read all adjList chunks for a vertex chunk
- val adj_list_df_chunk_2 = reader.readAdjListForVertexChunk(2)
- // ...
- // read all edge chunks (including adjList and all properties)
- val edge_df = reader.readEdges()
-
-See `TestReader.scala`_ for the complete example.
-
-
-Graph-level APIs
-``````````````````
-To improve the usability of the GraphAr Spark library, a set of APIs are provided to allow users to easily perform operations such as reading, writing, and transforming data at the graph level. These APIs are fairly easy to use, while the previous methods of using reader, writer and information classes are more flexibly and can be highly customized.
-
-The Graph Reader is a helper object which enables users to read all the chunk files from GraphAr for a single graph. The only input required is a GraphInfo object or the path to the information yaml file. On successful completion, it returns a set of vertex DataFrames and edge DataFrames, each of which can be accessed by specifying the vertex/edge label. The Graph Writer is used for writing all vertex DataFrames and edge DataFrames of a graph to generate GraphAr chunk files. For more det [...]
-
-The Graph Transformer is a helper object in the GraphAr Spark library, designed to assist with data transformation at the graph level. It takes two GraphInfo objects (or paths of two yaml files) as inputs: one for the source graph, and one for the destination graph. The transformer will then load data from existing GAR files for the source graph, utilizing the GraphAr Spark Reader and the meta data defined in the source GraphInfo. After reorganizing the data according to the destination [...]
-
-.. code-block:: scala
-
- // transform graphs by yaml paths
- val spark = ... // the Spark session
- val source_path = ... // e.g., /tmp/source.graph.yml
- val dest_path = ... // e.g., /tmp/dest.graph.yml
- GraphTransformer.transform(source_path, dest_path, spark)
-
- // transform graphs by information objects
- val source_info = ...
- val dest_info = ...
- GraphTransformer.transform(source_info, dest_info, spark)
-
-
-We provide an example in `TestGraphTransformer.scala`_, which demonstrates how to conduct data transformation from the `source graph <https://github.com/GraphScope/gar-test/blob/main/ldbc_sample/parquet/ldbc_sample.graph.yml>`_ to the `destination graph <https://github.com/GraphScope/gar-test/blob/main/transformer/ldbc_sample.graph.yml>`_.
-
-The Graph Transformer can be used for various purposes, including transforming GAR data between different file types (e.g. from ORC to Parquet), transforming between different adjList types (e.g. from COO to CSR), selecting properties or regrouping them, and setting a new chunk size.
-
-.. note::
- There are certain limitations while using the Graph Transformer:
-
- - The vertices (or edges) of the source and destination graphs are aligned by labels, meaning each vertex/edge label included in the destination graph must have an equivalent in the source graph, in order for the related chunks to be loaded as the data source.
- - For each group of vertices/edges (i.e., each single label), each property included in the destination graph (defined in the relevant VertexInfo/EdgeInfo) must also be present in the source graph.
-
- In addition, users can use the GraphAr Spark Reader/Writer to conduct data transformation more flexibly at the vertex/edge table level, as opposed to the graph level. This allows for a more granular approach to transforming data, as `TransformExample.scala`_ shows.
-
-
-More examples
-``````````````````
-For more information on usage, please refer to the examples:
-
-- `ComputeExample.scala`_ includes an example for constructing the GraphX graph from GAR files and executing a connected-components computation.
-- `TransformExample.scala`_ shows an example for graph data conversion between different file types or different adjList types.
-- `Neo4j2GraphAr.scala`_ and `GraphAr2Neo4j.scala`_ are examples to conduct data importing/exporting for Neo4j.
-
-
-.. _TestGraphInfo.scala: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TestGraphInfo.scala
-
-.. _TestIndexGenerator.scala: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TestIndexGenerator.scala
-
-.. _TestWriter.scala: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TestWriter.scala
-
-.. _TestReader.scala: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TestReader.scala
-
-.. _TestGraphTransformer.scala: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TestGraphTransformer.scala
-
-.. _ComputeExample.scala: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/ComputeExample.scala
-
-.. _TransformExample.scala: https://github.com/apache/incubator-graphar/blob/main/spark/src/test/scala/com/apache/incubator-graphar/TransformExample.scala
-
-.. _Neo4j2GraphAr.scala: https://github.com/apache/incubator-graphar/blob/main/spark/src/main/scala/com/apache/incubator-graphar/example/Neo4j2GraphAr.scala
-
-.. _GraphAr2Neo4j.scala: https://github.com/apache/incubator-graphar/blob/main/spark/src/main/scala/com/apache/incubator-graphar/example/GraphAr2Neo4j.scala
diff --git a/.github/workflows/license.yml b/docs/specification/_category_.yml
similarity index 52%
copy from .github/workflows/license.yml
copy to docs/specification/_category_.yml
index ab2dea7..bf658b9 100644
--- a/.github/workflows/license.yml
+++ b/docs/specification/_category_.yml
@@ -15,34 +15,9 @@
# specific language governing permissions and limitations
# under the License.
-name: license
-
-on:
- push:
- branches:
- - main
- pull_request:
- branches:
- - main
-
-concurrency:
- group: ${{ github.repository }}-${{ github.event.number || github.head_ref || github.sha }}-${{ github.workflow }}
- cancel-in-progress: true
-
-jobs:
- build:
- runs-on: ubuntu-latest
- steps:
-
- - uses: actions/checkout@v4
- with:
- repository: ${{ github.event.pull_request.head.repo.full_name }}
- ref: ${{ github.event.pull_request.head.ref }}
- submodules: false
- fetch-depth: 0
-
- - name: Check License Header
- uses: apache/skywalking-eyes/header@main
- with:
- config: .licenserc.yaml
- mode: check
+position: 2
+label: 'Specification'
+collapsible: true
+collapsed: false
+link:
+ type: generated-index
diff --git a/docs/format/file-format.rst b/docs/specification/format.md
similarity index 70%
rename from docs/format/file-format.rst
rename to docs/specification/format.md
index 197e226..e482cde 100644
--- a/docs/format/file-format.rst
+++ b/docs/specification/format.md
@@ -1,8 +1,9 @@
-GraphAr File Format
-============================
+---
+title: Format Specification
+sidebar_position: 1
+---
-Property Graph
----------------
+## Property Graph
GraphAr is designed for representing and storing the property graphs. Graph (in discrete mathematics) is a structure made of vertices and edges.
Property graph is then a type of graph model where the vertices/edges could carry a name (also called as type or label) and some properties.
@@ -26,14 +27,10 @@ Each edge contains a unique identifier and:
The following is an example property graph containing two types of vertices ("person" and "comment") and three types of edges.
-.. image:: ../images/property_graph.png
- :width: 700
- :align: center
- :alt: property graph
+<img src="../images/property_graph.png" alt="property graph" width="700" align="center"/>
+## Property Data Types
-Property Data Types
--------------------
GraphAr support a set of built-in property data types that are common in real use cases and supported by most file types (CSV, ORC, Parquet), includes:
- **Boolean**
@@ -50,11 +47,10 @@ GraphAr support a set of built-in property data types that are common in real us
GraphAr also supports the user-defined data types, which can be used to represent complex data structures,
such as the struct, map, and union types.
-Configurations
---------------
+## Configurations
+
+### Vertex Chunk Size
-Vertex Chunk Size
-`````````````````
The vertex chunk size is a configuration parameter that determines the number of vertices in a vertex chunk
and used to partition the logical vertex table into multiple physical vertex tables.
@@ -62,8 +58,7 @@ The vertex chunk size should be set to a value that is large enough to reduce th
but small enough to avoid reading/writing too many vertices at once. We recommend setting the vertex chunk size to
empirical value 2^18 (262,144) for most cases.
-Edge Chunk Size
-````````````````
+### Edge Chunk Size
The edge chunk size is a configuration parameter that determines the number of edges in an edge chunk
and used to partition the logical edge table into multiple physical edge tables.
@@ -72,8 +67,8 @@ The edge chunk size should be set to a value that is large enough to reduce the
but small enough to avoid reading/writing too many edges at once. We recommend setting the edge chunk size to
empirical value 2^22 (4,194,304) for most cases.
-Data File Format
-````````````````
+### Data File Format
+
GraphAr supports multiple file formats for storing the actual data of vertices and edges,
including Apache ORC, Apache Parquet, CSV, and JSON.
@@ -81,8 +76,8 @@ The file format should be chosen based on the specific use case and the data pro
process the graph data. For example, if the graph data will be processed using Apache Spark,
then the Apache Parquet file format is recommended.
-Adjacency List Type
-````````````````````
+## Adjacency List Type
+
Adjacency list is a data structure used to represent the edges of a graph. GraphAr supports multiple types of adjacency lists for a given group of edges, including:
- **ordered_by_source**: all the edges in the logical table are ordered and further partitioned by the internal vertex id of the source, which can be seen as the CSR format.
@@ -91,55 +86,48 @@ Adjacency list is a data structure used to represent the edges of a graph. Graph
- **unordered_by_dest**: the internal id of the destination vertex is used as the partition key to divide the edges into different sub-logical-tables, and the edges in each sub-logical-table are unordered, which can also be seen as the COO format.
-Vertex Chunks in GraphAr
-------------------------
+## Vertex Chunks in GraphAr
+
+### Logical table of vertices
-Logical table of vertices
-`````````````````````````
Each type of vertices (with the same label) constructs a logical vertex table, with each vertex assigned with a global index inside this type (called internal vertex id) starting from 0, corresponding to the row number of the vertex in the logical vertex table. An example layout for a logical table of vertices under the label "person" is provided for reference.
Given an internal vertex id and the vertex label, a vertex is uniquely identifiable and its respective properties can be accessed from this table. The internal vertex id is further used to identify the source and destination vertices when maintaining the topology of the graph.
-.. image:: ../images/vertex_logical_table.png
- :width: 650
- :align: center
- :alt: vertex logical table
+<img src="../images/vertex_logical_table.png" alt="vertex logical table" width="700" align="center"/>
+
+:::note
-.. note::
+In the logical vertex table, some property can be marked as the primary key, such as the "id" column of the "person" table.
- In the logical vertex table, some property can be marked as the primary key, such as the "id" column of the "person" table.
+:::
+### Physical table of vertices
-Physical table of vertices
-``````````````````````````
The logical vertex table will be partitioned into multiple continuous vertex chunks for enhancing the reading/writing efficiency. To maintain the ability of random access, the size of vertex chunks for the same label is fixed. To support to access required properties avoiding reading all properties from the files, and to add properties for vertices without modifying the existing files, the columns of the logical table will be divided into several column groups.
Take the "person" vertex table as an example, if the chunk size is set to be 500, the logical table will be separated into sub-logical-tables of 500 rows with the exception of the last one, which may have less than 500 rows. The columns for maintaining properties will also be divided into distinct groups (e.g., 2 for our example). As a result, a total of 4 physical vertex tables are created for storing the example logical table, which can be seen from the following figure.
-.. image:: ../images/vertex_physical_table.png
- :width: 650
- :align: center
- :alt: vertex physical table
+<img src="../images/vertex_physical_table.png" alt="vertex physical table" width="700" align="center"/>
+
+::: note
+For efficiently utilize the filter push-down of the payload file format like Parquet, the internal vertex id is stored in the payload file as a column. And since the internal vertex id is continuous, the payload file format can use the delta encoding for the internal vertex id column, which would not bring too much overhead for the storage.
-**Note**: For efficiently utilize the filter push-down of the payload file format like Parquet, the internal vertex id is stored in the payload file as a column. And since the internal vertex id is continuous, the payload file format can use the delta encoding for the internal vertex id column, which would not bring too much overhead for the storage.
+:::
-Edge Chunks in GraphAr
-------------------------
+## Edge Chunks in GraphAr
-Logical table of edges
-``````````````````````
-For maintaining a type of edges (that with the same triplet of the source label, edge label, and destination label), a logical edge table is established. And in order to support quickly creating a graph from the graph storage file, the logical edge table could maintain the topology information in a way similar to CSR/CSC (learn more about `CSR/CSC <https://en.wikipedia.org/wiki/Sparse_matrix>`_), that is, the edges are ordered by the internal vertex id of either source or destination. I [...]
+### Logical table of edges
+
+For maintaining a type of edges (that with the same triplet of the source label, edge label, and destination label), a logical edge table is established. And in order to support quickly creating a graph from the graph storage file, the logical edge table could maintain the topology information in a way similar to CSR/CSC (learn more about [CSR/CSC](https://en.wikipedia.org/wiki/Sparse_matrix)), that is, the edges are ordered by the internal vertex id of either source or destination. In [...]
Take the logical table for "person likes person" edges as an example, the logical edge table looks like:
-.. image:: ../images/edge_logical_table.png
- :width: 650
- :align: center
- :alt: edge logical table
+<img src="../images/edge_logical_table.png" alt="edge logical table" width="700" align="center"/>
+
+### Physical table of edges
-Physical table of edges
-```````````````````````
As same with the vertex table, the logical edge table is also partitioned into some sub-logical-tables, with each sub-logical-table contains edges that the source (or destination) vertices are in the same vertex chunk. According to the partition strategy and the order of the edges, edges can be stored in GraphAr following the setting adjacency list type.
After that, the whole logical table of edges will be divided into multiple sub-logical-tables with each sub-logical-table contains edges that the source (or destination) vertices are in the same vertex chunk. Then, a sub-logical-table is further divided into edge chunks of a predefined, fixed number of rows (referred to as edge chunk size). Finally, an edge chunk is separated into physical tables in the following way:
@@ -151,24 +139,16 @@ Additionally, there would be an offset table for **ordered_by_source** or **orde
Take the "person knows person" edges to illustrate. Suppose the vertex chunk size is set to 500 and the edge chunk size is 1024, and the edges are **ordered_by_source**, then the edges could be saved in the following physical tables:
-.. image:: ../images/edge_physical_table1.png
- :width: 650
- :align: center
- :alt: edge physical table1
+<img src="../images/edge_physical_table1.png" alt="edge physical table1" width="700" align="center"/>
-.. image:: ../images/edge_physical_table2.png
- :width: 650
- :align: center
- :alt: edge physical table2
+<img src="../images/edge_physical_table2.png" alt="edge physical table2" width="700" align="center"/>
-.. tip::
+> **Tip:** When the edge type is **ordered_by_source**, the sorted adjList table together with the offset table can be used as CSR, supporting the fast access of the outgoing edges for a given vertex. Similarly, a CSC view can be constructed by sorting the edges by destination and recording corresponding offsets, supporting the fast access of the incoming edges for a given vertex.
- When the edge type is **ordered_by_source**, the sorted adjList table together with the offset table can be used as CSR, supporting the fast access of the outgoing edges for a given vertex. Similarly, a CSC view can be constructed by sorting the edges by destination and recording corresponding offsets, supporting the fast access of the incoming edges for a given vertex.
+## Information files
-Information files
-------------------
GraphAr uses two kinds of files to store a graph: a group of Yaml files to describe metadata information; and data files to store actual data for vertices and edges.
-A graph information file which named "<name>.graph.yml" describes the meta information for a graph whose name is <name>. The content of this file includes:
+A graph information file which named "\<name\>.graph.yml" describes the meta information for a graph whose name is \<name\>. The content of this file includes:
- the graph name;
- the root directory path of the data files;
@@ -176,7 +156,7 @@ A graph information file which named "<name>.graph.yml" describes the meta infor
- the version of GraphAr.
- extra information for the graph, could be used for user defined information.
-A vertex information file which named "<label>.vertex.yml" defines a single group of vertices with the same vertex label <label>, and all vertices in this group have the same schema. The file defines:
+A vertex information file which named "\<label\>.vertex.yml" defines a single group of vertices with the same vertex label \<label\>, and all vertices in this group have the same schema. The file defines:
- the vertex label;
- the vertex chunk size;
@@ -184,7 +164,7 @@ A vertex information file which named "<label>.vertex.yml" defines a single grou
- the property groups attached: each property group has its own file type and the prefix for the path of its data files, it also lists all properties in this group, with every property containing its own name, data type, flagging of whether it is the primary key or not and flagging of whether it is nullable or not for non-primary key properties;
- the version of GraphAr.
-An edge information file which named "<source label>_<edge label>_<destination label>.edge.yml" defines a single group of edges with specific label for source vertex, destination vertex and the edge. It describes the meta information for these edges, includes:
+An edge information file which named "\<source label\>_\<edge label\>_\<destination label\>.edge.yml" defines a single group of edges with specific label for source vertex, destination vertex and the edge. It describes the meta information for these edges, includes:
- the source/edge/destination labels;
- the edge chunk size, the source vertex chunk size and the destination vertex chunk size;
@@ -194,25 +174,27 @@ An edge information file which named "<source label>_<edge label>_<destination l
- the property groups attached to the edge for all adjLists;
- the version of GraphAr.
-.. note::
- Please note that GraphAr supports the storage of multiple types of adjLists for a given group of edges, e.g., a group of edges could be accessed in both CSR and CSC way when two copies (one is **ordered_by_source** and the other is **ordered_by_dest**) of the relevant data are present in GraphAr.
+:::note
+
+Please note that GraphAr supports the storage of multiple types of adjLists for a given group of edges, e.g., a group of edges could be accessed in both CSR and CSC way when two copies (one is **ordered_by_source** and the other is **ordered_by_dest**) of the relevant data are present in GraphAr.
+
+:::
+
+See also [Gar Information Files](https://graphar.apache.org/docs/libraries/cpp/getting-started#gar-information-files) for an example.
-See also `Gar Information Files <cpp/getting-started.html#gar-information-files>`_ for an example.
+## Data files
-Data files
-----------
As previously mentioned, each logical vertex/edge table is divided into multiple physical tables stored in one of the following file formats:
-- `Apache ORC <https://orc.apache.org/>`_
-- `Apache Parquet <https://parquet.apache.org/>`_
+- [Apache ORC](https://orc.apache.org/)
+- [Apache Parquet](https://parquet.apache.org/)
- CSV
- JSON
-Both of Apache ORC and Apache Parquet are column-oriented data storage formats. In practice of graph processing, it is common to only query a subset of columns of the properties. Thus, the column-oriented formats are more efficient, which eliminate the need to read columns that are not relevant. They are also used by a large number of data processing frameworks like `Apache Spark <https://spark.apache.org/>`_, `Apache Hive <https://hive.apache.org/>`_, `Apache Flink <https://flink.apache [...]
+Both of Apache ORC and Apache Parquet are column-oriented data storage formats. In practice of graph processing, it is common to only query a subset of columns of the properties. Thus, the column-oriented formats are more efficient, which eliminate the need to read columns that are not relevant. They are also used by a large number of data processing frameworks like [Apache Spark](https://spark.apache.org/), [Apache Hive](https://hive.apache.org/), [Apache Flink](https://flink.apache.org [...]
-See also `Gar Data Files <cpp/getting-started.html#gar-data-files>`_ for an example.
+See also [GraphAr Data Files](https://graphar.apache.org/docs/libraries/cpp/getting-started#gar-data-files) for an example.
-Implementation
---------------
-The GraphAr libraries may implement part of the GraphAr format. The implementation status of the GraphAr libraries can refer to the `GraphAr implementation status <status.html>`_.
+## Implementation
+The GraphAr libraries may implement part of the GraphAr format. The implementation status of the GraphAr libraries can refer to the [GraphAr implementation status](implementation-status.md).
diff --git a/docs/specification/implementation-status.md b/docs/specification/implementation-status.md
new file mode 100644
index 0000000..a849119
--- /dev/null
+++ b/docs/specification/implementation-status.md
@@ -0,0 +1,253 @@
+---
+title: Implementation Status
+sidebar_position: 2
+---
+
+The following tables summarize the features available in the various official GraphAr libraries.
+All libraries currently follow version 1.0.0 of the GraphAr format.
+
+## Data Types
+
+| Data type (primitive) | C++ | Java | Scala | Python |
+| --------------------- | --- | ---- | ----- | ------ |
+| Boolean | ✓ | ✓ | ✓ | ✓ |
+| Int32 | ✓ | ✓ | ✓ | ✓ |
+| Int64 | ✓ | ✓ | ✓ | ✓ |
+| Float | ✓ | ✓ | ✓ | ✓ |
+| Double | ✓ | ✓ | ✓ | ✓ |
+| String | ✓ | ✓ | ✓ | ✓ |
+| Date | ✓ | | | |
+| Timestamp | ✓ | | | |
+| Time | | | | |
+
+| Data type (nested) | C++ | Java | Scala | Python |
+| --------------------- | --- | ---- | ----- | ------ |
+| List (*) | ✓ | | | |
+
+:::note
+
+- \(\*) The data type of List is not supported by the CSV payload file format.
+
+:::
+
+## Payload Data File Formats
+
+| Format | C++ | Java | Scala | Python |
+|-----------------------------|---------|---------|-------|------------|
+| CSV | R/W | R (1) | R/W | R/W (2) |
+| ORC | R/W | R (1) | R/W | R/W (2) |
+| Parquet | R/W | R (1) | R/W | R/W (2) |
+| Avro | | | | |
+| HDF5 | | | | |
+| JSON | | | | |
+
+:::note
+
+- *R* - Read supported
+- *W* - Write supported
+
+:::
+
+Supported compression methods for the file formats:
+
+| Compression | C++ | Java | Scala | Python |
+|-----------------------------|---------|---------|-------|------------|
+| ZSTD (*) | ✓ | ✓ | ✓ | ✓ |
+
+:::note
+
+- \(\*) Compression is not supported by the CSV payload file format.
+
+:::
+
+
+## Property
+
+| Property feature | C++ | Java | Scala | Python |
+|-------------------|-------|-------|-------|------------|
+| primary key | ✓ | ✓ | ✓ | ✓ |
+| nullable | ✓ | | ✓ | ✓ |
+
+
+Supported operations in Property:
+
+| Property operation| C++ | Java | Scala | Python |
+|-------------------|-------|-------|-------|------------|
+| create | ✓ | ✓ (1) | ✓ | ✓ (2) |
+| get_name | ✓ | ✓ (1) | ✓ | ✓ (2) |
+| is_primary_key | ✓ | ✓ (1) | ✓ | ✓ (2) |
+| is_nullable | ✓ | | ✓ | ✓ (2) |
+
+
+## Property Group
+
+| Property Group (operation) | C++ |Java (1)| Scala | Python (2)|
+|-------------------|-------|--------|-------|------------|
+| create | ✓ | ✓ | ✓ | ✓ |
+| add property | ✓ | ✓ | ✓ | ✓ |
+| remove property | | | | |
+| get properties | ✓ | ✓ | ✓ | ✓ |
+| check property | ✓ | ✓ | | |
+| get file type | ✓ | ✓ | ✓ | ✓ |
+| get path prefix | ✓ | ✓ | ✓ | ✓ |
+| check validation | ✓ | | | |
+
+
+## Adjacency List
+
+| Adjacency List (type) | C++ | Java | Scala | Python |
+|-------------------|-------|-------|-------|------------|
+| CSR | ✓ | ✓ | ✓ | ✓ |
+| CSC | ✓ | ✓ | ✓ | ✓ |
+| COO | ✓ | ✓ | ✓ | ✓ |
+
+Supported operations in Adjacency List:
+
+| Adjacency List (operation) | C++ |Java (1)| Scala | Python (2)|
+|-------------------|-------|--------|-------|------------|
+| create | ✓ | | ✓ | ✓ |
+| get adjacency type| ✓ | | ✓ | ✓ |
+| get file type | ✓ | | ✓ | ✓ |
+| get path prefix | ✓ | | ✓ | ✓ |
+| check validation | ✓ | | | |
+
+
+## Vertex
+
+Vertex features:
+
+| Vertex feature | C++ | Java | Scala | Python |
+|-------------------|-------|-------|-------|------------|
+| label | ✓ | ✓ | ✓ | ✓ |
+| tag | | | | |
+| chunk based | ✓ | ✓ | ✓ | ✓ |
+| property group | ✓ | ✓ | ✓ | ✓ |
+
+:::note
+
+* *label* is the vertex label, which is a unique identifier for the vertex.
+* *tag* is the vertex tag, which is tag or category for the vertex.
+
+:::
+
+Supported operations in Vertex Info:
+
+| Vertex Info (operation) | C++ |Java (1)| Scala | Python (2) |
+|-------------------|-------|--------|-------|------------|
+| create | ✓ | ✓ | ✓ | ✓ |
+| add group | ✓ | ✓ | ✓ | ✓ |
+| remove group | | | | |
+| get label | ✓ | ✓ | ✓ | ✓ |
+| get chunk size | ✓ | ✓ | ✓ | ✓ |
+| get groups | ✓ | ✓ | ✓ | ✓ |
+| get path prefix | ✓ | ✓ | ✓ | ✓ |
+| check property | ✓ | ✓ | ✓ | ✓ |
+| check validation | ✓ | | ✓ | ✓ |
+| serialize | ✓ | ✓ | ✓ | ✓ |
+| deserialize | ✓ | ✓ | ✓ | ✓ |
+
+
+## Edge
+
+Edge features:
+
+| Edge feature | C++ | Java | Scala | Python |
+|-------------------|-------|-------|-------|------------|
+| label | ✓ | ✓ | ✓ | ✓ |
+| chunk based | ✓ | ✓ | ✓ | ✓ |
+| property group | ✓ | ✓ | ✓ | ✓ |
+| adjacent list | ✓ | ✓ | ✓ | ✓ |
+| directed | ✓ | ✓ | ✓ | ✓ |
+
+Supported operations in Edge Info:
+
+| Edge Info (operation) | C++ |Java (1)| Scala | Python (2) |
+|-------------------|-------|--------|-------|------------|
+| create | ✓ | ✓ | ✓ | ✓ |
+| add group | ✓ | ✓ | ✓ | ✓ |
+| remove group | | | | |
+| add adj list | ✓ | ✓ | ✓ | ✓ |
+| remove adj list | | | | |
+| get label | ✓ | ✓ | ✓ | ✓ |
+| get source label | ✓ | ✓ | ✓ | ✓ |
+| get dest label | ✓ | ✓ | ✓ | ✓ |
+| get chunk size | ✓ | ✓ | ✓ | ✓ |
+| get source chunk size | ✓ | ✓ | ✓ | ✓ |
+| get dest chunk size | ✓ | ✓ | ✓ | ✓ |
+| get groups | ✓ | ✓ | ✓ | ✓ |
+| check adj list | ✓ | ✓ | ✓ | ✓ |
+| check property | ✓ | ✓ | ✓ | ✓ |
+| get file type | ✓ | ✓ | ✓ | ✓ |
+| get path prefix | ✓ | ✓ | ✓ | ✓ |
+| is directed | ✓ | ✓ | ✓ | ✓ |
+| check validation | ✓ | | ✓ | ✓ |
+| serialize | ✓ | ✓ | ✓ | ✓ |
+| deserialize | ✓ | ✓ | ✓ | ✓ |
+
+:::note
+
+- *\<source label, label, dest label\>* is the unique identifier for the edge type.
+
+:::
+
+
+## Graph
+
+| Graph | C++ | Java | Scala | Python |
+|-------------------|-------|-------|-------|------------|
+| labeled vertex (with property) | ✓ | ✓ | ✓ | ✓ |
+| labeled edge (with property) | ✓ | ✓ | ✓ | ✓ |
+| extra info | ✓ | | | |
+
+Supported operations in Graph Info:
+
+| Graph Info (operation) | C++ |Java (1)| Scala | Python (2) |
+|-------------------|-------|--------|-------|------------|
+| create | ✓ | ✓ | ✓ | ✓ |
+| add vertex | ✓ | ✓ | ✓ | ✓ |
+| remove vertex | | | | |
+| add edge | ✓ | ✓ | ✓ | ✓ |
+| remove edge | | | | |
+| get name | ✓ | ✓ | ✓ | ✓ |
+| get vertex | ✓ | ✓ | ✓ | ✓ |
+| get edge | ✓ | ✓ | ✓ | ✓ |
+| add extra info | | | | |
+| remove extra info | | | | |
+| get extra info | ✓ | | | |
+| check validation | ✓ | | | |
+| serialize | ✓ | ✓ | ✓ | ✓ |
+| deserialize | ✓ | ✓ | ✓ | ✓ |
+
+:::note
+
+- \(1) Through fastFFI bindings to the GraphAr C++ library.
+
+- \(2) Through py4j bindings to the GraphAr Spark library.
+
+:::
+
+
+## Libraries Version Compatibility
+
+| GraphAr C++ Version | C++ | CMake | Format Version |
+|---------------------|-----|-------|----------------|
+| 0.11.x | 17+ | 2.8+ | 1.0.0 |
+
+| GraphAr Java Version | Java | Maven | Format Version |
+|----------------------|------|-------|----------------|
+| 0.1.0 | 1.8 | 3.6+ | 1.0.0 |
+
+| GraphAr Spark Version | Apache Spark Version | Scala Version | Java Version | Hadoop Version | Format Version |
+|-----------------------|----------------------|---------------|--------------|----------------|----------------|
+| 0.1.0 | 3.2.x-3.3.x | 2.12.x | 1.8, 11 | 3 | 1.0.0 |
+
+| GraphAr PySpark Version | Python Version | PySpark Version | Hadoop Version | Format Version |
+|-------------------------|----------------|-----------------|----------------|----------------|
+| 0.1.0 | 3.8+ | 3.2.x | 3 | 1.0.0 |
+
+:::note
+
+- Since the GraphAr PySpark library is bindings to the GraphAr Spark library,
+ the PySpark version should be compatible with the Spark version.
+
+:::
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org