You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by su...@apache.org on 2019/03/07 01:10:49 UTC
[incubator-pinot] branch docs created (now 862aa7e)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a change to branch docs
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
at 862aa7e readthedocs: Fill in dev-guide sections
This branch includes the following new commits:
new 862aa7e readthedocs: Fill in dev-guide sections
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 01/01: readthedocs: Fill in dev-guide sections
Posted by su...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch docs
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 862aa7e7bfbc4ad0aa3b56a9b44cb7625b930514
Author: Sunitha Beeram <sb...@sbeeram-ld2.linkedin.biz>
AuthorDate: Wed Mar 6 17:10:36 2019 -0800
readthedocs: Fill in dev-guide sections
---
docs/{README => README.md} | 0
docs/admin_guide.rst | 1 +
docs/client_api.rst | 6 +-
docs/code_modues.rst | 21 ------
docs/code_modules.rst | 141 +++++++++++++++++++++++++++++++++++++++
docs/coding_guidelines.rst | 30 ---------
docs/contribution_guidelines.rst | 66 ++++++++++++++++++
docs/dev_env.rst | 58 +++++++++++++---
docs/dev_guide.rst | 2 +-
docs/getting_involved.rst | 21 ------
docs/img/PinotBroker.png | Bin 0 -> 31737 bytes
docs/img/PinotController.png | Bin 0 -> 43621 bytes
docs/img/PinotFoundation.png | Bin 0 -> 35783 bytes
docs/img/PinotMinionHadoop.png | Bin 0 -> 38638 bytes
docs/img/PinotServer.png | Bin 0 -> 43540 bytes
docs/img/eclipse_style.png | Bin 0 -> 137578 bytes
docs/img/import_scheme.png | Bin 0 -> 164276 bytes
docs/index.rst | 4 ++
docs/pql_examples.rst | 2 +
19 files changed, 269 insertions(+), 83 deletions(-)
diff --git a/docs/README b/docs/README.md
similarity index 100%
rename from docs/README
rename to docs/README.md
diff --git a/docs/admin_guide.rst b/docs/admin_guide.rst
index be5b6c7..422f56a 100644
--- a/docs/admin_guide.rst
+++ b/docs/admin_guide.rst
@@ -22,5 +22,6 @@
in_production
management_api
+ pinot_hadoop
customizations
tuning_pinot
diff --git a/docs/client_api.rst b/docs/client_api.rst
index cab78ff..e9c1a16 100644
--- a/docs/client_api.rst
+++ b/docs/client_api.rst
@@ -157,8 +157,10 @@ Selection
}
-Java
-----
+.. _java-client:
+
+Executing queries via Java Client API
+=====================================
The Pinot client API is similar to JDBC, although there are some differences, due to how Pinot behaves. For example, a query with multiple aggregation function will return one result set per aggregation function, as they are computed in parallel.
diff --git a/docs/code_modues.rst b/docs/code_modues.rst
deleted file mode 100644
index e3a503c..0000000
--- a/docs/code_modues.rst
+++ /dev/null
@@ -1,21 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-Code Modules and organization
-=============================
diff --git a/docs/code_modules.rst b/docs/code_modules.rst
new file mode 100644
index 0000000..adb1fc7
--- /dev/null
+++ b/docs/code_modules.rst
@@ -0,0 +1,141 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+.. _code-modules:
+
+Code Modules and Organization
+=============================
+
+Before proceeding to contributing changes to Pinot, review the contents of this section.
+
+External Dependencies
+---------------------
+Pinot depends on a number of external projects, the most notable ones are:
+
+* Apache Zookeeper
+* Apache Helix
+* Apache Kafka
+* Apache Thrift
+* Netty
+* Google Guava
+* Yammer
+
+*Helix* is used for ClusterManagement, and Pinot code is tightly integrated with Helix and Zookeeper interfaces.
+
+*Kafka* is the default realtime stream provider, but can be replaced with others. See customizations section for more info.
+
+*Thrift* is used for message exchange between broker and server components, with *Netty* providing the server functionality
+for processing messages in a non-blocking fashion.
+
+*Guava* is used for number of auxiliary components such as Caches and RateLimiters.
+*Yammer* metrics is used to register and expose metrics from Pinot components.
+
+In addition, Pinot relies on several key external libraries for some of its core functionality:
+*Roaring Bitmaps*: Pinot's inverted indices are built using `this <https://github.com/RoaringBitmap/RoaringBitmap>`_ library.
+*t-Digest*: Pinot's digest based percentile calculations are based on `this <https://github.com/tdunning/t-digest>`_ library.
+
+Pinot Modules
+-------------
+Pinot is a multi-module project, with each module providing specific functionality that helps us to build services from
+a combination of modules. This helps keep clean interface contracts between different modules as well as reduce the
+overall executable size for individually deployable component.
+
+Each module has a ``src/main/java`` folder where the code resides and ``src/test/java`` where the *unit* tests corresponding to
+the module's code reside.
+
+The following figure provides a high-level overview of the foundational Pinot modules.
+
+.. figure:: img/PinotFoundation.png
+ :scale: 50 %
+
+pinot-filesystem
+^^^^^^^^^^^^^^^^
+``pinot-filesystem`` provides abstractions for working with ``segments`` on local or remote filesystems. This module
+allows for users to plugin filesystems specific to their usecase. Extensions to the base ``PinotFS`` should ideally be
+housed in their specific modules so as not pull in unnecessary dependencies for all users.
+
+pinot-common
+^^^^^^^^^^^^
+``pinot-common`` provides classes common to Pinot components. Some key classes you will find here are:
+
+* ``config``: Definitions for various elements of Pinot's table config.
+* ``metrics``: Definitions for base metrics provided by Controller, Broker and Server.
+
+* ``metadata``: Definitions of metadata stored in Zookeeper.
+
+* ``pql.parsers``: Code to compile PQL strings into corresponding AbstractSyntaxTrees (AST).
+* ``request``: Autogenerated thrift classes representing various parts of PQL requests.
+* ``response``: Definitions of response format returned by the Broker.
+
+pinot-transport
+^^^^^^^^^^^^^^^
+``pinot-transport`` module provides classes required to handle scatter-gather on Pinot Broker and netty wrapper classes
+used by Server to handle connections from Broker.
+
+pinot-core
+^^^^^^^^^^
+``pinot-core`` modules provides the core functionality of Pinot, specifically for handling segments, various index
+structures, query execution - filters, transformations, aggregations etc and support for realtime segments.
+
+pinot-server
+^^^^^^^^^^^^
+``pinot-server`` provides server specific functionality including server startup and REST APIs exposed by the server.
+
+.. figure:: img/PinotServer.png
+ :scale: 50 %
+
+pinot-controller
+^^^^^^^^^^^^^^^^
+``pinot-server`` houses all the controller specific functionality, including many cluster administration APIs, segment
+upload (for both offline and realtime), segment assignment, retention strategies etc.
+
+.. figure:: img/PinotController.png
+ :scale: 50 %
+
+pinot-broker
+^^^^^^^^^^^^
+``pinot-broker`` provides broker functionality that includes wiring the broker startup sequence, building broker routing
+tables, PQL request handling.
+
+.. figure:: img/PinotBroker.png
+ :scale: 50 %
+
+pinot-minion
+^^^^^^^^^^^^
+``pinot-minion`` provides functionality for running auxiliary/periodic tasks on a Pinot Cluster such as purging records
+for compliance with regulations like GDPR.
+
+pinot-hadoop
+^^^^^^^^^^^^
+``pinot-hadoop`` provides classes for segment generation jobs using Hadoop infrastructure.
+
+.. figure:: img/PinotMinionHadoop.png
+ :scale: 50 %
+
+Auxiliary modules
+-----------------
+In addition to the core modules described above, Pinot code provides the following modules:
+
+* ``pinot-tools``: This module is a collection of many tools useful for setting up Pinot cluster, creating/updating segments. It also houses the Pinot quick start guide code.
+
+* ``pinot-perf``: This module has a collection of benchmark test code used to evaluate design options.
+
+* ``pinot-client-api``: This module houses the Java client API. See :ref:`java-client` for more info.
+
+* ``pinot-integration-tests``: This module holds integration tests that test functionality across multiple classes or components. These tests typically do not rely on mocking and provide more end to end coverage for code.
diff --git a/docs/coding_guidelines.rst b/docs/coding_guidelines.rst
deleted file mode 100644
index 2432276..0000000
--- a/docs/coding_guidelines.rst
+++ /dev/null
@@ -1,30 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-Coding Guidelines
-=================
-Verifying code-style
-^^^^^^^^^^^^^^^^^^^^
-
-Run the following command to verify the code-style before posting a PR
-
-.. code-block:: none
-
- mvn checkstyle:check
-
diff --git a/docs/contribution_guidelines.rst b/docs/contribution_guidelines.rst
new file mode 100644
index 0000000..0ac2f86
--- /dev/null
+++ b/docs/contribution_guidelines.rst
@@ -0,0 +1,66 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Contribution Guidelines
+=======================
+
+Before you begin to contribute, make sure you have reviewed :ref:`dev-setup` and :ref:`code-modules` sections.
+
+Create an issue for the change
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Create a Pinot issue for the change you would like to make. Provide information on why the change is needed and how you
+plan to address it. Use the conversations on the issue as a way to validate assumptions and the right way to proceed.
+
+Once you have a good about what you want to do, proceed with the next steps.
+
+Create a branch for your change
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Creating a PR
+^^^^^^^^^^^^^
+
+* Verifying code-style
+
+
+Run the following command to verify the code-style before posting a PR
+
+.. code-block:: none
+
+ mvn checkstyle:check
+
+* License Headers for newly added files
+All source code files should have license headers. To automatically add the header for any new file you plan to checkin,
+ run:
+
+.. code-block:: none
+
+ mvn license:format
+
+.. note::
+
+If you checkin third-party code or files, do not add license headers for them. Follow these instructions to ensure we
+are compliant with Apache Licensing process. <TBD>
+
+* Run tests
+
+* Push changes and create a PR for review
+
+* Addressing comments and Rebasing
+
+* Once your change is merged, check to see if any documentation needs to be updated. If so, create a PR for documentation.
diff --git a/docs/dev_env.rst b/docs/dev_env.rst
index 5292703..d54bb45 100644
--- a/docs/dev_env.rst
+++ b/docs/dev_env.rst
@@ -17,23 +17,65 @@
.. under the License.
..
+.. _dev-setup:
+
+*********************
Dev Environment Setup
-=====================
+*********************
+
+.. contents:: Table of Contents
+
+
+To contribute to Pinot, please follow the instructions below.
+
+Getting the Source Code
+-----------------------
+
+Create a fork
+^^^^^^^^^^^^^
+To limit the number of branches created on the main Pinot repository, we recommend that you create a fork by clicking on the fork button `here <https://github.com/apache/incubator-pinot>`_.
+
+.. code-block:: none
-To contribute to Pinot, please follow the code-style as described below:
+Clone the repository locally
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ $mkdir workspace
+ $cd workspace
+ $git clone https://github.com/<username>/incubator-pinot.git
+ $cd incubator-pinot
+
+Maven
+-----
+Pinot is a Maven project and familiarity with Maven will help you work with Pinot code. If you are new to Maven, you can
+more `here <maven.apache.org>`_ and `here <http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html>`_.
+
+Run the following maven command to setup the project.
+.. code-block:: none
+
+ # compile, download sources
+ $mvn install package -DskipTests -Pbin-dist -DdownloadSources -DdownloadJavadocs
+
+Setup IDE
+---------
+Import the project into your favorite IDE. Setup stylesheet according to your IDE. We have provided instructions for
+intellij and eclipse. If you are using other IDEs, please ensure you use stylesheet based on
+`this <https://github.com/apache/incubator-pinot/blob/master/config/codestyle-intellij.xml>`_.
Intellij
^^^^^^^^
+To import the Pinot stylesheet this launch intellij and navigate to ``Preferences`` (on Mac) or ``Settings`` on Linux.
-* `Import <https://www.jetbrains.com/help/idea/settings-code-style.html>`_ code style to Intellij
-* Navigate to ``Preferences`` -> ``Editor`` -> ``Code Style`` -> ``Java``
+* Navigate to ``Editor`` -> ``Code Style`` -> ``Java``
* Select ``Import Scheme`` -> ``Intellij IDES code style XML``
-* Choose ``codestyle-intellij.xml``
+* Choose ``codestyle-intellij.xml`` from ``incubator-pinot/config`` folder of your workspace. Click Apply.
+
+.. figure:: img/import_scheme.png
Eclipse
^^^^^^^
-.. todo::
-
- Add instructions for eclipse here
+To import the Pinot stylesheet this launch eclipse and navigate to ``Preferences`` (on Mac) or ``Settings`` on Linux.
+* Navigate to Java->Code Style->Formatter
+* Choose ``codestyle-eclipse.xml`` from ``incubator-pinot/config folder`` of your workspace. Click Apply.
+.. figure:: img/eclipse_style.png
diff --git a/docs/dev_guide.rst b/docs/dev_guide.rst
index 6c76e3b..1d1a05a 100644
--- a/docs/dev_guide.rst
+++ b/docs/dev_guide.rst
@@ -21,6 +21,6 @@
:maxdepth: 1
dev_env
- coding_guidelines
code_modules
+ contribution_guidelines
extensions
diff --git a/docs/getting_involved.rst b/docs/getting_involved.rst
deleted file mode 100644
index 11a0d33..0000000
--- a/docs/getting_involved.rst
+++ /dev/null
@@ -1,21 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. toctree::
- :maxdepth: 1
diff --git a/docs/img/PinotBroker.png b/docs/img/PinotBroker.png
new file mode 100644
index 0000000..675834d
Binary files /dev/null and b/docs/img/PinotBroker.png differ
diff --git a/docs/img/PinotController.png b/docs/img/PinotController.png
new file mode 100644
index 0000000..451e20b
Binary files /dev/null and b/docs/img/PinotController.png differ
diff --git a/docs/img/PinotFoundation.png b/docs/img/PinotFoundation.png
new file mode 100644
index 0000000..889f2fc
Binary files /dev/null and b/docs/img/PinotFoundation.png differ
diff --git a/docs/img/PinotMinionHadoop.png b/docs/img/PinotMinionHadoop.png
new file mode 100644
index 0000000..40a008d
Binary files /dev/null and b/docs/img/PinotMinionHadoop.png differ
diff --git a/docs/img/PinotServer.png b/docs/img/PinotServer.png
new file mode 100644
index 0000000..f2789bc
Binary files /dev/null and b/docs/img/PinotServer.png differ
diff --git a/docs/img/eclipse_style.png b/docs/img/eclipse_style.png
new file mode 100644
index 0000000..bc660e7
Binary files /dev/null and b/docs/img/eclipse_style.png differ
diff --git a/docs/img/import_scheme.png b/docs/img/import_scheme.png
new file mode 100644
index 0000000..092debf
Binary files /dev/null and b/docs/img/import_scheme.png differ
diff --git a/docs/index.rst b/docs/index.rst
index 728fee5..4a4d5b0 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -26,6 +26,7 @@
Pinot
#####
+.. _introduction:
.. toctree::
:caption: Introduction
:maxdepth: 1
@@ -34,18 +35,21 @@ Pinot
architecture
getting_started
+.. _user-guide:
.. toctree::
:caption: User Guide
:maxdepth: 1
user_guide
+.. _admin_guide:
.. toctree::
:caption: Admin Guide
:maxdepth: 1
admin_guide
+.. _dev_guide:
.. toctree::
:caption: Developer Guide
:maxdepth: 1
diff --git a/docs/pql_examples.rst b/docs/pql_examples.rst
index 8ff4764..ef69609 100644
--- a/docs/pql_examples.rst
+++ b/docs/pql_examples.rst
@@ -17,6 +17,8 @@
.. under the License.
..
+.. _pql:
+
PQL
===
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org