You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by xi...@apache.org on 2021/04/07 08:57:47 UTC
[incubator-pinot] branch release-0.7.1-rc created (now a54f561)
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a change to branch release-0.7.1-rc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
at a54f561 Run tests when pushing new commit (#6722)
This branch includes the following new commits:
new b892729 Fix bug #6671: RealtimeTableDataManager shuts down SegmentBuildTimeLeaseExtender for all tables in the host (#6682)
new 7d6b247 Fix log message in ControllerPeriodicTask (#6709)
new 5b373b0 Fix the default map return value in DictionaryBasedGroupKeyGenerator (#6712)
new 69d01cc Remove legacy docs (#6747)
new 6878584 fixing license headers and plugin checks (#6748)
new 5769866 Delete Travis CI (#6616)
new a54f561 Run tests when pushing new commit (#6722)
The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 04/07: Remove legacy docs (#6747)
Posted by xi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch release-0.7.1-rc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 69d01ccbda3f5e65ac366c32b170533ab3ce4233
Author: Xiang Fu <fx...@gmail.com>
AuthorDate: Tue Apr 6 12:17:58 2021 -0700
Remove legacy docs (#6747)
---
docs/Makefile | 230 --------------------
docs/README.md | 28 ---
docs/admin_guide.rst | 39 ----
docs/architecture.rst | 214 -------------------
docs/batch_data_ingestion.rst | 223 --------------------
docs/client_api.rst | 252 ----------------------
docs/code_modules.rst | 153 --------------
docs/conf.py | 313 ---------------------------
docs/contribution_guidelines.rst | 263 -----------------------
docs/customizations.rst | 172 ---------------
docs/dev_env.rst | 96 ---------
docs/dev_guide.rst | 36 ----
docs/docutils.conf | 2 -
docs/extensions.rst | 32 ---
docs/getting_started.rst | 317 ----------------------------
docs/img/CustomizingPinot.png | Bin 421572 -> 0 bytes
docs/img/Pinot - Hybrid flow.png | Bin 15756 -> 0 bytes
docs/img/Pinot-Offline-only-flow.png | Bin 8179 -> 0 bytes
docs/img/PinotBroker.png | Bin 40314 -> 0 bytes
docs/img/PinotController.png | Bin 61419 -> 0 bytes
docs/img/PinotFoundation.png | Bin 29973 -> 0 bytes
docs/img/PinotMinionHadoop.png | Bin 38638 -> 0 bytes
docs/img/PinotServer.png | Bin 73944 -> 0 bytes
docs/img/dictionary.png | Bin 80599 -> 0 bytes
docs/img/eclipse_style.png | Bin 137578 -> 0 bytes
docs/img/hlc-arch.png | Bin 47924 -> 0 bytes
docs/img/image2014-11-12-1-54-56.png | Bin 112436 -> 0 bytes
docs/img/image2014-11-12-19-54-12.png | Bin 54744 -> 0 bytes
docs/img/image2014-11-12-20-59-50.png | Bin 36925 -> 0 bytes
docs/img/image2014-11-13-11-21-7.png | Bin 99075 -> 0 bytes
docs/img/image2014-11-13-11-22-57.png | Bin 27562 -> 0 bytes
docs/img/image2015-4-14-194451.png | Bin 83171 -> 0 bytes
docs/img/image2015-5-12-7-24-58.png | Bin 38969 -> 0 bytes
docs/img/image2015-5-13 15-2-9.png | Bin 41882 -> 0 bytes
docs/img/image2015-5-17 13-32-28.png | Bin 89700 -> 0 bytes
docs/img/image2015-5-17 17-59-10.png | Bin 82542 -> 0 bytes
docs/img/image2015-5-19 0-29-34.png | Bin 39085 -> 0 bytes
docs/img/image2015-5-19 0-58-54.png | Bin 54859 -> 0 bytes
docs/img/image2015-5-19 1-28-35.png | Bin 34374 -> 0 bytes
docs/img/image2015-5-19 1-47-44.png | Bin 34374 -> 0 bytes
docs/img/image2015-5-19 1-59-51.png | Bin 37098 -> 0 bytes
docs/img/import_scheme.png | Bin 164276 -> 0 bytes
docs/img/list-schemas.png | Bin 247946 -> 0 bytes
docs/img/llc-arch.png | Bin 39899 -> 0 bytes
docs/img/no-dictionary.png | Bin 111237 -> 0 bytes
docs/img/partitioning.png | Bin 81303 -> 0 bytes
docs/img/pinot-architecture.png | Bin 58474 -> 0 bytes
docs/img/pinot-console.png | Bin 157310 -> 0 bytes
docs/img/pluggable_streams.png | Bin 136984 -> 0 bytes
docs/img/rebalance-table.png | Bin 164989 -> 0 bytes
docs/img/replica-group.png | Bin 126306 -> 0 bytes
docs/img/sorted-forward.png | Bin 70022 -> 0 bytes
docs/img/sorted-inverted.png | Bin 89473 -> 0 bytes
docs/img/znode_layout.png | Bin 43271 -> 0 bytes
docs/in_production.rst | 210 ------------------
docs/index.rst | 65 ------
docs/index_techniques.rst | 178 ----------------
docs/intro.rst | 68 ------
docs/introduction.rst | 35 ---
docs/make.bat | 300 --------------------------
docs/pinot_hadoop.rst | 354 -------------------------------
docs/pluggable_storage.rst | 128 -----------
docs/pluggable_streams.rst | 342 ------------------------------
docs/pql_examples.rst | 357 -------------------------------
docs/record_reader.rst | 117 ----------
docs/schema.rst | 140 ------------
docs/segment_fetcher.rst | 126 -----------
docs/star-tree/example.png | Bin 29005 -> 0 bytes
docs/star-tree/space-time.png | Bin 25225 -> 0 bytes
docs/star-tree/star-tree.rst | 351 ------------------------------
docs/star-tree/structure.png | Bin 25841 -> 0 bytes
docs/tableconfig_schema.rst | 387 ----------------------------------
docs/tuning_pinot.rst | 43 ----
docs/tuning_realtime_performance.rst | 115 ----------
docs/tuning_scatter_and_gather.rst | 184 ----------------
docs/user_guide.rst | 35 ---
76 files changed, 5905 deletions(-)
diff --git a/docs/Makefile b/docs/Makefile
deleted file mode 100644
index eee3f6e..0000000
--- a/docs/Makefile
+++ /dev/null
@@ -1,230 +0,0 @@
-# Makefile for Sphinx documentation
-#
-
-# You can set these variables from the command line.
-SPHINXOPTS =
-SPHINXBUILD = sphinx-build
-PAPER =
-BUILDDIR = _build
-
-# User-friendly check for sphinx-build
-ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
- $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don\'t have Sphinx installed, grab it from http://sphinx-doc.org/)
-endif
-
-# Internal variables.
-PAPEROPT_a4 = -D latex_paper_size=a4
-PAPEROPT_letter = -D latex_paper_size=letter
-ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
-# the i18n builder cannot share the environment and doctrees with the others
-I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
-
-.PHONY: help
-help:
- @echo "Please use \`make <target>' where <target> is one of"
- @echo " html to make standalone HTML files"
- @echo " dirhtml to make HTML files named index.html in directories"
- @echo " singlehtml to make a single large HTML file"
- @echo " pickle to make pickle files"
- @echo " json to make JSON files"
- @echo " htmlhelp to make HTML files and a HTML help project"
- @echo " qthelp to make HTML files and a qthelp project"
- @echo " applehelp to make an Apple Help Book"
- @echo " devhelp to make HTML files and a Devhelp project"
- @echo " epub to make an epub"
- @echo " epub3 to make an epub3"
- @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
- @echo " latexpdf to make LaTeX files and run them through pdflatex"
- @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
- @echo " text to make text files"
- @echo " man to make manual pages"
- @echo " texinfo to make Texinfo files"
- @echo " info to make Texinfo files and run them through makeinfo"
- @echo " gettext to make PO message catalogs"
- @echo " changes to make an overview of all changed/added/deprecated items"
- @echo " xml to make Docutils-native XML files"
- @echo " pseudoxml to make pseudoxml-XML files for display purposes"
- @echo " linkcheck to check all external links for integrity"
- @echo " doctest to run all doctests embedded in the documentation (if enabled)"
- @echo " coverage to run coverage check of the documentation (if enabled)"
- @echo " dummy to check syntax errors of document sources"
-
-.PHONY: clean
-clean:
- rm -rf $(BUILDDIR)
-
-.PHONY: html
-html:
- $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
- @echo
- @echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
-
-.PHONY: dirhtml
-dirhtml:
- $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
- @echo
- @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
-
-.PHONY: singlehtml
-singlehtml:
- $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
- @echo
- @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
-
-.PHONY: pickle
-pickle:
- $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
- @echo
- @echo "Build finished; now you can process the pickle files."
-
-.PHONY: json
-json:
- $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
- @echo
- @echo "Build finished; now you can process the JSON files."
-
-.PHONY: htmlhelp
-htmlhelp:
- $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
- @echo
- @echo "Build finished; now you can run HTML Help Workshop with the" \
- ".hhp project file in $(BUILDDIR)/htmlhelp."
-
-.PHONY: qthelp
-qthelp:
- $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
- @echo
- @echo "Build finished; now you can run "qcollectiongenerator" with the" \
- ".qhcp project file in $(BUILDDIR)/qthelp, like this:"
- @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Pinot.qhcp"
- @echo "To view the help file:"
- @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Pinot.qhc"
-
-.PHONY: applehelp
-applehelp:
- $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
- @echo
- @echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
- @echo "N.B. You won't be able to view it unless you put it in" \
- "~/Library/Documentation/Help or install it in your application" \
- "bundle."
-
-.PHONY: devhelp
-devhelp:
- $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
- @echo
- @echo "Build finished."
- @echo "To view the help file:"
- @echo "# mkdir -p $$HOME/.local/share/devhelp/Pinot"
- @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Pinot"
- @echo "# devhelp"
-
-.PHONY: epub
-epub:
- $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
- @echo
- @echo "Build finished. The epub file is in $(BUILDDIR)/epub."
-
-.PHONY: epub3
-epub3:
- $(SPHINXBUILD) -b epub3 $(ALLSPHINXOPTS) $(BUILDDIR)/epub3
- @echo
- @echo "Build finished. The epub3 file is in $(BUILDDIR)/epub3."
-
-.PHONY: latex
-latex:
- $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
- @echo
- @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
- @echo "Run \`make' in that directory to run these through (pdf)latex" \
- "(use \`make latexpdf' here to do that automatically)."
-
-.PHONY: latexpdf
-latexpdf:
- $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
- @echo "Running LaTeX files through pdflatex..."
- $(MAKE) -C $(BUILDDIR)/latex all-pdf
- @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
-
-.PHONY: latexpdfja
-latexpdfja:
- $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
- @echo "Running LaTeX files through platex and dvipdfmx..."
- $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
- @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
-
-.PHONY: text
-text:
- $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
- @echo
- @echo "Build finished. The text files are in $(BUILDDIR)/text."
-
-.PHONY: man
-man:
- $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
- @echo
- @echo "Build finished. The manual pages are in $(BUILDDIR)/man."
-
-.PHONY: texinfo
-texinfo:
- $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
- @echo
- @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
- @echo "Run \`make' in that directory to run these through makeinfo" \
- "(use \`make info' here to do that automatically)."
-
-.PHONY: info
-info:
- $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
- @echo "Running Texinfo files through makeinfo..."
- make -C $(BUILDDIR)/texinfo info
- @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
-
-.PHONY: gettext
-gettext:
- $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
- @echo
- @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
-
-.PHONY: changes
-changes:
- $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
- @echo
- @echo "The overview file is in $(BUILDDIR)/changes."
-
-.PHONY: linkcheck
-linkcheck:
- $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
- @echo
- @echo "Link check complete; look for any errors in the above output " \
- "or in $(BUILDDIR)/linkcheck/output.txt."
-
-.PHONY: doctest
-doctest:
- $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
- @echo "Testing of doctests in the sources finished, look at the " \
- "results in $(BUILDDIR)/doctest/output.txt."
-
-.PHONY: coverage
-coverage:
- $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
- @echo "Testing of coverage in the sources finished, look at the " \
- "results in $(BUILDDIR)/coverage/python.txt."
-
-.PHONY: xml
-xml:
- $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
- @echo
- @echo "Build finished. The XML files are in $(BUILDDIR)/xml."
-
-.PHONY: pseudoxml
-pseudoxml:
- $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
- @echo
- @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."
-
-.PHONY: dummy
-dummy:
- $(SPHINXBUILD) -b dummy $(ALLSPHINXOPTS) $(BUILDDIR)/dummy
- @echo
- @echo "Build finished. Dummy builder generates no files."
diff --git a/docs/README.md b/docs/README.md
deleted file mode 100644
index 5f23fcc..0000000
--- a/docs/README.md
+++ /dev/null
@@ -1,28 +0,0 @@
-<!--
-
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
--->
-
-Updating the docs:
-1. Edit or add files as needed.
-2. Run "make html"
-3. Open _build/html/index.html in your favorite browser and ensure contents and links work correctly.
-
-NOTE:
-You may see some differences locally as the version of sphinx-build on your local host might not be the same as the one used in readthedocs.io for building the pinot website docs.
\ No newline at end of file
diff --git a/docs/admin_guide.rst b/docs/admin_guide.rst
deleted file mode 100644
index 332da99..0000000
--- a/docs/admin_guide.rst
+++ /dev/null
@@ -1,39 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-:orphan:
-
-###########
-Admin Guide
-###########
-
-.. _admin-guide:
-
-.. toctree::
- :maxdepth: 1
-
- tableconfig_schema
- schema
- in_production
- batch_data_ingestion
- pinot_hadoop
- customizations
- tuning_pinot
diff --git a/docs/architecture.rst b/docs/architecture.rst
deleted file mode 100644
index 0348aa3..0000000
--- a/docs/architecture.rst
+++ /dev/null
@@ -1,214 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-.. _pinot-architecture-section:
-
-Architecture
-============
-
-.. _pinot-architecture-diagram:
-
-.. figure:: img/pinot-architecture.png
-
- Pinot Architecture Overview
-
-Terminology
------------
-
-*Table*
- A table is a logical abstraction to refer to a collection of related data. It consists of columns and rows (documents).
-*Segment*
- Data in table is divided into (horizontal) shards referred to as segments.
-
-Pinot Components
-----------------
-
-*Pinot Controller*
- Manages other pinot components (brokers, servers) as well as controls assignment of tables/segments to servers.
-*Pinot Server*
- Hosts one or more segments and serves queries from those segments
-*Pinot Broker*
- Accepts queries from clients and routes them to one or more servers, and returns consolidated response to the client.
-
-Pinot leverages `Apache Helix <http://helix.apache.org>`_ for cluster management.
-Helix is a cluster management framework to manage replicated, partitioned resources in a distributed system.
-Helix uses Zookeeper to store cluster state and metadata.
-
-Briefly, Helix divides nodes into three logical components based on their responsibilities:
-
-*Participant*
- The nodes that host distributed, partitioned resources
-*Spectator*
- The nodes that observe the current state of each Participant and use that information to access the resources.
- Spectators are notified of state changes in the cluster (state of a participant, or that of a partition in a participant).
-*Controller*
- The node that observes and controls the Participant nodes. It is responsible for coordinating all transitions
- in the cluster and ensuring that state constraints are satisfied while maintaining cluster stability
-
-Pinot Controller hosts Helix Controller, in addition to hosting REST APIs for Pinot cluster administration and data ingestion.
-There can be multiple instances of Pinot controller for redundancy. If there are multiple controllers, Pinot expects that all
-of them are configured with the same back-end storage system so that they have a common view of the segments (*e.g.* NFS).
-Pinot can use other storage systems such as HDFS or `ADLS <https://azure.microsoft.com/en-us/services/storage/data-lake-storage/>`_.
-
-Pinot Servers are modeled as Helix Participants, hosting Pinot tables (referred to as *resources* in helix terminology).
-Segments of a table are modeled as Helix partitions (of a resource). Thus, a Pinot server hosts one or more helix partitions of one
-or more helix resources (*i.e.* one or more segments of one or more tables).
-
-Pinot Brokers are modeled as Spectators. They need to know the location of each segment of a table (and each replica of the
-segments)
-and route requests to the
-appropriate server that hosts the segments of the table being queried. The broker ensures that all the rows of the table
-are queried exactly once so as to return correct, consistent results for a query. The brokers (or servers) may optimize
-to prune some of the segments as long as accuracy is not satisfied. In case of hybrid tables, the brokers ensure that
-the overlap between realtime and offline segment data is queried exactly once.
-Helix provides the framework by which spectators can learn the location (*i.e.* participant) in which each partition
-of a resource resides. The brokers use this mechanism to learn the servers that host specific segments of a table.
-
-Pinot Tables
-------------
-
-Pinot supports realtime, or offline, or hybrid tables. Data in Pinot tables is contained in the segments
-belonging to that table. A Pinot table is modeled as a Helix resource. Each segment of a table is modeled as a Helix Partition.
-
-Table Schema defines column names and their metadata. Table configuration and schema is stored in zookeeper.
-
-Offline tables ingest pre-built pinot-segments from external data stores, whereas Reatime tables
-ingest data from streams (such as Kafka) and build segments.
-
-A hybrid Pinot table essentially has both realtime as well as offline tables.
-In such a table, offline segments may be pushed periodically (say, once a day). The retention on the offline table
-can be set to a high value (say, a few years) since segments are coming in on a periodic basis, whereas the retention
-on the realtime part can be small (say, a few days). Once an offline segment is pushed to cover a recent time period,
-the brokers automatically switch to using the offline table for segments in *that* time period, and use realtime table
-only to cover later segments for which offline data may not be available yet.
-
-Note that the query does not know the existence of offline or realtime tables. It only specifies the table name
-in the query.
-
-See section on :ref:`table-config-section` for how to customize table configuration as per requirements.
-
-
-Ingesting Offline data
-^^^^^^^^^^^^^^^^^^^^^^
-Segments for offline tables are constructed outside of Pinot, typically in Hadoop via map-reduce jobs
-and ingested into Pinot via REST API provided by the Controller.
-Pinot provides libraries to create Pinot segments out of input files in AVRO, JSON or CSV formats in a hadoop job, and push
-the constructed segments to the controllers via REST APIs.
-
-When an Offline segment is ingested, the controller looks up the table's configuration and assigns the segment
-to the servers that host the table. It may assign multiple servers for each segment depending on the number of replicas
-configured for that table.
-
-Pinot supports different segment assignment strategies that are optimized for various use cases.
-
-Once segments are assigned, Pinot servers get notified via Helix to "host" the segment. The servers download the segments
-(as a cached local copy to serve queries) and load them into local memory. All segment data is maintained in memory as long
-as the server hosts that segment.
-
-Once the server has loaded the segment, Helix notifies brokers of the availability of these segments. The brokers
-start include the new
-segments for queries. Brokers support different routing strategies depending on the type of table, the segment assignment
-strategy and the use case.
-
-Data in offline segments are immutable (Rows cannot be added, deleted, or modified). However, segments may be replaced with modified data.
-
-.. _ingesting-realtime-data:
-
-Ingesting Realtime Data
-^^^^^^^^^^^^^^^^^^^^^^^
-Segments for realtime tables are constructed by Pinot servers with rows ingested from data streams such as Kafka.
-Rows ingested from streams are made available for query processing as soon as they are ingested, thus enabling
-applications such as those that need real-time charts on analytics.
-
-In large scale installations, data in streams is typically split across multiple stream partitions. The underlying
-stream may provide consumer implementations that allow applications to consume data from any subset of partitions,
-including all partitions (or, just from one partition).
-
-A pinot table can be configured to consume from streams in one of two modes:
-
- * ``LowLevel``: This is the preferred mode of consumption. Pinot creates independent partition-level consumers for
- each partition. Depending on the the configured number of replicas, multiple consumers may be created for
- each partition, taking care that no two replicas exist on the same server host. Therefore you need to provision
- *at least* as many hosts as the number of replcias configured.
-
- * ``HighLevel``: Pinot creates *one* stream-level consumer that consumes from all partitions. Each message consumed
- could be from any of the partitions of the stream. Depending on the configured number of replicas, multiple
- stream-level consumers are created, taking care that no two replicas exist on the same server host. Therefore
- you need to provision exactly as many hosts as the number of replicas configured.
-
-Of course, the underlying stream should support either mode of consumption in order for a Pinot table to use that
-mode. Kafka has support for both of these modes. See :ref:`pluggable-streams` for more information on support of other
-data streams in Pinot.
-
-In either mode, Pinot servers store the ingested rows in volatile memory until either one of the following conditions are met:
-
- #. A certain number of rows are consumed
- #. The consumption has gone on for a certain length of time
-
-(See :ref:`stream-config-description` on how to set these values, or have pinot compute them for you)
-
-Upon reaching either one of these limits, the servers do the following:
-
- * Pause consumption
- * Persist the rows consumed so far into non-volatile storage
- * Continue consuming new rows into volatile memory again.
-
-The persisted rows form what we call a *completed* segment (as opposed to a *consuming*
-segment that resides in volatile memory).
-
-In ``LowLevel`` mode, the completed segments are persisted the into local non-volatile store of pinot server
-*as well as* the segment store of the pinot cluster (See :ref:`pinot-architecture-diagram`). This allows for
-easy and automated mechanisms for replacing pinot servers, or expanding capacity, etc. Pinot has
-`special mechanisms <https://cwiki.apache.org/confluence/display/PINOT/Consuming+and+Indexing+rows+in+Realtime#ConsumingandIndexingrowsinRealtime-Segmentcompletionprotocol>`_
-that ensure that the completed segment is equivalent across all replicas.
-
-During segment completion, one winner is chosen by the controller from all the replicas as the ``committer server``. The ``committer server`` builds the segment and uploads it to the controller. All the other ``non-committer servers`` follow one of these two paths:
-
-1. If the in-memory segment is equivalent to the committed segment, the ``non-committer`` server also builds the segment locally and replaces the in-memory segment
-2. If the in-memory segment is non equivalent to the committed segment, the ``non-committer`` server downloads the segment from the controller.
-
-For more details on this protocol, please refer to `this doc <https://cwiki.apache.org/confluence/display/PINOT/Consuming+and+Indexing+rows+in+Realtime#ConsumingandIndexingrowsinRealtime-Segmentcompletionprotocol>`_.
-
-In ``HighLevel`` mode, the servers persist the consumed rows into local store (and **not** the segment store). Since consumption of rows
-can be from any partition, it is not possible to guarantee equivalence of segments across replicas.
-
-See `Consuming and Indexing rows in Realtime <https://cwiki.apache.org/confluence/display/PINOT/Consuming+and+Indexing+rows+in+Realtime>`_ for details.
-
-
-Pinot Segments
---------------
-
-A segment is laid out in a columnar format so that it can be directly mapped into memory for serving queries.
-
-Columns may be single or multi-valued. Column types may be
-STRING, INT, LONG, FLOAT, DOUBLE or BYTES. Columns may be declared to be metric or dimension (or specifically as a time dimension)
-in the schema. Columns can have default null value. For example, the default null value of a integer column can be 0.
-Note: The default value of byte column has to be hex-encoded before adding to the schema.
-
-Pinot uses dictionary encoding to store values as a dictionary ID. Columns may be configured to be "no-dictionary" column in which
-case raw values are stored. Dictionary IDs are encoded using minimum number of bits for efficient storage (*e.g.* a column with cardinality
-of 3 will use only 3 bits for each dictionary ID).
-
-There is a forward index built for each column and compressed appropriately for efficient memory use. In addition, optional inverted indices can be
-configured for any set of columns. Inverted indices, while take up more storage, offer better query performance.
-
-Specialized indexes like Star-Tree index is also supported.
-
diff --git a/docs/batch_data_ingestion.rst b/docs/batch_data_ingestion.rst
deleted file mode 100644
index e573d16..0000000
--- a/docs/batch_data_ingestion.rst
+++ /dev/null
@@ -1,223 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-.. _batch-data-ingestion:
-
-Batch Data Ingestion
-====================
-
-In practice, we need to run Pinot data ingestion as a pipeline or a scheduled job.
-
-Assuming `pinot-distribution` is already built, inside `examples` directory, you could find several sample table layouts.
-
-Table Layout
-------------
-
-Usually each table deserves its own directory, like `airlineStats`.
-
-Inside the table directory, `rawdata` is created to put all the input data.
-
-Typically, for data events with timestamp, we partition those data and store them into a daily folder.
-E.g. a typically layout is like: `rawdata/%yyyy%/%mm%/%dd%/[daily_input_files]`.
-
-
-Configuring batch ingestion job
--------------------------------
-
-Create a batch ingestion job spec file to describe how to ingest the data.
-
-Below is an example (also located at `examples/batch/airlineStats/ingestionJobSpec.yaml`)
-
-.. code-block:: none
-
- # executionFrameworkSpec: Defines ingestion jobs to be running.
- executionFrameworkSpec:
-
- # name: execution framework name
- name: 'standalone'
-
- # segmentGenerationJobRunnerClassName: class name implements org.apache.pinot.spi.batch.ingestion.runner.SegmentGenerationJobRunner interface.
- segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
-
- # segmentTarPushJobRunnerClassName: class name implements org.apache.pinot.spi.batch.ingestion.runner.SegmentTarPushJobRunner interface.
- segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
-
- # segmentUriPushJobRunnerClassName: class name implements org.apache.pinot.spi.batch.ingestion.runner.SegmentUriPushJobRunner interface.
- segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
-
- # jobType: Pinot ingestion job type.
- # Supported job types are:
- # 'SegmentCreation'
- # 'SegmentTarPush'
- # 'SegmentUriPush'
- # 'SegmentCreationAndTarPush'
- # 'SegmentCreationAndUriPush'
- jobType: SegmentCreationAndTarPush
-
- # inputDirURI: Root directory of input data, expected to have scheme configured in PinotFS.
- inputDirURI: 'examples/batch/airlineStats/rawdata'
-
- # includeFileNamePattern: include file name pattern, supported glob pattern.
- # Sample usage:
- # 'glob:*.avro' will include all avro files just under the inputDirURI, not sub directories;
- # 'glob:**/*.avro' will include all the avro files under inputDirURI recursively.
- includeFileNamePattern: 'glob:**/*.avro'
-
- # excludeFileNamePattern: exclude file name pattern, supported glob pattern.
- # Sample usage:
- # 'glob:*.avro' will exclude all avro files just under the inputDirURI, not sub directories;
- # 'glob:**/*.avro' will exclude all the avro files under inputDirURI recursively.
- # _excludeFileNamePattern: ''
-
- # outputDirURI: Root directory of output segments, expected to have scheme configured in PinotFS.
- outputDirURI: 'examples/batch/airlineStats/segments'
-
- # overwriteOutput: Overwrite output segments if existed.
- overwriteOutput: true
-
- # pinotFSSpecs: defines all related Pinot file systems.
- pinotFSSpecs:
-
- - # scheme: used to identify a PinotFS.
- # E.g. local, hdfs, dbfs, etc
- scheme: file
-
- # className: Class name used to create the PinotFS instance.
- # E.g.
- # org.apache.pinot.spi.filesystem.LocalPinotFS is used for local filesystem
- # org.apache.pinot.plugin.filesystem.AzurePinotFS is used for Azure Data Lake
- # org.apache.pinot.plugin.filesystem.HadoopPinotFS is used for HDFS
- className: org.apache.pinot.spi.filesystem.LocalPinotFS
-
- # recordReaderSpec: defines all record reader
- recordReaderSpec:
-
- # dataFormat: Record data format, e.g. 'avro', 'parquet', 'orc', 'csv', 'json', 'thrift' etc.
- dataFormat: 'avro'
-
- # className: Corresponding RecordReader class name.
- # E.g.
- # org.apache.pinot.plugin.inputformat.avro.AvroRecordReader
- # org.apache.pinot.plugin.inputformat.csv.CSVRecordReader
- # org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader
- # org.apache.pinot.plugin.inputformat.json.JSONRecordReader
- # org.apache.pinot.plugin.inputformat.orc.ORCRecordReader
- # org.apache.pinot.plugin.inputformat.thrift.ThriftRecordReader
- className: 'org.apache.pinot.plugin.inputformat.avro.AvroRecordReader'
-
- # tableSpec: defines table name and where to fetch corresponding table config and table schema.
- tableSpec:
-
- # tableName: Table name
- tableName: 'airlineStats'
-
- # schemaURI: defines where to read the table schema, supports PinotFS or HTTP.
- # E.g.
- # hdfs://path/to/table_schema.json
- # http://localhost:9000/tables/myTable/schema
- schemaURI: 'http://localhost:9000/tables/airlineStats/schema'
-
- # tableConfigURI: defines where to reade the table config.
- # Supports using PinotFS or HTTP.
- # E.g.
- # hdfs://path/to/table_config.json
- # http://localhost:9000/tables/myTable
- # Note that the API to read Pinot table config directly from pinot controller contains a JSON wrapper.
- # The real table config is the object under the field 'OFFLINE'.
- tableConfigURI: 'http://localhost:9000/tables/airlineStats'
-
- # pinotClusterSpecs: defines the Pinot Cluster Access Point.
- pinotClusterSpecs:
- - # controllerURI: used to fetch table/schema information and data push.
- # E.g. http://localhost:9000
- controllerURI: 'http://localhost:9000'
-
- # pushJobSpec: defines segment push job related configuration.
- pushJobSpec:
-
- # pushAttempts: number of attempts for push job, default is 1, which means no retry.
- pushAttempts: 2
-
- # pushRetryIntervalMillis: retry wait Ms, default to 1 second.
- pushRetryIntervalMillis: 1000
-
-Executing the job
------------------
-Below command will create example table into Pinot cluster.
-
-.. code-block:: bash
-
- bin/pinot-admin.sh AddTable -schemaFile examples/batch/airlineStats/airlineStats_schema.json -tableConfigFile examples/batch/airlineStats/airlineStats_offline_table_config.json -exec
-
-Below command will kick off the ingestion job to generate Pinot segments and push them into the cluster.
-
-.. code-block:: bash
-
- bin/pinot-ingestion-job.sh -jobSpec examples/batch/airlineStats/ingestionJobSpec.yaml
-
-After job finished, segments are stored in ` examples/batch/airlineStats/segments` following same layout of input directory layout.
-
-
-Executing the job using Spark
------------------------------
-Below example is running in a spark local mode. You can download spark distribution and start it by running:
-
-.. code-block:: bash
-
- $ wget http://apache-mirror.8birdsvideo.com/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
- $ tar xvf spark-2.4.4-bin-hadoop2.7.tgz
- $ cd spark-2.4.4-bin-hadoop2.7
- $ ./bin/spark-shell --master 'local[2]'
-
-Below command shows how to use `spark-submit` command to submit a spark job using pinot-all-${PINOT_VERSION}-jar-with-dependencies jar.
-
-Please ensure parameter `PINOT_ROOT_DIR` and `PINOT_VERSION` are set properly.
-
-.. code-block:: bash
-
- export PINOT_VERSION=0.3.0-SNAPSHOT
- export PINOT_DISTRIBUTION_DIR=${PINOT_ROOT_DIR}/pinot-distribution/target/apache-pinot-incubating-${PINOT_VERSION}-bin/apache-pinot-incubating-${PINOT_VERSION}-bin
- ./bin/spark-submit \
- --class org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher \
- --master "local[2]" \
- --deploy-mode client \
- --conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml" \
- --conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar" \
- local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar \
- ${PINOT_DISTRIBUTION_DIR}/examples/batch/airlineStats/sparkIngestionJobSpec.yaml
-
-
-Executing the job using Hadoop
-------------------------------
-
-Below command shows how to use `hadoop jar` command to run a hadoop job using pinot-all-${PINOT_VERSION}-jar-with-dependencies jar.
-
-Please ensure parameter `PINOT_ROOT_DIR` and `PINOT_VERSION` are set properly.
-
-.. code-block:: bash
-
- export PINOT_VERSION=0.3.0-SNAPSHOT
- export PINOT_DISTRIBUTION_DIR=${PINOT_ROOT_DIR}/pinot-distribution/target/apache-pinot-incubating-${PINOT_VERSION}-bin/apache-pinot-incubating-${PINOT_VERSION}-bin
- export HADOOP_CLIENT_OPTS="-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml"
- hadoop jar \
- ${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar \
- org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher \
- ${PINOT_DISTRIBUTION_DIR}/examples/batch/airlineStats/hadoopIngestionJobSpec.yaml
diff --git a/docs/client_api.rst b/docs/client_api.rst
deleted file mode 100644
index 4764c5c..0000000
--- a/docs/client_api.rst
+++ /dev/null
@@ -1,252 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-Executing queries via REST API on the Broker
-============================================
-
-The Pinot REST API can be accessed by invoking ``POST`` operation with a JSON body containing the parameter ``pql``
-to the ``/query`` URI endpoint on a broker. Depending on the type of query, the results can take different shapes.
-The examples below use curl.
-
-Aggregation
------------
-
-.. code-block:: none
-
- curl -X POST -d '{"pql":"select count(*) from flights"}' http://localhost:8099/query
-
-
- {
- "traceInfo":{},
- "numDocsScanned":17,
- "aggregationResults":[
- {
- "function":"count_star",
- "value":"17"
- }
- ],
- "timeUsedMs":27,
- "segmentStatistics":[],
- "exceptions":[],
- "totalDocs":17
- }
-
-
-Aggregation with grouping
--------------------------
-
-.. code-block:: none
-
- curl -X POST -d '{"pql":"select count(*) from flights group by Carrier"}' http://localhost:8099/query
-
-
- {
- "traceInfo":{},
- "numDocsScanned":23,
- "aggregationResults":[
- {
- "groupByResult":[
- {
- "value":"10",
- "group":["AA"]
- },
- {
- "value":"9",
- "group":["VX"]
- },
- {
- "value":"4",
- "group":["WN"]
- }
- ],
- "function":"count_star",
- "groupByColumns":["Carrier"]
- }
- ],
- "timeUsedMs":47,
- "segmentStatistics":[],
- "exceptions":[],
- "totalDocs":23
- }
-
-
-Selection
----------
-
-.. code-block:: none
-
- curl -X POST -d '{"pql":"select * from flights limit 3"}' http://localhost:8099/query
-
-
- {
- "selectionResults":{
- "columns":[
- "Cancelled",
- "Carrier",
- "DaysSinceEpoch",
- "Delayed",
- "Dest",
- "DivAirports",
- "Diverted",
- "Month",
- "Origin",
- "Year"
- ],
- "results":[
- [
- "0",
- "AA",
- "16130",
- "0",
- "SFO",
- [],
- "0",
- "3",
- "LAX",
- "2014"
- ],
- [
- "0",
- "AA",
- "16130",
- "0",
- "LAX",
- [],
- "0",
- "3",
- "SFO",
- "2014"
- ],
- [
- "0",
- "AA",
- "16130",
- "0",
- "SFO",
- [],
- "0",
- "3",
- "LAX",
- "2014"
- ]
- ]
- },
- "traceInfo":{},
- "numDocsScanned":3,
- "aggregationResults":[],
- "timeUsedMs":10,
- "segmentStatistics":[],
- "exceptions":[],
- "totalDocs":102
- }
-
-
-.. _java-client:
-
-Executing queries via Java Client API
-=====================================
-
-The Pinot client API is similar to JDBC, although there are some differences, due to how Pinot behaves. For example, a query with multiple aggregation function will return one result set per aggregation function, as they are computed in parallel.
-
-Connections to Pinot are created using the ConnectionFactory class' utility methods to create connections to a Pinot cluster given a Zookeeper URL, a Java Properties object or a list of broker addresses to connect to.
-
-.. code-block:: java
-
- Connection connection = ConnectionFactory.fromZookeeper
- ("some-zookeeper-server:2191/zookeeperPath");
-
- Connection connection = ConnectionFactory.fromProperties("demo.properties");
-
- Connection connection = ConnectionFactory.fromHostList
- ("some-server:1234", "some-other-server:1234", ...);
-
-
-Queries can be sent directly to the Pinot cluster using the Connection.execute(java.lang.String) and Connection.executeAsync(java.lang.String) methods of Connection.
-
-.. code-block:: java
-
- ResultSetGroup resultSetGroup = connection.execute("select * from foo...");
- Future<ResultSetGroup> futureResultSetGroup = connection.executeAsync
- ("select * from foo...");
-
-
-Queries can also use a PreparedStatement to escape query parameters:
-
-.. code-block:: java
-
- PreparedStatement statement = connection.prepareStatement
- ("select * from foo where a = ?");
- statement.setString(1, "bar");
-
- ResultSetGroup resultSetGroup = statement.execute();
- Future<ResultSetGroup> futureResultSetGroup = statement.executeAsync();
-
-
-In the case of a selection query, results can be obtained with the various get methods in the first ResultSet, obtained through the getResultSet(int) method:
-
-.. code-block:: java
-
- ResultSet resultSet = connection.execute
- ("select foo, bar from baz where quux = 'quuux'").getResultSet(0);
-
- for (int i = 0; i < resultSet.getRowCount(); ++i) {
- System.out.println("foo: " + resultSet.getString(i, 0));
- System.out.println("bar: " + resultSet.getInt(i, 1));
- }
-
- resultSet.close();
-
-
-In the case of aggregation, each aggregation function is within its own ResultSet:
-
-.. code-block:: java
-
- ResultSetGroup resultSetGroup = connection.execute("select count(*) from foo");
-
- ResultSet resultSet = resultSetGroup.getResultSet(0);
- System.out.println("Number of records: " + resultSet.getInt(0));
- resultSet.close();
-
-
-There can be more than one ResultSet, each of which can contain multiple results grouped by a group key.
-
-.. code-block:: java
-
- ResultSetGroup resultSetGroup = connection.execute
- ("select min(foo), max(foo) from bar group by baz");
-
- System.out.println("Number of result groups:" +
- resultSetGroup.getResultSetCount(); // 2, min(foo) and max(foo)
-
- ResultSet minResultSet = resultSetGroup.getResultSet(0);
- for(int i = 0; i < minResultSet.length(); ++i) {
- System.out.println("Minimum foo for " + minResultSet.getGroupKeyString(i, 1) +
- ": " + minResultSet.getInt(i));
- }
-
- ResultSet maxResultSet = resultSetGroup.getResultSet(1);
- for(int i = 0; i < maxResultSet.length(); ++i) {
- System.out.println("Maximum foo for " + maxResultSet.getGroupKeyString(i, 1) +
- ": " + maxResultSet.getInt(i));
- }
-
- resultSet.close();
-
diff --git a/docs/code_modules.rst b/docs/code_modules.rst
deleted file mode 100644
index e855799..0000000
--- a/docs/code_modules.rst
+++ /dev/null
@@ -1,153 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-.. _code-modules:
-
-
-*****************************
-Code Modules and Organization
-*****************************
-
-.. contents:: Table of Contents
-
-Before proceeding to contributing changes to Pinot, review the contents of this section.
-
-External Dependencies
----------------------
-Pinot depends on a number of external projects, the most notable ones are:
-
-* Apache Zookeeper
-* Apache Helix
-* Apache Kafka
-* Apache Thrift
-* Netty
-* Google Guava
-* Yammer
-
-*Helix* is used for ClusterManagement, and Pinot code is tightly integrated with Helix and Zookeeper interfaces.
-
-*Kafka* is the default realtime stream provider, but can be replaced with others. See customizations section for more info.
-
-*Thrift* is used for message exchange between broker and server components, with *Netty* providing the server functionality
-for processing messages in a non-blocking fashion.
-
-*Guava* is used for number of auxiliary components such as Caches and RateLimiters.
-*Yammer* metrics is used to register and expose metrics from Pinot components.
-
-In addition, Pinot relies on several key external libraries for some of its core functionality:
-*Roaring Bitmaps*: Pinot's inverted indices are built using `RoaringBitmap <https://github.com/RoaringBitmap/RoaringBitmap>`_ library.
-*t-Digest*: Pinot's digest based percentile calculations are based on `T-Digest <https://github.com/tdunning/t-digest>`_ library.
-
-Pinot Modules
--------------
-Pinot is a multi-module project, with each module providing specific functionality that helps us to build services from
-a combination of modules. This helps keep clean interface contracts between different modules as well as reduce the
-overall executable size for individually deployable component.
-
-Each module has a ``src/main/java`` folder where the code resides and ``src/test/java`` where the *unit* tests corresponding to
-the module's code reside.
-
-.. _pinot-foundation:
-
-Foundational modules
---------------------
-The following figure provides a high-level overview of the foundational Pinot modules.
-
-.. figure:: img/PinotFoundation.png
- :scale: 50 %
-
-pinot-common
-^^^^^^^^^^^^
-``pinot-common`` provides classes common to Pinot components. Some key classes you will find here are:
-
-* ``config``: Definitions for various elements of Pinot's table config.
-* ``metrics``: Definitions for base metrics provided by Controller, Broker and Server.
-
-* ``metadata``: Definitions of metadata stored in Zookeeper.
-
-* ``pql.parsers``: Code to compile PQL strings into corresponding AbstractSyntaxTrees (AST).
-* ``request``: Autogenerated thrift classes representing various parts of PQL requests.
-* ``response``: Definitions of response format returned by the Broker.
-* ``filesystem``: provides abstractions for working with ``segments`` on local or remote filesystems. This module allows for users to plugin filesystems specific to their usecase. Extensions to the base ``PinotFS`` should ideally be housed in their specific modules so as not pull in unnecessary dependencies for all users.
-
-pinot-core
-^^^^^^^^^^
-``pinot-core`` modules provides the core functionality of Pinot, specifically for handling segments, various index
-structures, query execution - filters, transformations, aggregations etc and support for realtime segments.
-
-pinot-server
-^^^^^^^^^^^^
-``pinot-server`` provides server specific functionality including server startup and REST APIs exposed by the server.
-
-.. figure:: img/PinotServer.png
- :scale: 50 %
-
-pinot-controller
-^^^^^^^^^^^^^^^^
-``pinot-server`` houses all the controller specific functionality, including many cluster administration APIs, segment
-upload (for both offline and realtime), segment assignment, retention strategies etc.
-
-.. figure:: img/PinotController.png
- :scale: 50 %
-
-pinot-broker
-^^^^^^^^^^^^
-``pinot-broker`` provides broker functionality that includes wiring the broker startup sequence, building broker routing
-tables, PQL request handling.
-
-.. figure:: img/PinotBroker.png
- :scale: 50 %
-
-pinot-minion
-^^^^^^^^^^^^
-``pinot-minion`` provides functionality for running auxiliary/periodic tasks on a Pinot Cluster such as purging records
-for compliance with regulations like GDPR.
-
-pinot-hadoop
-^^^^^^^^^^^^
-``pinot-hadoop`` provides classes for segment generation jobs using Hadoop infrastructure.
-
-.. figure:: img/PinotMinionHadoop.png
- :scale: 50 %
-
-Auxiliary modules
------------------
-In addition to the core modules described above, Pinot code provides the following modules:
-
-* ``pinot-tools``: This module is a collection of many tools useful for setting up Pinot cluster, creating/updating segments.
- It also houses the Pinot quick start guide code.
-
-* ``pinot-perf``: This module has a collection of benchmark test code used to evaluate design options.
-
-* ``pinot-client-api``: This module houses the Java client API. See :ref:`java-client` for more info.
-
-* ``pinot-integration-tests``: This module holds integration tests that test functionality across multiple classes or components.
-
-These tests typically do not rely on mocking and provide more end to end coverage for code.
-
-.. _extension-modules:
-
-Extension modules
------------------
-``pinot-hadoop-filesystem`` and ``pinot-azure-filesystem`` are module added to support extensions to Pinot filesystem.
-The functionality is broken down into modules of their own to avoid polluting the common modules with additional large libraries.
-These libraries bring in transitive dependencies of their own that can cause classpath conflicts at runtime. We would like to
-avoid this for the common usage of Pinot as much as possible.
diff --git a/docs/conf.py b/docs/conf.py
deleted file mode 100644
index cda332b..0000000
--- a/docs/conf.py
+++ /dev/null
@@ -1,313 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-# -*- coding: utf-8 -*-
-#
-# Pinot documentation build configuration file, created by
-# sphinx-quickstart on Fri Feb 9 14:54:43 2018.
-#
-# This file is execfile()d with the current directory set to its
-# containing dir.
-#
-# Note that not all possible configuration values are present in this
-# autogenerated file.
-#
-# All configuration values have a default; values that are commented out
-# serve to show the default.
-
-
-# If extensions (or modules to document with autodoc) are in another directory,
-# add these directories to sys.path here. If the directory is relative to the
-# documentation root, use os.path.abspath to make it absolute, like shown here.
-#sys.path.insert(0, os.path.abspath('.'))
-
-# -- General configuration ------------------------------------------------
-
-# If your documentation needs a minimal Sphinx version, state it here.
-#needs_sphinx = '1.0'
-
-# Add any Sphinx extension module names here, as strings. They can be
-# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
-# ones.
-extensions = [
- 'sphinx.ext.todo',
- 'sphinx.ext.mathjax',
- 'sphinx.ext.githubpages',
- 'sphinx.ext.intersphinx',
-]
-
-# Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
-
-# The suffix(es) of source filenames.
-# You can specify multiple suffix as a list of string:
-# source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
-
-# The encoding of source files.
-#source_encoding = 'utf-8-sig'
-
-# The master toctree document.
-master_doc = 'index'
-
-# General information about the project.
-project = u'Pinot'
-copyright = u'2018, Pinot development team'
-author = u'Pinot development team'
-
-# The version info for the project you're documenting, acts as replacement for
-# |version| and |release|, also used in various other places throughout the
-# built documents.
-#
-# The short X.Y version.
-version = u'0.1.0'
-# The full version, including alpha/beta/rc tags.
-release = u'0.1.0'
-
-# The language for content autogenerated by Sphinx. Refer to documentation
-# for a list of supported languages.
-#
-# This is also used if you do content translation via gettext catalogs.
-# Usually you set "language" from the command line for these cases.
-language = None
-
-# There are two options for replacing |today|: either, you set today to some
-# non-false value, then it is used:
-#today = ''
-# Else, today_fmt is used as the format for a strftime call.
-#today_fmt = '%B %d, %Y'
-
-# List of patterns, relative to source directory, that match files and
-# directories to ignore when looking for source files.
-# This patterns also effect to html_static_path and html_extra_path
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
-
-# The reST default role (used for this markup: `text`) to use for all
-# documents.
-#default_role = None
-
-# If true, '()' will be appended to :func: etc. cross-reference text.
-#add_function_parentheses = True
-
-# If true, the current module name will be prepended to all description
-# unit titles (such as .. function::).
-#add_module_names = True
-
-# If true, sectionauthor and moduleauthor directives will be shown in the
-# output. They are ignored by default.
-#show_authors = False
-
-# The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'sphinx'
-
-# A list of ignored prefixes for module index sorting.
-#modindex_common_prefix = []
-
-# If true, keep warnings as "system message" paragraphs in the built documents.
-#keep_warnings = False
-
-# If true, `todo` and `todoList` produce output, else they produce nothing.
-todo_include_todos = True
-
-
-# -- Options for HTML output ----------------------------------------------
-
-# The theme to use for HTML and HTML Help pages. See the documentation for
-# a list of builtin themes.
-html_theme = 'sphinx_rtd_theme'
-
-# Theme options are theme-specific and customize the look and feel of a theme
-# further. For a list of options available for each theme, see the
-# documentation.
-html_theme_options = {
- 'logo_only': True,
- # Toc options
- 'collapse_navigation': False
- }
-
-# Add any paths that contain custom themes here, relative to this directory.
-#html_theme_path = ["_themes",]
-
-# The name for this set of Sphinx documents.
-# "<project> v<release> documentation" by default.
-#html_title = u'Pinot v0.1.0'
-
-# A shorter title for the navigation bar. Default is the same as html_title.
-#html_short_title = None
-
-# The name of an image file (relative to this directory) to place at the top
-# of the sidebar.
-#html_logo = None
-
-# The name of an image file (relative to this directory) to use as a favicon of
-# the docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
-# pixels large.
-#html_favicon = None
-
-# Add any paths that contain custom static files (such as style sheets) here,
-# relative to this directory. They are copied after the builtin static files,
-# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = []
-
-# Add any extra paths that contain custom files (such as robots.txt or
-# .htaccess) here, relative to this directory. These files are copied
-# directly to the root of the documentation.
-#html_extra_path = []
-
-# If not None, a 'Last updated on:' timestamp is inserted at every page
-# bottom, using the given strftime format.
-# The empty string is equivalent to '%b %d, %Y'.
-#html_last_updated_fmt = None
-
-# If true, SmartyPants will be used to convert quotes and dashes to
-# typographically correct entities.
-#html_use_smartypants = True
-
-# Custom sidebar templates, maps document names to template names.
-#html_sidebars = {}
-
-# Additional templates that should be rendered to pages, maps page names to
-# template names.
-#html_additional_pages = {}
-
-# If false, no module index is generated.
-#html_domain_indices = True
-
-# If false, no index is generated.
-#html_use_index = True
-
-# If true, the index is split into individual pages for each letter.
-#html_split_index = False
-
-# If true, links to the reST sources are added to the pages.
-html_show_sourcelink = False
-
-# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
-#html_show_sphinx = True
-
-# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
-html_show_copyright = False
-
-# If true, an OpenSearch description file will be output, and all pages will
-# contain a <link> tag referring to it. The value of this option must be the
-# base URL from which the finished HTML is served.
-#html_use_opensearch = ''
-
-# This is the file name suffix for HTML files (e.g. ".xhtml").
-#html_file_suffix = None
-
-# Language to be used for generating the HTML full-text search index.
-# Sphinx supports the following languages:
-# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
-# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr', 'zh'
-#html_search_language = 'en'
-
-# A dictionary with options for the search language support, empty by default.
-# 'ja' uses this config value.
-# 'zh' user can custom change `jieba` dictionary path.
-#html_search_options = {'type': 'default'}
-
-# The name of a javascript file (relative to the configuration directory) that
-# implements a search results scorer. If empty, the default will be used.
-#html_search_scorer = 'scorer.js'
-
-# Output file base name for HTML help builder.
-htmlhelp_basename = 'Pinotdoc'
-
-# -- Options for LaTeX output ---------------------------------------------
-
-latex_elements = {
-# The paper size ('letterpaper' or 'a4paper').
-#'papersize': 'letterpaper',
-
-# The font size ('10pt', '11pt' or '12pt').
-#'pointsize': '10pt',
-
-# Additional stuff for the LaTeX preamble.
-#'preamble': '',
-
-# Latex figure (float) alignment
-#'figure_align': 'htbp',
-}
-
-# Grouping the document tree into LaTeX files. List of tuples
-# (source start file, target name, title,
-# author, documentclass [howto, manual, or own class]).
-latex_documents = [
- (master_doc, 'Pinot.tex', u'Pinot Documentation',
- u'Pinot development team', 'manual'),
-]
-
-# The name of an image file (relative to this directory) to place at the top of
-# the title page.
-#latex_logo = None
-
-# For "manual" documents, if this is true, then toplevel headings are parts,
-# not chapters.
-#latex_use_parts = False
-
-# If true, show page references after internal links.
-#latex_show_pagerefs = False
-
-# If true, show URL addresses after external links.
-#latex_show_urls = False
-
-# Documents to append as an appendix to all manuals.
-#latex_appendices = []
-
-# If false, no module index is generated.
-#latex_domain_indices = True
-
-
-# -- Options for manual page output ---------------------------------------
-
-# One entry per manual page. List of tuples
-# (source start file, name, description, authors, manual section).
-man_pages = [
- (master_doc, 'pinot', u'Pinot Documentation',
- [author], 1)
-]
-
-# If true, show URL addresses after external links.
-#man_show_urls = False
-
-
-# -- Options for Texinfo output -------------------------------------------
-
-# Grouping the document tree into Texinfo files. List of tuples
-# (source start file, target name, title, author,
-# dir menu entry, description, category)
-texinfo_documents = [
- (master_doc, 'Pinot', u'Pinot Documentation',
- author, 'Pinot', 'One line description of project.',
- 'Miscellaneous'),
-]
-
-
-# Documents to append as an appendix to all manuals.
-#texinfo_appendices = []
-
-# If false, no module index is generated.
-#texinfo_domain_indices = True
-
-# How to display URL addresses: 'footnote', 'no', or 'inline'.
-#texinfo_show_urls = 'footnote'
-
-# If true, do not generate a @detailmenu in the "Top" node's menu.
-#texinfo_no_detailmenu = False
diff --git a/docs/contribution_guidelines.rst b/docs/contribution_guidelines.rst
deleted file mode 100644
index 2d7d4b9..0000000
--- a/docs/contribution_guidelines.rst
+++ /dev/null
@@ -1,263 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-***********************
-Contribution Guidelines
-***********************
-
-.. contents:: Table of Contents
-
-Before you begin to contribute, make sure you have reviewed :ref:`dev-setup` and :ref:`code-modules` sections and that you have created your own fork of the pinot source code.
-
-Create a design document
-------------------------
-If your change is relatively minor, you can skip this step. If you are
-adding new major feature, we suggest that you add a design document
-and solicit comments from the community before submitting any code.
-
-`Here <https://cwiki.apache.org/confluence/display/PINOT/Design+Documents>`_
-is a list of current design documents.
-
-
-Create an issue for the change
-------------------------------
-Create a Pinot issue for the change you would like to make. Provide information on why the change is needed and how you
-plan to address it. Use the conversations on the issue as a way to validate assumptions and the right way to proceed.
-Be sure to review sections on :ref:`compatibility-changes` and :ref:`external-libs`.
-
-If you have a design document, please refer to the design documents
-in your Issue. You may even want to create multiple
-issues depending on the extent of your change.
-
-Once you are clear about what you want to do, proceed with the next steps listed below.
-
-Create a branch for your change
--------------------------------
-
-.. code-block:: none
-
- $ cd pinot
- #
- # ensure you are starting from the latest code base
- # the following steps, ensure your fork's (origin's) master is up-to-date
- #
- $ git fetch upstream
- $ git checkout master
- $ git merge upstream/master
- # create a branch for your issue
- $ git checkout -b <your issue branch>
-
-Make the necessary changes. If the changes you plan to make are too big, make sure you break it down into smaller tasks.
-
-Making the changes
-------------------
-
-Follow the recommendations/best-practices noted here when you are making changes.
-
-Code documentation
-^^^^^^^^^^^^^^^^^^
-Please ensure your code is adequately documented. Some things to consider for documentation:
-
-* Always include class level java docs.
- At the top class level, we are looking for information about what functionality is provided by the class,
- what state is maintained by the class, whether there are concurrency/thread-safety concerns and any exceptional behavior that the class might exhibit.
-
-* Document public methods and their parameters.
-
-Logging
-^^^^^^^
-
-* Ensure there is adequate logging for positive paths as well as exceptional paths. As a corollary to this, ensure logs are not noisy.
-* Do not use System.out.println to log messages. Use the ``slf4j`` loggers.
-* Use logging levels correctly: set level to ``debug`` for verbose logs useful for only for debugging.
-* Do not log stack traces via ``printStackTrace`` method of the exception.
-
-Exceptions and Exception-Handling
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-* Where possible, throw specific exceptions, preferably checked exceptions, so the callers can easily determine what the erroneous conditions that need to be handled are.
-* Avoid catching broad exceptions (ie, ``catch (Exception e)`` blocks, except for when this is in the run() method of a thread/runnable.
-
-Current Pinot code does not strictly adhere to this, but we would like to change this over time and adopt best practices around exception handling.
-
-.. _compatibility-changes:
-
-Backward and Forward compatibility changes
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-If you are making any changes to state stored, either in Zookeeper or in segments, make sure you consider both backward and forward compatibility issues.
-
-* For backward compatibility, consider cases where one component is using the new version and another is still on the old version. E.g., when the request format between broker and server is updated, consider resulting behaviors when a new broker is talking to an older server. Will it break?
-* For forward compatibility, consider rollback cases. E.g., consider what happens when state persisted by new code is handled by old code. Does the old code skip over new fields?
-
-.. _external-libs:
-
-External libraries
-^^^^^^^^^^^^^^^^^^
-Be cautious about pulling in external dependencies. You will need to consider multiple things when faced with a need to pull in a new library.
-
-* What capability is the addition of the library providing you with? Can existing libraries provide this functionality (may be with a little bit of effort)?
-* Is the external library maintained by an active community of contributors?
-* What are the licensing terms for the library. For more information about handling licenses, see :ref:`handling-licenses`.
-* Are you adding the library to :ref:`pinot-foundation` modules? This will affect the rest of the Pinot code base.
- If the new library pulls in a lot of transitive dependencies, then we might encounter unexpected issues with multiple classes in the classpath.
- These issues are hard to catch with tests as the order of loading the libraries at runtime matters. If you absolutely need the support, consider adding it via extension modules, see :ref:`extension-modules`.
-
-Testing your changes
-^^^^^^^^^^^^^^^^^^^^
-Automated tests are always recommended for contributions. Make sure you write tests so that:
-
-#. You verify the correctness of your contribution. This serves as proof to you as well as the reviewers.
-
-#. You future proof your contributions against code refactors or other changes. While this may not always be possible (see :ref:`testing_guidelines`), its a good goal to aim for.
-
-Identify a list of tests for the changes you have made. Depending on the scope of changes, you may need one or more of the following tests:
-
-* Unit Tests
-
- Make sure your code has the necessary class or method level unit tests. It is important to write both positive case as well as negative case tests.
- Document your tests well and add meaningful assertions in the tests; when the assertions fail, ensure that the right messages are logged with information that allows other to debug.
-
-
-* Integration Tests
-
- Add integration tests to cover End-to-End paths without relying on *mocking* (see note below). You ``MUST`` add integration tests for REST APIs, and must include tests that cover different error codes; i.e., 200 OK, 4xx or 5xx errors that are explicit contracts of the API.
-
-.. _testing_guidelines:
-
-Testing Guidelines
-^^^^^^^^^^^^^^^^^^
-* Mocking
-
- Use `Mockito <https://site.mockito.org/>`_ to mock classes to control specific behaviors - e.g., simulate various error conditions.
-
-.. note::
- DO NOT use advanced mock libraries such as `PowerMock <https://github.com/powermock/powermock>`_. They make bytecode level changes to allow tests for static/private members but this typically results in other tools like jacoco to fail. They also promote incorrect implementation choices that make it harder to test additional changes. When faced with a choice to use PowerMock or advanced mocking options, you might either need to refactor the code to work better with mocking or you actual [...]
-
-* Validate assumptions in tests
-
- Make sure that adequate asserts are added in the tests to verify that the tests are passing for the right reasons.
-
-
-* Write reliable tests
-
- Make sure you are writing tests that are reliable. If the tests depend on asynchronous events to be fired, do not add ``sleep`` to your tests. Where possible, use appropriate mocking or condition based triggers.
-
-.. _handling-licenses:
-
-License Headers for newly added files
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-All source code files should have license headers. To automatically add the header for any new file you plan to checkin, run in ``pinot`` top-level folder:
-
-.. code-block:: none
-
- mvn license:format
-
-.. note::
-
- If you checkin third-party code or files, please make sure you review Apache guidelines:
-
- * `Licences that can be included <https://www.apache.org/legal/resolved.html#what-can-we-include-in-an-asf-project-category-a>`_
-
- * `Licences that may be included <https://www.apache.org/legal/resolved.html#what-can-we-maybe-include-in-an-asf-project-category-b>`_
-
- * `Licenses that should not be included <https://www.apache.org/legal/resolved.html#what-can-we-not-include-in-an-asf-project-category-x>`_
-
-Once you determine the code you are pulling in adhere to the guidelines above, go ahead pull the changes in.
-Do not add license headers for them. Follow these instructions to ensure we are compliant with Apache Licensing process:
-
-* Under ``pinot/licenses`` add a LICENSE-<newlib> file that has the license terms of the included library.
-* Update the ``pinot/LICENSE`` file to indicate the newly added library file paths under the corresponding supported Licenses.
-* Update the exclusion rules for ``license`` and ``rat`` maven plugins in the parent pom: ``pinot/pom.xml``.
-
-If attention to the licensing terms in not paid early on, they will be caught much later in the process, when we prepare to make a new release.
-Updating code at that time to work with the right libraries at that time might require bigger refactoring changes and delay the release process.
-
-Creating a Pull Request (PR)
-----------------------------
-
-* Verifying code-style
-
- Run the following command to verify the code-style before posting a PR
-
-.. code-block:: none
-
- mvn checkstyle:check
-
-* Run tests
-
- Before you create a review request for the changes, make sure you have run the corresponding unit tests for your changes.
- You can run individual tests via the IDE or via maven command-line. Finally run all tests locally by running ``mvn clean install -Pbin-dist``.
-
- For changes that are related to performance issues or race conditions, it is hard to write reliable tests, so we recommend running manual stress tests to validate the changes. You ``MUST`` note the manual tests done in the PR description.
-
-* Push changes and create a PR for review
-
- Commit your changes with a meaningful commit message.
-
-.. code-block:: none
-
- $ git add <files required for the change>
- $ git commit -m "Meaningful oneliner for the change"
- $ git push origin <your issue branch>
-
- After this, create a PullRequest in `github <https://github.com/apache/incubator-pinot/pulls>`_. Include the following information in the description:
-
- * The changes that are included in the PR.
-
- * Design document, if any.
-
- * Information on any implementation choices that were made.
-
- * Evidence of sufficient testing. You ``MUST`` indicate the tests done, either manually or automated.
-
- Once the PR is created, the code base is compiled and all tests are run via ``travis``. Make sure you followup on any issues flagged by travis and address them.
- If you see test failures that are intermittent, ``please`` create an issue to track them.
-
- Once the ``travis`` run is clear, request reviews from atleast 2 committers on the project and be sure to gently to followup on the issue with the reviewers.
-
-* Once you receive comments on github on your changes, be sure to respond to them on github and address the concerns.
- If any discussions happen offline for the changes in question, make sure to capture the outcome of the discussion, so others can follow along as well.
-
- It is possible that while your change is being reviewed, other changes were made to the master branch. Be sure to pull rebase your change on the new changes thus:
-
-.. code-block:: none
-
- # commit your changes
- $ git add <updated files>
- $ git commit -m "Meaningful message for the udpate"
- # pull new changes
- $ git checkout master
- $ git merge upstream/master
- $ git checkout <your issue branch>
- $ git rebase master
-
- At this time, if rebase flags any conflicts, resolve the conflicts and follow the instructions provided by the rebase command.
-
- Run additional tests/validations for the new changes and update the PR by pushing your changes:
-
-.. code-block:: none
-
- $ git push origin <your issue branch>
-
-* When you have addressed all comments and have an approved PR, one of the committers can merge your PR.
-
-* After your change is merged, check to see if any documentation needs to be updated. If so, create a PR for documentation.
diff --git a/docs/customizations.rst b/docs/customizations.rst
deleted file mode 100644
index 0f496bf..0000000
--- a/docs/customizations.rst
+++ /dev/null
@@ -1,172 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-.. _customizing-pinot:
-
-Customizing Pinot
-===================
-
-There are a lot of places in Pinot which can be customized depending on the infrastructure or the use case. Below is a list of such customization points.
-
-
-.. image:: img/CustomizingPinot.png
-
-
-1. Generating Pinot segments
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Typically, data files will be available on some offline data storage, such as HDFS, and a Hadoop job can be written to read the data and create the segment. The `SegmentCreationJob <https://github.com/apache/incubator-pinot/blob/master/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java>`_ class contains a hadoop job for creating segments. This is a map only job, and the mapper can be found in `SegmentCreationMapper <https://github.com/apache/incubator-pinot/bl [...]
-
-Depending on how frequently new data is generated, you can schedule your jobs to run periodically using either cron or a scheduler such as `Azkaban <https://azkaban.github.io/>`_.
-
-
-2. Pluggable storage
-^^^^^^^^^^^^^^^^^^^^
-We expect the storage to be shared across controllers of the same cluster, such as NFS. You can write your own implementation of PinotFS to store segments in a data layer of your choice, for example Azure or S3. Please refer to the :ref:`pluggable-storage` doc for more details.
-
-
-3. Pluggable streams
-^^^^^^^^^^^^^^^^^^^^
-We provide out of the box support for consumption from Kafka stream. You can write your own plugin in order to consume from another pub-sub stream such as Azure EventHubs or Amazon Kinesis. Refer to the :ref:`pluggable-streams` doc for more details.
-
-
-4. Encrypting segments
-^^^^^^^^^^^^^^^^^^^^^^
-The `PinotCrypter <https://github.com/apache/incubator-pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/crypt/PinotCrypter.java>`_ will encrypt and decrypt segments when they are uploaded to the controller or downloaded by the server. This class is especially useful in cases where segments cannot be stored unencrypted in storage. By default, we use `NoOpPinotCrypter <https://github.com/apache/incubator-pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/crypt [...]
-
-You can write your own implementation by extending the ``org.apache.pinot.core.crypt.PinotCrypter`` interface in a similar fashion. The crypter can be used by passing the crypter class name in the header for the upload request as the header string ``CRYPTER``.
-
-
-5. Segment assignment strategies
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-We have various strategies for assigning segments to the available servers. These can be found under the `SegmentAssignmentStrategy <https://github.com/apache/incubator-pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/sharding/SegmentAssignmentStrategy.java>`_ interface. More details about which one to use depending on your usecase can be found in :ref:`tuning-pinot`. By default, the `BalanceNumSegmentAssignmentStrategy <https://github.com/apache/in [...]
-
-You can also write your own by implementing the ``org.apache.pinot.controller.helix.core.sharding.SegmentAssignmentStrategy`` interface. The segment assignment strategy can be configured for a table by setting it in the table config as
-
-
-.. code-block:: none
-
- {
- "segmentsConfig": {
- "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
- ...
- }
- }
-
-
-6. Data partitioning strategies
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-We have various algorithms to partition data during segment creation, as listed under the interface `PartitionFunction <https://github.com/apache/incubator-pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/data/partition/PartitionFunction.java>`_. No partitioning is done by default. You can write your own partitioning function by implementing the interface for ``org.apache.pinot.core.data.partition.PartitionFunction``. The partitioning function can be configured on the req [...]
-
-
-.. code-block:: none
-
- {
- "segmentPartitionConfig": {
- "columnPartitionMap": {
- "userId": {
- "functionName": "murmur",
- "numPartitions": 10
- }
- }
- }
- }
-
-
-7. Routing strategies
-^^^^^^^^^^^^^^^^^^^^^
-We have many routing strategies which you can find under the `RoutingTableBuilder <https://github.com/apache/incubator-pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/routing/builder/RoutingTableBuilder.java>`_ interface. More details about which one to use depending on your usecase can be found in :ref:`tuning-pinot`. By default we will use `DefaultOfflineRoutingTableBuilder <https://github.com/apache/incubator-pinot/blob/master/pinot-broker/src/main/java/org/apache [...]
-
-You can write your own routing table builder by implementing the ``org.apache.pinot.broker.routing.builder.RoutingTableBuilder`` interface. The routing table builder can be set in the table config as
-
-
-.. code-block:: none
-
- {
- "routing": {
- "routingTableBuilderName": "PartitionAwareRealtime",
- "routingTableBuilderOptions": {}
- }
- }
-
-
-8. Broker endpoint
-^^^^^^^^^^^^^^^^^^
-If you setup a usecase to have multiple brokers, you will have to develop your restful service to accept queries and distribute them across the brokers.
-
-
-9. Access Control
-^^^^^^^^^^^^^^^^^
-Access control can be setup at various points in Pinot, such as controller endpoints and broker query endpoints. By default we will use `AllowAllAccessFactory <https://github.com/apache/incubator-pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/api/access/AllowAllAccessFactory.java>`_ and hence not be enforcing any access controls. You can add access control by implementing the `AccessControlFactory <https://github.com/apache/incubator-pinot/blob/master/pinot- [...]
-
-The access control factory can be configured in the controller configs by setting the fully qualified class name of the AccessControlFactory in the property
-``controller.admin.access.control.factory.class``
-The access control factory can be configured in the broker configs by setting the fully qualified class name of the AccessControlFactory in the property
-``pinot.broker.access.control.class``
-Any other properties required for initializing the factory can be set in the broker configs as properties with the prefix ``pinot.broker.access.control``.
-
-
-10. Minion tasks
-^^^^^^^^^^^^^^^^
-Minion tasks can be configured for background activities such as purging data, adding indexes, merging segments. The existing minion tasks can be found under the `PinotTaskExecutor <https://github.com/apache/incubator-pinot/blob/master/pinot-minion/src/main/java/org/apache/pinot/minion/executor/PinotTaskExecutor.java>`_ interface. You can create your own minion task by implementing the PinotTaskExecutor and the `PinotTaskExecutorFactory <https://github.com/apache/incubator-pinot/blob/mas [...]
-
-
-.. code-block:: none
-
- {
- "task": {
- "taskTypeConfigsMap": {
- "PurgeTask": {}
- }
- }
- }
-
-
-11. Custom configs
-^^^^^^^^^^^^^^^^^^
-Custom configs can be injected into Pinot by adding the customConfigs field in the table config. This field accepts key value pairs.
-
-
-.. code-block:: none
-
- {
- "customConfigs": {
- "specialConfig": "testValue",
- "anotherSpecialConfig": "value"
- }
- }
-
-
-12. Metrics
-^^^^^^^^^^^
-We use `yammer MetricsRegistry <https://metrics.dropwizard.io/4.0.0/>`_ to collect metrics within our application components. These metrics can be published to a metrics server with the help of `MetricsRegistryRegistrationListener <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/MetricsRegistryRegistrationListener.java>`_ interface. By default, we publish metrics to JMX using the `JmxReporterMetricsRegistryRegistrationListe [...]
-
-You can write a listener to publish metrics to another metrics server by implementing the ``MetricsRegistryRegistrationListener`` interface. This listener can be injected into the controller by setting the fully qualified name of the class in the controller configs for the property ``pinot.controller.metrics.metricsRegistryRegistrationListeners``.
-
-You would have to design your own systems to view and monitor these metrics. A list of all the metrics published for each component can be found in `ControllerMeter <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerMeter.java>`_, `ControllerGauge <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerGauge.java>`_, `BrokerMeter <https://github.com [...]
-
-
-13. Deployables
-^^^^^^^^^^^^^^^
-You can deploy pinot server, broker, controller and minion individually. You can either use out of the box jars and start the components via `PinotAdministrator <https://github.com/apache/incubator-pinot/blob/master/pinot-tools/src/main/java/org/apache/pinot/tools/admin/PinotAdministrator.java>`_, or run via scripts as described in the :ref:`getting-started`.
-
-The starter classes for pinot controller, broker, server and minion are `ControllerStarter <https://github.com/apache/incubator-pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerStarter.java>`_, `HelixBrokerStarter <https://github.com/apache/incubator-pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/HelixBrokerStarter.java>`_, `HelixServerStarter <https://github.com/apache/incubator-pinot/blob/master/pinot-server/src/m [...]
-
-
diff --git a/docs/dev_env.rst b/docs/dev_env.rst
deleted file mode 100644
index b5c1930..0000000
--- a/docs/dev_env.rst
+++ /dev/null
@@ -1,96 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-.. _dev-setup:
-
-*********************
-Dev Environment Setup
-*********************
-
-.. contents:: Table of Contents
-
-
-To contribute to Pinot, please follow the instructions below.
-
-Git
----
-Pinot uses git for source code management. If you are new to Git, it will be good to review `basics <https://git-scm.com/book/en/v1/Getting-Started-Git-Basics>`_ of Git and a common tasks like `managing branches <https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell>`_ and `rebasing <https://git-scm.com/book/en/v2/Git-Branching-Rebasing>`_.
-
-Getting the Source Code
------------------------
-Create a fork
-^^^^^^^^^^^^^
-To limit the number of branches created on the Apache Pinot repository, we recommend that you create a fork by clicking on the fork button `in this page <https://github.com/apache/incubator-pinot>`_.
-Read more about `fork workflow here <https://www.atlassian.com/git/tutorials/comparing-workflows/forking-workflow>`_
-
-Clone the repository locally
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. code-block:: none
-
- $ mkdir workspace
- $ cd workspace
- $ git clone git@github.com:<github username>/pinot.git
- $ cd pinot
- # set upstream
- $ git remote add upstream https://github.com/apache/incubator-pinot
- # check that the upstream shows up correctly
- $ git remote -v
-
-Maven
------
-Pinot is a Maven project and familiarity with Maven will help you work with Pinot code. If you are new to Maven, you can
-read about Maven `here <maven.apache.org>`_ and `get a quick overview here <http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html>`_.
-
-Run the following maven command to setup the project.
-
-.. code-block:: none
-
- # compile, download sources
- $mvn install package -DskipTests -Pbin-dist -DdownloadSources -DdownloadJavadocs
-
-Setup IDE
----------
-Import the project into your favorite IDE. Setup stylesheet according to your IDE. We have provided instructions for
-intellij and eclipse. If you are using other IDEs, please ensure you use stylesheet based on
-`this <https://github.com/apache/incubator-pinot/blob/master/config/codestyle-intellij.xml>`_.
-
-Intellij
-^^^^^^^^
-To import the Pinot stylesheet this launch intellij and navigate to ``Preferences`` (on Mac) or ``Settings`` on Linux.
-
-* Navigate to ``Editor`` -> ``Code Style`` -> ``Java``
-* Select ``Import Scheme`` -> ``Intellij IDES code style XML``
-* Choose ``codestyle-intellij.xml`` from ``incubator-pinot/config`` folder of your workspace. Click Apply.
-
-.. figure:: img/import_scheme.png
-
-* Pinot uses TestNG for unit testing, thus to enable intellij to use them, make sure you have the testNG plugin
-installed and enabled.
-
-Eclipse
-^^^^^^^
-To import the Pinot stylesheet this launch eclipse and navigate to ``Preferences`` (on Mac) or ``Settings`` on Linux.
-
-* Navigate to Java->Code Style->Formatter
-* Choose ``codestyle-eclipse.xml`` from ``incubator-pinot/config folder`` of your workspace. Click Apply.
-
-.. figure:: img/eclipse_style.png
diff --git a/docs/dev_guide.rst b/docs/dev_guide.rst
deleted file mode 100644
index 79c2fd6..0000000
--- a/docs/dev_guide.rst
+++ /dev/null
@@ -1,36 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-:orphan:
-
-###############
-Developer Guide
-###############
-
-.. _dev-guide:
-
-.. toctree::
- :maxdepth: 1
-
- dev_env
- code_modules
- contribution_guidelines
- extensions
diff --git a/docs/docutils.conf b/docs/docutils.conf
deleted file mode 100644
index a4555de..0000000
--- a/docs/docutils.conf
+++ /dev/null
@@ -1,2 +0,0 @@
-[restructuredtext parser]
-syntax_highlight = short
\ No newline at end of file
diff --git a/docs/extensions.rst b/docs/extensions.rst
deleted file mode 100644
index 109849b..0000000
--- a/docs/extensions.rst
+++ /dev/null
@@ -1,32 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-Extending Pinot
-===============
-This section provides an overview of options to extend Pinot code to make Pinot work for environments not covered by default.
-
-.. toctree::
- :maxdepth: 1
-
- pluggable_streams
- segment_fetcher
- record_reader
- pluggable_storage
diff --git a/docs/getting_started.rst b/docs/getting_started.rst
deleted file mode 100644
index 8ca3235..0000000
--- a/docs/getting_started.rst
+++ /dev/null
@@ -1,317 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-.. _getting-started:
-
-Getting Started
-===============
-
-A quick way to get familiar with Pinot is to run the Pinot examples. The examples can be run either by compiling the
-code or by running the prepackaged Docker images.
-
-To demonstrate Pinot, let's start a simple one node cluster, along with the required Zookeeper. This demo setup also
-creates a table, generates some Pinot segments, then uploads them to the cluster in order to make them queryable.
-
-All of the setup is automated, so the only thing required at the beginning is to start the demonstration cluster.
-
-
-.. _compiling-code-section:
-
-Compiling the code
-~~~~~~~~~~~~~~~~~~
-
-One can also run the Pinot demonstration by checking out the code on GitHub, compiling it, and running it. Compiling
-Pinot requires JDK 8 or later and Apache Maven 3.
-
-#. Check out the code from GitHub (https://github.com/apache/incubator-pinot)
-#. With Maven installed, run ``mvn install package -DskipTests -Pbin-dist`` in the directory in which you checked out Pinot.
-#. Make the generated scripts executable:
-
-.. code-block:: none
-
- cd pinot-distribution/target/apache-pinot-incubating-<version>-SNAPSHOT-bin/apache-pinot-incubating-<version>-SNAPSHOT-bin; chmod +x bin/*.sh
-
-Trying out Batch quickstart demo
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-To run the demo with compiled code:
- ``bin/quick-start-batch.sh``
-
-Once the Pinot cluster is running, you can query it by going to http://localhost:9000
-
-You can also use the REST API to query Pinot, as well as the Java client. As this is outside of the scope of this
-introduction, the reference documentation to use the Pinot client APIs is in the :doc:`client_api` section.
-
-Pinot uses PQL, a SQL-like query language, to query data. Here are some sample queries:
-
-.. code-block:: sql
-
- /*Total number of documents in the table*/
- SELECT count(*) FROM baseballStats LIMIT 0
-
- /*Top 5 run scorers of all time*/
- SELECT sum('runs') FROM baseballStats GROUP BY playerName TOP 5 LIMIT 0
-
- /*Top 5 run scorers of the year 2000*/
- SELECT sum('runs') FROM baseballStats WHERE yearID=2000 GROUP BY playerName TOP 5 LIMIT 0
-
- /*Top 10 run scorers after 2000*/
- SELECT sum('runs') FROM baseballStats WHERE yearID>=2000 GROUP BY playerName
-
- /*Select playerName,runs,homeRuns for 10 records from the table and order them by yearID*/
- SELECT playerName,runs,homeRuns FROM baseballStats ORDER BY yearID LIMIT 10
-
-The full reference for the PQL query language is present in the :ref:`pql` section of the Pinot documentation.
-
-Trying out Streaming quickstart demo
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Pinot can ingest data from streaming sources such as Kafka.
-
-To run the demo with compiled code:
- ``bin/quick-start-streaming.sh``
-
-Once started, the demo will start Kafka, create a Kafka topic, and create a realtime Pinot table. Once created, Pinot
-will start ingesting events from the Kafka topic into the table. The demo also starts a consumer that consumes events
-from the Meetup API and pushes them into the Kafka topic that was created, causing new events modified on Meetup to
-show up in Pinot.
-
-.. role:: sql(code)
- :language: sql
-
-To show new events appearing, one can run :sql:`SELECT * FROM meetupRsvp ORDER BY mtime DESC LIMIT 50` repeatedly, which shows the
-last events that were ingested by Pinot.
-
-Experimenting with Pinot
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-Now we have a quick start Pinot cluster running locally. Below are step-by-step instructions on
-how to add a simple table to the Pinot system, how to upload a segment, and how to query the segment.
-
-Suppose we have a transcript in CSV format containing students' basic info and their scores for each subject.
-
-+------------+------------+-----------+-----------+-----------+-----------+
-| studentID | firstName | lastName | gender | subject | score |
-+============+============+===========+===========+===========+===========+
-| 200 | Lucy | Smith | Female | Maths | 3.8 |
-+------------+------------+-----------+-----------+-----------+-----------+
-| 200 | Lucy | Smith | Female | English | 3.5 |
-+------------+------------+-----------+-----------+-----------+-----------+
-| 201 | Bob | King | Male | Maths | 3.2 |
-+------------+------------+-----------+-----------+-----------+-----------+
-| 202 | Nick | Young | Male | Physics | 3.6 |
-+------------+------------+-----------+-----------+-----------+-----------+
-
-When we create a CSV file, we will also need a separate CSV config JSON file.
-
-First, however, we will create a working directory called ``getting-started`` (in this example, it is on ``Desktop``), and create two additional directories within ``getting-started`` called ``data``
-and ``config``.
-
-Note that we can create a variable for the working directory called ``WORKING_DIR``.
-
-.. code-block:: none
-
- $ mkdir getting-started
- $ WORKING_DIR=/Users/host1/Desktop/getting-started
- $ cd $WORKING_DIR
- $ mkdir getting-started/data
- $ mkdir getting started/config
-
-We will create the transcript CSV file in ``data``, and the CSV config file in ``config``.
-
-.. code-block:: none
-
- $ touch getting-started/data/test.csv
- $ touch getting-started/config/csv-record-reader-config.json
-
-The ``test.csv`` file should look like this, with no header line at the top:
-
-.. code-block:: none
-
- 200,Lucy,Smith,Female,Maths,3.8
- 200,Lucy,Smith,Female,English,3.5
- 201,Bob,King,Male,Maths,3.2
- 202,Nick,Young,Male,Physics,3.6
-
-Instead of using a header line, we will use the CSV config JSON file ``csv-record-reader-config.json`` to specify the header:
-
-.. code-block:: none
-
- {
- "header":"studentID,firstName,lastName,gender,subject,score",
- "fileFormat":"CSV"
- }
-
-In order to set up a table, we need to specify the schema of this transcript in ``transcript-schema.json``, which we will store in ``config``:
-
-.. code-block:: none
-
- $ touch getting-started/config/transcript-schema.json
-
-``transcript-schema.json`` should look like this:
-
-.. code-block:: none
-
- {
- "schemaName": "transcript",
- "dimensionFieldSpecs": [
- {
- "name": "studentID",
- "dataType": "STRING"
- },
- {
- "name": "firstName",
- "dataType": "STRING"
- },
- {
- "name": "lastName",
- "dataType": "STRING"
- },
- {
- "name": "gender",
- "dataType": "STRING"
- },
- {
- "name": "subject",
- "dataType": "STRING"
- }
- ],
- "metricFieldSpecs": [
- {
- "name": "score",
- "dataType": "FLOAT"
- }
- ]
- }
-
-Then, we need to specify the table config in another JSON file (also stored in ``config``), which links the schema to the table:
-
-.. code-block:: none
-
- $ touch getting-started/config/transcript-table-config.json
-
-``transcript-table-config.json`` should look like this:
-
-.. code-block:: none
-
- {
- "tableName": "transcript",
- "segmentsConfig" : {
- "replication" : "1",
- "schemaName" : "transcript",
- "segmentAssignmentStrategy" : "BalanceNumSegmentAssignmentStrategy"
- },
- "tenants" : {
- "broker":"DefaultTenant",
- "server":"DefaultTenant"
- },
- "tableIndexConfig" : {
- "invertedIndexColumns" : [],
- "loadMode" : "HEAP",
- "lazyLoad" : "false"
- },
- "tableType":"OFFLINE",
- "metadata": {}
- }
-
-
-To create pinot table, we can navigate to the directory in ``pinot-distribution`` that contains
-``pinot-admin.sh``, and use the command below:
-
-.. code-block:: none
-
- $ ./pinot-admin.sh AddTable -schemaFile $WORKING_DIR/config/transcript-schema.json -tableConfigFile $WORKING_DIR/config/transcript-table-config.json -exec
- Executing command: AddTable -tableConfigFile /Users/host1/Desktop/getting-started/config/transcript-table-config.json -schemaFile /Users/host1/Desktop/getting-started/config/transcript-schema.json -controllerHost [controller_host] -controllerPort 9000 -exec
- {"status":"Table transcript_OFFLINE successfully added"}
-
-At this point, the directory tree for our ``getting-started`` should look like this:
-
-.. code-block:: none
-
- |-- getting-started
- |-- data
- |-- test.csv
- |-- config
- |-- csv-record-reader-config.json
- |-- transcript-schema.json
- |-- transcript-table-config.json
-
-
-In order to upload our data to the Pinot cluster, we need to convert our CSV file into a Pinot Segment, which will be put in a new directory $WORKING_DIR/test2:
-
-.. code-block:: none
-
- $ ./pinot-admin.sh CreateSegment -dataDir $WORKING_DIR/data -format CSV -outDir $WORKING_DIR/test2 -tableName transcript -segmentName transcript_0 -overwrite -schemaFile $WORKING_DIR/config/transcript-schema.json -readerConfigFile $WORKING_DIR/config/csv-record-reader-config.json
- Executing command: CreateSegment -generatorConfigFile null -dataDir /Users/host1/Desktop/getting-started/data -format CSV -outDir /Users/host1/Desktop/getting-started/test2 -overwrite true -tableName transcript -segmentName transcript_0 -timeColumnName null -schemaFile /Users/host1/Desktop/getting-started/config/transcript-schema.json -readerConfigFile /Users/host1/Desktop/getting-started/config/csv-record-reader-config.json -enableStarTreeIndex false -starTreeIndexSpecFile null -hllS [...]
- Accepted files: [file:/Users/host1/Desktop/getting-started/data/test.csv]
- Finished building StatsCollector!
- Collected stats for 4 documents
- Created dictionary for STRING column: studentID with cardinality: 1, max length in bytes: 4, range: null to null
- Created dictionary for STRING column: firstName with cardinality: 3, max length in bytes: 4, range: Bob to Nick
- Created dictionary for STRING column: lastName with cardinality: 3, max length in bytes: 5, range: King to Young
- Created dictionary for FLOAT column: score with cardinality: 4, range: 3.2 to 3.8
- Created dictionary for STRING column: gender with cardinality: 2, max length in bytes: 6, range: Female to Male
- Created dictionary for STRING column: subject with cardinality: 3, max length in bytes: 7, range: English to Physics
- Start building IndexCreator!
- Finished records indexing in IndexCreator!
- Finished segment seal!
- Converting segment: /Users/host1/Desktop/getting-started/test2/transcript_0_0 to v3 format
- v3 segment location for segment: transcript_0_0 is /Users/host1/Desktop/getting-started/test2/transcript_0_0/v3
- Deleting files in v1 segment directory: /Users/host1/Desktop/getting-started/test2/transcript_0_0
- Driver, record read time : 1
- Driver, stats collector time : 0
- Driver, indexing time : 0
-
-Once we have the Pinot Segment, we can upload it to our cluster:
-
-.. code-block:: none
-
- $ ./pinot-admin.sh UploadSegment -segmentDir $WORKING_DIR/test2/
- Executing command: UploadSegment -controllerHost [controller_host] -controllerPort 9000 -segmentDir /Users/host1/Desktop/test2/
- Compressing segment transcript_0_0
- Uploading segment transcript_0_0.tar.gz
- Sending request: http://[controller_host]:9000/v2/segments to controller: [controller_host], version: 0.2.0-SNAPSHOT-68092ab9eb83af173d725ec685c22ba4eb5bacf9
-
-You did it! Now we can query the data in Pinot.
-
-To get all the number of rows in the table:
-
-.. code-block:: none
-
- $ ./pinot-admin.sh PostQuery -brokerPort 8000 -query "select count(*) from transcript"
- Executing command: PostQuery -brokerHost [controller_host] -brokerPort 8000 -query select count(*) from transcript
- Result: {"aggregationResults":[{"function":"count_star","value":"4"}],"exceptions":[],"numServersQueried":1,"numServersResponded":1,"numSegmentsQueried":1,"numSegmentsProcessed":1,"numSegmentsMatched":1,"numDocsScanned":4,"numEntriesScannedInFilter":0,"numEntriesScannedPostFilter":0,"numGroupsLimitReached":false,"totalDocs":4,"timeUsedMs":7,"segmentStatistics":[],"traceInfo":{}}
-
-To get the average score of subject Maths:
-
-.. code-block:: none
-
- $ ./pinot-admin.sh PostQuery -brokerPort 8000 -query "select avg(score) from transcript where subject = \"Maths\""
- Executing command: PostQuery -brokerHost [controller_host] -brokerPort 8000 -query select avg(score) from transcript where subject = "Maths"
- Result: {"aggregationResults":[{"function":"avg_score","value":"3.50000"}],"exceptions":[],"numServersQueried":1,"numServersResponded":1,"numSegmentsQueried":1,"numSegmentsProcessed":1,"numSegmentsMatched":1,"numDocsScanned":2,"numEntriesScannedInFilter":4,"numEntriesScannedPostFilter":2,"numGroupsLimitReached":false,"totalDocs":4,"timeUsedMs":33,"segmentStatistics":[],"traceInfo":{}}
-
-To get the average score for Lucy Smith:
-
-.. code-block:: none
-
- $ ./pinot-admin.sh PostQuery -brokerPort 8000 -query "select avg(score) from transcript where firstName = \"Lucy\" and lastName = \"Smith\""
- Executing command: PostQuery -brokerHost [controller_host] -brokerPort 8000 -query select avg(score) from transcript where firstName = "Lucy" and lastName = "Smith"
- Result: {"aggregationResults":[{"function":"avg_score","value":"3.65000"}],"exceptions":[],"numServersQueried":1,"numServersResponded":1,"numSegmentsQueried":1,"numSegmentsProcessed":1,"numSegmentsMatched":1,"numDocsScanned":2,"numEntriesScannedInFilter":6,"numEntriesScannedPostFilter":2,"numGroupsLimitReached":false,"totalDocs":4,"timeUsedMs":67,"segmentStatistics":[],"traceInfo":{}}
diff --git a/docs/img/CustomizingPinot.png b/docs/img/CustomizingPinot.png
deleted file mode 100644
index 1991d66..0000000
Binary files a/docs/img/CustomizingPinot.png and /dev/null differ
diff --git a/docs/img/Pinot - Hybrid flow.png b/docs/img/Pinot - Hybrid flow.png
deleted file mode 100644
index 8c9364f..0000000
Binary files a/docs/img/Pinot - Hybrid flow.png and /dev/null differ
diff --git a/docs/img/Pinot-Offline-only-flow.png b/docs/img/Pinot-Offline-only-flow.png
deleted file mode 100644
index a3fcc5c..0000000
Binary files a/docs/img/Pinot-Offline-only-flow.png and /dev/null differ
diff --git a/docs/img/PinotBroker.png b/docs/img/PinotBroker.png
deleted file mode 100644
index f14badd..0000000
Binary files a/docs/img/PinotBroker.png and /dev/null differ
diff --git a/docs/img/PinotController.png b/docs/img/PinotController.png
deleted file mode 100644
index f62d5ed..0000000
Binary files a/docs/img/PinotController.png and /dev/null differ
diff --git a/docs/img/PinotFoundation.png b/docs/img/PinotFoundation.png
deleted file mode 100644
index c2547d4..0000000
Binary files a/docs/img/PinotFoundation.png and /dev/null differ
diff --git a/docs/img/PinotMinionHadoop.png b/docs/img/PinotMinionHadoop.png
deleted file mode 100644
index 40a008d..0000000
Binary files a/docs/img/PinotMinionHadoop.png and /dev/null differ
diff --git a/docs/img/PinotServer.png b/docs/img/PinotServer.png
deleted file mode 100644
index 9cbb2cb..0000000
Binary files a/docs/img/PinotServer.png and /dev/null differ
diff --git a/docs/img/dictionary.png b/docs/img/dictionary.png
deleted file mode 100644
index d4c3d20..0000000
Binary files a/docs/img/dictionary.png and /dev/null differ
diff --git a/docs/img/eclipse_style.png b/docs/img/eclipse_style.png
deleted file mode 100644
index bc660e7..0000000
Binary files a/docs/img/eclipse_style.png and /dev/null differ
diff --git a/docs/img/hlc-arch.png b/docs/img/hlc-arch.png
deleted file mode 100644
index 0bc2974..0000000
Binary files a/docs/img/hlc-arch.png and /dev/null differ
diff --git a/docs/img/image2014-11-12-1-54-56.png b/docs/img/image2014-11-12-1-54-56.png
deleted file mode 100644
index 083c063..0000000
Binary files a/docs/img/image2014-11-12-1-54-56.png and /dev/null differ
diff --git a/docs/img/image2014-11-12-19-54-12.png b/docs/img/image2014-11-12-19-54-12.png
deleted file mode 100644
index 5531d53..0000000
Binary files a/docs/img/image2014-11-12-19-54-12.png and /dev/null differ
diff --git a/docs/img/image2014-11-12-20-59-50.png b/docs/img/image2014-11-12-20-59-50.png
deleted file mode 100644
index 842c755..0000000
Binary files a/docs/img/image2014-11-12-20-59-50.png and /dev/null differ
diff --git a/docs/img/image2014-11-13-11-21-7.png b/docs/img/image2014-11-13-11-21-7.png
deleted file mode 100644
index 1b8ebbe..0000000
Binary files a/docs/img/image2014-11-13-11-21-7.png and /dev/null differ
diff --git a/docs/img/image2014-11-13-11-22-57.png b/docs/img/image2014-11-13-11-22-57.png
deleted file mode 100644
index a7ea8bc..0000000
Binary files a/docs/img/image2014-11-13-11-22-57.png and /dev/null differ
diff --git a/docs/img/image2015-4-14-194451.png b/docs/img/image2015-4-14-194451.png
deleted file mode 100644
index 2b6269b..0000000
Binary files a/docs/img/image2015-4-14-194451.png and /dev/null differ
diff --git a/docs/img/image2015-5-12-7-24-58.png b/docs/img/image2015-5-12-7-24-58.png
deleted file mode 100644
index 8dec967..0000000
Binary files a/docs/img/image2015-5-12-7-24-58.png and /dev/null differ
diff --git a/docs/img/image2015-5-13 15-2-9.png b/docs/img/image2015-5-13 15-2-9.png
deleted file mode 100644
index b42d9ba..0000000
Binary files a/docs/img/image2015-5-13 15-2-9.png and /dev/null differ
diff --git a/docs/img/image2015-5-17 13-32-28.png b/docs/img/image2015-5-17 13-32-28.png
deleted file mode 100644
index 4bda362..0000000
Binary files a/docs/img/image2015-5-17 13-32-28.png and /dev/null differ
diff --git a/docs/img/image2015-5-17 17-59-10.png b/docs/img/image2015-5-17 17-59-10.png
deleted file mode 100644
index 01438e5..0000000
Binary files a/docs/img/image2015-5-17 17-59-10.png and /dev/null differ
diff --git a/docs/img/image2015-5-19 0-29-34.png b/docs/img/image2015-5-19 0-29-34.png
deleted file mode 100644
index a9ff13d..0000000
Binary files a/docs/img/image2015-5-19 0-29-34.png and /dev/null differ
diff --git a/docs/img/image2015-5-19 0-58-54.png b/docs/img/image2015-5-19 0-58-54.png
deleted file mode 100644
index d20a94a..0000000
Binary files a/docs/img/image2015-5-19 0-58-54.png and /dev/null differ
diff --git a/docs/img/image2015-5-19 1-28-35.png b/docs/img/image2015-5-19 1-28-35.png
deleted file mode 100644
index fbfd864..0000000
Binary files a/docs/img/image2015-5-19 1-28-35.png and /dev/null differ
diff --git a/docs/img/image2015-5-19 1-47-44.png b/docs/img/image2015-5-19 1-47-44.png
deleted file mode 100644
index fbfd864..0000000
Binary files a/docs/img/image2015-5-19 1-47-44.png and /dev/null differ
diff --git a/docs/img/image2015-5-19 1-59-51.png b/docs/img/image2015-5-19 1-59-51.png
deleted file mode 100644
index b016f09..0000000
Binary files a/docs/img/image2015-5-19 1-59-51.png and /dev/null differ
diff --git a/docs/img/import_scheme.png b/docs/img/import_scheme.png
deleted file mode 100644
index 092debf..0000000
Binary files a/docs/img/import_scheme.png and /dev/null differ
diff --git a/docs/img/list-schemas.png b/docs/img/list-schemas.png
deleted file mode 100644
index 3bd0685..0000000
Binary files a/docs/img/list-schemas.png and /dev/null differ
diff --git a/docs/img/llc-arch.png b/docs/img/llc-arch.png
deleted file mode 100644
index a8928ad..0000000
Binary files a/docs/img/llc-arch.png and /dev/null differ
diff --git a/docs/img/no-dictionary.png b/docs/img/no-dictionary.png
deleted file mode 100644
index 33877f7..0000000
Binary files a/docs/img/no-dictionary.png and /dev/null differ
diff --git a/docs/img/partitioning.png b/docs/img/partitioning.png
deleted file mode 100644
index 243e9f9..0000000
Binary files a/docs/img/partitioning.png and /dev/null differ
diff --git a/docs/img/pinot-architecture.png b/docs/img/pinot-architecture.png
deleted file mode 100644
index 338953c..0000000
Binary files a/docs/img/pinot-architecture.png and /dev/null differ
diff --git a/docs/img/pinot-console.png b/docs/img/pinot-console.png
deleted file mode 100644
index f73405b..0000000
Binary files a/docs/img/pinot-console.png and /dev/null differ
diff --git a/docs/img/pluggable_streams.png b/docs/img/pluggable_streams.png
deleted file mode 100644
index b5a3cd7..0000000
Binary files a/docs/img/pluggable_streams.png and /dev/null differ
diff --git a/docs/img/rebalance-table.png b/docs/img/rebalance-table.png
deleted file mode 100644
index 65de953..0000000
Binary files a/docs/img/rebalance-table.png and /dev/null differ
diff --git a/docs/img/replica-group.png b/docs/img/replica-group.png
deleted file mode 100644
index 186477d..0000000
Binary files a/docs/img/replica-group.png and /dev/null differ
diff --git a/docs/img/sorted-forward.png b/docs/img/sorted-forward.png
deleted file mode 100644
index 9c2e9a6..0000000
Binary files a/docs/img/sorted-forward.png and /dev/null differ
diff --git a/docs/img/sorted-inverted.png b/docs/img/sorted-inverted.png
deleted file mode 100644
index 9d607a6..0000000
Binary files a/docs/img/sorted-inverted.png and /dev/null differ
diff --git a/docs/img/znode_layout.png b/docs/img/znode_layout.png
deleted file mode 100644
index 81552fc..0000000
Binary files a/docs/img/znode_layout.png and /dev/null differ
diff --git a/docs/in_production.rst b/docs/in_production.rst
deleted file mode 100644
index d237707..0000000
--- a/docs/in_production.rst
+++ /dev/null
@@ -1,210 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-Running Pinot in Production
-===========================
-
-Requirements
-~~~~~~~~~~~~
-
-You will need the following in order to run pinot in production:
-
-* Hardware for controller/broker/servers as per your load
-* Working installation of Zookeeper that Pinot can use. We recommend setting aside a path within zookpeer and including that path in pinot.controller.zkStr. Pinot will create its own cluster under this path (cluster name decided by pinot.controller.helixClusterName)
-* Shared storage mounted on controllers (if you plan to have multiple controllers for the same cluster). Alternatively, an implementation of PinotFS that the Pinot hosts have access to.
-* HTTP load balancers for spraying queries across brokers (or other mechanism to balance queries)
-* HTTP load balancers for spraying controller requests (e.g. segment push, or other controller APIs) or other mechanisms for distribution of these requests.
-
-Deploying Pinot
-~~~~~~~~~~~~~~~
-
-In general, when deploying Pinot services, it is best to adhere to a specific ordering in which the various components should be deployed. This deployment order is recommended in case of the scenario that there might be protocol or other significant differences, the deployments go out in a predictable order in which failure due to these changes can be avoided.
-
-The ordering is as follows:
-
-#. pinot-controller
-#. pinot-broker
-#. pinot-server
-#. pinot-minion
-
-Managing Pinot
-~~~~~~~~~~~~~~
-
-Pinot provides a web-based management console and a command-line utility (``pinot-admin.sh``) in order to help provision and manage pinot clusters.
-
-Pinot Management Console
-------------------------
-
-The web based management console allows operations on tables, tenants, segments and schemas.
-You can access the console via ``http://controller-host:port/help``.
-The console also allows you to enter queries for interactive debugging.
-Here are some screen-shots from the console.
-
- .. figure:: img/pinot-console.png
-
-Listing all the schemas in the Pinot cluster:
-
- .. figure:: img/list-schemas.png
-
-Rebalancing segments of a table:
-
- .. figure:: img/rebalance-table.png
-
-Command line utility (pinot-admin.sh)
--------------------------------------
-
-The command line utility (``pinot-admin.sh``) can be generated by running
-``mvn install package -DskipTests -Pbin-dist`` in the directory in which you checked out Pinot.
-
-Here is an example of invoking the command to create a pinot segment:
-
-.. code-block:: none
-
- $ ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/bin/pinot-admin.sh CreateSegment -dataDir /Users/host1/Desktop/test/ -format CSV -outDir /Users/host1/Desktop/test2/ -tableName baseballStats -segmentName baseballStats_data -overwrite -schemaFile ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/examples/batch/baseballStats/baseballStats_schema.json
- Executing command: CreateSegment -generatorConfigFile null -dataDir /Users/host1/Desktop/test/ -format CSV -outDir /Users/host1/Desktop/test2/ -overwrite true -tableName baseballStats -segmentName baseballStats_data -timeColumnName null -schemaFile ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/examples/batch/baseballStats/baseballStats_schema.json -readerConfigFile null -enableStarTreeIndex false -starTreeIndexSpecFil [...]
- Accepted files: [/Users/host1/Desktop/test/baseballStats_data.csv]
- Finished building StatsCollector!
- Collected stats for 97889 documents
- Created dictionary for INT column: homeRuns with cardinality: 67, range: 0 to 73
- Created dictionary for INT column: playerStint with cardinality: 5, range: 1 to 5
- Created dictionary for INT column: groundedIntoDoublePlays with cardinality: 35, range: 0 to 36
- Created dictionary for INT column: numberOfGames with cardinality: 165, range: 1 to 165
- Created dictionary for INT column: AtBatting with cardinality: 699, range: 0 to 716
- Created dictionary for INT column: stolenBases with cardinality: 114, range: 0 to 138
- Created dictionary for INT column: tripples with cardinality: 32, range: 0 to 36
- Created dictionary for INT column: hitsByPitch with cardinality: 41, range: 0 to 51
- Created dictionary for STRING column: teamID with cardinality: 149, max length in bytes: 3, range: ALT to WSU
- Created dictionary for INT column: numberOfGamesAsBatter with cardinality: 166, range: 0 to 165
- Created dictionary for INT column: strikeouts with cardinality: 199, range: 0 to 223
- Created dictionary for INT column: sacrificeFlies with cardinality: 20, range: 0 to 19
- Created dictionary for INT column: caughtStealing with cardinality: 36, range: 0 to 42
- Created dictionary for INT column: baseOnBalls with cardinality: 154, range: 0 to 232
- Created dictionary for STRING column: playerName with cardinality: 11976, max length in bytes: 43, range: to Zoilo Casanova
- Created dictionary for INT column: doules with cardinality: 64, range: 0 to 67
- Created dictionary for STRING column: league with cardinality: 7, max length in bytes: 2, range: AA to UA
- Created dictionary for INT column: yearID with cardinality: 143, range: 1871 to 2013
- Created dictionary for INT column: hits with cardinality: 250, range: 0 to 262
- Created dictionary for INT column: runsBattedIn with cardinality: 175, range: 0 to 191
- Created dictionary for INT column: G_old with cardinality: 166, range: 0 to 165
- Created dictionary for INT column: sacrificeHits with cardinality: 54, range: 0 to 67
- Created dictionary for INT column: intentionalWalks with cardinality: 45, range: 0 to 120
- Created dictionary for INT column: runs with cardinality: 167, range: 0 to 192
- Created dictionary for STRING column: playerID with cardinality: 18107, max length in bytes: 9, range: aardsda01 to zwilldu01
- Start building IndexCreator!
- Finished records indexing in IndexCreator!
- Finished segment seal!
- Converting segment: /Users/host1/Desktop/test2/baseballStats_data_0 to v3 format
- v3 segment location for segment: baseballStats_data_0 is /Users/host1/Desktop/test2/baseballStats_data_0/v3
- Deleting files in v1 segment directory: /Users/host1/Desktop/test2/baseballStats_data_0
- Driver, record read time : 369
- Driver, stats collector time : 0
- Driver, indexing time : 373
-
-Here is an example of executing a query on a Pinot table:
-
-.. code-block:: none
-
- $ ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/bin/pinot-admin.sh PostQuery -query "select count(*) from baseballStats" ./pinot-distribution/target/apache-pinot-incubaExecuting command: PostQuery -brokerHost [broker_host] -brokerPort [broker_port] -query select count(*) from baseballStats
- Result: {"aggregationResults":[{"function":"count_star","value":"97889"}],"exceptions":[],"numServersQueried":1,"numServersResponded":1,"numSegmentsQueried":1,"numSegmentsProcessed":1,"numSegmentsMatched":1,"numDocsScanned":97889,"numEntriesScannedInFilter":0,"numEntriesScannedPostFilter":0,"numGroupsLimitReached":false,"totalDocs":97889,"timeUsedMs":107,"segmentStatistics":[],"traceInfo":{}}
-
-Monitoring Pinot
-~~~~~~~~~~~~~~~~
-
-Pinot exposes several metrics to monitor the service and ensure that pinot users are not experiencing issues. In this section we discuss some of the key metrics that are useful to monitor. A full list of metrics is available in the `Metrics <customizations.html#metrics>`_ section.
-
-Pinot Server
-------------
-
-* Missing Segments - `NUM_MISSING_SEGMENTS <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMeter.java>`_
-
- * Number of missing segments that the broker queried for (expected to be on the server) but the server didn't have. This can be due to retention or stale routing table.
-
-* Query latency - `TOTAL_QUERY_TIME <https://github.com/apache/incubator-pinot/blob/ce2d9ee9dc73b2d7273a63a4eede774eb024ea8f/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerQueryPhase.java>`_
-
- * Total time to take from receiving to finishing executing the query.
-
-* Query Execution Exceptions - `QUERY_EXECUTION_EXCEPTIONS <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMeter.java>`_
-
- * The number of exception which might have occurred during query execution.
-
-* Realtime Consumption Status - `LLC_PARTITION_CONSUMING <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerGauge.java>`_
-
- * This gives a binary value based on whether low-level consumption is healthy (1) or unhealthy (0). It's important to ensure at least a single replica of each partition is consuming.
-
-* Realtime Highest Offset Consumed - `HIGHEST_STREAM_OFFSET_CONSUMED <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerGauge.java>`_
-
- * The highest offset which has been consumed so far.
-
-Pinot Broker
-------------
-
-* Incoming QPS (per broker) - `QUERIES <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMeter.java>`_
-
- * The rate which an individual broker is receiving queries. Units are in QPS.
-
-* Dropped Requests - `REQUEST_DROPPED_DUE_TO_SEND_ERROR <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMeter.java>`_, `REQUEST_DROPPED_DUE_TO_CONNECTION_ERROR <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMeter.java>`_, `REQUEST_DROPPED_DUE_TO_ACCESS_ERROR <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/ap [...]
-
- * These multiple metrics will indicate if a query is dropped, ie the processing of that query has been forfeited for some reason.
-
-* Partial Responses - `BROKER_RESPONSES_WITH_PARTIAL_SERVERS_RESPONDED <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMeter.java>`_
-
- * Indicates a count of partial responses. A partial response is when at least 1 of the requested servers fails to respond to the query.
-
-* Table QPS quota exceeded - `QUERY_QUOTA_EXCEEDED <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMeter.java>`_
-
- * Binary metric which will indicate when the configured QPS quota for a table is exceeded (1) or if there is capacity remaining (0).
-
-* Table QPS quota usage percent - `QUERY_QUOTA_CAPACITY_UTILIZATION_RATE <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerGauge.java>`_
-
- * Percentage of the configured QPS quota being utilized.
-
-Pinot Controller
-----------------
-
-Many of the controller metrics include a table name and thus are dynamically generated in the code. The metrics below point to the classes which generate the corresponding metrics.
-
-To get the real metric name, the easiest route is to spin up a controller instance, create a table with the desired name and look through the generated metrics.
-
-.. todo::
-
- Give a more detailed explanation of how metrics are generated, how to identify real metrics names and where to find them in the code.
-
-* Percent Segments Available - `PERCENT_SEGMENTS_AVAILABLE <https://github.com/apache/incubator-pinot/blob/ce2d9ee9dc73b2d7273a63a4eede774eb024ea8f/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerGauge.java>`_
-
- * Percentage of complete online replicas in external view as compared to replicas in ideal state.
-
-* Segments in Error State - `SEGMENTS_IN_ERROR_STATE <https://github.com/apache/incubator-pinot/blob/ce2d9ee9dc73b2d7273a63a4eede774eb024ea8f/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerGauge.java>`_
-
- * Number of segments in an ``ERROR`` state for a given table.
-
-* Last push delay - Generated in the `ValidationMetrics <https://github.com/apache/incubator-pinot/blob/ce2d9ee9dc73b2d7273a63a4eede774eb024ea8f/pinot-common/src/main/java/org/apache/pinot/common/metrics/ValidationMetrics.java>`_ class.
-
- * The time in hours since the last time an offline segment has been pushed to the controller.
-
-* Percent of replicas up - `PERCENT_OF_REPLICAS <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerGauge.java>`_
-
- * Percentage of complete online replicas in external view as compared to replicas in ideal state.
-
-* Table storage quota usage percent - `TABLE_STORAGE_QUOTA_UTILIZATION <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerGauge.java>`_
-
- * Shows how much of the table's storage quota is currently being used, metric will a percentage of a the entire quota.
-
-
diff --git a/docs/index.rst b/docs/index.rst
deleted file mode 100644
index e1dbafc..0000000
--- a/docs/index.rst
+++ /dev/null
@@ -1,65 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. Pinot documentation master file, created by
- sphinx-quickstart on Fri Feb 9 14:54:43 2018.
- You can adapt this file completely to your liking, but it should at least
- contain the root `toctree` directive.
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-#####
-Pinot
-#####
-
-.. toctree::
- :maxdepth: 1
- :caption: Introduction
-
- intro
- architecture
- getting_started
-
-.. toctree::
- :maxdepth: 1
- :caption: User Guide
-
- pql_examples
- client_api
-
-.. toctree::
- :maxdepth: 1
- :caption: Admin Guide
-
- tableconfig_schema
- schema
- in_production
- batch_data_ingestion
- pinot_hadoop
- customizations
- tuning_pinot
-
-.. toctree::
- :maxdepth: 1
- :caption: Developer Guide
-
- dev_env
- code_modules
- contribution_guidelines
- extensions
diff --git a/docs/index_techniques.rst b/docs/index_techniques.rst
deleted file mode 100644
index c44028d..0000000
--- a/docs/index_techniques.rst
+++ /dev/null
@@ -1,178 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-.. TODO: add more details
-
-
-Index Techniques
-================
-
-.. contents:: Table of Contents
-
-Pinot currently supports the following index techniques, where each of them have their own advantages in different query
-scenarios. By default, Pinot will use ``dictionary-encoded forward index`` for each column.
-
-Forward Index
--------------
-
-Dictionary-Encoded Forward Index with Bit Compression (Default)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-For each unique value from a column, we assign an id to it, and build a dictionary from the id to the value. Then in the
-forward index, we only store the bit-compressed ids instead of the values.
-
-With few number of unique values, dictionary-encoding can significantly improve the space efficiency of the storage.
-
-The below diagram shows the dictionary encoding for two columns with ``integer`` and ``string`` types. As seen in the
-``colA``, dictionary encoding will save significant amount of space for duplicated values. On the other hand, ``colB``
-has no duplicated data. Dictionary encoding will not compress much data in this case where there are a lot of unique
-values in the column. For ``string`` type, we pick the length of the longest value and use it as the length for
-dictionary's fixed length value array. In this case, padding overhead can be high if there are a large number of unique
-values for a column.
-
-.. image:: img/dictionary.png
-
-
-Raw Value Forward Index
-~~~~~~~~~~~~~~~~~~~~~~~
-
-In contrast to the dictionary-encoded forward index, raw value forward index directly stores values instead of ids.
-
-Without the dictionary, the dictionary lookup step can be skipped for each value fetch. Also, the index can take
-advantage of the good locality of the values, thus improve the performance of scanning large number of values.
-
-A typical use case to apply raw value forward index is when the column has a large number of unique values and the
-dictionary does not provide much compression. As seen the above diagram for dictionary encoding, scanning values
-with a dictionary involves a lot of random access because we need to perform dictionary look up. On the other hand,
-we can scan values sequentially with raw value forward index and this can improve performance a lot when applied
-appropriately.
-
-.. image:: img/no-dictionary.png
-
-Raw value forward index can be configured for a table by setting it in the table config as
-
-.. code-block:: none
-
- {
- "tableIndexConfig": {
- "noDictionaryColumns": [
- "column_name",
- ...
- ],
- ...
- }
- }
-
-
-Sorted Forward Index with Run-Length Encoding
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-When a column is physically sorted, Pinot uses a sorted forward index with run-length encoding on top of the
-dictionary-encoding. Instead of saving dictionary ids for each document id, we store a pair of start and end
-document id for each value. (The below diagram does not include dictionary encoding layer for simplicity.)
-
-.. image:: img/sorted-forward.png
-
-
-Sorted forward index has the advantages of both good compression and data locality. Sorted forward index can
-also be used as inverted index.
-
-Sorted index can be configured for a table by setting it in the table config as
-
-.. code-block:: none
-
- {
- "tableIndexConfig": {
- "sortedColumn": [
- "column_name"
- ],
- ...
- }
- }
-
-Realtime server will sort data on ``sortedColumn`` when generating segment internally. For offline push, input data
-needs to be sorted before running Pinot segment conversion and push job.
-
-When applied correctly, one can find the following information on the segment metadata.
-
-.. code-block:: none
-
- $ grep memberId <segment_name>/v3/metadata.properties | grep isSorted
- column.memberId.isSorted = true
-
-
-Inverted Index (only available with dictionary-encoded indexes)
----------------------------------------------------------------
-
-Bitmap Inverted Index
-~~~~~~~~~~~~~~~~~~~~~
-
-When inverted index is enabled for a column, Pinot maintains a map from each value to a bitmap, which makes value
-lookup to be constant time. When you have a column that is used for filtering frequently, adding an inverted index
-will improve the performance greatly.
-
-Inverted index can be configured for a table by setting it in the table config as
-
-.. code-block:: none
-
- {
- "tableIndexConfig": {
- "invertedIndexColumns": [
- "column_name",
- ...
- ],
- ...
- }
- }
-
-
-Sorted Inverted Index
-~~~~~~~~~~~~~~~~~~~~~
-Sorted forward index can directly be used as inverted index, with ``log(n)`` time lookup and it can benefit from data locality.
-
-For the below example, if the query has a filter on ``memberId``, Pinot will perform binary search on ``memberId`` values
-to find the range pair of docIds for corresponding filtering value. If the query requires to scan values for other columns
-after filtering, values within the range docId pair will be located together; therefore, we can benefit a lot from data locality.
-
-.. image:: img/sorted-inverted.png
-
-Sorted index performs much better than inverted index; however, it can only be applied to one column. When the query performance
-with inverted index is not good enough and most of queries have a filter on a specific column (e.g. memberId), sorted index can
-improve the query performance.
-
-
-Advanced Index
---------------
-
-Star-Tree Index
-~~~~~~~~~~~~~~~
-
-Unlike other index techniques which work on single column, Star-Tree index is built on multiple columns, and utilize the
-pre-aggregated results to significantly reduce the number of values to be processed, thus improve the query performance.
-
-
-Notes on Index Tuning
----------------------
-
-If your use case is not site facing with a strict low latency requirement, inverted index will perform good enough for
-the most of use cases. We recommend to start with adding inverted index and if the query does not perform good enough,
-a user can consider to use more advanced indices such as sorted column and star-tree index.
-
diff --git a/docs/intro.rst b/docs/intro.rst
deleted file mode 100644
index b6212d5..0000000
--- a/docs/intro.rst
+++ /dev/null
@@ -1,68 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-About Pinot
-===========
-
-Pinot is a realtime distributed OLAP datastore, which is used at LinkedIn to deliver scalable real time analytics with low latency. It can ingest data
-from offline data sources (such as Hadoop and flat files) as well as streaming events (such as Kafka). Pinot is designed to scale horizontally,
-so that it can scale to larger data sets and higher query rates as needed.
-
-What is it for (and not)?
--------------------------
-
-Pinot is well suited for analytical use cases on immutable append-only data that require low latency between an event being ingested and it being available to be queried.
-
-Key Features
-------------
-
-* A column-oriented database with various compression schemes such as Run Length, Fixed Bit Length
-* Pluggable indexing technologies - Sorted Index, Bitmap Index, Inverted Index
-* Ability to optimize query/execution plan based on query and segment metadata .
-* Near real time ingestion from streams and batch ingestion from Hadoop
-* SQL like language that supports selection, aggregation, filtering, group by, order by, distinct queries on data.
-* Support for multivalued fields
-* Horizontally scalable and fault tolerant
-
-Because of the design choices we made to achieve these goals, there are certain limitations in Pinot:
-
-* Pinot is not a replacement for database i.e it cannot be used as source of truth store, cannot mutate data
-* Not a replacement for search engine i.e Full text search, relevance not supported
-* Query cannot span across multiple tables.
-
-Pinot works very well for querying time series data with lots of Dimensions and Metrics. For example:
-
-.. code-block:: sql
-
- SELECT sum(clicks), sum(impressions) FROM AdAnalyticsTable
- WHERE ((daysSinceEpoch >= 17849 AND daysSinceEpoch <= 17856)) AND accountId IN (123456789)
- GROUP BY daysSinceEpoch TOP 100
-
-.. code-block:: sql
-
- SELECT sum(impressions) FROM AdAnalyticsTable
- WHERE (daysSinceEpoch >= 17824 and daysSinceEpoch <= 17854) AND adveriserId = '1234356789'
- GROUP BY daysSinceEpoch,advertiserId TOP 100
-
-.. code-block:: sql
-
- SELECT sum(cost) FROM AdAnalyticsTable GROUP BY advertiserId TOP 50
-
diff --git a/docs/introduction.rst b/docs/introduction.rst
deleted file mode 100644
index 8e1df67..0000000
--- a/docs/introduction.rst
+++ /dev/null
@@ -1,35 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-:orphan:
-
-############
-Introduction
-############
-
-.. _introduction:
-
-.. toctree::
- :maxdepth: 1
-
- intro
- architecture
- getting_started
diff --git a/docs/make.bat b/docs/make.bat
deleted file mode 100644
index 319060e..0000000
--- a/docs/make.bat
+++ /dev/null
@@ -1,300 +0,0 @@
-@REM
-@REM Licensed to the Apache Software Foundation (ASF) under one
-@REM or more contributor license agreements. See the NOTICE file
-@REM distributed with this work for additional information
-@REM regarding copyright ownership. The ASF licenses this file
-@REM to you under the Apache License, Version 2.0 (the
-@REM "License"); you may not use this file except in compliance
-@REM with the License. You may obtain a copy of the License at
-@REM
-@REM http://www.apache.org/licenses/LICENSE-2.0
-@REM
-@REM Unless required by applicable law or agreed to in writing,
-@REM software distributed under the License is distributed on an
-@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-@REM KIND, either express or implied. See the License for the
-@REM specific language governing permissions and limitations
-@REM under the License.
-@REM
-
-@ECHO OFF
-
-REM Command file for Sphinx documentation
-
-if "%SPHINXBUILD%" == "" (
- set SPHINXBUILD=sphinx-build
-)
-set BUILDDIR=_build
-set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% .
-set I18NSPHINXOPTS=%SPHINXOPTS% .
-if NOT "%PAPER%" == "" (
- set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS%
- set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS%
-)
-
-if "%1" == "" goto help
-
-if "%1" == "help" (
- :help
- echo.Please use `make ^<target^>` where ^<target^> is one of
- echo. html to make standalone HTML files
- echo. dirhtml to make HTML files named index.html in directories
- echo. singlehtml to make a single large HTML file
- echo. pickle to make pickle files
- echo. json to make JSON files
- echo. htmlhelp to make HTML files and a HTML help project
- echo. qthelp to make HTML files and a qthelp project
- echo. devhelp to make HTML files and a Devhelp project
- echo. epub to make an epub
- echo. epub3 to make an epub3
- echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter
- echo. text to make text files
- echo. man to make manual pages
- echo. texinfo to make Texinfo files
- echo. gettext to make PO message catalogs
- echo. changes to make an overview over all changed/added/deprecated items
- echo. xml to make Docutils-native XML files
- echo. pseudoxml to make pseudoxml-XML files for display purposes
- echo. linkcheck to check all external links for integrity
- echo. doctest to run all doctests embedded in the documentation if enabled
- echo. coverage to run coverage check of the documentation if enabled
- echo. dummy to check syntax errors of document sources
- goto end
-)
-
-if "%1" == "clean" (
- for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i
- del /q /s %BUILDDIR%\*
- goto end
-)
-
-
-REM Check if sphinx-build is available and fallback to Python version if any
-%SPHINXBUILD% 1>NUL 2>NUL
-if errorlevel 9009 goto sphinx_python
-goto sphinx_ok
-
-:sphinx_python
-
-set SPHINXBUILD=python -m sphinx.__init__
-%SPHINXBUILD% 2> nul
-if errorlevel 9009 (
- echo.
- echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
- echo.installed, then set the SPHINXBUILD environment variable to point
- echo.to the full path of the 'sphinx-build' executable. Alternatively you
- echo.may add the Sphinx directory to PATH.
- echo.
- echo.If you don't have Sphinx installed, grab it from
- echo.http://sphinx-doc.org/
- exit /b 1
-)
-
-:sphinx_ok
-
-
-if "%1" == "html" (
- %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished. The HTML pages are in %BUILDDIR%/html.
- goto end
-)
-
-if "%1" == "dirhtml" (
- %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml.
- goto end
-)
-
-if "%1" == "singlehtml" (
- %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml.
- goto end
-)
-
-if "%1" == "pickle" (
- %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished; now you can process the pickle files.
- goto end
-)
-
-if "%1" == "json" (
- %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished; now you can process the JSON files.
- goto end
-)
-
-if "%1" == "htmlhelp" (
- %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished; now you can run HTML Help Workshop with the ^
-.hhp project file in %BUILDDIR%/htmlhelp.
- goto end
-)
-
-if "%1" == "qthelp" (
- %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished; now you can run "qcollectiongenerator" with the ^
-.qhcp project file in %BUILDDIR%/qthelp, like this:
- echo.^> qcollectiongenerator %BUILDDIR%\qthelp\Pinot.qhcp
- echo.To view the help file:
- echo.^> assistant -collectionFile %BUILDDIR%\qthelp\Pinot.ghc
- goto end
-)
-
-if "%1" == "devhelp" (
- %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished.
- goto end
-)
-
-if "%1" == "epub" (
- %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished. The epub file is in %BUILDDIR%/epub.
- goto end
-)
-
-if "%1" == "epub3" (
- %SPHINXBUILD% -b epub3 %ALLSPHINXOPTS% %BUILDDIR%/epub3
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished. The epub3 file is in %BUILDDIR%/epub3.
- goto end
-)
-
-if "%1" == "latex" (
- %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished; the LaTeX files are in %BUILDDIR%/latex.
- goto end
-)
-
-if "%1" == "latexpdf" (
- %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex
- cd %BUILDDIR%/latex
- make all-pdf
- cd %~dp0
- echo.
- echo.Build finished; the PDF files are in %BUILDDIR%/latex.
- goto end
-)
-
-if "%1" == "latexpdfja" (
- %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex
- cd %BUILDDIR%/latex
- make all-pdf-ja
- cd %~dp0
- echo.
- echo.Build finished; the PDF files are in %BUILDDIR%/latex.
- goto end
-)
-
-if "%1" == "text" (
- %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished. The text files are in %BUILDDIR%/text.
- goto end
-)
-
-if "%1" == "man" (
- %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished. The manual pages are in %BUILDDIR%/man.
- goto end
-)
-
-if "%1" == "texinfo" (
- %SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo.
- goto end
-)
-
-if "%1" == "gettext" (
- %SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished. The message catalogs are in %BUILDDIR%/locale.
- goto end
-)
-
-if "%1" == "changes" (
- %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes
- if errorlevel 1 exit /b 1
- echo.
- echo.The overview file is in %BUILDDIR%/changes.
- goto end
-)
-
-if "%1" == "linkcheck" (
- %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck
- if errorlevel 1 exit /b 1
- echo.
- echo.Link check complete; look for any errors in the above output ^
-or in %BUILDDIR%/linkcheck/output.txt.
- goto end
-)
-
-if "%1" == "doctest" (
- %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest
- if errorlevel 1 exit /b 1
- echo.
- echo.Testing of doctests in the sources finished, look at the ^
-results in %BUILDDIR%/doctest/output.txt.
- goto end
-)
-
-if "%1" == "coverage" (
- %SPHINXBUILD% -b coverage %ALLSPHINXOPTS% %BUILDDIR%/coverage
- if errorlevel 1 exit /b 1
- echo.
- echo.Testing of coverage in the sources finished, look at the ^
-results in %BUILDDIR%/coverage/python.txt.
- goto end
-)
-
-if "%1" == "xml" (
- %SPHINXBUILD% -b xml %ALLSPHINXOPTS% %BUILDDIR%/xml
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished. The XML files are in %BUILDDIR%/xml.
- goto end
-)
-
-if "%1" == "pseudoxml" (
- %SPHINXBUILD% -b pseudoxml %ALLSPHINXOPTS% %BUILDDIR%/pseudoxml
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished. The pseudo-XML files are in %BUILDDIR%/pseudoxml.
- goto end
-)
-
-if "%1" == "dummy" (
- %SPHINXBUILD% -b dummy %ALLSPHINXOPTS% %BUILDDIR%/dummy
- if errorlevel 1 exit /b 1
- echo.
- echo.Build finished. Dummy builder generates no files.
- goto end
-)
-
-:end
diff --git a/docs/pinot_hadoop.rst b/docs/pinot_hadoop.rst
deleted file mode 100644
index 5c045a9..0000000
--- a/docs/pinot_hadoop.rst
+++ /dev/null
@@ -1,354 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-.. _creating-segments:
-
-Creating Pinot segments
-=======================
-
-Pinot segments can be created offline on Hadoop, or via command line from data files. Controller REST endpoint
-can then be used to add the segment to the table to which the segment belongs. Pinot segments can also be created by
-ingesting data from realtime resources (such as Kafka).
-
-Creating segments using hadoop
-------------------------------
-
-.. figure:: img/Pinot-Offline-only-flow.png
-
- Offline Pinot workflow
-
-To create Pinot segments on Hadoop, a workflow can be created to complete the following steps:
-
-#. Pre-aggregate, clean up and prepare the data, writing it as Avro format files in a single HDFS directory
-#. Create segments
-#. Upload segments to the Pinot cluster
-
-Step one can be done using your favorite tool (such as Pig, Hive or Spark), Pinot provides two MapReduce jobs to do step two and three.
-
-Configuring the job
-^^^^^^^^^^^^^^^^^^^
-
-Create a job properties configuration file, such as one below:
-
-.. code-block:: none
-
- # === Index segment creation job config ===
-
- # path.to.input: Input directory containing Avro files
- path.to.input=/user/pinot/input/data
-
- # path.to.output: Output directory containing Pinot segments
- path.to.output=/user/pinot/output
-
- # path.to.schema: Schema file for the table, stored locally
- path.to.schema=flights-schema.json
-
- # segment.table.name: Name of the table for which to generate segments
- segment.table.name=flights
-
- # === Segment tar push job config ===
-
- # push.to.hosts: Comma separated list of controllers host names to which to push
- push.to.hosts=controller_host_0,controller_host_1
-
- # push.to.port: The port on which the controller runs
- push.to.port=8888
-
-Executing the job
-^^^^^^^^^^^^^^^^^
-
-The Pinot Hadoop module contains a job that you can incorporate into your
-workflow to generate Pinot segments.
-
-.. code-block:: bash
-
- mvn clean install -DskipTests -Pbuild-shaded-jar
- hadoop jar pinot-hadoop-<version>-SNAPSHOT-shaded.jar SegmentCreation job.properties
-
-You can then use the SegmentTarPush job to push segments via the controller REST API.
-
-.. code-block:: bash
-
- hadoop jar pinot-hadoop-<version>-SNAPSHOT-shaded.jar SegmentTarPush job.properties
-
-
-Creating Pinot segments using Spark
------------------------------------------
-
-Similar to Pinot hadoop, you can create Pinot segments in Spark.
-
-Configuring the job
-^^^^^^^^^^^^^^^^^^^
-
-Pinot Spark keeps same format of job configuration file, such as one below:
-
-.. code-block:: none
-
- # === Index segment creation job config ===
-
- # path.to.input: Input directory containing Avro files
- path.to.input=/user/pinot/input/data
-
- # path.to.output: Output directory containing Pinot segments
- path.to.output=/user/pinot/output
-
- # path.to.schema: Schema file for the table, stored locally
- path.to.schema=flights-schema.json
-
- # segment.table.name: Name of the table for which to generate segments
- segment.table.name=flights
-
- # use.relative.path: Match output segments hierarchy along with input file hierarchy.
- # E.g. data files layout is:
- # /user/pinot/input/data/2019/10/24/part-0.avro
- # /user/pinot/input/data/2019/10/24/part-1.avro
- # /user/pinot/input/data/2019/10/25/part-0.avro
- # /user/pinot/input/data/2019/10/25/part-1.avro
- # Then output directory layout would be:
- # /user/pinot/output/2019/10/24/flights_2019-10-24_2019-10-24_0.tar.gz
- # /user/pinot/output/2019/10/24/flights_2019-10-24_2019-10-24_1.tar.gz
- # /user/pinot/output/2019/10/25/flights_2019-10-25_2019-10-25_2.tar.gz
- # /user/pinot/output/2019/10/25/flights_2019-10-25_2019-10-25_3.tar.gz
- #
- # use.relative.path=true
-
- # look.back.period.in.days: only process files within recent days.
- # For segment creation job, it creates segments for data files been modified within recent configured days.
- # For segment push job, it pushes segments created/updated within recent configured days.
- #
- # look.back.period.in.days=2
-
- # local.directory.sequence.id: when enabled, segment sequence id is assigned based on local directory,
- # not globally.
- # E.g. data files layout is:
- # /user/pinot/input/data/2019/10/24/part-0.avro
- # /user/pinot/input/data/2019/10/24/part-1.avro
- # /user/pinot/input/data/2019/10/25/part-0.avro
- # /user/pinot/input/data/2019/10/25/part-1.avro
- # Then sequence ids for
- # `/user/pinot/input/data/2019/10/24/part-0.avro` is 0,
- # `/user/pinot/input/data/2019/10/24/part-1.avro` is 1,
- # `/user/pinot/input/data/2019/10/25/part-0.avro` is 0,
- # `/user/pinot/input/data/2019/10/25/part-1.avro` is 1.
- # This is result segment name to be
- # /user/pinot/output/2019/10/24/flights_2019-10-24_2019-10-24_0.tar.gz
- # /user/pinot/output/2019/10/24/flights_2019-10-24_2019-10-24_1.tar.gz
- # /user/pinot/output/2019/10/25/flights_2019-10-25_2019-10-25_0.tar.gz
- # /user/pinot/output/2019/10/25/flights_2019-10-25_2019-10-25_1.tar.gz
- #
- # local.directory.sequence.id=true
-
- # === Segment tar push job config ===
-
- # push.to.hosts: Comma separated list of controllers host names to which to push
- push.to.hosts=controller_host_0,controller_host_1
-
- # push.to.port: The port on which the controller runs
- push.to.port=8888
-
- # enable.parallel.push: Push Segments in parallel
- enable.parallel.push=true
-
- # push.job.parallelism: Push job parallelism, works when `enable.parallel.push=true`
- push.job.parallelism=4
-
- # push.job.retry: How many retries for segment push failure before throw exceptions
- push.job.retry=3
-
-Executing the job
-^^^^^^^^^^^^^^^^^
-
-The Pinot Spark module contains a job that you can incorporate into your
-workflow to generate Pinot segments.
-
-.. code-block:: bash
-
- mvn clean install -DskipTests -Pbuild-shaded-jar
- spark-submit --class org.apache.pinot.spark.PinotSparkJobLauncher \
- pinot-spark-<version>-SNAPSHOT-shaded.jar SegmentCreation job.properties
-
-You can then use the SegmentTarPush job to push segments via the controller REST API.
-
-.. code-block:: bash
-
- spark-submit --class org.apache.pinot.spark.PinotSparkJobLauncher \
- pinot-spark-<version>-SNAPSHOT-shaded.jar SegmentTarPush job.properties
-
-
-Creating Pinot segments outside of Hadoop
------------------------------------------
-
-Here is how you can create Pinot segments from standard formats like CSV/JSON/AVRO.
-
-#. Follow the steps described in the section on :ref:`compiling-code-section` to build pinot. Locate ``pinot-admin.sh`` in ``pinot-tools/target/pinot-tools=pkg/bin/pinot-admin.sh``.
-#. Create a top level directory containing all the CSV/JSON/AVRO files that need to be converted into segments.
-#. The file name extensions are expected to be the same as the format name (*i.e* ``.csv``, ``.json`` or ``.avro``), and are case insensitive. Note that the converter expects the ``.csv`` extension even if the data is delimited using tabs or spaces instead.
-#. Prepare a schema file describing the schema of the input data. The schema needs to be in JSON format. See example later in this section.
-#. Specifically for CSV format, an optional csv config file can be provided (also in JSON format). This is used to configure parameters like the delimiter/header for the CSV file etc. A detailed description of this follows below.
-
-Run the pinot-admin command to generate the segments. The command can be invoked as follows. Options within "[ ]" are optional. For -format, the default value is AVRO.
-
-.. code-block:: bash
-
- bin/pinot-admin.sh CreateSegment -dataDir <input_data_dir> [-format [CSV/JSON/AVRO]] [-readerConfigFile <csv_config_file>] [-generatorConfigFile <generator_config_file>] -segmentName <segment_name> -schemaFile <input_schema_file> -tableName <table_name> -outDir <output_data_dir> [-overwrite]
-
-To configure various parameters for CSV a config file in JSON format can be provided. This file is optional, as are each of its parameters. When not provided, default values used for these parameters are described below:
-
-#. fileFormat: Specify one of the following. Default is EXCEL.
-
- #. EXCEL
- #. MYSQL
- #. RFC4180
- #. TDF
-
-#. header: If the input CSV file does not contain a header, it can be specified using this field. Note, if this is specified, then the input file is expected to not contain the header row, or else it will result in parse error. The columns in the header must be delimited by the same delimiter character as the rest of the CSV file.
-#. delimiter: Use this to specify a delimiter character. The default value is ",".
-#. multiValueDelimiter: Use this to specify a delimiter character for each value in multi-valued columns. The default value is ";".
-
-Below is a sample config file.
-
-.. code-block:: json
-
- {
- "fileFormat": "EXCEL",
- "header": "col1,col2,col3,col4",
- "delimiter": "\t",
- "multiValueDelimiter": ","
- }
-
-Sample Schema:
-
-.. code-block:: json
-
- {
- "schemaName": "flights",
- "dimensionFieldSpecs": [
- {
- "name": "flightNumber",
- "dataType": "LONG"
- },
- {
- "name": "tags",
- "dataType": "STRING",
- "singleValueField": false
- }
- ],
- "metricFieldSpecs": [
- {
- "name": "price",
- "dataType": "DOUBLE"
- }
- ],
- "timeFieldSpec": {
- "incomingGranularitySpec": {
- "name": "daysSinceEpoch",
- "dataType": "INT",
- "timeType": "DAYS"
- }
- }
- }
-
-Pushing offline segments to Pinot
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-You can use curl to push a segment to pinot:
-
-.. code-block:: bash
-
- curl -X POST -F segment=@<segment-tar-file-path> http://controllerHost:controllerPort/segments
-
-
-Alternatively you can use the pinot-admin.sh utility to upload one or more segments:
-
-.. code-block:: bash
-
- pinot-tools/target/pinot-tools-pkg/bin//pinot-admin.sh UploadSegment -controllerHost <hostname> -controllerPort <port> -segmentDir <segmentDirectoryPath>
-
-The command uploads all the segments found in ``segmentDirectoryPath``.
-The segments could be either tar-compressed (in which case it is a file under ``segmentDirectoryPath``)
-or uncompressed (in which case it is a directory under ``segmentDirectoryPath``).
-
-Realtime segment generation
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To consume in realtime, we simply need to create a table with the same name as the schema and point to the Kafka topic
-to consume from, using a table definition such as this one:
-
-.. code-block:: json
-
- {
- "tableName": "flights",
- "tableType": "REALTIME",
- "segmentsConfig": {
- "retentionTimeUnit": "DAYS",
- "retentionTimeValue": "7",
- "segmentPushFrequency": "daily",
- "segmentPushType": "APPEND",
- "replication": "1",
- "timeColumnName": "daysSinceEpoch",
- "timeType": "DAYS",
- "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
- },
- "tableIndexConfig": {
- "invertedIndexColumns": [
- "flightNumber",
- "tags",
- "daysSinceEpoch"
- ],
- "loadMode": "MMAP",
- "streamConfigs": {
- "streamType": "kafka",
- "stream.kafka.consumer.type": "highLevel",
- "stream.kafka.topic.name": "flights-realtime",
- "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
- "stream.kafka.zk.broker.url": "localhost:2181",
- "stream.kafka.hlc.zk.connect.string": "localhost:2181"
- }
- },
- "tenants": {
- "broker": "brokerTenant",
- "server": "serverTenant"
- },
- "metadata": {
- }
- }
-
-First, we'll start a local instance of Kafka and start streaming data into it:
-
-.. code-block:: bash
-
- bin/pinot-admin.sh StartKafka &
- bin/pinot-admin.sh StreamAvroIntoKafka -avroFile flights-2014.avro -kafkaTopic flights-realtime &
-
-This will stream one event per second from the Avro file to the Kafka topic. Then, we'll create a realtime table, which
-will start consuming from the Kafka topic.
-
-.. code-block:: bash
-
- bin/pinot-admin.sh AddTable -schemaFile flights-schema.json -tableConfigFile flights-definition-realtime.json -exec
-
-We can then query the table with the following query to see the events stream in:
-
-.. code-block:: sql
-
- SELECT COUNT(*) FROM flights
-
-Repeating the query multiple times should show the events slowly being streamed into the table.
diff --git a/docs/pluggable_storage.rst b/docs/pluggable_storage.rst
deleted file mode 100644
index ed61892..0000000
--- a/docs/pluggable_storage.rst
+++ /dev/null
@@ -1,128 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-.. _pluggable-storage:
-
-Pluggable Storage
-=================
-
-Pinot enables its users to write a PinotFS abstraction layer to store data in a data layer of their choice for realtime and offline segments.
-
-Some examples of storage backends(other than local storage) currently supported are:
-
-* `HadoopFS <https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html>`_
-* `Azure Data Lake <https://azure.microsoft.com/en-us/solutions/data-lake/>`_
-
-If the above two filesystems do not meet your needs, you can extend the current `PinotFS <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/filesystem/PinotFS.java>`_ to customize for your needs.
-
-New Storage Type implementation
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-In order to add a new type of storage backend (say, Amazon s3) implement the following class:
-
-S3FS extends `PinotFS <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/filesystem/PinotFS.java>`_
-
-Configurations for Realtime Tables
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-The example here uses the existing org.apache.pinot.filesystem.HadoopPinotFS to store realtime segments in a HDFS filesytem. In the Pinot controller config, add the following new configs:
-
-.. code-block:: none
-
- "controller.data.dir": "SET_TO_YOUR_HDFS_ROOT_DIR"
- "controller.local.temp.dir": "SET_TO_A_LOCAL_FILESYSTEM_DIR"
- "pinot.controller.storage.factory.class.hdfs": "org.apache.pinot.filesystem.HadoopPinotFS"
- "pinot.controller.storage.factory.hdfs.hadoop.conf.path": "SET_TO_YOUR_HDFS_CONFIG_DIR"
- "pinot.controller.storage.factory.hdfs.hadoop.kerberos.principle": "SET_IF_YOU_USE_KERBEROS"
- "pinot.controller.storage.factory.hdfs.hadoop.kerberos.keytab": "SET_IF_YOU_USE_KERBEROS"
- "controller.enable.split.commit": "true"
-
-In the Pinot controller config, add the following new configs:
-
-.. code-block:: none
-
- "pinot.server.instance.enable.split.commit": "true"
-
-
-Note: currently there is a bug in the controller (`issue <https://github.com/apache/incubator-pinot/issues/3847>`), for now you can cherrypick the PR https://github.com/apache/incubator-pinot/pull/3849 to fix the issue as tested already. The PR is under review now.
-
-Configurations for Offline Tables
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-These properties for the stream implementation are to be set in your controller and server configurations.
-
-In your controller and server configs, please set the FS class you would like to support. pinot.controller.storage.factory.class.${YOUR_URI_SCHEME} to the full path of the FS class you would like to include
-
-You also need to configure pinot.controller.local.temp.dir for the local dir on the controller machine.
-
-For filesystem specific configs, you can pass in the following with either the pinot.controller prefix or the pinot.server prefix.
-
-All the following configs need to be prefixed with storage.factory.
-
-AzurePinotFS requires the following configs according to your environment:
-
-adl.accountId, adl.authEndpoint, adl.clientId, adl.clientSecret
-
-Sample Controller Config
-
-.. code-block:: none
-
- "pinot.controller.storage.factory.class.adl": "org.apache.pinot.filesystem.AzurePinotFS"
- "pinot.controller.storage.factory.adl.accountId": "xxxx"
- "pinot.controller.storage.factory.adl.authEndpoint": "xxxx"
- "pinot.controller.storage.factory.adl.clientId": "xxxx"
- "pinot.controller.storage.factory.adl.clientId": "xxxx"
- "pinot.controller.segment.fetcher.protocols": "adl"
-
-
-Sample Server Config
-
-.. code-block:: none
-
- "pinot.server.storage.factory.class.adl": "org.apache.pinot.filesystem.AzurePinotFS"
- "pinot.server.storage.factory.adl.accountId": "xxxx"
- "pinot.server.storage.factory.adl.authEndpoint": "xxxx"
- "pinot.server.storage.factory.adl.clientId": "xxxx"
- "pinot.server.storage.factory.adl.clientId": "xxxx"
- "pinot.server.segment.fetcher.protocols": "adl"
-
-
-You can find the parameters in your account as follows:
-https://stackoverflow.com/questions/56349040/what-is-clientid-authtokenendpoint-clientkey-for-accessing-azure-data-lake
-
-Please also make sure to set the following config with the value "adl"
-
-.. code-block:: none
-
- "segment.fetcher.protocols" : "adl"
-
-
-To see how to upload segments to different storage systems, check
-:file:`../segment_fetcher.rst`.
-
-HadoopPinotFS requires the following configs according to your environment:
-
-hadoop.kerberos.principle, hadoop.kerberos.keytab, hadoop.conf.path
-
-Please make sure to also set the following config with the value "hdfs"
-
-.. code-block:: none
-
- "segment.fetcher.protocols" : "hdfs"
-
-
diff --git a/docs/pluggable_streams.rst b/docs/pluggable_streams.rst
deleted file mode 100644
index 309c840..0000000
--- a/docs/pluggable_streams.rst
+++ /dev/null
@@ -1,342 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-.. _pluggable-streams:
-
-Pluggable Streams
-=================
-
-.. note::
-
- This section is a pre-read if you are planning to develop plug-ins for streams other than Kafka. Pinot supports Kafka out of the box.
-
-Prior to commit `ba9f2d <https://github.com/apache/incubator-pinot/commit/ba9f2ddfc0faa42fadc2cc48df1d77fec6b174fb>`_, Pinot was only able to support consuming
-from `Kafka <https://kafka.apache.org/documentation/>`_ stream.
-
-Pinot now enables its users to write plug-ins to consume from pub-sub streams
-other than Kafka. (Please refer to `Issue #2583 <https://github.com/apache/incubator-pinot/issues/2583>`_)
-
-Some of the streams for which plug-ins can be added are:
-
-* `Amazon kinesis <https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-kcl.html>`_
-* `Azure Event Hubs <https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-java-get-started-receive-eph>`_
-* `LogDevice <https://code.fb.com/core-data/logdevice-a-distributed-data-store-for-logs/>`_
-* `Pravega <http://pravega.io/docs/latest/javadoc/>`_
-* `Pulsar <https://pulsar.apache.org/docs/en/client-libraries-java/>`_
-
-
-You may encounter some limitations either in Pinot or in the stream system while developing plug-ins.
-Please feel free to get in touch with us when you start writing a stream plug-in, and we can help you out.
-We are open to receiving PRs in order to improve these abstractions if they do not work for a certain stream implementation.
-
-Refer to `Consuming and Indexing rows in Realtime <https://cwiki.apache.org/confluence/display/PINOT/Consuming+and+Indexing+rows+in+Realtime>`_
-for details on how Pinot consumes streaming data.
-
-Requirements to support Stream Level (High Level) consumers
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The stream should provide the following guarantees:
-
-* Exactly once delivery (unless restarting from a checkpoint) for each consumer of the stream.
-* (Optionally) support mechanism to split events (in some arbitrary fashion) so that each event in the stream is delivered exactly to one host out of set of hosts.
-* Provide ways to save a checkpoint for the data consumed so far. If the stream is partitioned, then this checkpoint is a vector of checkpoints for events consumed from individual partitions.
-* The checkpoints should be recorded only when Pinot makes a call to do so.
-* The consumer should be able to start consumption from one of:
-
- * latest avaialble data
- * earliest available data
- * last saved checkpoint
-
-Requirements to support Partition Level (Low Level) consumers
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-While consuming rows at a partition level, the stream should support the following
-properties:
-
-* Stream should provide a mechanism to get the current number of partitions.
-* Each event in a partition should have a unique offset that is not more than 64 bits long.
-* Refer to a partition as a number not exceeding 32 bits long.
-* Stream should provide the following mechanisms to get an offset for a given partition of the stream:
-
- * get the offset of the oldest event available (assuming events are aged out periodically) in the partition.
- * get the offset of the most recent event published in the partition
- * (optionally) get the offset of an event that was published at a specified time
-
-* Stream should provide a mechanism to consume a set of events from a partition starting from a specified offset.
-* Pinot assumes that the offsets of incoming events are monotonically increasing; *i.e.*, if Pinot
- consumes an event at offset ``o1``, then the offset ``o2`` of the following event should be such that
- ``o2 > o1``.
-
-In addition, we have an operational requirement that the number of partitions should not be
-reduced over time.
-
-Stream plug-in implementation
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-In order to add a new type of stream (say,Foo) implement the following classes:
-
-#. FooConsumerFactory extends `StreamConsumerFactory <https://github.com/apache/incubator-pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/realtime/stream/StreamConsumerFactory.java>`_
-#. FooPartitionLevelConsumer implements `PartitionLevelConsumer <https://github.com/apache/incubator-pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/realtime/stream/PartitionLevelConsumer.java>`_
-#. FooStreamLevelConsumer implements `StreamLevelConsumer <https://github.com/apache/incubator-pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/realtime/stream/StreamLevelConsumer.java>`_
-#. FooMetadataProvider implements `StreamMetadataProvider <https://github.com/apache/incubator-pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/realtime/stream/StreamMetadataProvider.java>`_
-#. FooMessageDecoder implements `StreamMessageDecoder <https://github.com/apache/incubator-pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/realtime/stream/StreamMessageDecoder.java>`_
-
-Depending on stream level or partition level, your implementation needs to include StreamLevelConsumer or PartitionLevelConsumer.
-
-
-The properties for the stream implementation are to be set in the table configuration, inside `streamConfigs <https://github.com/apache/incubator-pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/realtime/stream/StreamConfig.java>`_ section.
-
-Use the ``streamType`` property to define the stream type. For example, for the implementation of stream ``foo``, set the property ``"streamType" : "foo"``.
-
-The rest of the configuration properties for your stream should be set with the prefix ``"stream.foo"``. Be sure to use the same suffix for: (see examples below):
-
-* topic
-* consumer type
-* stream consumer factory
-* offset
-* decoder class name
-* decoder properties
-* connnection timeout
-* fetch timeout
-
-All values should be strings. For example:
-
-
-.. code-block:: none
-
- "streamType" : "foo",
- "stream.foo.topic.name" : "SomeTopic",
- "stream.foo.consumer.type": "LowLevel",
- "stream.foo.consumer.factory.class.name": "fully.qualified.pkg.ConsumerFactoryClassName",
- "stream.foo.consumer.prop.auto.offset.reset": "largest",
- "stream.foo.decoder.class.name" : "fully.qualified.pkg.DecoderClassName",
- "stream.foo.decoder.prop.a.decoder.property" : "decoderPropValue",
- "stream.foo.connection.timeout.millis" : "10000", // default 30_000
- "stream.foo.fetch.timeout.millis" : "10000" // default 5_000
-
-
-You can have additional properties that are specific to your stream. For example:
-
-.. code-block:: none
-
- "stream.foo.some.buffer.size" : "24g"
-
-In addition to these properties, you can define thresholds for the consuming segments:
-
-* rows threshold
-* time threshold
-
-The properties for the thresholds are as follows:
-
-.. code-block:: none
-
- "realtime.segment.flush.threshold.size" : "100000"
- "realtime.segment.flush.threshold.time" : "6h"
-
-
-An example of this implementation can be found in the `KafkaConsumerFactory <https://github.com/apache/incubator-pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaConsumerFactory.java>`_, which is an implementation for the kafka stream.
-
-
-Kafka 2.x Plugin
-^^^^^^^^^^^^^^^^
-
-Pinot provides stream plugin support for Kafka 2.x version.
-Although the version used in this implementation is kafka 2.0.0, it's possible to compile it with higher kafka lib version, e.g. 2.1.1.
-
-How to build and release Pinot package with Kafka 2.x connector
----------------------------------------------------------------
-
-.. code-block:: none
-
- mvn clean package -DskipTests -Pbin-dist -Dkafka.version=2.0
-
-How to use Kafka 2.x connector
-------------------------------
-
-- **Use Kafka Stream(High) Level Consumer**
-
-Below is a sample ``streamConfigs`` used to create a realtime table with Kafka Stream(High) level consumer.
-
-Kafka 2.x HLC consumer uses ``org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory`` in config ``stream.kafka.consumer.factory.class.name``.
-
-.. code-block:: none
-
- "streamConfigs": {
- "streamType": "kafka",
- "stream.kafka.consumer.type": "highLevel",
- "stream.kafka.topic.name": "meetupRSVPEvents",
- "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
- "stream.kafka.hlc.zk.connect.string": "localhost:2191/kafka",
- "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
- "stream.kafka.zk.broker.url": "localhost:2191/kafka",
- "stream.kafka.hlc.bootstrap.server": "localhost:19092"
- }
-
-
-- **Use Kafka Partition(Low) Level Consumer**
-
-Below is a sample table config used to create a realtime table with Kafka Partition(Low) level consumer:
-
-.. code-block:: none
-
- {
- "tableName": "meetupRsvp",
- "tableType": "REALTIME",
- "segmentsConfig": {
- "timeColumnName": "mtime",
- "timeType": "MILLISECONDS",
- "segmentPushType": "APPEND",
- "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
- "schemaName": "meetupRsvp",
- "replication": "1",
- "replicasPerPartition": "1"
- },
- "tenants": {},
- "tableIndexConfig": {
- "loadMode": "MMAP",
- "streamConfigs": {
- "streamType": "kafka",
- "stream.kafka.consumer.type": "LowLevel",
- "stream.kafka.topic.name": "meetupRSVPEvents",
- "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
- "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
- "stream.kafka.zk.broker.url": "localhost:2191/kafka",
- "stream.kafka.broker.list": "localhost:19092"
- }
- },
- "metadata": {
- "customConfigs": {}
- }
- }
-
-Please note:
-
-1. Config ``replicasPerPartition`` under ``segmentsConfig`` is required to specify table replication.
-#. Config ``stream.kafka.consumer.type`` should be specified as ``LowLevel`` to use partition level consumer. (The use of ``simple`` instead of ``LowLevel`` is deprecated)
-#. Configs ``stream.kafka.zk.broker.url`` and ``stream.kafka.broker.list`` are required under ``tableIndexConfig.streamConfigs`` to provide kafka related information.
-
-Here is another example which is used to read avro messages using the schema id from confluent schema registry.
-The only difference between this and the KafkaAvroMessageDecoder is that the KafkaConfluentSchemaRegistryAvroMessageDecoder uses
-confluent schema registry:
-
-.. code-block:: none
-
- {
- "tableName": "meetupRsvp",
- "tableType": "REALTIME",
- "segmentsConfig": {
- "timeColumnName": "mtime",
- "timeType": "MILLISECONDS",
- "segmentPushType": "APPEND",
- "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
- "schemaName": "meetupRsvp",
- "replication": "1",
- "replicasPerPartition": "1"
- },
- "tenants": {},
- "tableIndexConfig": {
- "loadMode": "MMAP",
- "streamConfigs": {
- "streamType": "kafka",
- "stream.kafka.consumer.type": "LowLevel",
- "stream.kafka.topic.name": "meetupRSVPEvents",
- "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
- "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
- "stream.kafka.zk.broker.url": "localhost:2191/kafka",
- "stream.kafka.broker.list": "localhost:19092"
- }
- },
- "metadata": {
- "customConfigs": {}
- }
- }
-
-Here is another example which uses SSL based authentication to talk with kafka
-and schema-registry. Notice there are two sets of SSL options, ones starting with
-`ssl.` are for kafka consumer and ones with `stream.kafka.decoder.prop.schema.registry.`
-are for `SchemaRegistryClient` used by `KafkaConfluentSchemaRegistryAvroMessageDecoder`.
-
-
-.. code-block:: none
-
- {
- "tableName": "meetupRsvp",
- "tableType": "REALTIME",
- "segmentsConfig": {
- "timeColumnName": "mtime",
- "timeType": "MILLISECONDS",
- "segmentPushType": "APPEND",
- "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
- "schemaName": "meetupRsvp",
- "replication": "1",
- "replicasPerPartition": "1"
- },
- "tenants": {},
- "tableIndexConfig": {
- "loadMode": "MMAP",
- "streamConfigs": {
- "streamType": "kafka",
- "stream.kafka.consumer.type": "LowLevel",
- "stream.kafka.topic.name": "meetupRSVPEvents",
- "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
- "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
- "stream.kafka.zk.broker.url": "localhost:2191/kafka",
- "stream.kafka.broker.list": "localhost:19092",
-
- "schema.registry.url": "",
- "security.protocol": "",
- "ssl.truststore.location": "",
- "ssl.keystore.location": "",
- "ssl.truststore.password": "",
- "ssl.keystore.password": "",
- "ssl.key.password": "",
-
- "stream.kafka.decoder.prop.schema.registry.rest.url": "",
- "stream.kafka.decoder.prop.schema.registry.ssl.truststore.location": "",
- "stream.kafka.decoder.prop.schema.registry.ssl.keystore.location": "",
- "stream.kafka.decoder.prop.schema.registry.ssl.truststore.password": "",
- "stream.kafka.decoder.prop.schema.registry.ssl.keystore.password": "",
- "stream.kafka.decoder.prop.schema.registry.ssl.keystore.type": "",
- "stream.kafka.decoder.prop.schema.registry.ssl.truststore.type": "",
- "stream.kafka.decoder.prop.schema.registry.ssl.key.password": "",
- "stream.kafka.decoder.prop.schema.registry.ssl.protocol": "",
- }
- },
- "metadata": {
- "customConfigs": {}
- }
- }
-
-Upgrade from Kafka 0.9 connector to Kafka 2.x connector
--------------------------------------------------------
-
-* Update table config for both high level and low level consumer:
- Update config: ``stream.kafka.consumer.factory.class.name`` from ``org.apache.pinot.plugin.stream.kafka09.KafkaConsumerFactory`` to ``org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory``.
-
-* If using Stream(High) level consumer:
- Please also add config ``stream.kafka.hlc.bootstrap.server`` into ``tableIndexConfig.streamConfigs``.
- This config should be the URI of Kafka broker lists, e.g. ``localhost:9092``.
-
-
-How to use this plugin with higher Kafka version?
--------------------------------------------------
-
-This connector is also suitable for Kafka lib version higher than ``2.0.0``.
-In ``pinot-connector-kafka-2.0/pom.xml`` change the ``kafka.lib.version`` from ``2.0.0`` to ``2.1.1`` will make this Connector working with Kafka ``2.1.1``.
diff --git a/docs/pql_examples.rst b/docs/pql_examples.rst
deleted file mode 100644
index 9362944..0000000
--- a/docs/pql_examples.rst
+++ /dev/null
@@ -1,357 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-.. _pql:
-
-PQL
-===
-
-* PQL is a derivative of SQL derivative that supports selection, projection, aggregation, grouping aggregation.
- There is no support for Joins or Subqueries.
-
-* Specifically, for Pinot:
-
- * Grouping keys always appear in query results, even if not requested
- * Aggregations are computed in parallel
- * Results of aggregations with large amounts of group keys (>1M) are approximated
- * ``ORDER BY`` only works for selection queries, for aggregations one must use the ``TOP`` keyword
-
-PQL Examples
-------------
-
-The Pinot Query Language (PQL) is very similar to standard SQL:
-
-.. code-block:: sql
-
- SELECT COUNT(*) FROM myTable
-
-Aggregation
------------
-
-.. code-block:: sql
-
- SELECT COUNT(*), MAX(foo), SUM(bar) FROM myTable
-
-Grouping on Aggregation
------------------------
-
-.. code-block:: sql
-
- SELECT MIN(foo), MAX(foo), SUM(foo), AVG(foo) FROM myTable
- GROUP BY bar, baz TOP 50
-
-Filtering
----------
-
-.. code-block:: sql
-
- SELECT COUNT(*) FROM myTable
- WHERE foo = 'foo'
- AND bar BETWEEN 1 AND 20
- OR (baz < 42 AND quux IN ('hello', 'goodbye') AND quuux NOT IN (42, 69))
-
-Selection (Projection)
-----------------------
-
-.. code-block:: sql
-
- SELECT * FROM myTable
- WHERE quux < 5
- LIMIT 50
-
-Ordering on Selection
----------------------
-
-.. code-block:: sql
-
- SELECT foo, bar FROM myTable
- WHERE baz > 20
- ORDER BY bar DESC
- LIMIT 100
-
-Pagination on Selection
------------------------
-Note: results might not be consistent if column ordered by has same value in multiple rows.
-
-.. code-block:: sql
-
- SELECT foo, bar FROM myTable
- WHERE baz > 20
- ORDER BY bar DESC
- LIMIT 50, 100
-
-Wild-card match (in WHERE clause only)
---------------------------------------
-
-To count rows where the column ``airlineName`` starts with ``U``
-
-.. code-block:: sql
-
- SELECT count(*) FROM SomeTable
- WHERE regexp_like(airlineName, '^U.*')
- GROUP BY airlineName TOP 10
-
-Examples with UDF
------------------
-
-As of now, functions have to be implemented within Pinot. Injecting functions is not allowed yet.
-The examples below demonstrate the use of UDFs
-
-.. code-block:: sql
-
- SELECT count(*) FROM myTable
- GROUP BY timeConvert(timeColumnName, 'SECONDS', 'DAYS')
-
-Examples with BYTES column
---------------------------
-
-Pinot supports queries on BYTES column using HEX string. The query response also uses hex string to represent bytes value.
-
-E.g. the query below fetches all the rows for a given UID.
-
-.. code-block:: sql
-
- SELECT * FROM myTable
- WHERE UID = "c8b3bce0b378fc5ce8067fc271a34892"
-
-PQL Specification
------------------
-
-SELECT
-^^^^^^
-
-The select statement is as follows
-
-.. code-block:: sql
-
- SELECT <outputColumn> (, outputColumn, outputColumn,...)
- FROM <tableName>
- (WHERE ... | GROUP BY ... | ORDER BY ... | TOP ... | LIMIT ...)
-
-``outputColumn`` can be ``*`` to project all columns, columns (``foo``, ``bar``, ``baz``) or aggregation functions like (``MIN(foo)``, ``MAX(bar)``, ``AVG(baz)``).
-
-Supported aggregations on single-value columns
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-* ``COUNT``
-* ``MIN``
-* ``MAX``
-* ``SUM``
-* ``AVG``
-* ``MINMAXRANGE``
-* ``DISTINCTCOUNT``
-* ``DISTINCTCOUNTHLL``
-* ``DISTINCTCOUNTRAWHLL``: Returns HLL response serialized as string. The serialized HLL can be converted back into an HLL (see `pinot-core/\*\*/HllUtil.java` as an example) and then aggregated with other HLLs. A common use case may be to merge HLL responses from different Pinot tables, or to allow aggregation after client-side batching.
-* ``FASTHLL`` (**WARN**: will be deprecated soon. ``FASTHLL`` stores serialized HyperLogLog in String format, which performs
- worse than ``DISTINCTCOUNTHLL``, which supports serialized HyperLogLog in BYTES (byte array) format)
-* ``PERCENTILE[0-100]``: e.g. ``PERCENTILE5``, ``PERCENTILE50``, ``PERCENTILE99``, etc.
-* ``PERCENTILEEST[0-100]``: e.g. ``PERCENTILEEST5``, ``PERCENTILEEST50``, ``PERCENTILEEST99``, etc.
-
-Supported aggregations on multi-value columns
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-* ``COUNTMV``
-* ``MINMV``
-* ``MAXMV``
-* ``SUMMV``
-* ``AVGMV``
-* ``MINMAXRANGEMV``
-* ``DISTINCTCOUNTMV``
-* ``DISTINCTCOUNTHLLMV``
-* ``DISTINCTCOUNTRAWHLLMV``: Returns HLL response serialized as string. The serialized HLL can be converted back into an HLL (see `pinot-core/**/HllUtil.java` as an example) and then aggregated with other HLLs. A common use case may be to merge HLL responses from different Pinot tables, or to allow aggregation after client-side batching.
-* ``FASTHLLMV`` (**WARN**: will be deprecated soon. It does not make lots of sense to configure serialized HyperLogLog
- column as a dimension)
-* ``PERCENTILE[0-100]MV``: e.g. ``PERCENTILE5MV``, ``PERCENTILE50MV``, ``PERCENTILE99MV``, etc.
-* ``PERCENTILEEST[0-100]MV``: e.g. ``PERCENTILEEST5MV``, ``PERCENTILEEST50MV``, ``PERCENTILEEST99MV``, etc.
-
-WHERE
-^^^^^
-
-Supported predicates are comparisons with a constant using the standard SQL operators (``=``, ``<``, ``<=``, ``>``, ``>=``, ``<>``, '!=') , range comparisons using ``BETWEEN`` (``foo BETWEEN 42 AND 69``), set membership (``foo IN (1, 2, 4, 8)``) and exclusion (``foo NOT IN (1, 2, 4, 8)``). For ``BETWEEN``, the range is inclusive.
-
-Comparison with a regular expression is supported using the regexp_like function, as in ``WHERE regexp_like(columnName, 'regular expression')``
-
-GROUP BY
-^^^^^^^^
-
-The ``GROUP BY`` clause groups aggregation results by a list of columns, or transform functions on columns (see below)
-
-
-ORDER BY
-^^^^^^^^
-
-The ``ORDER BY`` clause orders selection results by a list of columns. PQL supports ordering ``DESC`` or ``ASC``.
-
-TOP
-^^^
-
-The ``TOP n`` clause causes the 'n' largest group results to be returned. If not specified, the top 10 groups are returned.
-
-LIMIT
-^^^^^
-
-The ``LIMIT n`` clause causes the selection results to contain at most 'n' results.
-The ``LIMIT a, b`` clause paginate the selection results from the 'a' th results and return at most 'b' results.
-
-Transform Function in Aggregation and Grouping
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-In aggregation and grouping, each column can be transformed from one or multiple columns.
-For example, the following query will calculate the maximum value of column ``foo`` divided by column ``bar`` grouping on the column ``time`` converted form time unit ``MILLISECONDS`` to ``SECONDS``:
-
-.. code-block:: sql
-
- SELECT MAX(DIV(foo, bar) FROM myTable
- GROUP BY TIMECONVERT(time, 'MILLISECONDS', 'SECONDS')
-
-Supported transform functions
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-``ADD``
- Sum of at least two values
-
-``SUB``
- Difference between two values
-
-``MULT``
- Product of at least two values
-
-``DIV``
- Quotient of two values
-
-``TIMECONVERT``
- Takes 3 arguments, converts the value into another time unit. *e.g.* ``TIMECONVERT(time, 'MILLISECONDS', 'SECONDS')``
- This expression converts the value of coulumn ``time`` (taken to be in milliseconds) to the nearest seconds
- (*i.e.* the nearest seconds that is lower than the value of ``date`` column)
-
-``DATETIMECONVERT``
- Takes 4 arguments, converts the value into another date time format, and buckets time based on the given time granularity.
- *e.g.* ``DATETIMECONVERT(date, '1:MILLISECONDS:EPOCH', '1:SECONDS:EPOCH', '15:MINUTES')``
- This expression converts the column ``date`` which is formatted as ``1:MILLISECONDS:EPOCH``, and converts it into
- format ``1:SECONDS:EPOCH`` with a granularity of ``15:MINUTES`` (*i.e.* nearest 15-minute value lower than the value
- of ``date`` column.
-
-``DATETRUNC``
- (Presto) SQL compatible date truncation, equivalent to the Presto function `date_trunc <https://mode.com/blog/date-trunc-sql-timestamp-function-count-on>`_.
- Takes atleast 3 and upto 5 arguments, converts the value into a specified output granularity seconds since UTC epoch that is bucketed on a unit in a specified timezone.
-
- *e.g.* ``DATETRUNC('week', time_in_seconds, 'SECONDS')`` This expression
- converts the column ``time_in_seconds``, which is a long containing seconds
- since UTC epoch truncated at ``WEEK`` (where a Week starts at Monday UTC
- midnight). The output is a long seconds since UTC epoch.
-
- *e.g.* ``DATETRUNC('quarter', DIV(time_milliseconds/1000), 'SECONDS',
- 'America/Los_Angeles', 'HOURS')`` This expression converts the expression
- ``time_in_milliseconds/1000`` (which is thus in seconds) into hours that are
- truncated at ``QUARTER`` at the Los Angeles time zone (where a Quarter
- begins on 1/1, 4/1, 7/1, 10/1 in Los Angelese timezone). The output is
- expressed as hours since UTC epoch (note that the output is not Los Angeles
- timezone)
-
-``ARRAYLENGTH``
- Takes a multi-valued column and returns the length of the column
-
-``VALUEIN``
- Takes at least 2 arguments, where the first argument is a multi-valued column, and the following arguments are constant values.
- The transform function will filter the value from the multi-valued column with the given constant values.
- The ``VALUEIN`` transform function is especially useful when the same multi-valued column is both filtering column and grouping column.
- *e.g.* ``VALUEIN(mvColumn, 3, 5, 15)``
-
-
-Differences with SQL
---------------------
-
-* ``JOIN`` is not supported
-* Use ``TOP`` instead of ``LIMIT`` for truncation
-* ``LIMIT n`` has no effect in grouping queries, should use ``TOP n`` instead. If no ``TOP n`` defined, PQL will use ``TOP 10`` as default truncation setting.
-* No need to select the columns to group with.
-
-The following two queries are both supported in PQL, where the non-aggregation columns are ignored.
-
-.. code-block:: sql
-
- SELECT MIN(foo), MAX(foo), SUM(foo), AVG(foo) FROM mytable
- GROUP BY bar, baz
- TOP 50
-
- SELECT bar, baz, MIN(foo), MAX(foo), SUM(foo), AVG(foo) FROM mytable
- GROUP BY bar, baz
- TOP 50
-
-* The results will always order by the aggregated value (descending).
-
-The results for query:
-
-.. code-block:: sql
-
- SELECT MIN(foo), MAX(foo) FROM myTable
- GROUP BY bar
- TOP 50
-
-will be the same as the combining results from the following queries:
-
-.. code-block:: sql
-
- SELECT MIN(foo) FROM myTable
- GROUP BY bar
- TOP 50
- SELECT MAX(foo) FROM myTable
- GROUP BY bar
- TOP 50
-
-where we don't put the results for the same group together.
-
-
-* We are beginning work on standard sql support. As a first step, we have introduced ``ORDER BY``.
-
-In order to use ``ORDER BY`` certain options need to be set in the request json payload:
-
-1. ``groupByMode`` - Setting this to ``sql`` will take the code path of standard sql, and hence accept ``ORDER BY``. By default, this is ``pql``
-
-.. code-block:: json
-
- {
- "pql" : "SELECT COUNT(*) from myTable GROUP BY foo ORDER BY foo DESC TOP 100",
- "queryOptions" : "groupByMode=sql"
- }
-
-2. ``responseFormat`` - Setting this to ``sql`` will present results in the standard sql way i.e. tabular, with same keys across all aggregations. This only works when used in combination with ``groupByMode=sql``. By default, this is ``pql``
-
-.. code-block:: json
-
- {
- "pql" : "SELECT SUM(foo), SUM(bar) from myTable GROUP BY moo ORDER BY SUM(bar) ASC, moo DESC TOP 10",
- "queryOptions" : "groupByMode=sql;responseFormat=sql"
- }
-
-ResultTable looks as follows:
-
-.. code-block:: json
-
- {
- "resultTable": {
- "columns":["moo", "SUM(foo)","SUM(bar)"],
- "results":[["abc", 10, 100],
- ["pqr", 20, 200],
- ["efg", 20, 200],
- ["lmn", 30, 300]]
- }
-
-These options are also available on the query console (checkboxes ``Group By Mode: SQL`` and ``Response Format: SQL``)
diff --git a/docs/record_reader.rst b/docs/record_reader.rst
deleted file mode 100644
index e02fbd2..0000000
--- a/docs/record_reader.rst
+++ /dev/null
@@ -1,117 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-Record Reader
-=============
-
-Pinot supports indexing data from various file formats. To support reading from a file format, a record reader need to
-be provided to read the file and convert records into the general format which the indexing engine can understand. The
-record reader serves as the connector from each individual file format to Pinot record format.
-
-Pinot package provides the following record readers out of the box:
-
-- Avro record reader: record reader for Avro format files
-- CSV record reader: record reader for CSV format files
-- JSON record reader: record reader for JSON format files
-- ORC record reader: record reader for ORC format files
-- Thrift record reader: record reader for Thrift format files
-- Pinot segment record reader: record reader for Pinot segment
-
-Initialize Record Reader
-------------------------
-
-To initialize a record reader, the data file and table schema should be provided (for Pinot segment record reader, only
-need to provide the index directory because schema can be derived from the segment). The output record will follow the
-table schema provided.
-
-For Avro/JSON/ORC/Pinot segment record reader, no extra configuration is required as column names and multi-values are
-embedded in the data file.
-
-For CSV/Thrift record reader, extra configuration might be provided to determine the column names and multi-values for
-the data.
-
-CSV Record Reader Config
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-The CSV record reader config contains the following settings:
-
-- Header: the header for the CSV file (column names)
-- Column delimiter: delimiter for each column
-- Multi-value delimiter: delimiter for each value for a multi-valued column
-
-If no config provided, use the default setting:
-
-- Use the first row in the data file as the header
-- Use ',' as the column delimiter
-- Use ';' as the multi-value delimiter
-
-Thrift Record Reader Config
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The Thrift record reader config is mandatory. It contains the Thrift class name for the record reader to de-serialize
-the Thrift objects.
-
-ORC Record Reader Config
-~~~~~~~~~~~~~~~~~~~~~~~~
-The following property is to be set during segment generation in your Hadoop properties.
-
-record.reader.path: ${FULL_PATH_OF_YOUR_RECORD_READER_CLASS}
-
-For ORC, it would be:
-
-record.reader.path: org.apache.pinot.orc.data.readers.ORCRecordReader
-
-
-Implement Your Own Record Reader
---------------------------------
-
-For other file formats, we provide a general interface for record reader - `RecordReader <https://github.com/apache/incubator-pinot/blob/master/pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/RecordReader.java>`_.
-To index the file into Pinot segment, simply implement the interface and plug it into the index engine - `SegmentCreationDriverImpl <https://github.com/apache/incubator-pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentIndexCreationDriverImpl.java>`_.
-We use a 2-passes algorithm to index the file into Pinot segment, hence the *rewind()* method is required for the record
-reader.
-
-Generic Row
-~~~~~~~~~~~
-
-`GenericRow <https://github.com/apache/incubator-pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/data/GenericRow.java>`_
-is the record abstraction which the index engine can read and index with. It is a map from column name (String) to
-column value (Object). For multi-valued column, the value should be an object array (Object[]).
-
-Contracts for Record Reader
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-There are several contracts for record readers that developers should follow when implementing their own record readers:
-
-- The output GenericRow should follow the table schema provided, in the sense that:
-
- - All the columns in the schema should be preserved (if column does not exist in the original record, put default
- value instead)
- - Columns not in the schema should not be included
- - Values for the column should follow the field spec from the schema (data type, single-valued/multi-valued)
-
-- For the time column (refer to `TimeFieldSpec <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/data/TimeFieldSpec.java>`_),
- record reader should be able to read both incoming and outgoing time (we allow *incoming time - time value from the
- original data* to *outgoing time - time value stored in Pinot* conversion during index creation).
-
- - If incoming and outgoing time column name are the same, use incoming time field spec
- - If incoming and outgoing time column name are different, put both of them as time field spec
- - We keep both incoming and outgoing time column to handle cases where the input file contains time values that are
- already converted
diff --git a/docs/schema.rst b/docs/schema.rst
deleted file mode 100644
index a0ba761..0000000
--- a/docs/schema.rst
+++ /dev/null
@@ -1,140 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-.. _schema-section:
-
-Pinot Schema
-============
-
-Pinot Schema consists of columns that can be categorized as dimensions, metric or time.
-
-**Dimensions** - These are columns that organize the data. For example, ``accountId``, ``country``, ``industry``, etc. These columns are used to slice/dice the data and typically appear in the ``selection``, ``filter`` and ``group-by`` sections in queries.
-
-**Metrics** - These are columns that represent quantative measurements. For example, ``numClicks``, ``pageViews``, etc. These columns typically appear in the aggregation section of the query, e.g., ``select sum(pageViews) from...``.
-
-**Time** - The time column represents the timestamp of the data. There is only one time column in a schema, and typically appears in the ``filter`` and ``group-by`` sections in queries.
-
-A sample schema is shown below and will be used as an example to described the various fields.
-
-Sample Schema:
-~~~~~~~~~~~~~~
-
-.. code-block:: json
-
- {
- "schemaName": "flights",
- "dimensionFieldSpecs": [
- {
- "name": "flightNumber",
- "dataType": "LONG"
- },
- {
- "name": "tags",
- "dataType": "STRING",
- "singleValueField": false,
- "defaultNullValue": "null"
- }
- ],
- "metricFieldSpecs": [
- {
- "name": "price",
- "dataType": "DOUBLE",
- "defaultNullValue": 0
- }
- ],
- "timeFieldSpec": {
- "incomingGranularitySpec": {
- "name": "secondsSinceEpoch",
- "dataType": "INT",
- "timeFormat" : "EPOCH",
- "timeType": "SECONDS"
- },
- "outgoingGranularitySpec": {
- "name": "messageTimeHours",
- "dataType": "INT",
- "timeFormat" : "EPOCH",
- "timeType": "DAYS"
- }
- }
- }
-
-Schema Name:
-~~~~~~~~~~~~
-
-Every Pinot schema has a name that is used to identify it. For example, schema for a table is specified in the table conig by its name.
-
-
-FieldSpecs:
-~~~~~~~~~~~
-
-Each column in the schema is described using a ``fieldSpec`` that captures various attributes of the column, such as name, data-type, etc. A schema may contain an array of ``dimensionFieldSpecs`` that describe all the dimension columns, ``metricFieldSpecs`` that describe all the metric columns, and a ``timeFieldSpec`` that describes the time column.
-
-Dimensions:
-~~~~~~~~~~~
-
-The schema example above has two dimensions ``flightNumber`` of type ``Long`` and ``tags`` of type ``String``. Data types supported for dimension columns are:
-
-* **INT**
-* **FLOAT**
-* **LONG**
-* **DOUBLE**
-* **STRING**
-* **BYTES**
-
-Dimension columns also have an optional attribute named ``singleValuedFiled`` with a default value of ``true``. This attribute describes whether the column can take a single value or multiple values for a row. In the example above, the dimension ``tags`` is multi-valued. This means that it can have multiple values for a particular row, say ``tag1, tag2, tag3``. For a multi-valued column, individual rows don't necessarily need to have the same number of values. Typical use case for this w [...]
-
-
-Metrics:
-~~~~~~~~
-
-The column ``price`` in the schema is a metric column, as one can query aggregations such as ``average``, ``min``, ``max``, etc on it and are typically single-valued. Metric columns can have the following data types:
-
-* **INT**
-* **FLOAT**
-* **LONG**
-* **DOUBLE**
-* **BYTES**
-
-Metric columns are typically numeric. However, note that ``BYTES`` is an allowed data type for metric columns. This is typically used in cases of specialized representations such as HLL, TDigest, etc, where the column actually stores byte serialized version of the value.
-
-Time:
-~~~~~
-
-The schema above also contains a ``timeFieldSpec`` that is used to specify the attributes of the time column:
-
-* **incomingGranularitySpec** : Specifies the name, data type and time type for the time stamp present in the incoming data into Pinot.
-* **outgoingGranularitySpec** : Specifies the name, data type and time type for the time stamp as desired to be stored in Pinot.
-
-In this example, the input timestamp specified in ``SECONDS`` will be automatically converted into ``DAYS`` before storing into Pinot. The ``timeFieldSpec`` also has an optional attribute ``timeFormat`` that can take values ``EPOCH`` (default) and ``SIMPLE_DATE_FORMAT:<format>``.
-
-Time columns are mandatory for ``APPEND`` (incremental data push) use cases but optional for ``REFRESH`` (data refresh with each push) use cases. More details on this can be found at the `Segment Config <tableconfig_schema.html#segments-config-section>`_ section.
-
-
-Default Null Value:
-~~~~~~~~~~~~~~~~~~~
-
-Pinot does not store null values natively, so null values are stored as placeholder values as specified in the fieldspec, or a default value chosen by the system. In the schema example above, all null values in the input for column ``flights`` are converted to String ``"null"`` and stored internally. Simiarly, null values in the input for column ``price`` is converted to integer ``0`` and stored internally. Since this is an optional field, if not specified, default null values are chosen [...]
-
-* **Dimensions**: Numeric fields get a default null value of MIN_VALUE (e.g Integer.MIN_VALUE, Double.MIN_VALUE, etc), and String fields get a default null value of String ``"null"``, if not specified in the field spec.
-* **Metrics**: All metrics get a default null value of numeric ``0`` if not specified in the field spec.
-
-Dimension and metric columns of data type ``BYTES`` get a defaulit null value of ``byte[0]``, if not specified in the field spec. For specifiying a custom defaultNullValue for column of ``BYTES`` type, use the Hex String representation of the byte[] using `this <https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Hex.html>`_ library.
-
diff --git a/docs/segment_fetcher.rst b/docs/segment_fetcher.rst
deleted file mode 100644
index 78e843c..0000000
--- a/docs/segment_fetcher.rst
+++ /dev/null
@@ -1,126 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-.. _segment-fetcher:
-
-Segment Fetchers
-================
-
-When pinot segment files are created in external systems (hadoop/spark/etc), there are several ways to push those data to pinot Controller and Server:
-
-#. push segment to shared NFS and let pinot pull segment files from the location of that NFS.
-#. push segment to a Web server and let pinot pull segment files from the Web server with http/https link.
-#. push segment to HDFS and let pinot pull segment files from HDFS with hdfs location uri.
-#. push segment to other system and implement your own segment fetcher to pull data from those systems.
-
-The first two options should be supported out of the box with pinot package. As long your remote jobs send Pinot controller with the corresponding URI to the files it will pick up the file and allocate it to proper Pinot Servers and brokers. To enable Pinot support for HDFS, you will need to provide Pinot Hadoop configuration and proper Hadoop dependencies.
-
-HDFS segment fetcher configs
------------------------------
-
-In your Pinot controller/server configuration, you will need to provide the following configs:
-
-.. code-block:: none
-
- pinot.controller.segment.fetcher.hdfs.hadoop.conf.path=`<file path to hadoop conf folder>
-
-
-or
-
-.. code-block:: none
-
-
- pinot.server.segment.fetcher.hdfs.hadoop.conf.path=`<file path to hadoop conf folder>
-
-
-This path should point the local folder containing ``core-site.xml`` and ``hdfs-site.xml`` files from your Hadoop installation
-
-.. code-block:: none
-
- pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.principle=`<your kerberos principal>
- pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.keytab=`<your kerberos keytab>
-
-or
-
-.. code-block:: none
-
- pinot.server.segment.fetcher.hdfs.hadoop.kerberos.principle=`<your kerberos principal>
- pinot.server.segment.fetcher.hdfs.hadoop.kerberos.keytab=`<your kerberos keytab>
-
-
-These two configs should be the corresponding Kerberos configuration if your Hadoop installation is secured with Kerberos. Please check Hadoop Kerberos guide on how to generate Kerberos security identification.
-
-You will also need to provide proper Hadoop dependencies jars from your Hadoop installation to your Pinot startup scripts.
-
-Push HDFS segment to Pinot Controller
--------------------------------------
-
-To push HDFS segment files to Pinot controller, you just need to ensure you have proper Hadoop configuration as we mentioned in the previous part. Then your remote segment creation/push job can send the HDFS path of your newly created segment files to the Pinot Controller and let it download the files.
-
-For example, the following curl requests to Controller will notify it to download segment files to the proper table:
-
-.. code-block:: none
-
- curl -X POST -H "UPLOAD_TYPE:URI" -H "DOWNLOAD_URI:hdfs://nameservice1/hadoop/path/to/segment/file.gz" -H "content-type:application/json" -d '' localhost:9000/v2/segments
-
-
-Note that we have two versions of the segment endpoint. V1 is deprecated and in most cases, should not be used. In this endpoint, the segment is not moved. It is
-assumed that where the segment is currently stored in HDFS is where it will be served from.
-
-.. code-block:: none
-
- curl -X POST -H "UPLOAD_TYPE:URI" -H "DOWNLOAD_URI:hdfs://nameservice1/hadoop/path/to/segment/file.gz" -H "content-type:application/json" -d '' localhost:9000/segments
-
-In Pinot, users can also store encrypted segments in the deep storage of their choice. This is accomplished by the following.
-We have an option for users to implement their own crypter class based on the PinotCrypter interface.
-The simple class name of the crypter is declared inside the header during segment upload.
-
-After you implement your own crypter, you will be able to upload segments with the below command.
-
-.. code-block:: none
-
- curl -X POST -F "UPLOAD_TYPE:URI" -H "DOWNLOAD_URI:hdfs://nameservice1/hadoop/path/to/segment/file.gz" -H "content-type:application/json" -d '' localhost:9000/segments --header "CRYPTER: ${CRYPTER_SIMPLE_CLASS_NAME}"
-
-On the server and controller side, you will need to configure several properties for your particular crypter class. All
-crypter classes are extended from the PinotCrypter interface - feel free to implement your own as well.
-
-.. code-block:: none
-
- pinot.server.crypter.class.${SIMPLE_NAME_OF_CRYPTER}=${FULL_PATH_TO_CRYPTER}
- pinot.server.crypter.lipinotcrypter.keyMap.${FIRST_KEY}=${KEY_VALUE}
-
-Implement your own segment fetcher for other systems
-----------------------------------------------------
-
-You can also implement your own segment fetchers for other file systems and load into Pinot system with an external jar. All you need to do is to implement a class that extends the interface of `SegmentFetcher <https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/segment/fetcher/SegmentFetcher.java>`_ and provides config to Pinot Controller and Server as follows:
-
-.. code-block:: none
-
- pinot.controller.segment.fetcher.`<protocol>`.class =`<class path to your implementation>
-
-or
-
-.. code-block:: none
-
- pinot.server.segment.fetcher.`<protocol>`.class =`<class path to your implementation>
-
-You can also provide other configs to your fetcher under config-root ``pinot.server.segment.fetcher.<protocol>``
-
diff --git a/docs/star-tree/example.png b/docs/star-tree/example.png
deleted file mode 100755
index 2ab957d..0000000
Binary files a/docs/star-tree/example.png and /dev/null differ
diff --git a/docs/star-tree/space-time.png b/docs/star-tree/space-time.png
deleted file mode 100755
index c684a80..0000000
Binary files a/docs/star-tree/space-time.png and /dev/null differ
diff --git a/docs/star-tree/star-tree.rst b/docs/star-tree/star-tree.rst
deleted file mode 100644
index 630982e..0000000
--- a/docs/star-tree/star-tree.rst
+++ /dev/null
@@ -1,351 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-Star-Tree: A Specialized Index for Fast Aggregations
-====================================================
-
-One of the biggest challenges in realtime OLAP systems is achieving and maintaining tight SLA’s on latency and
-throughput on large data sets.
-
-Existing techniques such as sorted index or inverted index help improve query latencies, but speed-ups are still limited
-by number of documents necessary to process for computing the results. On the other hand, pre-aggregating the results
-ensures a constant upper bound on query latencies, but can lead to storage space explosion.
-
-Here we introduce **Star-Tree** index to utilize the pre-aggregated documents in a smart way to achieve low query
-latencies but also use the storage space efficiently for aggregation/group-by queries.
-
-Existing Solutions
-------------------
-
-Consider the following data set as an example to discuss the existing approaches:
-
-========= ========= ======== =============
- Country Browser Locale Impressions
-========= ========= ======== =============
-CA Chrome en 400
-CA Firefox fr 200
-MX Safari es 300
-MX Safari en 100
-USA Chrome en 600
-USA Firefox es 200
-USA Firefox en 400
-========= ========= ======== =============
-
-Sorted Index
-~~~~~~~~~~~~
-
-In this approach, data is sorted on a primary key, which is likely to appear as filter in most queries in the query set.
-
-This reduces the time to search the documents for a given primary key value from linear scan *O(n)* to binary search
-*O(logn)*, and also keeps good locality for the documents selected.
-
-While this is a good improvement over linear scan, there are still a few issues with this approach:
-
-- While sorting on one column does not require additional space, sorting on additional columns would require additional
- storage space to re-index the records for the various sort orders.
-
-- While search time is reduced from *O(n)* to *O(logn)*, overall latency is still a function of total number of
- documents need to be processed to answer a query.
-
-Inverted Index
-~~~~~~~~~~~~~~
-
-In this approach, for each value of a given column, we maintain a list of document id’s where this value appears.
-
-Below are the inverted indexes for columns ‘Browser’ and ‘Locale’ for our example data set:
-
-========= ========
- Browser Doc Id
-========= ========
-Firefox 1,5,6
-Chrome 0,4
-Safari 2,3
-========= ========
-
-======== ========
- Locale Doc Id
-======== ========
-en 0,3,4,6
-es 2,5
-fr 1
-======== ========
-
-For example, if we want to get all the documents where ‘Browser’ is ‘Firefox’, we can simply look up the inverted index
-for ‘Browser’ and identify that it appears in documents [1, 5, 6].
-
-Using inverted index, we can reduce the search time to constant time *O(1)*. However, the query latency is still a
-function of the selectivity of the query, i.e. increases with the number of documents need to be processed to answer the
-query.
-
-Pre-aggregation
-~~~~~~~~~~~~~~~
-
-In this technique, we pre-compute the answer for a given query set upfront.
-
-In the example below, we have pre-aggregated the total impressions for each country:
-
-========= =============
- Country Impressions
-========= =============
-CA 600
-MX 400
-USA 1200
-========= =============
-
-Doing so makes answering queries about total impressions for a country just a value lookup, by eliminating the need of
-processing a large number of documents. However, to be able to answer with multiple predicates implies pre-aggregating
-for various combinations of different dimensions. This leads to exponential explosion in storage space.
-
-Star-Tree Solution
-------------------
-
-On one end of the spectrum we have indexing techniques that improve search times with limited increase in space, but do
-not guarantee a hard upper bound on query latencies. On the other end of the spectrum we have pre-aggregation techniques
-that offer hard upper bound on query latencies, but suffer from exponential explosion of storage space.
-
-.. figure:: space-time.png
-
- Space-Time Trade Off Between Different Techniques
-
-We propose the Star-Tree data structure that offers a configurable trade-off between space and time and allows us to
-achieve hard upper bound for query latencies for a given use case. In the following sections we will define the
-Star-Tree data structure, and discuss how it is utilized within Pinot for achieving low latencies with high throughput.
-
-Definition
-~~~~~~~~~~
-
-Tree Structure
-``````````````
-
-Star-Tree is a tree data structure that is consisted of the following properties:
-
-.. figure:: structure.png
-
- Star-Tree Structure
-
-- **Root Node** (Orange):
- Single root node, from which the rest of the tree can be traversed.
-
-- **Leaf Node** (Blue):
- A leaf node can containing at most *T* records, where *T* is configurable.
-
-- **Non-leaf Node** (Green):
- Nodes with more than *T* records are further split into children nodes.
-
-- **Star-Node** (Yellow):
- Non-leaf nodes can also have a special child node called the Star-Node. This node contains the pre-aggregated records
- after removing the dimension on which the data was split for this level.
-
-- **Dimensions Split Order** ([D1, D2]):
- Nodes at a given level in the tree are split into children nodes on all values of a particular dimension. The
- dimensions split order is an ordered list of dimensions that is used to determine the dimension to split on for a
- given level in the tree.
-
-Node Properties
-```````````````
-
-The properties stored in each node are as follows:
-
-- **Dimension**:
- The dimension which the node is split on
-
-- **Start/End Document Id**:
- The range of documents this node points to
-
-- **Aggregated Document Id**:
- One single document which is the aggregation result of all documents pointed by this node
-
-Index Generation
-~~~~~~~~~~~~~~~~
-
-Star-Tree index is generated in the following steps:
-
-- The data is first projected as per the *dimensionsSplitOrder*. Only the dimensions from the split order are reserved,
- others are dropped. For each unique combination of reserved dimensions, metrics are aggregated per configuration. The
- aggregated documents are written to a file and served as the initial Star-Tree documents (separate from the original
- documents).
-
-- Sort the Star-Tree documents based on the *dimensionsSplitOrder*. It is primary-sorted on the first dimension in this
- list, and then secondary sorted on the rest of the dimensions based on their order in the list. Each node in the tree
- points to a range in the sorted documents.
-
-- The tree structure can be created recursively (starting at root node) as follows:
-
- - If a node has more than *T* records, it is split into multiple children nodes, one for each value of the dimension
- in the split order corresponding to current level in the tree.
-
- - A Star-Node can be created (per configuration) for the current node, by dropping the dimension being split on, and
- aggregating the metrics for rows containing dimensions with identical values. These aggregated documents are
- appended to the end of the Star-Tree documents.
-
- If there is only one value for the current dimension, Star-Node won't be created because the documents under the
- Star-Node are identical to the single node.
-
-- The above step is repeated recursively until there are no more nodes to split.
-
-- Multiple Star-Trees can be generated based on different configurations (*dimensionsSplitOrder*, *aggregations*, *T*)
-
-Aggregation
-~~~~~~~~~~~
-
-Aggregation is configured as a pair of aggregation function and the column to apply the aggregation.
-
-All types of aggregation function with bounded-sized intermediate result are supported.
-
-Supported Functions
-```````````````````
-
-- COUNT
-- MIN
-- MAX
-- SUM
-- AVG
-- MINMAXRANGE
-- DISTINCTCOUNTHLL
-- PERCENTILEEST
-- PERCENTILETDIGEST
-
-Unsupported Functions
-`````````````````````
-
-- DISTINCTCOUNT:
- Intermediate result *Set* is unbounded
-
-- PERCENTILE:
- Intermediate result *List* is unbounded
-
-Index Generation Configuration
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Multiple index generation configurations can be provided to generate multiple Star-Trees. Each configuration should
-contain the following properties:
-
-- **dimensionsSplitOrder**:
- An ordered list of dimension names can be specified to configure the split order. Only the dimensions in this list are
- reserved in the aggregated documents. The nodes will be split based on the order of this list. For example, split at
- level *i* is performed on the values of dimension at index *i* in the list.
-
-- **skipStarNodeCreationForDimensions** (Optional, default empty):
- A list of dimension names for which to not create the Star-Node.
-
-- **functionColumnPairs**:
- A list of aggregation function and column pairs (split by double underscore "__").
- E.g. **SUM__Impressions** (*SUM* of column *Impressions*)
-
-- **maxLeafRecords** (Optional, default 10000):
- The threshold *T* to determine whether to further split each node.
-
-Example
-~~~~~~~
-
-For our example data set, with the following example configuration, the tree and documents should be something like
-below.
-
-StarTreeIndexConfig
-```````````````````
-
-.. code-block:: json
-
- {
- "dimensionsSplitOrder": [
- "Country",
- "Browser",
- "Locale"
- ],
- "skipStarNodeCreationForDimensions": [],
- "functionColumnPairs": [
- "SUM__Impressions"
- ],
- "maxLeafRecords": 1
- }
-
-Tree Structure
-``````````````
-The values in the parentheses are the aggregated sum of *Impressions* for all the documents under the node.
-
-.. figure:: example.png
-
- Star-Tree Example
-
-Star-Tree documents
-```````````````````
-
-========= ========= ======== ==================
- Country Browser Locale SUM__Impressions
-========= ========= ======== ==================
-CA Chrome en 400
-CA Firefox fr 200
-MX Safari en 100
-MX Safari es 300
-USA Chrome en 600
-USA Firefox en 400
-USA Firefox es 200
-CA \* en 400
-CA \* fr 200
-CA \* \* 600
-MX Safari \* 400
-USA Firefox \* 600
-USA \* en 1000
-USA \* es 200
-USA \* \* 1200
-\* Chrome en 1000
-\* Firefox en 400
-\* Firefox es 200
-\* Firefox fr 200
-\* Firefox \* 800
-\* Safari en 100
-\* Safari es 300
-\* Safari \* 400
-\* \* en 1500
-\* \* es 500
-\* \* fr 200
-\* \* \* 2200
-========= ========= ======== ==================
-
-Query Execution
-~~~~~~~~~~~~~~~
-
-For query execution, the idea is to first check metadata to determine whether the query can be solved with the Star-Tree
-documents, then traverse the Star-Tree to identify documents that satisfy all the predicates. After applying any
-remaining predicates that were missed while traversing the Star-Tree to the identified documents, apply
-aggregation/group-by on the qualified documents.
-
-
-The algorithm to traverse the tree can be described as follows:
-
-- Start from root node.
-
-- For each level, what child node(s) to select depends on whether there are any predicates/group-by on the split
- dimension for the level in the query.
-
- - If there is no predicate or group-by on the split dimension, select the Star-Node if exists, or all child nodes to
- traverse further.
-
- - If there are predicate(s) on the split dimension, select the child node(s) that satisfy the predicate(s).
-
- - If there is no predicate, but there is a group-by on the split dimension, select all child nodes except Star-Node.
-
-- Recursively repeat the previous step until all leaf nodes are reached, or all predicates are satisfied.
-
-- Collect all the documents pointed to by the selected nodes.
-
- - If all predicates and group-bys are satisfied, pick the single aggregated document from each selected node.
-
- - Otherwise, collect all the documents in the document range from each selected node.
diff --git a/docs/star-tree/structure.png b/docs/star-tree/structure.png
deleted file mode 100755
index 98009a8..0000000
Binary files a/docs/star-tree/structure.png and /dev/null differ
diff --git a/docs/tableconfig_schema.rst b/docs/tableconfig_schema.rst
deleted file mode 100644
index 7ebe804..0000000
--- a/docs/tableconfig_schema.rst
+++ /dev/null
@@ -1,387 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-.. _table-config-section:
-
-Table Config
-============
-
-Sample table config and descriptions
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-A sample table config is shown below which has sub-sections collasped. The sub-sections will be described individually in following sections. Further links to feature specific documention will be included where available.
-
-``tableName`` - Should only contain alpha-numeric characters, hyphens ('-'), or underscores ('_'). Though using a double-underscore ('__') is not allowed and reserved for other features within Pinot.
-
-``tableType`` - Indicates the type of the table. There are some settings specific to each type. This will be clarified below as each sub-section is explained.
-
- * Allowed values:
-
- * ``OFFLINE`` - An offline table is used to host data which might be periodically uploaded - daily, weekly, etc. More information on `Offline Tables <architecture.html#ingesting-offline-data>`_
- * ``REALTIME`` - A realtime table is used to consume data from incoming data streams and serve this data in a near-realtime manner. More information on `Realtime Tables <architecture.html#ingesting-realtime-data>`_
-
-.. code-block:: none
-
- {
- "tableName": "myPinotTable",
- "tableType": "REALTIME"
- "segmentsConfig": {...},
- "tableIndexConfig": {...},
- "tenants": {...},
- "routing": {...},
- "task": {...},
- "metadata": {...}
- }
-
-Some sections are required, otherwise the table config will be rejected by pinot-controller. The required sections are:
-
-* ``tableName``
-* ``tabletype``
-* ``"segmentsConfig": {...}``
-* ``"tableIndexConfig": {...}``
-* ``"tenants": {...}``
-* ``"metadata": {...}``
-
-In case of realtime tables, the ``streamConfig`` section is required.
-
-Segments Config Section
-~~~~~~~~~~~~~~~~~~~~~~~
-
-The ``segmentsConfig`` section has information about configuring the following:
-
-* Segment Retention - with the ``retentionTimeUnit`` and ``retentionTimeValue`` options. Retention is only applicable to tables of type ``APPEND``.
-
- * Allowed values:
-
- * ``retentionTimeUnit`` - ``DAYS``
- * ``retentionTimeValue`` - Positive integers
-
-* ``segmentPushFrequency`` - to indicate how frequently segments are uploaded. Ignored for tables of type ``REALTIME``.
-
- * Allowed values - ``daily``, ``hourly``
-
-* ``segmentPushType`` - Indicates the type of push to the table. Ignored for tables of type ``REALTIME``.
-
- * Allowed values:
-
- * ``APPEND`` means new data will be pushed and appended to the current data in the table, all realtime tables *must* be explicity set to ``APPEND``.
- * ``REFRESH`` will refresh the entire dataset contained within the table. Segment retention is ignored when set to ``REFRESH``.
-
-* ``replication`` - Number of replicas of data in a table, used for tables of type ``OFFLINE`` and tables of type ``REALIME`` if ``stream.<consumername>.consumer.type`` is set to ``HighLevel`` (See :ref:`stream-config-description`)
-
- * Allowed values - Positive integers
-
-* ``replicasPerPartition`` - Number of replicas that consume a single partition of streaming data of a table, used for tables of type ``REALTIME`` if ``stream.<consumername>.consumer.type`` is set to ``LowLevel`` (See :ref:`stream-config-description`)
-
- * Allowed values - Positive integers
-
-* Time column - using ``timeColumnName`` and ``timeType``, this must match what's configured in the preceeding schema. This is a special column that Pinot uses to manage retention (removing old segments), split query between ``REALTIME`` and ``OFFLINE`` tables in a hybrid table, etc.
-
- * Allowed values - String, this must match the ``timeFieldSpec`` section in the schema
-
-* Segment assignment strategy - Described more on the page `Customizing Pinot <customizations.html#segment-assignment-strategies>`_
-
-
-.. code-block:: none
-
- "segmentsConfig": {
- "retentionTimeUnit": "DAYS",
- "retentionTimeValue": "5",
- "segmentPushFrequency": "daily",
- "segmentPushType": "APPEND",
- "replication": "3",
- "replicasPerPartition": "3",
- "schemaName": "myPinotSchmea",
- "timeColumnName": "daysSinceEpoch",
- "timeType": "DAYS",
- "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
- },
-
-* Completion Config
-
- You can also add a ``completionConfig`` section under the ``segmentsConfig`` section. Completion config holds information related to realtime segment completion. There is just one field in this config as of now, which is the ``completionMode``. The value of the ``completioMode`` decides how non-committers servers should replace the in-memory segment during realtime segment completion. Refer to the `Architecture <architecture.html#ingesting-realtime-data>`_ for description about committ [...]
-
- By default, if the in memory segment in the non-winner server is equivalent to the committed segment, then the non-committer server builds and replaces the segment, else it download the segment from the controller.
-
- Currently, the supported value for ``completionMode`` is
-
- * ``DOWNLOAD``: In certain scenarios, segment build can get very memory intensive. It might become desirable to enforce the non-committer servers to just download the segment from the controller, instead of building it again. Setting this completionMode ensures that the non-committer servers always download the segment.
-
-
-For example:
-
-.. code-block:: none
-
- "segmentsConfig": {
- ..
- ..
- "completionConfig": {
- "completionMode": "DOWNLOAD"
- }
- },
-
-Table Index Config Section
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The ``tableIndexConfig`` section has information about how to configure:
-
-* ``invertedIndexColumns`` - Indicates a list of real column names as specified in the schema to create inverted indexes for. More info on indexes can be found on the `Index Techniques <index_techniques.html>`_ page.
-
- * Allowed values - String; string must match the column name in the corresponding schema
-
-* ``noDictionaryColumns`` - Indicates a list of real column names as specified in the schema. Column names present will **not** have a dictionary created. More info on indexes can be found on the `Index Techniques <index_techniques.html>`_ page.
-
- * Allowed values - String; string must match the column name in the corresponding schema
-
-* ``sortedColumn`` - Indicates a list of real column names as specified in the schema. Data should be sorted based on the column names provided. This field needs to be set only for realtime tables. For offline, if the data at source is sorted, we will create a sorted index automatically. More info on indexes can be found on the `Index Techniques <index_techniques.html>`_ page.
-
- * Allowed values - String; string must match the column name in the corresponding schema
-
-* ``aggregateMetrics`` - Switch for the aggregate metrics feature. This feature will aggregate realtime stream data as it is consumed, where applicable, in order to reduce segment sizes. We sum the metric column values of all rows that have the same value for dimension columns and create one row in a realtime segment for all such rows. This feature is only available on REALTIME tables.
-
- * Allowed values - ``true`` to enable, ``false`` to disable.
-
-.. todo::
-
- Create a separate section to describe this feature and design, then link to it from this config description
-
-* ``segmentPartitionConfig`` - Configures the Data Partitioning Strategy. Further documentation on this feather available in the `Data Partitioning Strategies <customizations.html#data-partitioning-strategies>`_ section.
-* ``loadMode`` - indicates how data will be loaded on pinot-server. either ``"MMAP"`` or ``"HEAP"`` can be configured.
-
- * Allowed values:
-
- * ``MMAP`` - Configures pinot-server to load data segments to off-heap memory.
- * ``HEAP`` - Configures pinot-server to load data directly into direct memory.
-
-* ``streamConfigs`` - This section is where the bulk of the settings specific to only ``REALTIME`` tables are found. See :ref:`stream-config-description`
-
-.. code-block:: none
-
- "tableIndexConfig": {
- "invertedIndexColumns": [],
- "noDictionaryColumns" : [],
- "sortedColumn": [
- "nameOfSortedColumn"
- ],
- "noDictionaryColumns": [
- "nameOfNoDictionaryColumn"
- ],
- "aggregateMetrics": "true",
- "segmentPartitionConfig": {
- "columnPartitionMap": {
- "contentId": {
- "functionName": "murmur",
- "numPartitions": 32
- }
- }
- },
- "loadMode": "MMAP",
- "lazyLoad": "false",
- "segmentFormatVersion": "v3",
- "streamConfigs": {}
- },
-
-Tenants Section
-~~~~~~~~~~~~~~~
-
-The ``tenants`` section has two main config fields in it. These fields are used to configure which tenants are used within Helix.
-
-.. code-block:: none
-
- "tenants": {
- "broker": "brokerTenantName",
- "server": "serverTenantName"
- },
-
-
-In the above example,
-
-* The table will be served by brokers that have been tagged as ``brokerTenantName_BROKER`` in Helix.
-
-* The offline segments for the table will be hosted in pinot servers tagged in helix as ``serverTenantName_OFFLINE``
-
-* The realtime segments (both consuming as well as completed ones) will be hosted in pinot servers tagged in helix as ``serverTeantName_REALTIME``.
-
-
-You can also add a ``tagOverrideConfig`` section under the ``tenants`` section. Currently, the only override allowed is to add additional tags for
-consuming and completed segments. For example:
-
-.. code-block:: none
-
- "tenants": {
- "broker": "brokerTenantName",
- "server": "serverTenantName",
- "tagOverrideConfig" : {
- "realtimeConsuming" : "serverTenantName_REALTIME"
- "realtimeCompleted" : "serverTenantName_OFFLINE"
- }
- }
-
-
-In the above example, the consuming segments will stil be assigned to ``serverTenantName_REALTIME`` hosts, but once they are completed, the
-segments will be moved to ``serverTeantnName_OFFLINE``. It is possible to specify the full name of *any* tag in this section (so, for example, you
-could decide that completed segments for this table should be in pinot servers tagged as ``allTables_COMPLETED``).
-
-See :ref:`ingesting-realtime-data` section for more details on consuming and completed segments.
-
-
-Routing Section
-~~~~~~~~~~~~~~~
-
-The ``routing`` section contains configurations on how which routingTableBuilder will be used and to pass options specific to that builder. There is more information in the `Routing Strategies <customizations.html#routing-strategies>`_ section.
-
-.. code-block:: none
-
- "routing": {
- "routingTableBuilderName": "PartitionAwareRealtime",
- "routingTableBuilderOptions": {}
- },
-
-Metadata Section
-~~~~~~~~~~~~~~~~
-
-The ``metadata`` section is used for passing special key-value pairs into Pinot which will be stored with the table config inside of Pinot. There's more info in the `Custom Configs <customizations.html#custom-configs>`_ section.
-
-.. code-block:: none
-
- "metadata": {
- "customConfigs": {
- "specialConfig": "testValue",
- "anotherSpecialConfig": "value"
- }
- }
-
-.. _stream-config-description:
-
-StreamConfigs Section
-~~~~~~~~~~~~~~~~~~~~~
-
-This section is specific to tables of type ``REALTIME`` and is ignored if the table type is any other.
-See section on :ref:`ingesting-realtime-data` for an overview of how realtime ingestion works.
-
-Here is a minimal example of what the ``streamConfigs`` section may look like:
-
-.. code-block:: none
-
- "streamConfigs" : {
- "realtime.segment.flush.threshold.size": "0",
- "realtime.segment.flush.threshold.time": "24h",
- "realtime.segment.flush.desired.size": "150M",
- "streamType": "kafka",
- "stream.kafka.consumer.type": "LowLevel",
- "stream.kafka.topic.name": "ClickStream",
- "stream.kafka.consumer.prop.auto.offset.reset" : "largest"
- }
-
-
-The ``streamType`` field is mandatory. In this case, it is set to ``kafka``.
-StreamType of ``kafka`` is supported natively in Pinot.
-You can use default decoder classes and consumer factory classes.
-Pinot allows you to use other stream types with their own consumer factory
-and decoder classes (or, even other decoder and consumer factory for ``kafka`` if
-your installation formats kafka messages differently).
-
-If you are considering adding support for streams other than Kafka,
-please see section :ref:`pluggable-streams`.
-
-There are some configurations that are generic to all stream types, and others that
-are specific to stream types.
-
-Configuration generic to all stream types
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-* ``realtime.segment.flush.threshold.size``: Maximum number of rows to consume before persisting the consuming segment.
-
- Note that in the example above, it is set to
- ``0``. In this case, Pinot automatically computes the row limit using the value of ``realtime.segment.flush.desired.size``
- described below. If the consumer type is ``HighLevel``, then this value will be the maximum per consuming
- segment. If the consumer type is ``LowLevel`` then this value will be divided across all consumers being hosted
- on any one pinot-server.
-
- Default is ``5000000``.
-
-* ``realtime.segment.flush.threshold.time``: Maximum elapsed time after which a consuming segment should be persisted.
-
- The value can be set as a human readable string, such as ``"1d"``,
- ``"4h30m"``, etc. This value should be set such that it is not below the retention of
- messages in the underlying stream, but is not so long that it may cause the server to run out of memory.
-
- Default is ``"6h"``
-
-
-* ``realtime.segment.flush.desired.size``: Desired size of the completed segments.
-
- This setting is supported only if consumer type is set to ``LowLevel``.
- This value can be set as a human readable string such as ``"150M"``, or ``"1.1G"``, etc.
- This value is used when ``realtime.segment.flush.threshold.size``
- is set to 0. Pinot learns and then estimates the number of rows that need to be consumed so that the
- persisted segment
- is approximately of this size. The learning phase starts by setting the number of rows to 100,000
- (can be changed with the setting ``realtime.segment.flush.autotune.initialRows``).
- and increasing to reach the desired segment size. Segment size may go over the desired size significantly
- during the learning phase.
- Pinot corrects the estimation as it
- goes along, so it is not guaranteed that the resulting completed segments are of the exact size
- as configured. You should set this value to optimize the performance of queries (i.e. neither
- too small nor too large)
-
- Default is ``"200M"``
-
-* ``realtime.segment.flush.autotune.initialRows``: Initial number of rows for learning.
-
- This value is used only if ``realtime.segment.flush.threshold.size`` is set o ``0`` and the consumer
- type is ``LowLevel``. See ``realtime.segment.flush.desired.size`` above.
-
- Default is ``"100K"``
-
-
-Configuration specific to stream types
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-All of these configuaration items have the prefix ``stream.<streamType>``. In the example above,
-the prefix is ``stream.kafka``.
-
-Important ones to note here are:
-
-* ``stream.kafka.consumer.type``: This should have a value of ``LowLevel`` (recommended) or ``HighLevel``.
-
- Be sure to set the value of ``replicasPerPartition`` correctly as described before in your table configuration.
-
-* ``stream.kafka.topic.name``: Name of the topic from which to consume.
-
-* ``stream.kafka.consumer.prop.auto.offset.reset``: Indicates where to start consumption from in the stream.
-
- If the consumer type is ``LowLevel``, This configuration is used only when the table is first provisioned.
- In ``HighLevel`` consumer type, it will also be used when new servers are rolled in, or when existing servers
- are replaced with new ones.
- You can specify values such as ``smallest`` or ``largest``, or even ``3d`` if your stream supports it.
- If you specify ``largest``, the consumption starts from the most recent events in the data stream.
- This is the recommended way to create a new table.
-
- If you specify ``smallest`` then the consumption starts from the earliest event avaiable in the
- data stream.
-
-All the configurations that are prefixed with the streamtype are expected to be used by the underlying
-stream. So, you can set any of the configurations described in the
-`Kafka configuraton page <https://kafka.apache.org/documentation/#consumerconfigs>`_ can be set using
-the prefix ``stream.kafka`` and Kafka should pay attention to it.
-
-More options are explained in the `Pluggable Streams <pluggable_streams.html#pluggable-streams>`_ section.
diff --git a/docs/tuning_pinot.rst b/docs/tuning_pinot.rst
deleted file mode 100644
index 31325ef..0000000
--- a/docs/tuning_pinot.rst
+++ /dev/null
@@ -1,43 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-.. _tuning-pinot:
-
-Tuning Pinot
-============
-
-This section provides information on various options to tune Pinot cluster for storage and query efficiency.
-Unlike Key-Value store, tuning Pinot sometimes can be tricky because the cost of query can vary depending on the
-workload and data characteristics.
-
-If you want to improve query latency for your use case, you can refer to ``Index Techniques`` section. If your
-use case faces the scalability issue after tuning index, you can refer ``Optimizing Scatter and Gather`` for
-improving query throughput for Pinot cluster. If you have identified a performance issue on the specific component
-(broker or server), you can refer to the ``Tuning Broker`` or ``Tuning Server`` section.
-
-
-.. toctree::
- :maxdepth: 1
-
- index_techniques
- star-tree/star-tree
- tuning_scatter_and_gather
- tuning_realtime_performance
diff --git a/docs/tuning_realtime_performance.rst b/docs/tuning_realtime_performance.rst
deleted file mode 100644
index 3c559d3..0000000
--- a/docs/tuning_realtime_performance.rst
+++ /dev/null
@@ -1,115 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-Tuning Realtime Performance
-===========================
-
-See the section on :ref:`ingesting-realtime-data` before reading this section.
-
-Pinot servers ingest rows into a consuming segment that resides in volatile memory.
-Therefore, pinot servers hosting consuming segments tend to be memory bound.
-They may also have long garbage collection cycles when the segment is completed
-and memory is released.
-
-Controlling memory allocation
------------------------------
-
-You can configure pinot servers to use off-heap memory for dictionary and forward
-indices of consuming segments by setting the value of ``pinot.server.instance.realtime.alloc.offheap``
-to ``true``. With this configuration in place, the server allocates off-heap memory by memory-mapping
-files. These files are never flushed to stable storage by Pinot (the Operating System may do so depending
-on demand for memory on the host). The files are discarded when the consuming segment is turned into
-a completed segment.
-
-By default the files are created under the directory where the table's segments are stored
-in local disk attached to the consuming server.
-You can set a specific directory for consuming segments with the configuration
-``pinot.server.consumerDir``. Given that there is no control over flushing of
-pages from the memory mapped for consuming segments, you may want to set the directory
-to point to a memory-based file system, eliminating wasteful disk I/O.
-
-If memory-mapping is not desirable, you can set ``pinot.server.instance.realtime.alloc.offheap.direct``
-to ``true``. In this case, pinot allocates direct
-`ByteBuffer <https://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html>`_ objects for
-consuming segments. Using direct allocation can potentially result in address space fragmentation.
-
-.. note::
-
- We still use heap memory to store inverted indices for consuming segments.
-
-
-Controlling number of rows in consuming segment
------------------------------------------------
-
-The number of rows in a consuming segment needs to be balanced. Having too many rows can result in
-memory pressure. On the other hand, having too few rows results in having too many small segments.
-Having too many segments can be detrimental to query performance, and also increase pressure on the Helix.
-
-The recommended way to do this is to use the ``realtime.segment.flush.desired.size`` setting as described in
-:ref:`stream-config-description`. You can run the administrative tool ``pinot-admin.sh RealtimeProvisioningHelper``
-that will help you to come up with an optimal setting for the segment size.
-
-Moving completed segments to different hosts
---------------------------------------------
-
-This feature is avaialble only if the consumption type is ``LowLevel``.
-
-The structure of the consuming segments and the completed segments are very different. The memory, CPU, I/O
-and GC characteristics could be very different while processing queries on these segments. Therefore it may be
-useful to move the completed segments onto differnt set of hosts in some use cases.
-
-You can host completed segments on a different set of hosts using the ``tagOverrideConfig`` as described in
-:ref:`table-config-section`. Pinot will automatically move them once the consuming segments are completed.
-
-Controlling segment build vs segment download on Realtime servers
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-This feature is available only if the consumption type is ``LowLevel``.
-
-When a realtime segment completes, a winner server is chosen as a committer amongst all replicas by the controller. That committer builds the segment and uploads to the controller. The non-committer servers are asked to catchup to the winning offset. If the non-committer servers are able to catch up, they are asked to build the segment and replace the in-memory segment. If they are unable to catchup, they are asked to download the segment from the controller.
-
-Building a segment can cause excessive garbage and may result in GC pauses on the server.
-Long GC pauses can affect query processing. In order to avoid this, we have a configuration
-that allows you to control whether
-
-It might become desirable to force the non-committer servers to download the segment from the controller, instead of building it again. The ``completionConfig`` as described in :ref:`table-config-section` can be used to configure this.
-
-Fine tuning the segment commit protocol
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-This feature is available only if the consumption type is ``LowLevel``.
-
-Once a committer is asked to commit the segment, it builds a segment, and issues an HTTP POST to the controller, with the segment.
-The controller than commits the segment in Zookeeper and starts the next consuming segment.
-
-It is possible to conifigure the servers to do a *split* commit, in which the committer performs the following steps:
-
- * Build the segment
- * Start a transaction with the lead controller to commit the segment (CommitStart phase)
- * Post the completed segment to any of the controllers (and the controller posts it to segment store)
- * End the transaction with the lead controller (CommentEnd phase). Optionally, this step can be done *with* the segment metadata.
-
-This method of committing can be useful if the network bandwidth on the lead controller is limiting segment uploads.
-
-In order to accomplish this, you will need to set the following configurations:
- * On the controller, set ``pinot.controller.enable.split.commit`` to ``true`` (default is ``false``).
- * On the server, set ``pinot.server.enable.split.commit`` to ``true`` (default is ``false``).
- * On the server, set ``pinot.server.enable.commitend.metadata`` to ``true`` (default is false).
diff --git a/docs/tuning_scatter_and_gather.rst b/docs/tuning_scatter_and_gather.rst
deleted file mode 100644
index 5429644..0000000
--- a/docs/tuning_scatter_and_gather.rst
+++ /dev/null
@@ -1,184 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-Optimizing Scatter and Gather
-=============================
-
-.. contents:: Table of Contents
-
-
-When the use case has very high qps along with low latency requirements (usually site facing use cases),
-we need to consider optimizing the scatter-and-gather.
-
-Below table summarizes the two issues with the default behavior of Pinot.
-
-.. csv-table::
- :header: "Problem", "Impact", "Solution"
- :widths: 15, 15, 15
-
- "Querying all servers", "Bad tail latency, not scalable", "Control the number of servers to fan out"
- "Querying all segments", "More CPU work on server", "Minimize the number of segment"
-
-Querying All Servers
---------------------
-By default, Pinot uses ``BalanceNumSegmentAssignmentStrategy`` for segment assignment. This scheme tries to distribute
-the number of segments uniformly to all servers. When we perform scatter-and-gather a query request, broker will try
-to uniformly distribute the workload among servers by assigning the balanced number of segments to each server. As
-a result, each query will span out to all servers under this scheme. It works pretty well when qps is low and you
-have small number of servers in the cluster. However, as we add more servers or have more qps, the probability of
-hitting slow server (e.g. gc) increases steeply and Pinot will suffer from a long tail latency.
-
-In order to address this issue, we have introduced a concept of ``Replica Group``, which allows us to control the
-number of servers to fan out for each query.
-
-
-Replica Group Segment Assignment and Query Routing
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-``Replica Group`` is a set of server that contains a 'complete' set of all segments of a table. Once we assign the
-segment based on replica group, each query can be answered by fanning out to a replica group instead of all servers.
-
-.. image:: img/replica-group.png
-
-``Replica Group`` based segment assignment can be configured for a table by setting it in the table config. Note that
-``ReplicaGroupSegmentAssignmentStrategy`` needs to be used along with ``PartitionAwareOffline`` for routing and this is
-currently available for **offline table** only.
-
-.. code-block:: none
-
- {
- "segmentsConfig": {
- ...
- "replication": "3",
- "segmentAssignmentStrategy": "ReplicaGroupSegmentAssignmentStrategy",
- "replicaGroupStrategyConfig": {
- "mirrorAssignmentAcrossReplicaGroups": true,
- "numInstancesPerPartition": 4
- }
- }
- ...
- "routing": {
- "routingTableBuilderName": "PartitionAwareOffline",
- "routingTableBuilderOptions": {}
- },
- ...
- }
-
-As seen above, you can use ``replication`` and ``numInstancesPerPartition`` to control the number of servers to span. For
-instance, let's say that you have 12 servers in the cluster. Above configuration will generate 3 replica groups (based on
-``replication=3``) and each replica group will contain 4 servers (``numInstancesPerPartition=4``). In this example, each
-query will span to a single replica group (4 servers).
-
-As you seen above, replica group gives you the control on the number of servers to span for each query. When you try to
-decide the proper number of ``replication`` and ``numInstancesPerPartition``, you should consider the trade-off between
-throughput and latency. Given a fixed number of servers, increasing ``replication`` factor while decreasing
-``numInstancesPerPartition`` will give you more throughput because each server requires to process less number of queries.
-However, each server will need to process more number of segments per query, thus increasing overall latency. Similarly,
-decreasing ``replication`` while increasing ``numInstancesPerPartition`` will make each server processing more number
-of queries but each server needs to process less number of segments per query. So, this number has to be decided based
-on the use case requirements.
-
-
-Querying All Segments
----------------------
-
-By default, Pinot broker will distribute all segments for query processing and segment pruning is happening in Server.
-In other words, Server will look at the segment metadata such as min/max time value and discard the segment if it does
-not contain any data that the query is asking for. Server side pruning works pretty well when the qps is low; however,
-it becomes the bottleneck if qps is very high (hundreds to thousands queries per second) because unnecessary segments
-still need to be scheduled for processing and consume cpu resources.
-
-Currently, we have two different mechanisms to prune segments on the broker side to minimize the number of segment for
-processing before scatter-and-gather.
-
-Partitioning
-^^^^^^^^^^^^
-When the data is partitioned on a dimension, each segment will contain all the rows with the same partition value for
-a partitioning dimension. In this case, a lot of segments can be pruned if a query requires to look at a single
-partition to compute the result. Below diagram gives the example of data partitioned on member id while the query
-includes an equality filter on member id.
-
-
-.. image:: img/partitioning.png
-
-``Partitoning`` can be enabled by setting the following configuration in the table config.
-
-.. code-block:: none
-
- {
- "tableIndexConfig": {
- "segmentPartitionConfig": {
- "columnPartitionMap": {
- "memberId": {
- "functionName": "modulo",
- "numPartitions": 4
- }
- }
- }
- }
- ...
- "routing": {
- "routingTableBuilderName": "PartitionAwareOffline",
- "routingTableBuilderOptions": {}
- },
- }
-
-Pinot currently supports ``modulo`` and ``murmur`` hash function. After setting the above config, data needs to be partitioned
-using the same partition function and the number of partition to partition before running Pinot segment conversion and push job
-for offline push. Realtime partitioning depends on the kafka for partitioning. When emitting an event to kafka, a user need to
-feed partitioning key and partition function for Kafka producer API.
-
-When applied correctly, partition information should be available in the segment metadata.
-
-.. code-block:: none
-
- $ column.memberId.partitionFunction = Murmur
- column.memberId.partitionValues = [9 9]
-
-
-Note that broker side pruning for partitioning only works with ``PartitionAwareOffline`` and ``PartitionAwareRealtime`` routing
-table builder strategies. Also note that the current implementation for partitioning only works for **EQUALITY** filter
-(e.g. memberId = xx).
-
-
-Bloom Filter for Dictionary
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Dictionary encoding provides the array of unique values. Pinot allows to create a bloom filter on this unique values for each
-column. Bloom filter can quickly determine whether the value exist in the segment.
-
-Bloom filter can be enabled by setting the following configuration in the table config.
-
-.. code-block:: none
-
- {
- "tableIndexConfig": {
- "bloomFilterColumns": [
- "column_name",
- ...
- ],
- ...
- }
- }
-
-Our implementation limits the size of bloom filter to be less than 1MB per segment along with max false positive of 5% to
-avoid consuming too much memory. We recommend to put bloom filter for the column with ``less than 1 million cardinality``.
-
-Note that the current implementation for bloom filter also works for **EQUALITY** filter only.
\ No newline at end of file
diff --git a/docs/user_guide.rst b/docs/user_guide.rst
deleted file mode 100644
index d14f5f4..0000000
--- a/docs/user_guide.rst
+++ /dev/null
@@ -1,35 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. warning:: The documentation is not up-to-date and has moved to `Apache Pinot Docs <https://docs.pinot.apache.org/>`_.
-
-:orphan:
-
-##########
-User Guide
-##########
-
-.. _user-guide:
-
-.. toctree::
- :maxdepth: 1
-
- pql_examples
- client_api
-
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 01/07: Fix bug #6671: RealtimeTableDataManager
shuts down SegmentBuildTimeLeaseExtender for all tables in the host (#6682)
Posted by xi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch release-0.7.1-rc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit b892729490d9db54b9eaa2d1899298902d669c74
Author: Jiapeng Tao <ji...@linkedin.com>
AuthorDate: Tue Mar 16 00:38:58 2021 -0700
Fix bug #6671: RealtimeTableDataManager shuts down SegmentBuildTimeLeaseExtender for all tables in the host (#6682)
Co-authored-by: Jiapeng Tao <ji...@jiatao-mn1.linkedin.biz>
---
.../realtime/LLRealtimeSegmentDataManager.java | 2 +-
.../manager/realtime/RealtimeTableDataManager.java | 2 +-
.../realtime/SegmentBuildTimeLeaseExtender.java | 59 +++++++++++++++-------
.../realtime/LLRealtimeSegmentDataManagerTest.java | 37 ++++++++++++--
.../starter/helix/HelixInstanceDataManager.java | 8 ++-
5 files changed, 83 insertions(+), 25 deletions(-)
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
index 84d4592..853fcf0 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
@@ -1115,7 +1115,7 @@ public class LLRealtimeSegmentDataManager extends RealtimeSegmentDataManager {
_serverMetrics = serverMetrics;
_segmentVersion = indexLoadingConfig.getSegmentVersion();
_instanceId = _realtimeTableDataManager.getServerInstance();
- _leaseExtender = SegmentBuildTimeLeaseExtender.getLeaseExtender(_instanceId);
+ _leaseExtender = SegmentBuildTimeLeaseExtender.getLeaseExtender(_tableNameWithType);
_protocolHandler = new ServerSegmentCompletionProtocolHandler(_serverMetrics, _tableNameWithType);
String timeColumnName = tableConfig.getValidationConfig().getTimeColumnName();
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
index 33283b9..35f040e 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
@@ -120,7 +120,7 @@ public class RealtimeTableDataManager extends BaseTableDataManager {
@Override
protected void doInit() {
- _leaseExtender = SegmentBuildTimeLeaseExtender.create(_instanceId, _serverMetrics, _tableNameWithType);
+ _leaseExtender = SegmentBuildTimeLeaseExtender.getOrCreate(_instanceId, _serverMetrics, _tableNameWithType);
File statsFile = new File(_tableDataDir, STATS_FILE_NAME);
try {
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentBuildTimeLeaseExtender.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentBuildTimeLeaseExtender.java
index 69d7e80..725dc95 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentBuildTimeLeaseExtender.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentBuildTimeLeaseExtender.java
@@ -18,8 +18,8 @@
*/
package org.apache.pinot.core.data.manager.realtime;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.util.concurrent.Uninterruptibles;
-import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Future;
@@ -45,42 +45,63 @@ public class SegmentBuildTimeLeaseExtender {
private static final int EXTRA_TIME_SECONDS = 120;
// Retransmit lease request 10% before lease expires.
private static final int REPEAT_REQUEST_PERIOD_SEC = (EXTRA_TIME_SECONDS * 9 / 10);
- private static Logger LOGGER = LoggerFactory.getLogger(SegmentBuildTimeLeaseExtender.class);
- private static final Map<String, SegmentBuildTimeLeaseExtender> INSTANCE_TO_LEASE_EXTENDER = new HashMap<>(1);
+ private static final Logger LOGGER = LoggerFactory.getLogger(SegmentBuildTimeLeaseExtender.class);
+ private static final Map<String, SegmentBuildTimeLeaseExtender> TABLE_TO_LEASE_EXTENDER = new ConcurrentHashMap<>();
+ private static ScheduledExecutorService _executor;
- private ScheduledExecutorService _executor;
private final Map<String, Future> _segmentToFutureMap = new ConcurrentHashMap<>();
private final String _instanceId;
+ private final String _tableNameWithType;
private final ServerSegmentCompletionProtocolHandler _protocolHandler;
- public static SegmentBuildTimeLeaseExtender getLeaseExtender(final String instanceId) {
- return INSTANCE_TO_LEASE_EXTENDER.get(instanceId);
+ public static void initExecutor() {
+ _executor = new ScheduledThreadPoolExecutor(1);
}
- public static synchronized SegmentBuildTimeLeaseExtender create(final String instanceId,
- ServerMetrics serverMetrics, String tableNameWithType) {
- SegmentBuildTimeLeaseExtender leaseExtender = INSTANCE_TO_LEASE_EXTENDER.get(instanceId);
- if (leaseExtender != null) {
- LOGGER.warn("Instance already exists");
- } else {
- leaseExtender = new SegmentBuildTimeLeaseExtender(instanceId, serverMetrics, tableNameWithType);
- INSTANCE_TO_LEASE_EXTENDER.put(instanceId, leaseExtender);
+ public static void shutdownExecutor() {
+ if (_executor != null) {
+ _executor.shutdownNow();
+ _executor = null;
}
- return leaseExtender;
+ }
+
+ @VisibleForTesting
+ public static boolean isExecutorShutdown() {
+ return _executor == null;
+ }
+
+ public static SegmentBuildTimeLeaseExtender getLeaseExtender(final String tableNameWithType) {
+ return TABLE_TO_LEASE_EXTENDER.get(tableNameWithType);
+ }
+
+ public static SegmentBuildTimeLeaseExtender getOrCreate(final String instanceId,
+ ServerMetrics serverMetrics, String tableNameWithType) {
+ return TABLE_TO_LEASE_EXTENDER.compute(tableNameWithType, (k, v) -> {
+ if (v == null) {
+ return new SegmentBuildTimeLeaseExtender(instanceId, serverMetrics, tableNameWithType);
+ } else {
+ LOGGER.warn("Lease extender for Table: {} already exists", tableNameWithType);
+ return v;
+ }
+ });
}
private SegmentBuildTimeLeaseExtender(String instanceId, ServerMetrics serverMetrics, String tableNameWithType) {
_instanceId = instanceId;
+ _tableNameWithType = tableNameWithType;
_protocolHandler = new ServerSegmentCompletionProtocolHandler(serverMetrics, tableNameWithType);
- _executor = new ScheduledThreadPoolExecutor(1);
}
public void shutDown() {
- if (_executor != null) {
- _executor.shutdownNow();
- _executor = null;
+ for (Map.Entry<String, Future> entry : _segmentToFutureMap.entrySet()) {
+ Future future = entry.getValue();
+ boolean cancelled = future.cancel(true);
+ if (!cancelled) {
+ LOGGER.warn("Task could not be cancelled for {}" + entry.getKey());
+ }
}
_segmentToFutureMap.clear();
+ TABLE_TO_LEASE_EXTENDER.remove(_tableNameWithType);
}
/**
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManagerTest.java b/pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManagerTest.java
index 0017c43..0e468a4 100644
--- a/pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManagerTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManagerTest.java
@@ -30,13 +30,19 @@ import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Semaphore;
import java.util.concurrent.atomic.AtomicReference;
import org.apache.commons.io.FileUtils;
+import org.apache.helix.HelixManager;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
import org.apache.pinot.common.metadata.instance.InstanceZKMetadata;
import org.apache.pinot.common.metadata.segment.LLCRealtimeSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.RealtimeSegmentZKMetadata;
import org.apache.pinot.common.metrics.ServerMetrics;
import org.apache.pinot.common.protocols.SegmentCompletionProtocol;
import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.config.TableConfigUtils;
+import org.apache.pinot.core.data.manager.TableDataManager;
import org.apache.pinot.core.data.manager.config.InstanceDataManagerConfig;
+import org.apache.pinot.core.data.manager.config.TableDataManagerConfig;
+import org.apache.pinot.core.data.manager.offline.TableDataManagerProvider;
import org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl;
import org.apache.pinot.core.realtime.impl.RealtimeSegmentStatsHistory;
import org.apache.pinot.core.realtime.impl.fakestream.FakeStreamConsumerFactory;
@@ -58,6 +64,8 @@ import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.Mockito.anyString;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
@@ -128,9 +136,9 @@ public class LLRealtimeSegmentDataManagerTest {
return JsonUtils.stringToObject(_tableConfigJson, TableConfig.class);
}
- private RealtimeTableDataManager createTableDataManager() {
+ private RealtimeTableDataManager createTableDataManager(TableConfig tableConfig) {
final String instanceId = "server-1";
- SegmentBuildTimeLeaseExtender.create(instanceId, new ServerMetrics(new MetricsRegistry()), _tableName);
+ SegmentBuildTimeLeaseExtender.getOrCreate(instanceId, new ServerMetrics(new MetricsRegistry()), _tableName);
RealtimeTableDataManager tableDataManager = mock(RealtimeTableDataManager.class);
when(tableDataManager.getServerInstance()).thenReturn(instanceId);
RealtimeSegmentStatsHistory statsHistory = mock(RealtimeSegmentStatsHistory.class);
@@ -154,7 +162,7 @@ public class LLRealtimeSegmentDataManagerTest {
LLCRealtimeSegmentZKMetadata segmentZKMetadata = createZkMetadata();
TableConfig tableConfig = createTableConfig();
InstanceZKMetadata instanceZKMetadata = new InstanceZKMetadata();
- RealtimeTableDataManager tableDataManager = createTableDataManager();
+ RealtimeTableDataManager tableDataManager = createTableDataManager(tableConfig);
String resourceDir = _segmentDir;
LLCSegmentName llcSegmentName = new LLCSegmentName(_segmentNameStr);
_partitionIdToSemaphoreMap.putIfAbsent(_partitionId, new Semaphore(1));
@@ -169,11 +177,13 @@ public class LLRealtimeSegmentDataManagerTest {
@BeforeClass
public void setUp() {
_segmentDirFile.deleteOnExit();
+ SegmentBuildTimeLeaseExtender.initExecutor();
}
@AfterClass
public void tearDown() {
FileUtils.deleteQuietly(_segmentDirFile);
+ SegmentBuildTimeLeaseExtender.shutdownExecutor();
}
@Test
@@ -765,6 +775,27 @@ public class LLRealtimeSegmentDataManagerTest {
Assert.assertEquals(secondSegmentDataManager.get().getPartitionConsumerSemaphore().availablePermits(), 1);
}
+ @Test
+ public void testShutdownTableDataManagerWillNotShutdownLeaseExtenderExecutor()
+ throws Exception {
+ TableConfig tableConfig = createTableConfig();
+ tableConfig.setUpsertConfig(null);
+ ZkHelixPropertyStore propertyStore = mock(ZkHelixPropertyStore.class);
+ when(propertyStore.get(anyString(), any(), anyInt())).thenReturn(TableConfigUtils.toZNRecord(tableConfig));
+
+ TableDataManagerConfig tableDataManagerConfig = mock(TableDataManagerConfig.class);
+ when(tableDataManagerConfig.getTableDataManagerType()).thenReturn("REALTIME");
+ when(tableDataManagerConfig.getTableName()).thenReturn(tableConfig.getTableName());
+ when(tableDataManagerConfig.getDataDir()).thenReturn(FileUtils.getTempDirectoryPath());
+
+ TableDataManager tableDataManager = TableDataManagerProvider
+ .getTableDataManager(tableDataManagerConfig, "testInstance", propertyStore, mock(ServerMetrics.class),
+ mock(HelixManager.class));
+ tableDataManager.start();
+ tableDataManager.shutDown();
+ Assert.assertFalse(SegmentBuildTimeLeaseExtender.isExecutorShutdown());
+ }
+
public static class FakeLLRealtimeSegmentDataManager extends LLRealtimeSegmentDataManager {
public Field _state;
diff --git a/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java b/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
index deb3a6a..30ba125 100644
--- a/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
+++ b/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
@@ -41,6 +41,7 @@ import org.apache.pinot.core.data.manager.SegmentDataManager;
import org.apache.pinot.core.data.manager.TableDataManager;
import org.apache.pinot.core.data.manager.config.TableDataManagerConfig;
import org.apache.pinot.core.data.manager.offline.TableDataManagerProvider;
+import org.apache.pinot.core.data.manager.realtime.SegmentBuildTimeLeaseExtender;
import org.apache.pinot.core.indexsegment.immutable.ImmutableSegment;
import org.apache.pinot.core.indexsegment.immutable.ImmutableSegmentLoader;
import org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl;
@@ -91,9 +92,12 @@ public class HelixInstanceDataManager implements InstanceDataManager {
Preconditions.checkState(instanceSegmentTarDir.mkdirs());
}
+ // Initialize segment build time lease extender executor
+ SegmentBuildTimeLeaseExtender.initExecutor();
+ LOGGER.info("Initialized segment build time lease extender executor");
+
// Initialize the table data manager provider
TableDataManagerProvider.init(_instanceDataManagerConfig);
-
LOGGER.info("Initialized Helix instance data manager");
}
@@ -108,6 +112,8 @@ public class HelixInstanceDataManager implements InstanceDataManager {
for (TableDataManager tableDataManager : _tableDataManagerMap.values()) {
tableDataManager.shutDown();
}
+ SegmentBuildTimeLeaseExtender.shutdownExecutor();
+ LOGGER.info("Segment build time lease extender executor shut down");
LOGGER.info("Helix instance data manager shut down");
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 03/07: Fix the default map return value in
DictionaryBasedGroupKeyGenerator (#6712)
Posted by xi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch release-0.7.1-rc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 5b373b0e77649e8c8406250a3bbd21e41c6cb247
Author: Xiaotian (Jackie) Jiang <17...@users.noreply.github.com>
AuthorDate: Tue Mar 23 09:01:19 2021 -0700
Fix the default map return value in DictionaryBasedGroupKeyGenerator (#6712)
---
.../groupby/DictionaryBasedGroupKeyGenerator.java | 22 ++++++++++++++++------
.../DefaultAggregationExecutorTest.java | 4 +---
.../DoubleAggregationResultHolderTest.java | 4 +---
.../AggregationGroupByTrimmingServiceTest.java | 4 +---
.../DictionaryBasedGroupKeyGeneratorTest.java | 17 ++++++++++++++---
.../groupby/DoubleGroupByResultHolderTest.java | 4 +---
.../groupby/NoDictionaryGroupKeyGeneratorTest.java | 5 +----
.../aggregation/groupby/StringGroupKeyTest.java | 3 +--
.../query/executor/QueryExecutorTest.java | 18 +++++++-----------
.../selection/SelectionOperatorServiceTest.java | 4 +---
10 files changed, 44 insertions(+), 41 deletions(-)
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/DictionaryBasedGroupKeyGenerator.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/DictionaryBasedGroupKeyGenerator.java
index 98f42a4..29dc067 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/DictionaryBasedGroupKeyGenerator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/DictionaryBasedGroupKeyGenerator.java
@@ -64,11 +64,20 @@ public class DictionaryBasedGroupKeyGenerator implements GroupKeyGenerator {
private static final int INITIAL_MAP_SIZE = (int) ((1 << 9) * 0.75f);
private static final int MAX_CACHING_MAP_SIZE = (int) ((1 << 20) * 0.75f);
- private static final ThreadLocal<IntGroupIdMap> THREAD_LOCAL_INT_MAP = ThreadLocal.withInitial(IntGroupIdMap::new);
- private static final ThreadLocal<Long2IntOpenHashMap> THREAD_LOCAL_LONG_MAP =
- ThreadLocal.withInitial(() -> new Long2IntOpenHashMap(INITIAL_MAP_SIZE));
- private static final ThreadLocal<Object2IntOpenHashMap<IntArray>> THREAD_LOCAL_INT_ARRAY_MAP =
- ThreadLocal.withInitial(() -> new Object2IntOpenHashMap<>(INITIAL_MAP_SIZE));
+ @VisibleForTesting
+ static final ThreadLocal<IntGroupIdMap> THREAD_LOCAL_INT_MAP = ThreadLocal.withInitial(IntGroupIdMap::new);
+ @VisibleForTesting
+ static final ThreadLocal<Long2IntOpenHashMap> THREAD_LOCAL_LONG_MAP = ThreadLocal.withInitial(() -> {
+ Long2IntOpenHashMap map = new Long2IntOpenHashMap(INITIAL_MAP_SIZE);
+ map.defaultReturnValue(INVALID_ID);
+ return map;
+ });
+ @VisibleForTesting
+ static final ThreadLocal<Object2IntOpenHashMap<IntArray>> THREAD_LOCAL_INT_ARRAY_MAP = ThreadLocal.withInitial(() -> {
+ Object2IntOpenHashMap<IntArray> map = new Object2IntOpenHashMap<>(INITIAL_MAP_SIZE);
+ map.defaultReturnValue(INVALID_ID);
+ return map;
+ });
private final ExpressionContext[] _groupByExpressions;
private final int _numGroupByExpressions;
@@ -1085,8 +1094,9 @@ public class DictionaryBasedGroupKeyGenerator implements GroupKeyGenerator {
/**
* Drop un-necessary checks for highest performance.
*/
+ @VisibleForTesting
@SuppressWarnings("EqualsWhichDoesntCheckParameterClass")
- private static class IntArray {
+ static class IntArray {
public int[] _elements;
public IntArray(int[] elements) {
diff --git a/pinot-core/src/test/java/org/apache/pinot/query/aggregation/DefaultAggregationExecutorTest.java b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/DefaultAggregationExecutorTest.java
similarity index 98%
rename from pinot-core/src/test/java/org/apache/pinot/query/aggregation/DefaultAggregationExecutorTest.java
rename to pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/DefaultAggregationExecutorTest.java
index 618514f..d655576 100644
--- a/pinot-core/src/test/java/org/apache/pinot/query/aggregation/DefaultAggregationExecutorTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/DefaultAggregationExecutorTest.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.query.aggregation;
+package org.apache.pinot.core.query.aggregation;
import java.io.File;
import java.util.ArrayList;
@@ -37,8 +37,6 @@ import org.apache.pinot.core.operator.blocks.TransformBlock;
import org.apache.pinot.core.operator.filter.MatchAllFilterOperator;
import org.apache.pinot.core.operator.transform.TransformOperator;
import org.apache.pinot.core.plan.DocIdSetPlanNode;
-import org.apache.pinot.core.query.aggregation.AggregationExecutor;
-import org.apache.pinot.core.query.aggregation.DefaultAggregationExecutor;
import org.apache.pinot.core.query.aggregation.function.AggregationFunction;
import org.apache.pinot.core.query.request.context.ExpressionContext;
import org.apache.pinot.core.query.request.context.QueryContext;
diff --git a/pinot-core/src/test/java/org/apache/pinot/query/aggregation/DoubleAggregationResultHolderTest.java b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/DoubleAggregationResultHolderTest.java
similarity index 91%
rename from pinot-core/src/test/java/org/apache/pinot/query/aggregation/DoubleAggregationResultHolderTest.java
rename to pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/DoubleAggregationResultHolderTest.java
index 1b1a97f..a28dd94 100644
--- a/pinot-core/src/test/java/org/apache/pinot/query/aggregation/DoubleAggregationResultHolderTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/DoubleAggregationResultHolderTest.java
@@ -16,11 +16,9 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.query.aggregation;
+package org.apache.pinot.core.query.aggregation;
import java.util.Random;
-import org.apache.pinot.core.query.aggregation.AggregationResultHolder;
-import org.apache.pinot.core.query.aggregation.DoubleAggregationResultHolder;
import org.testng.Assert;
import org.testng.annotations.Test;
diff --git a/pinot-core/src/test/java/org/apache/pinot/query/aggregation/groupby/AggregationGroupByTrimmingServiceTest.java b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/AggregationGroupByTrimmingServiceTest.java
similarity index 96%
rename from pinot-core/src/test/java/org/apache/pinot/query/aggregation/groupby/AggregationGroupByTrimmingServiceTest.java
rename to pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/AggregationGroupByTrimmingServiceTest.java
index eae4e30..ce1e0f4 100644
--- a/pinot-core/src/test/java/org/apache/pinot/query/aggregation/groupby/AggregationGroupByTrimmingServiceTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/AggregationGroupByTrimmingServiceTest.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.query.aggregation.groupby;
+package org.apache.pinot.core.query.aggregation.groupby;
import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
import java.util.ArrayList;
@@ -28,8 +28,6 @@ import java.util.Random;
import java.util.Set;
import org.apache.commons.lang.RandomStringUtils;
import org.apache.pinot.common.response.broker.GroupByResult;
-import org.apache.pinot.core.query.aggregation.groupby.AggregationGroupByTrimmingService;
-import org.apache.pinot.core.query.aggregation.groupby.GroupKeyGenerator;
import org.apache.pinot.core.query.request.context.QueryContext;
import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
import org.testng.Assert;
diff --git a/pinot-core/src/test/java/org/apache/pinot/query/aggregation/groupby/DictionaryBasedGroupKeyGeneratorTest.java b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/DictionaryBasedGroupKeyGeneratorTest.java
similarity index 96%
rename from pinot-core/src/test/java/org/apache/pinot/query/aggregation/groupby/DictionaryBasedGroupKeyGeneratorTest.java
rename to pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/DictionaryBasedGroupKeyGeneratorTest.java
index 1dd0b05..7a0e00c 100644
--- a/pinot-core/src/test/java/org/apache/pinot/query/aggregation/groupby/DictionaryBasedGroupKeyGeneratorTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/DictionaryBasedGroupKeyGeneratorTest.java
@@ -16,8 +16,10 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.query.aggregation.groupby;
+package org.apache.pinot.core.query.aggregation.groupby;
+import it.unimi.dsi.fastutil.longs.Long2IntOpenHashMap;
+import it.unimi.dsi.fastutil.objects.Object2IntOpenHashMap;
import java.io.File;
import java.util.ArrayList;
import java.util.Arrays;
@@ -40,8 +42,6 @@ import org.apache.pinot.core.operator.transform.TransformOperator;
import org.apache.pinot.core.plan.DocIdSetPlanNode;
import org.apache.pinot.core.plan.TransformPlanNode;
import org.apache.pinot.core.plan.maker.InstancePlanMakerImplV2;
-import org.apache.pinot.core.query.aggregation.groupby.DictionaryBasedGroupKeyGenerator;
-import org.apache.pinot.core.query.aggregation.groupby.GroupKeyGenerator;
import org.apache.pinot.core.query.request.context.ExpressionContext;
import org.apache.pinot.core.query.request.context.QueryContext;
import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
@@ -423,6 +423,17 @@ public class DictionaryBasedGroupKeyGeneratorTest {
assertEquals(groupKeySet.size(), numUniqueKeys, _errorMessage);
}
+ @Test
+ public void testMapDefaultValue() {
+ Long2IntOpenHashMap longMap = DictionaryBasedGroupKeyGenerator.THREAD_LOCAL_LONG_MAP.get();
+ assertEquals(longMap.get(0L), GroupKeyGenerator.INVALID_ID);
+
+ Object2IntOpenHashMap<DictionaryBasedGroupKeyGenerator.IntArray> intArrayMap =
+ DictionaryBasedGroupKeyGenerator.THREAD_LOCAL_INT_ARRAY_MAP.get();
+ assertEquals(intArrayMap.getInt(new DictionaryBasedGroupKeyGenerator.IntArray(new int[0])),
+ GroupKeyGenerator.INVALID_ID);
+ }
+
@AfterClass
public void tearDown() {
FileUtils.deleteQuietly(new File(INDEX_DIR_PATH));
diff --git a/pinot-core/src/test/java/org/apache/pinot/query/aggregation/groupby/DoubleGroupByResultHolderTest.java b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/DoubleGroupByResultHolderTest.java
similarity index 94%
rename from pinot-core/src/test/java/org/apache/pinot/query/aggregation/groupby/DoubleGroupByResultHolderTest.java
rename to pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/DoubleGroupByResultHolderTest.java
index 3d22087..a7d78e3 100644
--- a/pinot-core/src/test/java/org/apache/pinot/query/aggregation/groupby/DoubleGroupByResultHolderTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/DoubleGroupByResultHolderTest.java
@@ -16,11 +16,9 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.query.aggregation.groupby;
+package org.apache.pinot.core.query.aggregation.groupby;
import java.util.Random;
-import org.apache.pinot.core.query.aggregation.groupby.DoubleGroupByResultHolder;
-import org.apache.pinot.core.query.aggregation.groupby.GroupByResultHolder;
import org.testng.Assert;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Test;
diff --git a/pinot-core/src/test/java/org/apache/pinot/query/aggregation/groupby/NoDictionaryGroupKeyGeneratorTest.java b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/NoDictionaryGroupKeyGeneratorTest.java
similarity index 97%
rename from pinot-core/src/test/java/org/apache/pinot/query/aggregation/groupby/NoDictionaryGroupKeyGeneratorTest.java
rename to pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/NoDictionaryGroupKeyGeneratorTest.java
index 8a728a9..744087b 100644
--- a/pinot-core/src/test/java/org/apache/pinot/query/aggregation/groupby/NoDictionaryGroupKeyGeneratorTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/NoDictionaryGroupKeyGeneratorTest.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.query.aggregation.groupby;
+package org.apache.pinot.core.query.aggregation.groupby;
import java.io.File;
import java.io.IOException;
@@ -40,9 +40,6 @@ import org.apache.pinot.core.operator.transform.TransformOperator;
import org.apache.pinot.core.plan.DocIdSetPlanNode;
import org.apache.pinot.core.plan.TransformPlanNode;
import org.apache.pinot.core.plan.maker.InstancePlanMakerImplV2;
-import org.apache.pinot.core.query.aggregation.groupby.GroupKeyGenerator;
-import org.apache.pinot.core.query.aggregation.groupby.NoDictionaryMultiColumnGroupKeyGenerator;
-import org.apache.pinot.core.query.aggregation.groupby.NoDictionarySingleColumnGroupKeyGenerator;
import org.apache.pinot.core.query.request.context.ExpressionContext;
import org.apache.pinot.core.query.request.context.QueryContext;
import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
diff --git a/pinot-core/src/test/java/org/apache/pinot/query/aggregation/groupby/StringGroupKeyTest.java b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/StringGroupKeyTest.java
similarity index 94%
rename from pinot-core/src/test/java/org/apache/pinot/query/aggregation/groupby/StringGroupKeyTest.java
rename to pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/StringGroupKeyTest.java
index ce9606f..8f3fa87 100644
--- a/pinot-core/src/test/java/org/apache/pinot/query/aggregation/groupby/StringGroupKeyTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/StringGroupKeyTest.java
@@ -16,9 +16,8 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.query.aggregation.groupby;
+package org.apache.pinot.core.query.aggregation.groupby;
-import org.apache.pinot.core.query.aggregation.groupby.GroupKeyGenerator;
import org.testng.Assert;
import org.testng.annotations.Test;
diff --git a/pinot-core/src/test/java/org/apache/pinot/query/executor/QueryExecutorTest.java b/pinot-core/src/test/java/org/apache/pinot/core/query/executor/QueryExecutorTest.java
similarity index 95%
rename from pinot-core/src/test/java/org/apache/pinot/query/executor/QueryExecutorTest.java
rename to pinot-core/src/test/java/org/apache/pinot/core/query/executor/QueryExecutorTest.java
index fa00339..42884ac 100644
--- a/pinot-core/src/test/java/org/apache/pinot/query/executor/QueryExecutorTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/query/executor/QueryExecutorTest.java
@@ -16,10 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.query.executor;
-
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
+package org.apache.pinot.core.query.executor;
import java.io.File;
import java.net.URL;
@@ -27,7 +24,6 @@ import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
-
import org.apache.commons.configuration.PropertiesConfiguration;
import org.apache.commons.io.FileUtils;
import org.apache.helix.HelixManager;
@@ -44,8 +40,6 @@ import org.apache.pinot.core.indexsegment.IndexSegment;
import org.apache.pinot.core.indexsegment.generator.SegmentGeneratorConfig;
import org.apache.pinot.core.indexsegment.immutable.ImmutableSegment;
import org.apache.pinot.core.indexsegment.immutable.ImmutableSegmentLoader;
-import org.apache.pinot.core.query.executor.QueryExecutor;
-import org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl;
import org.apache.pinot.core.query.request.ServerQueryRequest;
import org.apache.pinot.core.segment.creator.SegmentIndexCreationDriver;
import org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl;
@@ -64,6 +58,8 @@ import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
import com.yammer.metrics.core.MetricsRegistry;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
public class QueryExecutorTest {
@@ -96,8 +92,8 @@ public class QueryExecutorTest {
TableConfig tableConfig = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
int i = 0;
for (; i < NUM_SEGMENTS_TO_GENERATE; i++) {
- SegmentGeneratorConfig config =
- SegmentTestUtils.getSegmentGeneratorConfig(avroFile, FileFormat.AVRO, INDEX_DIR, TABLE_NAME, tableConfig, schema);
+ SegmentGeneratorConfig config = SegmentTestUtils
+ .getSegmentGeneratorConfig(avroFile, FileFormat.AVRO, INDEX_DIR, TABLE_NAME, tableConfig, schema);
config.setSegmentNamePostfix(Integer.toString(i));
SegmentIndexCreationDriver driver = new SegmentIndexCreationDriverImpl();
driver.init(config);
@@ -114,8 +110,8 @@ public class QueryExecutorTest {
Assert.assertNotNull(resourceUrl);
File jsonFile = new File(resourceUrl.getFile());
for (; i < NUM_SEGMENTS_TO_GENERATE + NUM_EMPTY_SEGMENTS_TO_GENERATE; i++) {
- SegmentGeneratorConfig config =
- SegmentTestUtils.getSegmentGeneratorConfig(jsonFile, FileFormat.JSON, INDEX_DIR, TABLE_NAME, tableConfig, schema);
+ SegmentGeneratorConfig config = SegmentTestUtils
+ .getSegmentGeneratorConfig(jsonFile, FileFormat.JSON, INDEX_DIR, TABLE_NAME, tableConfig, schema);
config.setSegmentNamePostfix(Integer.toString(i));
SegmentIndexCreationDriver driver = new SegmentIndexCreationDriverImpl();
driver.init(config);
diff --git a/pinot-core/src/test/java/org/apache/pinot/query/selection/SelectionOperatorServiceTest.java b/pinot-core/src/test/java/org/apache/pinot/core/query/selection/SelectionOperatorServiceTest.java
similarity index 98%
rename from pinot-core/src/test/java/org/apache/pinot/query/selection/SelectionOperatorServiceTest.java
rename to pinot-core/src/test/java/org/apache/pinot/core/query/selection/SelectionOperatorServiceTest.java
index e8c321f..324d8b6 100644
--- a/pinot-core/src/test/java/org/apache/pinot/query/selection/SelectionOperatorServiceTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/query/selection/SelectionOperatorServiceTest.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.query.selection;
+package org.apache.pinot.core.query.selection;
import java.io.Serializable;
import java.util.ArrayList;
@@ -33,8 +33,6 @@ import org.apache.pinot.core.indexsegment.IndexSegment;
import org.apache.pinot.core.query.request.context.ExpressionContext;
import org.apache.pinot.core.query.request.context.QueryContext;
import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
-import org.apache.pinot.core.query.selection.SelectionOperatorService;
-import org.apache.pinot.core.query.selection.SelectionOperatorUtils;
import org.apache.pinot.spi.utils.BytesUtils;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 05/07: fixing license headers and plugin checks
(#6748)
Posted by xi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch release-0.7.1-rc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 68785847df682d5ff044d708c53eaf168f612584
Author: Xiang Fu <fx...@gmail.com>
AuthorDate: Wed Apr 7 01:26:47 2021 -0700
fixing license headers and plugin checks (#6748)
---
.codecov.yml | 19 +++
rstheaderdefinition.xml => headerdefinition.xml | 10 ++
pinot-common/src/main/proto/server.proto | 19 +++
.../src/main/resources/zone-index.properties | 19 +++
pinot-controller/src/main/resources/tsconfig.json | 2 +-
.../test/resources/conf/pinot-broker.properties | 19 +++
.../test/resources/conf/query-executor.properties | 19 +++
pinot-core/src/test/resources/data/README | 19 +++
.../src/test/resources/generate-test-data.py | 19 +++
pinot-distribution/pinot-source-assembly.xml | 11 +-
pinot-distribution/pom.xml | 2 +-
.../tests/BaseClusterIntegrationTestSet.java | 2 +-
.../pinot/integration/tests/QueryGenerator.java | 2 +-
...formance_2014_100k_subset.test_queries_10K.pql} | 19 +++
...rformance_2014_100k_subset.test_queries_10K.sql | 19 +++
...formance_2014_100k_subset.test_queries_500.pql} | 19 +++
...rformance_2014_100k_subset.test_queries_500.sql | 19 +++
.../test/resources/compat-tests/kafka/topic1.cfg | 19 +++
.../src/test/resources/index.html | 20 +++
.../org.mockito.plugins.MockMaker | 19 +++
.../pinot-s3/src/test/resources/log4j2.xml | 20 +++
.../src/test/resources/complex_types.proto | 19 +++
.../pinot-protobuf/src/test/resources/sample.proto | 19 +++
.../test/resources/conf/pinot-broker.properties | 19 +++
.../resources/pinot-configuration-1.properties | 19 +++
.../resources/pinot-configuration-2.properties | 19 +++
.../resources/pinot-configuration-3.properties | 21 ++-
.../conf/sample_filesystem_benchmark.conf | 19 +++
.../resources/conf/sample_query_comparison.conf | 19 +++
pom.xml | 178 ++++++++++-----------
30 files changed, 541 insertions(+), 107 deletions(-)
diff --git a/.codecov.yml b/.codecov.yml
index e66155e..d9bee7f 100644
--- a/.codecov.yml
+++ b/.codecov.yml
@@ -1,3 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
coverage:
ignore:
# Ignore all files under pinot-tools, pinot-perf and pinot-hadoop.
diff --git a/rstheaderdefinition.xml b/headerdefinition.xml
similarity index 76%
rename from rstheaderdefinition.xml
rename to headerdefinition.xml
index 309cf00..f3fd428 100644
--- a/rstheaderdefinition.xml
+++ b/headerdefinition.xml
@@ -30,4 +30,14 @@
<isMultiline>true</isMultiline>
<padLines>false</padLines>
</rst_style>
+ <javadoc_style>
+ <firstLine>/**</firstLine>
+ <beforeEachLine> * </beforeEachLine>
+ <endLine> */</endLine>
+ <firstLineDetectionPattern>(\s|\t)*/\*.*$</firstLineDetectionPattern>
+ <lastLineDetectionPattern>.*\*/(\s|\t)*$</lastLineDetectionPattern>
+ <allowBlankLines>false</allowBlankLines>
+ <isMultiline>true</isMultiline>
+ <padLines>false</padLines>
+ </javadoc_style>
</additionalHeaders>
diff --git a/pinot-common/src/main/proto/server.proto b/pinot-common/src/main/proto/server.proto
index 4b3c75a..0239ae1 100644
--- a/pinot-common/src/main/proto/server.proto
+++ b/pinot-common/src/main/proto/server.proto
@@ -1,3 +1,22 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
diff --git a/pinot-common/src/main/resources/zone-index.properties b/pinot-common/src/main/resources/zone-index.properties
index 3735834..f3bb967 100644
--- a/pinot-common/src/main/resources/zone-index.properties
+++ b/pinot-common/src/main/resources/zone-index.properties
@@ -1,4 +1,23 @@
#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+#
# DO NOT REMOVE OR MODIFY EXISTING ENTRIES
#
# This file contain the fixed numeric id of every supported time zone id.
diff --git a/pinot-controller/src/main/resources/tsconfig.json b/pinot-controller/src/main/resources/tsconfig.json
index ead39f2..0b8c72a 100644
--- a/pinot-controller/src/main/resources/tsconfig.json
+++ b/pinot-controller/src/main/resources/tsconfig.json
@@ -26,4 +26,4 @@
"esModuleInterop": true,
"jsx": "react"
}
-}
\ No newline at end of file
+}
diff --git a/pinot-core/src/test/resources/conf/pinot-broker.properties b/pinot-core/src/test/resources/conf/pinot-broker.properties
index 2efa525..25fba3f 100644
--- a/pinot-core/src/test/resources/conf/pinot-broker.properties
+++ b/pinot-core/src/test/resources/conf/pinot-broker.properties
@@ -1,3 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
pinot.broker.routing.tableName=testTable
# Number of nodes constituting a replica-set. A replica-set has 0-(n-1) nodes where n is the below config value.
pinot.broker.routing.testTable.numNodesPerReplica=1
diff --git a/pinot-core/src/test/resources/conf/query-executor.properties b/pinot-core/src/test/resources/conf/query-executor.properties
index 4659aff..547385c 100644
--- a/pinot-core/src/test/resources/conf/query-executor.properties
+++ b/pinot-core/src/test/resources/conf/query-executor.properties
@@ -1,3 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
pruner.class=ValidSegmentPruner,DataSchemaSegmentPruner,ColumnValueSegmentPruner,SelectionQuerySegmentPruner
class=org.apache.pinot.core.query.executor.ServerQueryExecutor
timeout=150000
diff --git a/pinot-core/src/test/resources/data/README b/pinot-core/src/test/resources/data/README
index 0abecf3..a8866dc 100644
--- a/pinot-core/src/test/resources/data/README
+++ b/pinot-core/src/test/resources/data/README
@@ -1,3 +1,22 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
paddingNull.tar.gz, paddingPercent.tar.gz and paddingOld.tar.gz are tar/gzip'ed versions of segments created
with a padding character of Null, padding character of "%" and a fixed default padding character
of "%" respectively. The old format was created until 08/2016
diff --git a/pinot-core/src/test/resources/generate-test-data.py b/pinot-core/src/test/resources/generate-test-data.py
index cc97fdc..03ffb1f 100755
--- a/pinot-core/src/test/resources/generate-test-data.py
+++ b/pinot-core/src/test/resources/generate-test-data.py
@@ -1,4 +1,23 @@
#!/usr/bin/env python
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
import argparse
import sys
from fastavro import writer
diff --git a/pinot-distribution/pinot-source-assembly.xml b/pinot-distribution/pinot-source-assembly.xml
index 4b06c82..bc61316 100644
--- a/pinot-distribution/pinot-source-assembly.xml
+++ b/pinot-distribution/pinot-source-assembly.xml
@@ -36,26 +36,25 @@
<exclude>**/*.iws</exclude>
<exclude>**/*.log</exclude>
<exclude>**/.idea/**</exclude>
- <exclude>thirdeye/**</exclude>
- <exclude>**/.travis/**</exclude>
+ <exclude>**/.github/**</exclude>
<exclude>.codecov*</exclude>
<exclude>.gitignore</exclude>
- <exclude>contrib/**</exclude>
<!-- Do not inclue node_modules in pinot-controller -->
<exclude>pinot-controller/src/main/resources/node_modules/**</exclude>
+ <exclude>pinot-controller/src/main/resources/dist/**</exclude>
+ <exclude>pinot-controller/src/main/resources/package-lock.json</exclude>
<!-- Do not include temp files created by maven-release-plugin -->
<exclude>**/*.releaseBackup</exclude>
<exclude>release.properties</exclude>
- <!-- Do not include build files from docs -->
- <exclude>docs/_build/**</exclude>
-
<!-- Do not include docker, kubernetes, website related files -->
<exclude>kubernetes/**</exclude>
<exclude>docker/**</exclude>
<exclude>website/**</exclude>
+ <exclude>contrib/**</exclude>
+ <exclude>thirdeye/**</exclude>
</excludes>
</fileSet>
</fileSets>
diff --git a/pinot-distribution/pom.xml b/pinot-distribution/pom.xml
index ba3bb1e..e189614 100644
--- a/pinot-distribution/pom.xml
+++ b/pinot-distribution/pom.xml
@@ -183,7 +183,7 @@
</build>
<profiles>
<profile>
- <id>apache-release</id>
+ <id>src-dist</id>
<build>
<plugins>
<plugin>
diff --git a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
index 7bf675d..8fc1296 100644
--- a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
+++ b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
@@ -60,7 +60,7 @@ public abstract class BaseClusterIntegrationTestSet extends BaseClusterIntegrati
// Default settings
private static final String DEFAULT_PQL_QUERY_FILE_NAME =
- "On_Time_On_Time_Performance_2014_100k_subset.test_queries_500";
+ "On_Time_On_Time_Performance_2014_100k_subset.test_queries_500.pql";
private static final String DEFAULT_SQL_QUERY_FILE_NAME =
"On_Time_On_Time_Performance_2014_100k_subset.test_queries_500.sql";
private static final int DEFAULT_NUM_QUERIES_TO_GENERATE = 100;
diff --git a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/QueryGenerator.java b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/QueryGenerator.java
index 323e6e0..58a2283 100644
--- a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/QueryGenerator.java
+++ b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/QueryGenerator.java
@@ -217,7 +217,7 @@ public class QueryGenerator {
File avroFile = new File("pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_1.avro");
QueryGenerator queryGenerator = new QueryGenerator(Collections.singletonList(avroFile), "mytable", "mytable");
File outputFile = new File(
- "pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_10K");
+ "pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_10K.pql");
try (BufferedWriter writer = new BufferedWriter(new FileWriter(outputFile))) {
for (int i = 0; i < 10000; i++) {
Query query = queryGenerator.generateQuery();
diff --git a/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_10K b/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_10K.pql
similarity index 99%
rename from pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_10K
rename to pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_10K.pql
index 43950ed..ecc5602 100644
--- a/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_10K
+++ b/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_10K.pql
@@ -1,3 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
{"hsqls":["SELECT DivReachedDest, FirstDepTime, DepDelayMinutes FROM mytable WHERE DestState IN ('VI', 'UT', 'VA') OR ArrTime < 806 ORDER BY OriginAirportSeqID, ArrivalDelayGroups LIMIT 10000"],"pql":"SELECT DivReachedDest, FirstDepTime, DepDelayMinutes FROM mytable WHERE DestState IN ('VI', 'UT', 'VA') OR ArrTime < 806 ORDER BY OriginAirportSeqID, ArrivalDelayGroups LIMIT 10"}
{"hsqls":["SELECT DistanceGroup, Carrier, DepTime FROM mytable WHERE Cancelled IN (1, 0) AND Flights BETWEEN 1 AND 1 OR TaxiOut <> 32 LIMIT 10000"],"pql":"SELECT DistanceGroup, Carrier, DepTime FROM mytable WHERE Cancelled IN (1, 0) AND Flights BETWEEN 1 AND 1 OR TaxiOut <> 32 LIMIT 7"}
{"hsqls":["SELECT ActualElapsedTime, CancellationCode, DivArrDelay, MIN(Month) FROM mytable WHERE DayofMonth BETWEEN 12 AND 24 GROUP BY ActualElapsedTime, CancellationCode, DivArrDelay LIMIT 10000","SELECT ActualElapsedTime, CancellationCode, DivArrDelay, AVG(CAST(DivAirportLandings AS DOUBLE)) FROM mytable WHERE DayofMonth BETWEEN 12 AND 24 GROUP BY ActualElapsedTime, CancellationCode, DivArrDelay LIMIT 10000"],"pql":"SELECT MIN(Month), AVG(DivAirportLandings) FROM mytable WHERE DayofMo [...]
diff --git a/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_10K.sql b/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_10K.sql
index bcb6033..6d56143 100644
--- a/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_10K.sql
+++ b/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_10K.sql
@@ -1,3 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
{"sql":"SELECT DivReachedDest, FirstDepTime, DepDelayMinutes FROM mytable WHERE DestState IN ('VI', 'UT', 'VA') OR ArrTime < 806 ORDER BY OriginAirportSeqID, ArrivalDelayGroups LIMIT 10"}
{"sql":"SELECT DistanceGroup, Carrier, DepTime FROM mytable WHERE Cancelled IN (1, 0) AND Flights BETWEEN 1 AND 1 OR TaxiOut <> 32 LIMIT 7"}
{"sql":"SELECT ActualElapsedTime, CancellationCode, DivArrDelay, MIN(\"Month\"), AVG(DivAirportLandings) FROM mytable WHERE DayofMonth BETWEEN 12 AND 24 GROUP BY ActualElapsedTime, CancellationCode, DivArrDelay LIMIT 29","hsqls":["SELECT ActualElapsedTime, CancellationCode, DivArrDelay, MIN(Month), AVG(DivAirportLandings) FROM mytable WHERE DayofMonth BETWEEN 12 AND 24 GROUP BY ActualElapsedTime, CancellationCode, DivArrDelay LIMIT 29"]}
diff --git a/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_500 b/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_500.pql
similarity index 99%
rename from pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_500
rename to pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_500.pql
index 1f884e3..0217fba 100644
--- a/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_500
+++ b/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_500.pql
@@ -1,3 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
# Aggregation
{"hsqls":["SELECT sum(DepartureDelayGroups) FROM mytable LIMIT 10000","SELECT MIN(FlightNum) FROM mytable LIMIT 10000","SELECT MAX(AirlineID) FROM mytable LIMIT 10000","SELECT AVG(CAST(Year AS DOUBLE)) FROM mytable LIMIT 10000"],"pql":"SELECT sum(DepartureDelayGroups), min(FlightNum), MAX(AirlineID), AVG(Year) FROM mytable"}
{"hsqls":["SELECT AVG(CAST(ActualElapsedTime AS DOUBLE)) FROM mytable LIMIT 10000","SELECT COUNT(DivDistance) FROM mytable LIMIT 10000","SELECT MIN(DepDelay) FROM mytable LIMIT 10000"],"pql":"SELECT AVG(ActualElapsedTime), COUNT(DivDistance), MIN(DepDelay) FROM mytable"}
diff --git a/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_500.sql b/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_500.sql
index ea02c95..425dc0e 100644
--- a/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_500.sql
+++ b/pinot-integration-tests/src/test/resources/On_Time_On_Time_Performance_2014_100k_subset.test_queries_500.sql
@@ -1,3 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
# Aggregation
{"sql":"SELECT AVG(ActualElapsedTime), COUNT(DivDistance), MIN(DepDelay) FROM mytable"}
{"sql":"SELECT AVG(ArrDelayMinutes) FROM mytable"}
diff --git a/pinot-integration-tests/src/test/resources/compat-tests/kafka/topic1.cfg b/pinot-integration-tests/src/test/resources/compat-tests/kafka/topic1.cfg
index e4eb246..5b15fe1 100644
--- a/pinot-integration-tests/src/test/resources/compat-tests/kafka/topic1.cfg
+++ b/pinot-integration-tests/src/test/resources/compat-tests/kafka/topic1.cfg
@@ -1,3 +1,22 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
topicName = topic1
numPartitions = 1
diff --git a/pinot-integration-tests/src/test/resources/index.html b/pinot-integration-tests/src/test/resources/index.html
index 1583599..43ba8ae 100644
--- a/pinot-integration-tests/src/test/resources/index.html
+++ b/pinot-integration-tests/src/test/resources/index.html
@@ -1,3 +1,23 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
<!DOCTYPE html>
<!DOCTYPE html>
<html lang="en">
diff --git a/pinot-plugins/pinot-file-system/pinot-adls/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker b/pinot-plugins/pinot-file-system/pinot-adls/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker
index 25c823c..7939fec 100644
--- a/pinot-plugins/pinot-file-system/pinot-adls/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker
+++ b/pinot-plugins/pinot-file-system/pinot-adls/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker
@@ -1,3 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
# This class defines Mockito plugins to be used during testing
# mock-maker inline enables mocking of 'final' classes in Mockito v2.
diff --git a/pinot-plugins/pinot-file-system/pinot-s3/src/test/resources/log4j2.xml b/pinot-plugins/pinot-file-system/pinot-s3/src/test/resources/log4j2.xml
index cc7b1f9..253b9ad 100644
--- a/pinot-plugins/pinot-file-system/pinot-s3/src/test/resources/log4j2.xml
+++ b/pinot-plugins/pinot-file-system/pinot-s3/src/test/resources/log4j2.xml
@@ -1,4 +1,24 @@
<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
<configuration>
<root level="warn"/>
diff --git a/pinot-plugins/pinot-input-format/pinot-protobuf/src/test/resources/complex_types.proto b/pinot-plugins/pinot-input-format/pinot-protobuf/src/test/resources/complex_types.proto
index 4fb6c27..0494732 100644
--- a/pinot-plugins/pinot-input-format/pinot-protobuf/src/test/resources/complex_types.proto
+++ b/pinot-plugins/pinot-input-format/pinot-protobuf/src/test/resources/complex_types.proto
@@ -1,3 +1,22 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
syntax = "proto3";
package org.apache.pinot.plugin.inputformat.protobuf;
diff --git a/pinot-plugins/pinot-input-format/pinot-protobuf/src/test/resources/sample.proto b/pinot-plugins/pinot-input-format/pinot-protobuf/src/test/resources/sample.proto
index e27377b..20dafb6 100644
--- a/pinot-plugins/pinot-input-format/pinot-protobuf/src/test/resources/sample.proto
+++ b/pinot-plugins/pinot-input-format/pinot-protobuf/src/test/resources/sample.proto
@@ -1,3 +1,22 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
syntax = "proto3";
package org.apache.pinot.plugin.inputformat.protobuf;
diff --git a/pinot-server/src/test/resources/conf/pinot-broker.properties b/pinot-server/src/test/resources/conf/pinot-broker.properties
index 2efa525..25fba3f 100644
--- a/pinot-server/src/test/resources/conf/pinot-broker.properties
+++ b/pinot-server/src/test/resources/conf/pinot-broker.properties
@@ -1,3 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
pinot.broker.routing.tableName=testTable
# Number of nodes constituting a replica-set. A replica-set has 0-(n-1) nodes where n is the below config value.
pinot.broker.routing.testTable.numNodesPerReplica=1
diff --git a/pinot-spi/src/test/resources/pinot-configuration-1.properties b/pinot-spi/src/test/resources/pinot-configuration-1.properties
index ab3547a..0f8ff51 100644
--- a/pinot-spi/src/test/resources/pinot-configuration-1.properties
+++ b/pinot-spi/src/test/resources/pinot-configuration-1.properties
@@ -1,3 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
controller.host=config-path-1-controller-host
controller.port=config-path-1-controller-port
controller.cluster-name=config-path-1-cluster-name
diff --git a/pinot-spi/src/test/resources/pinot-configuration-2.properties b/pinot-spi/src/test/resources/pinot-configuration-2.properties
index f06acc0..27d2e90 100644
--- a/pinot-spi/src/test/resources/pinot-configuration-2.properties
+++ b/pinot-spi/src/test/resources/pinot-configuration-2.properties
@@ -1,3 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
controller.host=config-path-2-controller-host
controller.port=config-path-2-controller-port
controller.cluster-name=config-path-2-cluster-name
diff --git a/pinot-spi/src/test/resources/pinot-configuration-3.properties b/pinot-spi/src/test/resources/pinot-configuration-3.properties
index 248e120..3045e9f 100644
--- a/pinot-spi/src/test/resources/pinot-configuration-3.properties
+++ b/pinot-spi/src/test/resources/pinot-configuration-3.properties
@@ -1,2 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
controller.config-paths-multi-value-test-1=config-path-3-config-paths-multi-value-test-1
-controller.config-paths-multi-value-test-2=config-path-3-config-paths-multi-value-test-2
\ No newline at end of file
+controller.config-paths-multi-value-test-2=config-path-3-config-paths-multi-value-test-2
diff --git a/pinot-tools/src/main/resources/conf/sample_filesystem_benchmark.conf b/pinot-tools/src/main/resources/conf/sample_filesystem_benchmark.conf
index 3583707..d9f573b 100644
--- a/pinot-tools/src/main/resources/conf/sample_filesystem_benchmark.conf
+++ b/pinot-tools/src/main/resources/conf/sample_filesystem_benchmark.conf
@@ -1,3 +1,22 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
// Azure Datalake Gen2
class.abfss=org.apache.pinot.plugin.filesystem.AzureGen2PinotFS
abfss.accountName=<account_name>
diff --git a/pinot-tools/src/main/resources/conf/sample_query_comparison.conf b/pinot-tools/src/main/resources/conf/sample_query_comparison.conf
index 142f8fa..ee93740 100644
--- a/pinot-tools/src/main/resources/conf/sample_query_comparison.conf
+++ b/pinot-tools/src/main/resources/conf/sample_query_comparison.conf
@@ -1,3 +1,22 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
start.zookeeper = true
zookeeper.address = localhost:2181
cluster.name = QueryComparisonCluster
diff --git a/pom.xml b/pom.xml
index e30e414..8f3d418 100644
--- a/pom.xml
+++ b/pom.xml
@@ -185,17 +185,6 @@
</properties>
</profile>
<profile>
- <id>travis</id>
- <activation>
- <activeByDefault>false</activeByDefault>
- </activation>
- <properties>
- <argLine>-Xms4g -Xmx4g -XX:MaxPermSize=512m -XX:MaxDirectMemorySize=10g
- -Dlog4j2.configurationFile=log4j2-fatal-only.xml
- </argLine>
- </properties>
- </profile>
- <profile>
<id>github-actions</id>
<activation>
<activeByDefault>false</activeByDefault>
@@ -1392,7 +1381,7 @@
<plugin>
<groupId>com.mycila</groupId>
<artifactId>license-maven-plugin</artifactId>
- <version>2.8</version>
+ <version>4.0</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
@@ -1535,52 +1524,58 @@
<plugin>
<groupId>com.mycila</groupId>
<artifactId>license-maven-plugin</artifactId>
- <version>2.8</version>
+ <version>4.0</version>
<configuration>
<header>${pinot.root}/HEADER</header>
<headerDefinitions>
- <headerDefinition>${pinot.root}/rstheaderdefinition.xml</headerDefinition>
+ <headerDefinition>${pinot.root}/headerdefinition.xml</headerDefinition>
</headerDefinitions>
<excludes>
- <!-- Exclude build files for docs -->
- <exclude>docs/_build/**</exclude>
- <exlude>docs/Makefile</exlude>
-
<!-- Text and log files -->
<exclude>**/*.txt</exclude>
<exclude>**/*.log</exclude>
+ <exclude>**/*.list</exclude>
+ <exclude>**/*.out</exclude>
+ <exclude>**/*.generated</exclude>
+ <exclude>**/*.json</exclude>
+ <exclude>**/*.schema</exclude>
- <exclude>**/*.pql</exclude>
- <exclude>**/*.config</exclude>
- <exclude>**/*.csv</exclude>
- <exclude>**/*.avro</exclude>
- <exclude>**/*.conf</exclude>
- <exclude>**/appAssemblerScriptTemplate</exclude>
+ <!-- JDBC Drivers -->
+ <exclude>**/java.sql.*</exclude>
- <exclude>**/zone-index.properties</exclude>
+ <!-- Spark datasource register -->
+ <exclude>**/org.apache.spark.sql.sources.DataSourceRegister</exclude>
- <!-- Resource directories under test -->
- <exclude>**/src/test/resources/**</exclude>
+ <!-- Binary files -->
+ <exclude>**/*.avro</exclude>
+ <exclude>**/*.avsc</exclude>
+ <exclude>**/*.csv</exclude>
+ <exclude>**/*.desc</exclude>
+ <exclude>**/*.parquet</exclude>
+ <exclude>**/*.gz</exclude>
+ <exclude>**/*.orc</exclude>
+ <exclude>**/*.dict</exclude>
+ <exclude>**/*.raw</exclude>
+ <exclude>**/*.mapping</exclude>
+ <exclude>**/*.ser</exclude>
+ <exclude>**/*.v1</exclude>
+ <exclude>**/*.v2</exclude>
<!-- Auto-generated target directories -->
<exclude>target/**</exclude>
<exclude>pinot-*/target/**</exclude>
- <!-- Top level directories -->
- <exclude>thirdeye/**</exclude>
+ <!-- Pinot Website (not part of the distribution) -->
<exclude>website/**</exclude>
- <exclude>docker/**</exclude>
- <exclude>kubernetes/**</exclude>
- <!-- Docker image configs -->
- <exclude>docker/images/pinot-thirdeye/**</exclude>
+ <!-- Docker and Kubernetes (not part of the distribution) -->
+ <exclude>kubernetes/**</exclude>
+ <exclude>docker/**</exclude>
- <!-- Note: this will exclude all single character directories under the root directory.
- We use this to exclude the temporary files generated during build -->
- <exclude>?/**</exclude>
+ <!-- Pinot-Druid Benchmark (not part of the distribution) -->
+ <exclude>contrib/**</exclude>
<!-- Top level files -->
- <exclude>.codecov*</exclude>
<exclude>HEADER</exclude>
<exclude>LICENSE*</exclude>
<exclude>NOTICE*</exclude>
@@ -1589,26 +1584,29 @@
<!-- js, css files that are exact copies of the third-party works. In this case, the original header has to
be kept. Please refer to 'Treatment of Third-party works' in https://www.apache.org/legal/src-headers.html
-->
- <exclude>**/codemirror/**</exclude>
- <exclude>**/codemirror*</exclude>
- <exclude>**/foundation*</exclude>
- <exclude>**/angular*</exclude>
- <exclude>**/underscore*</exclude>
- <exclude>**/jquery*</exclude>
- <exclude>**/normalize*</exclude>
- <exclude>**/handlebars*</exclude>
- <exclude>**/beautify*</exclude>
<exclude>**/node_modules/**</exclude>
<exclude>**/dist/**</exclude>
- <exclude>**/.factorypath</exclude>
- <exclude>**/app/**</exclude>
- <exclude>**/src/main/resources/dist/**</exclude>
- <exclude>**/src/main/resources/.*</exclude>
+ <exclude>**/src/main/resources/*.*rc</exclude>
</excludes>
<mapping>
+ <java>JAVADOC_STYLE</java>
+ <scala>JAVADOC_STYLE</scala>
<thrift>JAVADOC_STYLE</thrift>
<g4>JAVADOC_STYLE</g4>
+ <ts>JAVADOC_STYLE</ts>
+ <tsx>JAVADOC_STYLE</tsx>
+ <config>SCRIPT_STYLE</config>
+ <queries>SCRIPT_STYLE</queries>
+ <results>SCRIPT_STYLE</results>
+ <MockMaker>SCRIPT_STYLE</MockMaker>
+ <appAssemblerScriptTemplate>SCRIPT_STYLE</appAssemblerScriptTemplate>
+ <pql>SCRIPT_STYLE</pql>
+ <sql>SCRIPT_STYLE</sql>
+ <conf>DOUBLESLASH_STYLE</conf>
+ <cfg>DOUBLESLASH_STYLE</cfg>
+ <proto>DOUBLESLASH_STYLE</proto>
<rst>RST_STYLE</rst>
+ <readme>RST_STYLE</readme>
</mapping>
</configuration>
<executions>
@@ -1633,12 +1631,6 @@
</executions>
<configuration>
<excludes>
- <!-- Sphynx auto generated makefile -->
- <exclude>docs/Makefile</exclude>
-
- <!-- Exclude build files for docs -->
- <exclude>docs/_build/**</exclude>
-
<!-- Exclude license copies -->
<exclude>licenses/**</exclude>
<exclude>licenses-binary/**</exclude>
@@ -1646,50 +1638,48 @@
<!-- Exclude build targets -->
<exclude>**/target/**</exclude>
+ <!-- Text and log files -->
+ <exclude>**/*.txt</exclude>
+ <exclude>**/*.log</exclude>
+ <exclude>**/*.list</exclude>
+ <exclude>**/*.out</exclude>
+ <exclude>**/*.generated</exclude>
+ <exclude>**/*.json</exclude>
+ <exclude>**/*.schema</exclude>
- <!-- js, css files that are exact copies of the third-party works. In this case, the original header has to
- be kept. Please refer to 'Treatment of Third-party works' in https://www.apache.org/legal/src-headers.html
- -->
- <exclude>**/codemirror/**</exclude>
- <exclude>**/codemirror*</exclude>
- <exclude>**/foundation*</exclude>
- <exclude>**/angular*</exclude>
- <exclude>**/underscore*</exclude>
- <exclude>**/jquery*</exclude>
- <exclude>**/normalize*</exclude>
- <exclude>**/handlebars*</exclude>
- <exclude>**/beautify*</exclude>
- <exclude>**/src/main/resources/.*</exclude>
- <exclude>**/src/main/resources/dist/**</exclude>
- <exclude>**/src/main/resources/node_modules/**</exclude>
+ <!-- JDBC Drivers -->
+ <exclude>**/java.sql.*</exclude>
+
+ <!-- Spark datasource register -->
+ <exclude>**/org.apache.spark.sql.sources.DataSourceRegister</exclude>
<!-- Binary files -->
- <exclude>**/*.json</exclude>
- <exclude>**/*.pql</exclude>
- <exclude>**/*.log</exclude>
- <exclude>**/*.csv</exclude>
<exclude>**/*.avro</exclude>
- <exclude>**/*.conf</exclude>
- <exclude>**/*.config</exclude>
- <exclude>**/*.yml</exclude>
- <exclude>**/*.yaml</exclude>
- <exclude>**/*.properties</exclude>
+ <exclude>**/*.avsc</exclude>
+ <exclude>**/*.csv</exclude>
+ <exclude>**/*.desc</exclude>
+ <exclude>**/*.parquet</exclude>
+ <exclude>**/*.gz</exclude>
+ <exclude>**/*.orc</exclude>
+ <exclude>**/*.dict</exclude>
+ <exclude>**/*.raw</exclude>
+ <exclude>**/*.mapping</exclude>
+ <exclude>**/*.ser</exclude>
+ <exclude>**/*.v1</exclude>
+ <exclude>**/*.v2</exclude>
+ <exclude>**/appAssemblerScriptTemplate</exclude>
+
+ <!-- js, css files that are exact copies of the third-party works. In this case, the original header has to
+ be kept. Please refer to 'Treatment of Third-party works' in https://www.apache.org/legal/src-headers.html
+ -->
+ <exclude>**/node_modules/**</exclude>
+ <exclude>**/dist/**</exclude>
+ <exclude>**/*.*rc</exclude>
+ <exclude>**/src/main/resources/*.*rc</exclude>
<!-- IDE files -->
<exclude>**/*.iml</exclude>
- <!-- Top level -->
- <exclude>.codecov*</exclude>
-
- <!-- Test Resources -->
- <exclude>**/test/resources/**</exclude>
-
- <!-- JDBC Drivers -->
- <exclude>**/main/resources/java.sql.*</exclude>
-
- <!-- Spark datasource register -->
- <exclude>**/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister</exclude>
-
<!-- Pinot Website (not part of the distribution) -->
<exclude>website/**</exclude>
@@ -1700,10 +1690,6 @@
<!-- Pinot-Druid Benchmark (not part of the distribution) -->
<exclude>contrib/**</exclude>
- <!-- Third eye (not part of the distribution) -->
- <exclude>thirdeye/**</exclude>
- <exclude>docker/images/pinot-thirdeye/**</exclude>
-
<!-- Github template files -->
<exclude>.github/*.md</exclude>
</excludes>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 02/07: Fix log message in ControllerPeriodicTask
(#6709)
Posted by xi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch release-0.7.1-rc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 7d6b2474e6b5dc07181a30d43f1a978a8be5ee5f
Author: Jiapeng Tao <ji...@linkedin.com>
AuthorDate: Wed Mar 24 10:49:08 2021 -0700
Fix log message in ControllerPeriodicTask (#6709)
---
.../controller/helix/core/periodictask/ControllerPeriodicTask.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/periodictask/ControllerPeriodicTask.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/periodictask/ControllerPeriodicTask.java
index 9139f1d..5aff79c 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/periodictask/ControllerPeriodicTask.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/periodictask/ControllerPeriodicTask.java
@@ -94,7 +94,7 @@ public abstract class ControllerPeriodicTask<C> extends BasePeriodicTask {
try {
processTable(tableNameWithType, context);
} catch (Exception e) {
- LOGGER.error("Caught exception while processing table: {} in task: {}", tableNamesWithType, _taskName, e);
+ LOGGER.error("Caught exception while processing table: {} in task: {}", tableNameWithType, _taskName, e);
}
numTablesProcessed++;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 07/07: Run tests when pushing new commit (#6722)
Posted by xi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch release-0.7.1-rc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit a54f5618f5d7b4c191b42c66938884fdd841658f
Author: Xiaotian (Jackie) Jiang <17...@users.noreply.github.com>
AuthorDate: Sun Mar 28 23:11:38 2021 -0700
Run tests when pushing new commit (#6722)
Also remove the unused codecov script
---
.codecov_bash | 1285 -------------------------------------
.github/workflows/pinot_tests.yml | 12 +-
2 files changed, 11 insertions(+), 1286 deletions(-)
diff --git a/.codecov_bash b/.codecov_bash
deleted file mode 100644
index 523fb30..0000000
--- a/.codecov_bash
+++ /dev/null
@@ -1,1285 +0,0 @@
-#!/usr/bin/env bash
-
-set -e +o pipefail
-
-VERSION="1b98dd4"
-
-url="https://codecov.io"
-url_o=""
-env="$CODECOV_ENV"
-pr_o=""
-pr=""
-job=""
-build_url=""
-service=""
-build_o=""
-token=""
-commit_o=""
-search_in=""
-search_in_o=""
-tag_o=""
-tag=""
-flags=""
-exit_with=0
-branch_o=""
-slug_o=""
-dump="0"
-clean="0"
-branch=""
-commit=""
-include_cov=""
-exclude_cov=""
-ddp="$(echo ~)/Library/Developer/Xcode/DerivedData"
-xp=""
-files=""
-cacert="$CODECOV_CA_BUNDLE"
-gcov_ignore=""
-gcov_include=""
-ft_gcov="1"
-ft_coveragepy="1"
-ft_fix="1"
-ft_search="1"
-ft_s3="1"
-ft_xcode="1"
-ft_network="1"
-_git_root=$(git rev-parse --show-toplevel 2>/dev/null || hg root 2>/dev/null || echo $PWD)
-git_root="$_git_root"
-remote_addr=""
-if [ "$git_root" = "$PWD" ];
-then
- git_root="."
-fi
-
-proj_root="$git_root"
-gcov_exe="gcov"
-gcov_arg=""
-
-b="\033[0;36m"
-g="\033[0;32m"
-r="\033[0;31m"
-e="\033[0;90m"
-x="\033[0m"
-
-show_help() {
-cat << EOF
-
- Codecov Bash $VERSION
-
- Global report uploading tool for Codecov
- Contribute at https://github.com/codecov/codecov-bash
-
-
- -h Display this help and exit
- -f FILE Target file(s) to upload
-
- -f "path/to/file" only upload this file
- skips searching unless provided patterns below
- -f "!*.bar" ignore all files at pattern *.bar
- -f "*.foo" include all files at pattern *.foo
-
- -s DIR Directory to search for coverage reports.
- Already searches project root and artifact folders.
- -t TOKEN Set the private repository token
- (option) set environment variable CODECOV_TOKEN=:uuid
-
- -t @/path/to/token_file
- -t uuid
-
- -e ENV Specify environment variables to be included with this build
- Also accepting environment variables: CODECOV_ENV=VAR,VAR2
-
- -e VAR,VAR2
-
- -X feature Toggle functionalities
-
- -X gcov Disable gcov
- -X coveragepy Disable python coverage
- -X fix Disable report fixing
- -X search Disable searching for reports
- -X xcode Disable xcode processing
- -X network Disable uploading the file network
-
- -R root dir Used when not in git/hg project to identify project root directory
- -F flag Flag the upload to group coverage metrics
-
- -F unittests This upload is only unittests
- -F integration This upload is only integration tests
- -F ui,chrome This uplaod is Chrome - UI tests
-
- -c Move discovered coverage reports to the trash
- -Z Exit with 1 if not successful. Default will Exit with 0
-
- -- xcode --
- -D Custom Derived Data Path for Coverage.profdata and gcov processing
- Default '~/Library/Developer/Xcode/DerivedData'
- -J Specify packages to build coverage.
- This can significantly reduces time to build coverage reports.
-
- -J 'MyAppName' Will match "MyAppName" and "MyAppNameTests"
- -J '^ExampleApp$' Will match only "ExampleApp" not "ExampleAppTests"
-
- -- gcov --
- -g GLOB Paths to ignore during gcov gathering
- -G GLOB Paths to include during gcov gathering
- -p dir Project root directory
- Also used when preparing gcov
- -x gcovexe gcov executable to run. Defaults to 'gcov'
- -a gcovargs extra arguments to pass to gcov
-
- -- Override CI Environment Variables --
- These variables are automatically detected by popular CI providers
-
- -B branch Specify the branch name
- -C sha Specify the commit sha
- -P pr Specify the pull request number
- -b build Specify the build number
- -T tag Specify the git tag
-
- -- Enterprise --
- -u URL Set the target url for Enterprise customers
- Not required when retrieving the bash uploader from your CCE
- (option) Set environment variable CODECOV_URL=https://my-hosted-codecov.com
- -r SLUG owner/repo slug used instead of the private repo token in Enterprise
- (option) set environment variable CODECOV_SLUG=:owner/:repo
- (option) set in your codecov.yml "codecov.slug"
- -S PATH File path to your cacert.pem file used to verify ssl with Codecov Enterprise (optional)
- (option) Set environment variable: CODECOV_CA_BUNDLE="/path/to/ca.pem"
-
- -- Debugging --
- -d Dont upload and dump to stdin
- -K Remove color from the output
- -v Verbose mode
-
-EOF
-}
-
-
-say() {
- echo -e "$1"
-}
-
-
-urlencode() {
- echo "$1" | curl -Gso /dev/null -w %{url_effective} --data-urlencode @- "" | cut -c 3- | sed -e 's/%0A//'
-}
-
-
-swiftcov() {
- _dir=$(dirname "$1")
- for _type in app framework xctest
- do
- find "$_dir" -name "*.$_type" | while read f
- do
- _proj=${f##*/}
- _proj=${_proj%."$_type"}
- if [ "$2" = "" ] || [ "$(echo "$_proj" | grep -i "$2")" != "" ];
- then
- say " $g+$x Building reports for $_proj $_type"
- dest=$([ -f "$f/$_proj" ] && echo "$f/$_proj" || echo "$f/Contents/MacOS/$_proj")
- _proj_name=$(echo "$_proj" | sed -e 's/[[:space:]]//g')
- xcrun llvm-cov show -instr-profile "$1" "$dest" > "$_proj_name.$_type.coverage.txt" \
- || say " ${r}x>${x} llvm-cov failed to produce results for $dest"
- fi
- done
- done
-}
-
-
-# Credits to: https://gist.github.com/pkuczynski/8665367
-parse_yaml() {
- local prefix=$2
- local s='[[:space:]]*' w='[a-zA-Z0-9_]*' fs=$(echo @|tr @ '\034')
- sed -ne "s|^\($s\)\($w\)$s:$s\"\(.*\)\"$s\$|\1$fs\2$fs\3|p" \
- -e "s|^\($s\)\($w\)$s:$s\(.*\)$s\$|\1$fs\2$fs\3|p" $1 |
- awk -F$fs '{
- indent = length($1)/2;
- vname[indent] = $2;
- for (i in vname) {if (i > indent) {delete vname[i]}}
- if (length($3) > 0) {
- vn=""; if (indent > 0) {vn=(vn)(vname[0])("_")}
- printf("%s%s%s=\"%s\"\n", "'$prefix'",vn, $2, $3);
- }
- }'
-}
-
-
-if [ $# != 0 ];
-then
- while getopts "a:b:B:cC:dD:e:f:F:g:G:hJ:Kp:P:r:R:s:S:t:T:u:vx:X:Z" o
- do
- case "$o" in
- "a")
- gcov_arg=$OPTARG
- ;;
- "b")
- build_o="$OPTARG"
- ;;
- "B")
- branch_o="$OPTARG"
- ;;
- "c")
- clean="1"
- ;;
- "C")
- commit_o="$OPTARG"
- ;;
- "d")
- dump="1"
- ;;
- "D")
- ddp="$OPTARG"
- ;;
- "e")
- env="$env,$OPTARG"
- ;;
- "f")
- if [ "${OPTARG::1}" = "!" ];
- then
- exclude_cov="$exclude_cov -not -path '${OPTARG:1}'"
-
- elif [[ "$OPTARG" = *"*"* ]];
- then
- include_cov="$include_cov -or -name '$OPTARG'"
-
- else
- ft_search=0
- if [ "$files" = "" ];
- then
- files="$OPTARG"
- else
- files="$files
-$OPTARG"
- fi
- fi
- ;;
- "F")
- if [ "$flags" = "" ];
- then
- flags="$OPTARG"
- else
- flags="$flags,$OPTARG"
- fi
- ;;
- "g")
- gcov_ignore="$gcov_ignore -not -path '$OPTARG'"
- ;;
- "G")
- gcov_include="$gcov_include -path '$OPTARG'"
- ;;
- "h")
- show_help
- exit 0;
- ;;
- "J")
- if [ "$xp" = "" ];
- then
- xp="$OPTARG"
- else
- xp="$xp\|$OPTARG"
- fi
- ;;
- "K")
- b=""
- g=""
- r=""
- e=""
- x=""
- ;;
- "p")
- proj_root="$OPTARG"
- ;;
- "P")
- pr_o="$OPTARG"
- ;;
- "r")
- slug_o="$OPTARG"
- ;;
- "R")
- git_root="$OPTARG"
- ;;
- "s")
- if [ "$search_in_o" = "" ];
- then
- search_in_o="$OPTARG"
- else
- search_in_o="$search_in_o $OPTARG"
- fi
- ;;
- "S")
- cacert="--cacert \"$OPTARG\""
- ;;
- "t")
- if [ "${OPTARG::1}" = "@" ];
- then
- token=$(cat "${OPTARG:1}" | tr -d ' \n')
- else
- token="$OPTARG"
- fi
- ;;
- "T")
- tag_o="$OPTARG"
- ;;
- "u")
- url_o=$(echo "$OPTARG" | sed -e 's/\/$//')
- ;;
- "v")
- set -x
- ;;
- "x")
- gcov_exe=$OPTARG
- ;;
- "X")
- if [ "$OPTARG" = "gcov" ];
- then
- ft_gcov="0"
- elif [ "$OPTARG" = "coveragepy" ] || [ "$OPTARG" = "py" ];
- then
- ft_coveragepy="0"
- elif [ "$OPTARG" = "fix" ] || [ "$OPTARG" = "fixes" ];
- then
- ft_fix="0"
- elif [ "$OPTARG" = "xcode" ];
- then
- ft_xcode="0"
- elif [ "$OPTARG" = "search" ];
- then
- ft_search="0"
- elif [ "$OPTARG" = "network" ];
- then
- ft_network="0"
- fi
- ;;
- "Z")
- exit_with=1
- ;;
- esac
- done
-fi
-
-say "
- _____ _
- / ____| | |
-| | ___ __| | ___ ___ _____ __
-| | / _ \\ / _\` |/ _ \\/ __/ _ \\ \\ / /
-| |___| (_) | (_| | __/ (_| (_) \\ V /
- \\_____\\___/ \\__,_|\\___|\\___\\___/ \\_/
- Bash-$VERSION
-
-"
-
-search_in="$proj_root"
-
-if [ "$JENKINS_URL" != "" ];
-then
- say "$e==>$x Jenkins CI detected."
- # https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project
- # https://wiki.jenkins-ci.org/display/JENKINS/GitHub+pull+request+builder+plugin#GitHubpullrequestbuilderplugin-EnvironmentVariables
- service="jenkins"
- branch=$([ ! -z "$ghprbSourceBranch" ] && echo "$ghprbSourceBranch" || echo "$GIT_BRANCH")
- commit=$([ ! -z "$ghprbActualCommit" ] && echo "$ghprbActualCommit" || echo "$GIT_COMMIT")
- build="$BUILD_NUMBER"
- pr="$ghprbPullId"
- build_url=$(urlencode "$BUILD_URL")
-
-elif [ "$CI" = "true" ] && [ "$TRAVIS" = "true" ] && [ "$SHIPPABLE" != "true" ];
-then
- say "$e==>$x Travis CI detected."
- # http://docs.travis-ci.com/user/ci-environment/#Environment-variables
- service="travis"
- branch="$TRAVIS_BRANCH"
- commit="$TRAVIS_COMMIT"
- build="$TRAVIS_JOB_NUMBER"
- pr="$TRAVIS_PULL_REQUEST"
- job="$TRAVIS_JOB_ID"
- slug="$TRAVIS_REPO_SLUG"
- tag="$TRAVIS_TAG"
- env="$env,TRAVIS_OS_NAME"
-
- language=$(printenv | grep "TRAVIS_.*_VERSION" | head -1)
- if [ "$language" != "" ];
- then
- env="$env,${language%=*}"
- fi
-
-elif [ "$CI" = "true" ] && [ "$CI_NAME" = "codeship" ];
-then
- say "$e==>$x Codeship CI detected."
- # https://www.codeship.io/documentation/continuous-integration/set-environment-variables/
- service="codeship"
- branch="$CI_BRANCH"
- build="$CI_BUILD_NUMBER"
- build_url=$(urlencode "$CI_BUILD_URL")
- commit="$CI_COMMIT_ID"
-
-elif [ "$TEAMCITY_VERSION" != "" ];
-then
- say "$e==>$x TeamCity CI detected."
- # https://confluence.jetbrains.com/display/TCD8/Predefined+Build+Parameters
- # https://confluence.jetbrains.com/plugins/servlet/mobile#content/view/74847298
- if [ "$TEAMCITY_BUILD_BRANCH" = '' ];
- then
- echo " Teamcity does not automatically make build parameters available as environment variables."
- echo " Add the following environment parameters to the build configuration"
- echo " env.TEAMCITY_BUILD_BRANCH = %teamcity.build.branch%"
- echo " env.TEAMCITY_BUILD_ID = %teamcity.build.id%"
- echo " env.TEAMCITY_BUILD_URL = %teamcity.serverUrl%/viewLog.html?buildId=%teamcity.build.id%"
- echo " env.TEAMCITY_BUILD_COMMIT = %system.build.vcs.number%"
- echo " env.TEAMCITY_BUILD_REPOSITORY = %vcsroot.<YOUR TEAMCITY VCS NAME>.url%"
- fi
- service="teamcity"
- branch="$TEAMCITY_BUILD_BRANCH"
- build="$TEAMCITY_BUILD_ID"
- build_url=$(urlencode "$TEAMCITY_BUILD_URL")
- if [ "$TEAMCITY_BUILD_COMMIT" != "" ];
- then
- commit="$TEAMCITY_BUILD_COMMIT"
- else
- commit="$BUILD_VCS_NUMBER"
- fi
- remove_addr="$TEAMCITY_BUILD_REPOSITORY"
-
-elif [ "$CI" = "true" ] && [ "$CIRCLECI" = "true" ];
-then
- say "$e==>$x Circle CI detected."
- # https://circleci.com/docs/environment-variables
- service="circleci"
- branch="$CIRCLE_BRANCH"
- build="$CIRCLE_BUILD_NUM"
- job="$CIRCLE_NODE_INDEX"
- slug="$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME"
- pr="$CIRCLE_PR_NUMBER"
- commit="$CIRCLE_SHA1"
- search_in="$search_in $CIRCLE_ARTIFACTS $CIRCLE_TEST_REPORTS"
-
-elif [ "$BUDDYBUILD_BRANCH" != "" ];
-then
- say "$e==>$x buddybuild detected"
- # http://docs.buddybuild.com/v6/docs/custom-prebuild-and-postbuild-steps
- service="buddybuild"
- branch="$BUDDYBUILD_BRANCH"
- build="$BUDDYBUILD_BUILD_NUMBER"
- build_url="https://dashboard.buddybuild.com/public/apps/$BUDDYBUILD_APP_ID/build/$BUDDYBUILD_BUILD_ID"
- # BUDDYBUILD_TRIGGERED_BY
- if [ "$ddp" = "$(echo ~)/Library/Developer/Xcode/DerivedData" ];
- then
- ddp="/private/tmp/sandbox/${BUDDYBUILD_APP_ID}/bbtest"
- fi
-
-elif [ "${bamboo_planRepository_revision}" != "" ];
-then
- say "$e==>$x Bamboo detected"
- # https://confluence.atlassian.com/bamboo/bamboo-variables-289277087.html#Bamboovariables-Build-specificvariables
- service="bamboo"
- commit="${bamboo_planRepository_revision}"
- branch="${bamboo_planRepository_branch}"
- build="${bamboo_buildNumber}"
- build_url="${bamboo_buildResultsUrl}"
- remote_addr="${bamboo_planRepository_repositoryUrl}"
-
-elif [ "$CI" = "true" ] && [ "$BITRISE_IO" = "true" ];
-then
- # http://devcenter.bitrise.io/faq/available-environment-variables/
- say "$e==>$x Bitrise CI detected."
- service="bitrise"
- branch="$BITRISE_GIT_BRANCH"
- build="$BITRISE_BUILD_NUMBER"
- build_url=$(urlencode "$BITRISE_BUILD_URL")
- pr="$BITRISE_PULL_REQUEST"
- if [ "$GIT_CLONE_COMMIT_HASH" != "" ];
- then
- commit="$GIT_CLONE_COMMIT_HASH"
- fi
-
-elif [ "$CI" = "true" ] && [ "$SEMAPHORE" = "true" ];
-then
- say "$e==>$x Semaphore CI detected."
- # https://semaphoreapp.com/docs/available-environment-variables.html
- service="semaphore"
- branch="$BRANCH_NAME"
- build="$SEMAPHORE_BUILD_NUMBER.$SEMAPHORE_CURRENT_THREAD"
- pr="$PULL_REQUEST_NUMBER"
- slug="$SEMAPHORE_REPO_SLUG"
- commit="$REVISION"
- env="$env,SEMAPHORE_TRIGGER_SOURCE"
-
-elif [ "$CI" = "true" ] && [ "$BUILDKITE" = "true" ];
-then
- say "$e==>$x Buildkite CI detected."
- # https://buildkite.com/docs/guides/environment-variables
- service="buildkite"
- branch="$BUILDKITE_BRANCH"
- build="$BUILDKITE_BUILD_NUMBER.$BUILDKITE_JOB_ID"
- build_url=$(urlencode "$BUILDKITE_BUILD_URL")
- slug="$BUILDKITE_PROJECT_SLUG"
- commit="$BUILDKITE_COMMIT"
-
-elif [ "$CI" = "true" ] && [ "$DRONE" = "true" ];
-then
- say "$e==>$x Drone CI detected."
- # http://docs.drone.io/env.html
- # drone commits are not full shas
- service="drone.io"
- branch="$DRONE_BRANCH"
- build="$DRONE_BUILD_NUMBER"
- build_url=$(urlencode "${DRONE_BUILD_URL:-$CI_BUILD_URL}")
- pr="$DRONE_PULL_REQUEST"
- job="$DRONE_JOB_NUMBER"
- tag="$DRONE_TAG"
-
-elif [ "$CI" = "True" ] && [ "$APPVEYOR" = "True" ];
-then
- say "$e==>$x Appveyor CI detected."
- # http://www.appveyor.com/docs/environment-variables
- service="appveyor"
- branch="$APPVEYOR_REPO_BRANCH"
- build=$(urlencode "$APPVEYOR_JOB_ID")
- pr="$APPVEYOR_PULL_REQUEST_NUMBER"
- job="$APPVEYOR_ACCOUNT_NAME%2F$APPVEYOR_PROJECT_SLUG%2F$APPVEYOR_BUILD_VERSION"
- slug="$APPVEYOR_REPO_NAME"
- commit="$APPVEYOR_REPO_COMMIT"
-
-elif [ "$CI" = "true" ] && [ "$WERCKER_GIT_BRANCH" != "" ];
-then
- say "$e==>$x Wercker CI detected."
- # http://devcenter.wercker.com/articles/steps/variables.html
- service="wercker"
- branch="$WERCKER_GIT_BRANCH"
- build="$WERCKER_MAIN_PIPELINE_STARTED"
- slug="$WERCKER_GIT_OWNER/$WERCKER_GIT_REPOSITORY"
- commit="$WERCKER_GIT_COMMIT"
-
-elif [ "$CI" = "true" ] && [ "$MAGNUM" = "true" ];
-then
- say "$e==>$x Magnum CI detected."
- # https://magnum-ci.com/docs/environment
- service="magnum"
- branch="$CI_BRANCH"
- build="$CI_BUILD_NUMBER"
- commit="$CI_COMMIT"
-
-elif [ "$CI" = "true" ] && [ "$SNAP_CI" = "true" ];
-then
- say "$e==>$x Snap CI detected."
- # https://docs.snap-ci.com/environment-variables/
- service="snap"
- branch=$([ "$SNAP_BRANCH" != "" ] && echo "$SNAP_BRANCH" || echo "$SNAP_UPSTREAM_BRANCH")
- build="$SNAP_PIPELINE_COUNTER"
- job="$SNAP_STAGE_NAME"
- pr="$SNAP_PULL_REQUEST_NUMBER"
- commit=$([ "$SNAP_COMMIT" != "" ] && echo "$SNAP_COMMIT" || echo "$SNAP_UPSTREAM_COMMIT")
- env="$env,DISPLAY"
-
-elif [ "$SHIPPABLE" = "true" ];
-then
- say "$e==>$x Shippable CI detected."
- # http://docs.shippable.com/ci_configure/
- service="shippable"
- branch=$([ "$HEAD_BRANCH" != "" ] && echo "$HEAD_BRANCH" || echo "$BRANCH")
- build="$BUILD_NUMBER"
- build_url=$(urlencode "$BUILD_URL")
- pr="$PULL_REQUEST"
- slug="$REPO_FULL_NAME"
- commit="$COMMIT"
-
-elif [ "$TDDIUM" = "true" ];
-then
- say "Solano CI detected."
- # http://docs.solanolabs.com/Setup/tddium-set-environment-variables/
- service="solano"
- commit="$TDDIUM_CURRENT_COMMIT"
- branch="$TDDIUM_CURRENT_BRANCH"
- build="$TDDIUM_TID"
- pr="$TDDIUM_PR_ID"
-
-elif [ "$GREENHOUSE" = "true" ];
-then
- say "$e==>$x Greenhouse CI detected."
- # http://docs.greenhouseci.com/docs/environment-variables-files
- service="greenhouse"
- branch="$GREENHOUSE_BRANCH"
- build="$GREENHOUSE_BUILD_NUMBER"
- build_url=$(urlencode "$GREENHOUSE_BUILD_URL")
- pr="$GREENHOUSE_PULL_REQUEST"
- commit="$GREENHOUSE_COMMIT"
- search_in="$search_in $GREENHOUSE_EXPORT_DIR"
-
-elif [ "$GITLAB_CI" != "" ];
-then
- say "$e==>$x GitLab CI detected."
- # http://doc.gitlab.com/ce/ci/variables/README.html
- service="gitlab"
- branch="$CI_BUILD_REF_NAME"
- build="$CI_BUILD_ID"
- remove_addr="$CI_BUILD_REPO"
- commit="$CI_BUILD_REF"
-
-else
- say "${r}x>${x} No CI provider detected."
-
- commit="$VCS_COMMIT_ID"
- branch="$VCS_BRANCH_NAME"
- pr="$VCS_PULL_REQUEST"
- slug="$VCS_SLUG"
- build_url="$CI_BUILD_URL"
- build="$CI_BUILD_ID"
-
-fi
-
-say " ${e}project root:${x} $git_root"
-
-# find branch, commit, repo from git command
-if [ "$GIT_BRANCH" != "" ];
-then
- branch="$GIT_BRANCH"
-
-elif [ "$branch" = "" ];
-then
- branch=$(git rev-parse --abbrev-ref HEAD 2>/dev/null || hg branch 2>/dev/null || echo "")
- if [ "$branch" = "HEAD" ];
- then
- branch=""
- fi
-fi
-
-if [ "$commit_o" = "" ];
-then
- # merge commit -> actual commit
- mc=$(git log -1 --pretty=%B 2>/dev/null | tr -d '[[:space:]]' || true)
- if [[ "$mc" =~ ^Merge[[:space:]][a-z0-9]{40}[[:space:]]into[[:space:]][a-z0-9]{40}$ ]];
- then
- say " (DEPRECIATING SOON) Learn more at http://docs.codecov.io/docs/merge-commits"
- # Merge xxx into yyy
- say " Fixing merge commit sha"
- commit=$(echo "$mc" | cut -d' ' -f2)
- elif [ "$GIT_COMMIT" != "" ];
- then
- commit="$GIT_COMMIT"
- elif [ "$commit" = "" ];
- then
- commit=$(git log -1 --format="%H" 2>/dev/null || hg id -i --debug 2>/dev/null | tr -d '+' || echo "")
- fi
-else
- commit="$commit_o"
-fi
-
-if [ "$CODECOV_TOKEN" != "" ] && [ "$token" = "" ];
-then
- say "${e}-->${x} token set from env"
- token="$CODECOV_TOKEN"
-fi
-
-if [ "$CODECOV_URL" != "" ] && [ "$url_o" = "" ];
-then
- say "${e}-->${x} url set from env"
- url_o=$(echo "$CODECOV_URL" | sed -e 's/\/$//')
-fi
-
-if [ "$CODECOV_SLUG" != "" ];
-then
- say "${e}-->${x} slug set from env"
- slug_o="$CODECOV_SLUG"
-
-elif [ "$slug" = "" ];
-then
- if [ "$remote_addr" = "" ];
- then
- remote_addr=$(git config --get remote.origin.url || hg paths default || echo '')
- fi
- if [ "$remote_addr" != "" ];
- then
- if echo "$remote_addr" | grep -q "//"; then
- # https
- slug=$(echo "$remote_addr" | cut -d / -f 4,5 | sed -e 's/\.git$//')
- else
- # ssh
- slug=$(echo "$remote_addr" | cut -d : -f 2 | sed -e 's/\.git$//')
- fi
- fi
- if [ "$slug" = "/" ];
- then
- slug=""
- fi
-fi
-
-yaml=$(cd "$git_root" && \
- git ls-files "*codecov.yml" "*codecov.yaml" 2>/dev/null \
- || hg locate "*codecov.yml" "*codecov.yaml" 2>/dev/null \
- || echo '')
-yaml=$(echo "$yaml" | head -1)
-
-if [ "$yaml" != "" ];
-then
- say " ${e}Yaml found at:${x} $yaml"
- config=$(parse_yaml $yaml || echo '')
-
- # TODO validate the yaml here
-
- if [ "$(echo "$config" | grep 'codecov_token="')" != "" ] && [ "$token" = "" ];
- then
- say "${e}-->${x} token set from yaml"
- token="$(echo "$config" | grep 'codecov_token="' | sed -e 's/codecov_token="//' | sed -e 's/"\.*//')"
- fi
-
- if [ "$(echo "$config" | grep 'codecov_url="')" != "" ] && [ "$url_o" = "" ];
- then
- say "${e}-->${x} url set from yaml"
- url_o="$(echo "$config" | grep 'codecov_url="' | sed -e 's/codecov_url="//' | sed -e 's/"\.*//')"
- fi
-
- if [ "$(echo "$config" | grep 'codecov_slug="')" != "" ] && [ "$slug_o" = "" ];
- then
- say "${e}-->${x} slug set from yaml"
- slug_o="$(echo "$config" | grep 'codecov_slug="' | sed -e 's/codecov_slug="//' | sed -e 's/"\.*//')"
- fi
-else
- say " ${g}Yaml not found, that's ok! Learn more at${x} ${b}http://docs.codecov.io/docs/codecov-yaml${x}"
-
-fi
-
-if [ "$branch_o" != "" ];
-then
- branch=$(urlencode "$branch_o")
-else
- branch=$(urlencode "$branch")
-fi
-
-query="branch=$branch\
- &commit=$commit\
- &build=$([ "$build_o" = "" ] && echo "$build" || echo "$build_o")\
- &build_url=$build_url\
- &tag=$([ "$tag_o" = "" ] && echo "$tag" || echo "$tag_o")\
- &slug=$([ "$slug_o" = "" ] && urlencode "$slug" || urlencode "$slug_o")\
- &yaml=$(urlencode "$yaml")\
- &service=$service\
- &flags=$flags\
- &pr=$([ "$pr_o" = "" ] && echo "$pr" || echo "$pr_o")\
- &job=$job"
-
-if [ "$ft_search" = "1" ];
-then
- # detect bower comoponents location
- bower_components="bower_components"
- bower_rc=$(cd "$git_root" && cat .bowerrc 2>/dev/null || echo "")
- if [ "$bower_rc" != "" ];
- then
- bower_components=$(echo "$bower_rc" | tr -d '\n' | grep '"directory"' | cut -d'"' -f4 | sed -e 's/\/$//')
- if [ "$bower_components" = "" ];
- then
- bower_components="bower_components"
- fi
- fi
-
- # Swift Coverage
- if [ "$ft_xcode" = "1" ] && [ -d "$ddp" ];
- then
- say "${e}==>${x} Processing Xcode reports"
- say " DerivedData folder: $ddp"
- profdata_files=$(find "$ddp" -name '*.profdata' 2>/dev/null || echo '')
- if [ "$profdata_files" != "" ];
- then
- # xcode via profdata
- if [ "$xp" = "" ];
- then
- # xp=$(xcodebuild -showBuildSettings 2>/dev/null | grep -i "^\s*PRODUCT_NAME" | sed -e 's/.*= \(.*\)/\1/')
- # say " ${e}->${x} Speed up Xcode processing by adding ${e}-J '$xp'${x}"
- say " ${g}hint${x} Speed up Swift processing by using use -J 'AppName' (accepting regexp)"
- fi
- while read -r profdata;
- do
- if [ "$profdata" != "" ];
- then
- swiftcov "$profdata" "$xp"
- fi
- done <<< "$profdata_files"
- else
- say " ${e}->${x} No Swift coverage found"
- fi
-
- # Obj-C Gcov Coverage
- if [ "$ft_gcov" = "1" ];
- then
- say " ${e}->${x} Running $gcov_exe for Obj-C"
- bash -c "find $ddp -type f -name '*.gcda' $gcov_include $gcov_ignore -exec $gcov_exe -pbcu $gcov_arg {} +" || true
- fi
- fi
-
- # Gcov Coverage
- if [ "$ft_gcov" = "1" ];
- then
- say "${e}==>${x} Running gcov in $proj_root ${e}(disable via -X gcov)${x}"
- bash -c "find $proj_root -type f -name '*.gcno' $gcov_include $gcov_ignore -exec $gcov_exe -pb $gcov_arg {} +" || true
- else
- say "${e}==>${x} gcov disabled"
- fi
-
- # Python Coverage
- if [ "$ft_coveragepy" = "1" ];
- then
- if [ ! -f coverage.xml ];
- then
- if which coverage >/dev/null 2>&1;
- then
- say "${e}==>${x} Python coveragepy exists ${e}disable via -X coveragepy${x}"
-
- dotcoverage=$(find "$git_root" -name '.coverage' -or -name '.coverage.*' | head -1 || echo '')
- if [ "$dotcoverage" != "" ];
- then
- cd "$(dirname "$dotcoverage")"
- if [ ! -f .coverage ];
- then
- say " ${e}->${x} Running coverage combine"
- coverage combine
- fi
- say " ${e}->${x} Running coverage xml"
- if [ "$(coverage xml -i)" != "No data to report." ];
- then
- files="$files
-$PWD/coverage.xml"
- else
- say " ${r}No data to report.${x}"
- fi
- cd "$proj_root"
- else
- say " ${r}No .coverage file found.${x}"
- fi
- else
- say "${e}==>${x} Python coveragepy not found"
- fi
- fi
- else
- say "${e}==>${x} Python coveragepy disabled"
- fi
-
- if [ "$search_in_o" != "" ];
- then
- # location override
- search_in="$search_in_o"
- fi
-
- say "$e==>$x Searching for coverage reports in:"
- for _path in $search_in
- do
- say " ${g}+${x} $_path"
- done
-
- patterns="find $search_in -type f \( -name '*coverage*.*' \
- -or -name 'nosetests.xml' \
- -or -name 'jacoco*.xml' \
- -or -name 'clover.xml' \
- -or -name 'report.xml' \
- -or -name '*.codecov.*' \
- -or -name 'codecov.*' \
- -or -name 'cobertura.xml' \
- -or -name 'excoveralls.json' \
- -or -name 'luacov.report.out' \
- -or -name 'coverage-final.json' \
- -or -name 'naxsi.info' \
- -or -name 'lcov.info' \
- -or -name 'lcov.dat' \
- -or -name '*.lcov' \
- -or -name '*.clover' \
- -or -name 'cover.out' \
- -or -name 'gcov.info' \
- -or -name '*.gcov' \
- -or -name '*.lst' \
- $include_cov \) \
- $exclude_cov \
- -not -name '*.profdata' \
- -not -name 'coverage-summary.json' \
- -not -name 'phpunit-code-coverage.xml' \
- -not -name 'remapInstanbul.coverage*.json' \
- -not -name '*codecov.yml' \
- -not -name '*.serialized' \
- -not -name '.coverage*' \
- -not -name '.*coveragerc' \
- -not -name '*.sh' \
- -not -name '*.bat' \
- -not -name '*.ps1' \
- -not -name '*.rst' \
- -not -name '*.sbt' \
- -not -name '*.xcoverage.*' \
- -not -name '*.gz' \
- -not -name '*.conf' \
- -not -name '*.p12' \
- -not -name '*.csv' \
- -not -name '*.rsp' \
- -not -name '*.m4' \
- -not -name '*.am' \
- -not -name '*.template' \
- -not -name '*.cp' \
- -not -name '*.bw' \
- -not -name '*.crt' \
- -not -name '*.log' \
- -not -name '*.cmake' \
- -not -name '*.pth' \
- -not -name '*.in' \
- -not -name '*.jar*' \
- -not -name '*.pom*' \
- -not -name '*.png' \
- -not -name '*.jpg' \
- -not -name '*.sql' \
- -not -name '*.jpeg' \
- -not -name '*.svg' \
- -not -name '*.gif' \
- -not -name '*.csv' \
- -not -name '*.snapshot' \
- -not -name '*.mak*' \
- -not -name '*.bash' \
- -not -name '*.data' \
- -not -name '*.py' \
- -not -name '*.class' \
- -not -name '*.xcconfig' \
- -not -name '*.ec' \
- -not -name '*.coverage' \
- -not -name '*.pyc' \
- -not -name '*.cfg' \
- -not -name '*.egg' \
- -not -name '*.ru' \
- -not -name '*.css' \
- -not -name '*.less' \
- -not -name '*.pyo' \
- -not -name '*.whl' \
- -not -name '*.html' \
- -not -name '*.ftl' \
- -not -name '*.erb' \
- -not -name '*.rb' \
- -not -name '*.js' \
- -not -name '*.jade' \
- -not -name '*.db' \
- -not -name '*.md' \
- -not -name '*.cpp' \
- -not -name '*.gradle' \
- -not -name '*.tar.tz' \
- -not -name '*.scss' \
- -not -name 'include.lst' \
- -not -name 'fullLocaleNames.lst' \
- -not -name 'inputFiles.lst' \
- -not -name 'createdFiles.lst' \
- -not -name 'scoverage.measurements.*' \
- -not -name 'test_*_coverage.txt' \
- -not -name 'testrunner-coverage*' \
- -not -path '*/vendor/*' \
- -not -path '*/htmlcov/*' \
- -not -path '*/home/cainus/*' \
- -not -path '*/virtualenv/*' \
- -not -path '*/js/generated/coverage/*' \
- -not -path '*/.virtualenv/*' \
- -not -path '*/virtualenvs/*' \
- -not -path '*/.virtualenvs/*' \
- -not -path '*/.env/*' \
- -not -path '*/.envs/*' \
- -not -path '*/env/*' \
- -not -path '*/envs/*' \
- -not -path '*/.venv/*' \
- -not -path '*/.venvs/*' \
- -not -path '*/venv/*' \
- -not -path '*/venvs/*' \
- -not -path '*/.git/*' \
- -not -path '*/.hg/*' \
- -not -path '*/.tox/*' \
- -not -path '*/__pycache__/*' \
- -not -path '*/.egg-info*' \
- -not -path '*/$bower_components/*' \
- -not -path '*/node_modules/*' \
- -not -path '*/conftest_*.c.gcov' 2>/dev/null"
- files=$(eval "$patterns" || echo '')
-
-elif [ "$include_cov" != "" ];
-then
- files=$(eval "find $search_in -type f \( ${include_cov:5} \)$exclude_cov 2>/dev/null" || echo '')
-fi
-
-num_of_files=$(echo "$files" | wc -l | tr -d ' ')
-if [ "$num_of_files" != '' ] && [ "$files" != '' ];
-then
- say " ${e}->${x} Found $num_of_files reports"
-fi
-
-# no files found
-if [ "$files" = "" ];
-then
- say "${r}-->${x} No coverage report found."
- say " Please visit ${b}http://docs.codecov.io/docs/supported-languages${x}"
- exit ${exit_with};
-fi
-
-if [ "$ft_network" == "1" ];
-then
- say "${e}==>${x} Detecting git/mercurial file structure"
- network=$(cd "$git_root" && git ls-files 2>/dev/null || hg locate 2>/dev/null || echo "")
- if [ "$network" = "" ];
- then
- network=$(find "$git_root" -type f \
- -not -path '*/virtualenv/*' \
- -not -path '*/.virtualenv/*' \
- -not -path '*/virtualenvs/*' \
- -not -path '*/.virtualenvs/*' \
- -not -path '*.png' \
- -not -path '*.gif' \
- -not -path '*.jpg' \
- -not -path '*.jpeg' \
- -not -path '*.md' \
- -not -path '*/.env/*' \
- -not -path '*/.envs/*' \
- -not -path '*/env/*' \
- -not -path '*/envs/*' \
- -not -path '*/.venv/*' \
- -not -path '*/.venvs/*' \
- -not -path '*/venv/*' \
- -not -path '*/venvs/*' \
- -not -path '*/build/lib/*' \
- -not -path '*/.git/*' \
- -not -path '*/.egg-info/*' \
- -not -path '*/shunit2-2.1.6/*' \
- -not -path '*/vendor/*' \
- -not -path '*/js/generated/coverage/*' \
- -not -path '*/__pycache__/*' \
- -not -path '*/node_modules/*' \
- -not -path "*/$bower_components/*" 2>/dev/null || echo '')
- fi
-fi
-
-upload_file=`mktemp /tmp/codecov.XXXXXX`
-
-cleanup() {
- rm -f $upload_file
-}
-
-trap cleanup INT ABRT TERM
-
-if [ "$env" != "" ];
-then
- inc_env=""
- say "${e}==>${x} Appending build variables"
- for varname in $(echo "$env" | tr ',' ' ')
- do
- if [ "$varname" != "" ];
- then
- say " ${g}+${x} $varname"
- inc_env="${inc_env}${varname}=$(eval echo "\$${varname}")
-"
- fi
- done
-
-echo "$inc_env<<<<<< ENV" >> $upload_file
-fi
-
-if [ "$ft_network" == "1" ];
-then
- echo "$network
-<<<<<< network" >> $upload_file
-fi
-
-fr=0
-say "${e}==>${x} Reading reports"
-while IFS='' read -r file;
-do
- # read the coverage file
- if [ "$(echo "$file" | tr -d ' ')" != '' ];
- then
- if [ -f "$file" ];
- then
- report_len=$(wc -c < "$file")
- if [ "$report_len" -ne 0 ];
- then
- say " ${g}+${x} $file ${e}bytes=$(echo "$report_len" | tr -d ' ')${x}"
- # append to to upload
- echo "# path=$(echo "$file" | sed "s|^$git_root/||")" >> $upload_file
- cat "$file" >> $upload_file
- echo "<<<<<< EOF" >> $upload_file
- fr=1
- if [ "$clean" = "1" ];
- then
- rm "$file"
- fi
- else
- say " ${r}-${x} Skipping empty file $file"
- fi
- else
- say " ${r}-${x} file not found at $file"
- fi
- fi
-done <<< "$(echo -e "$files")"
-
-if [ "$fr" = "0" ];
-then
- say "${r}-->${x} No coverage data found."
- say " Please visit ${b}http://docs.codecov.io/docs/supported-languages${x}"
- say " search for your projects language to learn how to collect reports."
- exit ${exit_with};
-fi
-
-if [ "$ft_fix" = "1" ];
-then
- say "${e}==>${x} Appending adjustments"
- say " ${b}http://docs.codecov.io/docs/fixing-reports${x}"
- adjustments=""
-
- if echo "$network" | grep -m1 '.kt$' 1>/dev/null;
- then
- # skip brackets and comments
- adjustments="$adjustments
-$(find "$git_root" -type f -name '*.kt' -exec grep -nIH '^[[:space:]]*}$' {} \; | cut -f1-2 -d: 2>/dev/null || echo '')
-$(find "$git_root" -type f -name '*.kt' -exec grep -nIH '^/\*' {} \; | cut -f1-2 -d: 2>/dev/null || echo '')"
- fi
-
- if echo "$network" | grep -m1 '.go$' 1>/dev/null;
- then
- # skip empty lines, comments, and brackets
- adjustments="$adjustments
-$(find "$git_root" -type f -not -path '*/vendor/*' -name '*.go' -exec grep -nIH \
- -e '^[[:space:]]*$' \
- -e '^[[:space:]]*//.*' \
- -e '^[[:space:]]*/\*' \
- -e '^[[:space:]]*\*/' \
- -e '^[[:space:]]*}$' \
- {} \; | cut -f1-2 -d: 2>/dev/null || echo '')"
- fi
-
- if echo "$network" | grep -m1 '.jsx$' 1>/dev/null;
- then
- # skip empty lines, comments, and brackets
- adjustments="$adjustments
-$(find "$git_root" -type f -name '*.jsx' -exec grep -nIH \
- -e '^[[:space:]]*$' \
- -e '^[[:space:]]*//.*' \
- -e '^[[:space:]]*/\*' \
- -e '^[[:space:]]*\*/' \
- -e '^[[:space:]]*}$' \
- {} \; | cut -f1-2 -d: 2>/dev/null || echo '')"
- fi
-
- if echo "$network" | grep -m1 '.php$' 1>/dev/null;
- then
- # skip empty lines, comments, and brackets
- adjustments="$adjustments
-$(find "$git_root" -type f -not -path '*/vendor/*' -name '*.php' -exec grep -nIH \
- -e '^[[:space:]]*[\{\}\\[][[:space:]]*$' \
- -e '^[[:space:]]*);[[:space:]]*$' \
- -e '^[[:space:]]*][[:space:]]*$' \
- {} \; | cut -f1-2 -d: 2>/dev/null || echo '')"
- fi
-
- if echo "$network" | grep -m1 '\(.cpp\|.h\|.cxx\|.c\|.hpp\|.m\)$' 1>/dev/null;
- then
- # skip brackets
- adjustments="$adjustments
-$(find "$git_root" -type f \( -name '*.h' -or -name '*.cpp' -or -name '*.cxx' -or -name '*.m' -or -name '*.c' -or -name '*.hpp' \) -exec grep -nIH '^[[:space:]]*}' {} \; | cut -f1-2 -d: 2>/dev/null || echo '')"
- fi
-
- # join files into single line
- found=$(echo "$adjustments" | wc -l | tr -d ' ')
- adjustments=$(echo "$adjustments" | awk 'BEGIN { FS=":" }
- $1!=key { if (key!="") print out ; key=$1 ; out=$0 ; next }
- { out=out","$2 }
- END { print out }')
-
- if echo "$network" | grep -m1 '\(.cpp\|.h\|.cxx\|.c\|.hpp\|.m\)$' 1>/dev/null;
- then
- # skip LCOV_EXCL
- adjustments="$adjustments
-$(find "$git_root" -type f \( -name '*.h' -or -name '*.cpp' -or -name '*.cxx' -or -name '*.m' -or -name '*.c' -or -name '*.hpp' \) -exec grep -nIH '// LCOV_EXCL' {} \; 2>/dev/null || echo '')"
- fi
-
- if echo "$network" | grep -m1 '.kt$' 1>/dev/null;
- then
- # last line in file
- adjustments="$adjustments
-$(find "$git_root" -type f -name '*.kt' -exec wc -l {} \; | while read l; do echo "EOF: $l"; done 2>/dev/null || echo '')"
- fi
-
- if [ "$found" != "1" ];
- then
- say " ${g}+${x} Found $found adjustments"
- echo "# path=fixes
-$adjustments
-<<<<<< EOF" >> $upload_file
- else
- say " ${e}->${x} Found 0 adjustments"
- fi
-fi
-
-if [ "$url_o" != "" ];
-then
- url="$url_o"
-fi
-# trim whitespace from query
-
-if [ "$dump" != "0" ];
-then
- echo "$url/upload/v4?$(echo "package=bash-$VERSION&token=$token&$query" | tr -d ' ')"
- cat $upload_file
-else
-
- query=$(echo "${query}" | tr -d ' ')
- say "${e}==>${x} Uploading reports"
- say " ${e}url:${x} $url"
- say " ${e}query:${x} $query"
-
- # now add token to query
- query=$(echo "package=bash-$VERSION&token=$token&$query" | tr -d ' ')
-
- if [ "$ft_s3" = "1" ];
- then
- say " ${e}->${x} Pinging Codecov"
- res=$(curl -sX $cacert POST "$url/upload/v4?$query" -H 'Accept: text/plain' || true)
- # a good replay is "https://codecov.io" + "\n" + "https://codecov.s3.amazonaws.com/..."
- status=$(echo "$res" | head -1 | grep 'HTTP ' | cut -d' ' -f2)
- if [ "$status" = "" ];
- then
- s3target=$(echo "$res" | sed -n 2p)
- say " ${e}->${x} Uploading to S3 $(echo "$s3target" | cut -c1-32)"
- s3=$(curl -fisX PUT --data-binary @$upload_file \
- -H 'Content-Type: text/plain' \
- -H 'x-amz-acl: public-read' \
- -H 'x-amz-storage-class: REDUCED_REDUNDANCY' \
- "$s3target" || true)
- if [ "$s3" != "" ];
- then
- say " ${g}->${x} View reports at ${b}$(echo "$res" | sed -n 1p)${x}"
- exit 0
- else
- say " ${r}X>${x} Failed to upload to S3"
- fi
- elif [ "$status" = "400" ];
- then
- # 400 Error
- say "${g}${res}${x}"
- exit ${exit_with}
- fi
- fi
-
- say " ${e}->${x} Uploading to Codecov"
- i="0"
- while [ $i -lt 4 ]
- do
- i=$[$i+1]
-
- res=$(curl -sX POST $cacert --data-binary @$upload_file "$url/upload/v2?$query" -H 'Accept: text/plain' || echo 'HTTP 500')
- # HTTP 200
- # http://....
- status=$(echo "$res" | head -1 | cut -d' ' -f2)
- if [ "$status" = "" ];
- then
- say " View reports at ${b}$(echo "$res" | head -2 | tail -1)${x}"
- exit 0
-
- elif [ "${status:0:1}" = "5" ];
- then
- say " ${e}->${x} Sleeping for 10s and trying again..."
- sleep 10
-
- else
- say " ${g}${res}${x}"
- exit 0
- exit ${exit_with}
- fi
-
- done
-
-fi
-
-say " ${r}X> Failed to upload coverage reports${x}"
-exit ${exit_with}
-
-# EOF
diff --git a/.github/workflows/pinot_tests.yml b/.github/workflows/pinot_tests.yml
index fab7c01..9ffebba 100644
--- a/.github/workflows/pinot_tests.yml
+++ b/.github/workflows/pinot_tests.yml
@@ -20,8 +20,18 @@
name: Pinot Tests
on:
+ push:
+ branches:
+ - master
+ paths-ignore:
+ - "docs/**"
+ - "licenses/**"
+ - "licenses-binary/**"
+ - "website/**"
+ - "**.md"
pull_request:
- branches: [ master ]
+ branches:
+ - master
paths-ignore:
- "docs/**"
- "licenses/**"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 06/07: Delete Travis CI (#6616)
Posted by xi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch release-0.7.1-rc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 5769866528c1dcca5747a7aabb08e066974302ff
Author: Jialiang Li <jl...@linkedin.com>
AuthorDate: Thu Feb 25 14:56:39 2021 -0800
Delete Travis CI (#6616)
Co-authored-by: Jack Li(Analytics Engineering) <jl...@jlli-mn1.linkedin.biz>
---
.travis.yml | 91 -------------------
.travis/.ci.settings.xml | 31 -------
.travis/.travis_install.sh | 49 -----------
.travis/.travis_nightly_build.sh | 32 -------
.travis/.travis_quickstart.sh | 146 -------------------------------
.travis/.travis_quickstart_openjdk.sh | 32 -------
.travis/.travis_set_deploy_build_opts.sh | 27 ------
.travis/.travis_test.sh | 53 -----------
8 files changed, 461 deletions(-)
diff --git a/.travis.yml b/.travis.yml
deleted file mode 100644
index 8006c5b..0000000
--- a/.travis.yml
+++ /dev/null
@@ -1,91 +0,0 @@
-language: java
-os: linux
-dist: trusty
-
-# Prevent test build of a documentation-only change.
-#
-# if: conditions are preferred as they obviate job creation. However, they can only use
-# pre-defined env variables or globals that can be evaluated statically. The other way to skip
-# is to exit early. We do that here to prevent overhead of running tests that only include Markdown
-# (documentation) updates.
-_terminate_if_only_docs: &terminate_if_only_docs
- - |
- if [ -n "${TRAVIS_COMMIT_RANGE}" ] && ! git diff --name-only "${TRAVIS_COMMIT_RANGE}" -- | grep -qv '\.md$'; then
- echo "Stopping job as changes only affect documentation (ex. README.md)"
- travis_terminate 0
- fi
-
-before_install:
- - *terminate_if_only_docs
- - sudo apt-get update && sudo apt-get install -y --no-install-recommends gdb
- - mkdir -p ~/bin && curl -sSL -o ~/bin/jq https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 && chmod +x ~/bin/jq
- - export PATH=$PATH:~/bin
- - jq --version
- - curl -sSL -o ~/bin/install-jdk.sh https://raw.githubusercontent.com/sormuras/bach/master/install-jdk.sh && chmod +x ~/bin/install-jdk.sh
- - source ./.travis/.travis_set_deploy_build_opts.sh
-
-before_cache:
- - rm -rf $HOME/.m2/repository/org/apache/pinot
-
-cache:
- directories:
- - $HOME/.m2
- yarn: true
-
-addons:
- firefox: latest
-
-install:
- - ./.travis/.travis_install.sh
-
-#branches:
-# only:
-# - master
-
-stages:
- - test
- - name: deploy
-# if: branch = master
-
-jobs:
- include:
-# - name: "Integration Tests - OracleJDK 8"
-# jdk: oraclejdk8
-# script:
-# - ./.travis/.travis_test.sh
-# env:
-# - RUN_INTEGRATION_TESTS=true
-# - name: "Unit Tests - OracleJDK 8"
-# jdk: oraclejdk8
-# script:
-# - ./.travis/.travis_test.sh
-# env:
-# - RUN_INTEGRATION_TESTS=false
-# - name: "QuickStart - Java 8 & OpenJDK 14-15"
-# jdk: oraclejdk8
-# script:
-# - ./.travis/.travis_quickstart.sh
-# - jdk_switcher use openjdk8
-# - ./.travis/.travis_install.sh
-# - ./.travis/.travis_quickstart.sh
-# - ./.travis/.travis_quickstart_openjdk.sh 14
-# - ./.travis/.travis_quickstart_openjdk.sh 15
-# - name: "QuickStart - OpenJDK 10-13"
-# jdk: openjdk10
-# script:
-# - java -version
-# - ./.travis/.travis_quickstart.sh
-# - ./.travis/.travis_quickstart_openjdk.sh 11
-# - ./.travis/.travis_quickstart_openjdk.sh 12
-# - ./.travis/.travis_quickstart_openjdk.sh 13
- - stage: deploy
- script:
- - travis_wait 40 ./.travis/.travis_nightly_build.sh
-
-env:
- global:
- - secure: R3NdobUAThkk6BBzXapek6PiuWHWa1ldnJBvXpk2TAi7Lua/Ctgg8EHCSve7+MK4SU3XFJSi1th32+VSy0K/ARfdVcxiVh6EJSL/Nw0LRQSvGk7VjO382SsWCFoJuZBBqjy9DfBgRLdoqEPfulIBC0XBxjJ+4xedRptkZo7+TAAvtB8Y32EA5ve+Z4cdJipLF1tviL5q2rVbkNXucYUYO2XIv5HOzF2Xy3Hw6lsNrHpbes2CLEkoMiLOQ30zfbXZT/5Xd6OhQNmoZJ9gKziT1opiN2AJMPv269K8FFJZTrDZX74/thq9wmeF4X4f3wPA7EbTupooj0i9LPzb8Xev1QKAsxBtHyRm9NGYOXM06c2yKiGsDtT4HZYQGk4m8/89moe3oGnEwvw/JLu5RvLIZzFE2cBhHrcxRAJgHLIr80+kebNg/YstNlDL+rmP0b9NRrGIVw5OO2BT27ZBgsMFmjnujQtCUJ [...]
- - secure: Hj71ADg7ZgQpn8PUbEjaz1RrjkPfPCpGksp+WDjE6jf+oDvl55ILQhPgNcxIy81UcxS3Gmx6UWteVGkNV4Fi59TJIQ8KPbQ9RU2FFVXcxHH2TmkOvdNezwkJIFUw0TNMCbRAXgDj7ov75BM6lLi271GfqPVg9pB504Bc43WohnV6lDo/y93Z/xWbuFk5uJ+cogAJme3pvYHxBNlNzwIYnNAAqrEb+YKsRuux3qWLTlm1guT78g3C9wl/WCelTo1A3hGd12FqFWmowcoQlAyhbnFJ7u7qwfSNTSGCFLYbrUGooXqTZg2B+2QZORT2b5b0P9SXgB2a/UJxXmxkUzbmPGdUJipmQCEemlxwkPCqMW0YSscTDHU5WjSxqy4Cvwa8y+DEpLu06WxAPwjBje53025PvLjsMEfNJ0wsQbm9jkrH0alzICemXyyV5uGBCmuiMv5WqZ70WEpy7xlPlk5aFTwWQRpYJ5 [...]
- - secure: VfBYdV1EKmIBQlJj6yYUmucDU33HOm2dZLGcWC91QFVQx3FCOFzhSwTcby/gFM/9q/OzudKIRyTLV3iCtM1j1dEQA+oHLq6T1uV8DsOXnY908oShK1eil53aa3Ki4tYyMhR8WiXqVCoXglSB/Eakl6X2FnBfj0au+jvYLgPJE9IgY8hOvmp9ZdX5yFCbgcU2lmmJb7TZSnarrdtDBPuMG4dmRoqcUMxslvIZskOs38NVfkR4fj3gPGt2yZcauEKrqiiBF1nLGLfkigrifdHTW9fkIQtQmLwKrZx7+Mxz0GHAIrkGfoauQKfJy6fdFjyumtxWPxHKG1LlkmjrXEOustCefRy9S2bMkemgbCFT4jXsgHBCfmmV4BFADwvxcWSI6U9iSqeHfQMRgyvrJ36PaVi66UjVs+nPKPjMeHNcO32edB71/BKINEOgd38J1f4qTaUfnNexAf/lAT+rxl9PMfAuttDM5Q [...]
- - secure: Vp4Bv3B9RsImeo4zSiaiJMg4NvzgYRo/90ykozpoRjGQR1Ix7wBSsJvtFsf9iEVRBO5zrpT8GbdaV/Ed+UGi9XZVOnQEHS89pbscRAz2aIBfBlS0DCDrZAvxYP6XvwlJd1wz0zt1b87+dUeWyoEJWpZrzb3Z3knGTaNLnPYpW3HuG6BOv5bwo/n0ptY05IabsqPRi1yEiHdT/eSef6wLEhZuXtPXdUr7WMhjNNmIY6bwIH6ld2gUY3/1lVlz4ryd4+fPiPgaDc9ffWbn9JK8oHX+EEabkGXehkWbm/7dkBV+y0W+sV8Cw2+adhWX86FuS1J+2GoTcqpkB+Do36nAMccY9yKulpB6Kb0OInNSrZHLMtJl6p6Txm9VkihGTvGAlWEX8OhGFx/AEPf6cuVMpD4uzF7HS/L2fvlBYZbilUptaMzr/LDjfO0xdJF4wbtkBX0w2x7ISp2lIj2aVcKSERqisrTxaQ [...]
- - secure: Mg2mAcQRycTMKKU5g9xFvpe1DISMPEtGQgtCrnxpGMTLGUVaT1nUDa18OUgGuENjGy/slYeQZXQTkO/pQtxrPJcueXbG6ZMBa6Z0OizncX2G+FVtpuUGhy0LdSQFjAD5LAJy4DF5fQl4hlZ8Ucp0Er1o+j28UoETQ413jFj+hkevxpd+jIZ+VsLumedI3ChaRs9nZQG0cCv/RtHZPDQGuEJ6U4e4p/wTA1zCIVuqxS94pKq2IY3QTpQrBXdeIwaRkyhgZDhG43PSPqNGmSQU5WnLJRMzaQYc2+LowKhpjba+WouJ/PuyLiIUwOu2r7e+HGiXh9Wak0xM27m5g6kG1dqYEhVCIgTUBM2IJempcDJqZsw00m9t1Zv6PuYlMQDJPRoLVdpy7IIsR5OmZQbMegFw4ciNzjCPVV5xiYuBGg3VscDTLYosHLcYv3HjqV+d+vIQS39RjzWTKaMleBH5m8kYSeGZer [...]
diff --git a/.travis/.ci.settings.xml b/.travis/.ci.settings.xml
deleted file mode 100644
index 587dd4f..0000000
--- a/.travis/.ci.settings.xml
+++ /dev/null
@@ -1,31 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
-
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
--->
-<settings xsi:schemaLocation='http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd'
- xmlns='http://maven.apache.org/SETTINGS/1.0.0' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'>
- <servers>
- <server>
- <id>bintray-linkedin-maven</id>
- <username>${env.MVN_DEPLOY_BINTRAY_USER}</username>
- <password>${env.MVN_DEPLOY_BINTRAY_KEY}</password>
- </server>
- </servers>
-</settings>
diff --git a/.travis/.travis_install.sh b/.travis/.travis_install.sh
deleted file mode 100755
index 4c40d1a..0000000
--- a/.travis/.travis_install.sh
+++ /dev/null
@@ -1,49 +0,0 @@
-#!/bin/bash -x
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-# Ignore changes not related to pinot code
-echo 'Changed files:'
-git diff --name-only $TRAVIS_COMMIT_RANGE
-if [ $? -ne 0 ]; then
- echo 'Commit range is invalid.'
- exit 1
-fi
-
-KAFKA_BUILD_OPTS=""
-if [ "$KAFKA_VERSION" != '2.0' ] && [ "$KAFKA_VERSION" != '' ]; then
- KAFKA_BUILD_OPTS="-Dkafka.version=${KAFKA_VERSION}"
-fi
-
-# Java version
-java -version
-
-echo "Full Pinot build"
-if [ "$TRAVIS_JDK_VERSION" != 'oraclejdk8' ]; then
- # JDK 11 prints more logs exceeding Travis limits.
- mvn clean install -B -DskipTests=true -Pbin-dist -Dmaven.javadoc.skip=true ${DEPLOY_BUILD_OPTS} ${KAFKA_BUILD_OPTS} > /tmp/mvn_build_log
- if [ $? -eq 0 ]; then
- exit 0
- else
- tail -1000 /tmp/mvn_build_log
- exit 1
- fi
-else
- mvn clean install -B -DskipTests=true -Pbin-dist -Dmaven.javadoc.skip=true ${DEPLOY_BUILD_OPTS} ${KAFKA_BUILD_OPTS} || exit $?
-fi
diff --git a/.travis/.travis_nightly_build.sh b/.travis/.travis_nightly_build.sh
deleted file mode 100755
index d33e6a2..0000000
--- a/.travis/.travis_nightly_build.sh
+++ /dev/null
@@ -1,32 +0,0 @@
-#!/bin/bash -x
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-if [ -n "${DEPLOY_BUILD_OPTS}" ]; then
- echo "Deploying to bintray"
- echo "Current branch name: ${TRAVIS_BRANCH}"
-
- BUILD_VERSION=$(grep -E "<version>(.*)-SNAPSHOT</version>" pom.xml | cut -d'>' -f2 | cut -d'<' -f1 | cut -d'-' -f1)
- echo "Current build version: $BUILD_VERSION${DEV_VERSION}"
- mvn versions:set -DnewVersion="$BUILD_VERSION${DEV_VERSION}" -q -B
- mvn versions:commit -q -B
-
- # Deploy to bintray
- mvn deploy -s .travis/.ci.settings.xml -DscmBranch="${TRAVIS_BRANCH}" -DskipTests -q -DretryFailedDeploymentCount=5 -DaltDeploymentRepository=bintray-linkedin-maven::default::'https://api.bintray.com/maven/linkedin/maven/pinot/;publish=1;override=1'
-fi
diff --git a/.travis/.travis_quickstart.sh b/.travis/.travis_quickstart.sh
deleted file mode 100755
index f3c231c..0000000
--- a/.travis/.travis_quickstart.sh
+++ /dev/null
@@ -1,146 +0,0 @@
-#!/bin/bash -x
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-# Java version
-java -version
-
-# Quickstart
-DIST_BIN_DIR=`ls -d pinot-distribution/target/apache-pinot-*/apache-pinot-*`
-cd "${DIST_BIN_DIR}"
-
-# Test quick-start-batch
-bin/quick-start-batch.sh &
-PID=$!
-
-PASS=0
-
-# Wait for 30 seconds for table to be set up, then at most 5 minutes to reach the desired state
-sleep 30
-for i in $(seq 1 150)
-do
- QUERY_RES=`curl -X POST --header 'Accept: application/json' -d '{"sql":"select count(*) from baseballStats limit 1","trace":false}' http://localhost:8000/query/sql`
- if [ $? -eq 0 ]; then
- COUNT_STAR_RES=`echo "${QUERY_RES}" | jq '.resultTable.rows[0][0]'`
- if [[ "${COUNT_STAR_RES}" =~ ^[0-9]+$ ]] && [ "${COUNT_STAR_RES}" -eq 97889 ]; then
- PASS=1
- break
- fi
- fi
- sleep 2
-done
-
-cleanup () {
- # Terminate the process and wait for the clean up to be done
- kill "$1"
- while true;
- do
- kill -0 "$1" && sleep 1 || break
- done
-
- # Delete ZK directory
- rm -rf '/tmp/PinotAdmin/zkData'
-}
-
-cleanup "${PID}"
-if [ "${PASS}" -eq 0 ]; then
- echo 'Batch Quickstart failed: Cannot get correct result for count star query.'
- exit 1
-fi
-
-# Test quick-start-streaming
-bin/quick-start-streaming.sh &
-PID=$!
-
-PASS=0
-RES_1=0
-
-# Wait for 30 seconds for table to be set up, then at most 5 minutes to reach the desired state
-sleep 30
-for i in $(seq 1 150)
-do
- QUERY_RES=`curl -X POST --header 'Accept: application/json' -d '{"sql":"select count(*) from meetupRsvp limit 1","trace":false}' http://localhost:8000/query/sql`
- if [ $? -eq 0 ]; then
- COUNT_STAR_RES=`echo "${QUERY_RES}" | jq '.resultTable.rows[0][0]'`
- if [[ "${COUNT_STAR_RES}" =~ ^[0-9]+$ ]] && [ "${COUNT_STAR_RES}" -gt 0 ]; then
- if [ "${RES_1}" -eq 0 ]; then
- RES_1="${COUNT_STAR_RES}"
- continue
- elif [ "${COUNT_STAR_RES}" -gt "${RES_1}" ]; then
- PASS=1
- break
- fi
- fi
- fi
- sleep 2
-done
-
-cleanup "${PID}"
-if [ "${PASS}" -eq 0 ]; then
- if [ "${RES_1}" -eq 0 ]; then
- echo 'Streaming Quickstart test failed: Cannot get correct result for count star query.'
- exit 1
- fi
- echo 'Streaming Quickstart test failed: Cannot get incremental counts for count star query.'
- exit 1
-fi
-
-# Test quick-start-hybrid
-cd bin
-./quick-start-hybrid.sh &
-PID=$!
-
-PASS=0
-RES_1=0
-
-# Wait for 30 seconds for table to be set up, then at most 5 minutes to reach the desired state
-sleep 30
-for i in $(seq 1 150)
-do
- QUERY_RES=`curl -X POST --header 'Accept: application/json' -d '{"sql":"select count(*) from airlineStats limit 1","trace":false}' http://localhost:8000/query/sql`
- if [ $? -eq 0 ]; then
- COUNT_STAR_RES=`echo "${QUERY_RES}" | jq '.resultTable.rows[0][0]'`
- if [[ "${COUNT_STAR_RES}" =~ ^[0-9]+$ ]] && [ "${COUNT_STAR_RES}" -gt 0 ]; then
- if [ "${RES_1}" -eq 0 ]; then
- RES_1="${COUNT_STAR_RES}"
- continue
- elif [ "${COUNT_STAR_RES}" -gt "${RES_1}" ]; then
- PASS=1
- break
- fi
- fi
- fi
- sleep 2
-done
-
-cleanup "${PID}"
-if [ "${PASS}" -eq 0 ]; then
- if [ "${RES_1}" -eq 0 ]; then
- echo 'Hybrid Quickstart test failed: Cannot get correct result for count star query.'
- exit 1
- fi
- echo 'Hybrid Quickstart test failed: Cannot get incremental counts for count star query.'
- exit 1
-fi
-
-cd ../../../../../
-pwd
-mvn clean > /dev/null
-
-exit 0
diff --git a/.travis/.travis_quickstart_openjdk.sh b/.travis/.travis_quickstart_openjdk.sh
deleted file mode 100755
index fd395ca..0000000
--- a/.travis/.travis_quickstart_openjdk.sh
+++ /dev/null
@@ -1,32 +0,0 @@
-#!/bin/bash -x
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-JDK_VERSION=$1
-JAVA_HOME=$HOME/openjdk${JDK_VERSION}
-rm -rf "${JAVA_HOME}"
-. ~/bin/install-jdk.sh --feature "${JDK_VERSION}" --cacerts --target "${JAVA_HOME}"
-gdb --batch-silent --pid=$$ --eval-command='call unbind_variable("script_name")'
-gdb --batch-silent --pid=$$ --eval-command='call unbind_variable("script_version")'
-rm -rf jdk.tar.gz
-echo "${JAVA_HOME}"
-java -version
-./.travis/.travis_install.sh
-./.travis/.travis_quickstart.sh
-rm -rf "${JAVA_HOME}"
diff --git a/.travis/.travis_set_deploy_build_opts.sh b/.travis/.travis_set_deploy_build_opts.sh
deleted file mode 100755
index 1d5d1a7..0000000
--- a/.travis/.travis_set_deploy_build_opts.sh
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/bin/bash -x
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-if [ "$TRAVIS_EVENT_TYPE" = "cron" ]; then
- export DEV_VERSION="-dev-${TRAVIS_BUILD_NUMBER}"
- export DEPLOY_BUILD_OPTS="-Dsha1=-dev-${TRAVIS_BUILD_NUMBER}"
- npm install -g npm-login-noninteractive
-else
- export DEPLOY_BUILD_OPTS=""
-fi
diff --git a/.travis/.travis_test.sh b/.travis/.travis_test.sh
deleted file mode 100755
index d692b61..0000000
--- a/.travis/.travis_test.sh
+++ /dev/null
@@ -1,53 +0,0 @@
-#!/bin/bash -x
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-# Only run tests for JDK 8
-if [ "$TRAVIS_JDK_VERSION" != 'oraclejdk8' ]; then
- echo 'Skip tests for version other than oracle jdk8.'
- exit 0
-fi
-
-passed=0
-
-KAFKA_BUILD_OPTS=""
-if [ "$KAFKA_VERSION" != '2.0' ] && [ "$KAFKA_VERSION" != '' ]; then
- KAFKA_BUILD_OPTS="-Dkafka.version=${KAFKA_VERSION}"
-fi
-
-# Only run integration tests if needed
-if [ "$RUN_INTEGRATION_TESTS" != 'false' ]; then
- mvn test -B -P travis,integration-tests-only ${DEPLOY_BUILD_OPTS} ${KAFKA_BUILD_OPTS}
- if [ $? -eq 0 ]; then
- passed=1
- fi
-else
- mvn test -B -P travis,no-integration-tests ${DEPLOY_BUILD_OPTS} ${KAFKA_BUILD_OPTS}
- if [ $? -eq 0 ]; then
- passed=1
- fi
-fi
-
-if [ $passed -eq 1 ]; then
- # Only send code coverage data if passed
- bash <(cat .codecov_bash)
- exit 0
-else
- exit 1
-fi
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org