You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by jl...@apache.org on 2019/06/07 17:21:17 UTC
[incubator-pinot] branch master updated: Put managing Pinot under
in_production (#4284)
This is an automated email from the ASF dual-hosted git repository.
jlli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new f608b38 Put managing Pinot under in_production (#4284)
f608b38 is described below
commit f608b38d40394be616c1c36f808403b1c8d1a695
Author: Jialiang Li <jl...@linkedin.com>
AuthorDate: Fri Jun 7 10:21:11 2019 -0700
Put managing Pinot under in_production (#4284)
* Put managing Pinot under in_production
---
docs/admin_guide.rst | 1 -
docs/in_production.rst | 87 +++++++++++++++++++++++++++++++++++++------
docs/management_api.rst | 98 -------------------------------------------------
3 files changed, 76 insertions(+), 110 deletions(-)
diff --git a/docs/admin_guide.rst b/docs/admin_guide.rst
index c69ada6..2a50913 100644
--- a/docs/admin_guide.rst
+++ b/docs/admin_guide.rst
@@ -28,7 +28,6 @@ Admin Guide
tableconfig_schema
in_production
- management_api
pinot_hadoop
customizations
tuning_pinot
diff --git a/docs/in_production.rst b/docs/in_production.rst
index a9779ee..08b0605 100644
--- a/docs/in_production.rst
+++ b/docs/in_production.rst
@@ -46,17 +46,82 @@ The ordering is as follows:
Managing Pinot
~~~~~~~~~~~~~~
-Creating tables
----------------
-
-Updating tables
----------------
-
-Uploading data
---------------
-
-Configuring realtime data ingestion
------------------------------------
+Pinot provides a web-based management console and a command-line utility (``pinot-admin.sh``) in order to help provision and manage pinot clusters.
+
+Pinot Management Console
+------------------------
+
+The web based management console allows operations on tables, tenants, segments and schemas.
+You can access the console via ``http://controller-host:port/help``.
+The console also allows you to enter queries for interactive debugging.
+Here are some screen-shots from the console.
+
+ .. figure:: img/pinot-console.png
+
+Listing all the schemas in the Pinot cluster:
+
+ .. figure:: img/list-schemas.png
+
+Rebalancing segments of a table:
+
+ .. figure:: img/rebalance-table.png
+
+Command line utility (pinot-admin.sh)
+-------------------------------------
+
+The command line utility (``pinot-admin.sh``) can be generated by running
+``mvn install package -DskipTests -Pbin-dist`` in the directory in which you checked out Pinot.
+
+Here is an example of invoking the command to create a pinot segment:
+
+.. code-block:: none
+
+ $ ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/bin/pinot-admin.sh CreateSegment -dataDir /Users/host1/Desktop/test/ -format CSV -outDir /Users/host1/Desktop/test2/ -tableName baseballStats -segmentName baseballStats_data -overwrite -schemaFile ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/sample_data/baseballStats_schema.json
+ Executing command: CreateSegment -generatorConfigFile null -dataDir /Users/host1/Desktop/test/ -format CSV -outDir /Users/host1/Desktop/test2/ -overwrite true -tableName baseballStats -segmentName baseballStats_data -timeColumnName null -schemaFile ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/sample_data/baseballStats_schema.json -readerConfigFile null -enableStarTreeIndex false -starTreeIndexSpecFile null -hllSize 9 [...]
+ Accepted files: [/Users/host1/Desktop/test/baseballStats_data.csv]
+ Finished building StatsCollector!
+ Collected stats for 97889 documents
+ Created dictionary for INT column: homeRuns with cardinality: 67, range: 0 to 73
+ Created dictionary for INT column: playerStint with cardinality: 5, range: 1 to 5
+ Created dictionary for INT column: groundedIntoDoublePlays with cardinality: 35, range: 0 to 36
+ Created dictionary for INT column: numberOfGames with cardinality: 165, range: 1 to 165
+ Created dictionary for INT column: AtBatting with cardinality: 699, range: 0 to 716
+ Created dictionary for INT column: stolenBases with cardinality: 114, range: 0 to 138
+ Created dictionary for INT column: tripples with cardinality: 32, range: 0 to 36
+ Created dictionary for INT column: hitsByPitch with cardinality: 41, range: 0 to 51
+ Created dictionary for STRING column: teamID with cardinality: 149, max length in bytes: 3, range: ALT to WSU
+ Created dictionary for INT column: numberOfGamesAsBatter with cardinality: 166, range: 0 to 165
+ Created dictionary for INT column: strikeouts with cardinality: 199, range: 0 to 223
+ Created dictionary for INT column: sacrificeFlies with cardinality: 20, range: 0 to 19
+ Created dictionary for INT column: caughtStealing with cardinality: 36, range: 0 to 42
+ Created dictionary for INT column: baseOnBalls with cardinality: 154, range: 0 to 232
+ Created dictionary for STRING column: playerName with cardinality: 11976, max length in bytes: 43, range: to Zoilo Casanova
+ Created dictionary for INT column: doules with cardinality: 64, range: 0 to 67
+ Created dictionary for STRING column: league with cardinality: 7, max length in bytes: 2, range: AA to UA
+ Created dictionary for INT column: yearID with cardinality: 143, range: 1871 to 2013
+ Created dictionary for INT column: hits with cardinality: 250, range: 0 to 262
+ Created dictionary for INT column: runsBattedIn with cardinality: 175, range: 0 to 191
+ Created dictionary for INT column: G_old with cardinality: 166, range: 0 to 165
+ Created dictionary for INT column: sacrificeHits with cardinality: 54, range: 0 to 67
+ Created dictionary for INT column: intentionalWalks with cardinality: 45, range: 0 to 120
+ Created dictionary for INT column: runs with cardinality: 167, range: 0 to 192
+ Created dictionary for STRING column: playerID with cardinality: 18107, max length in bytes: 9, range: aardsda01 to zwilldu01
+ Start building IndexCreator!
+ Finished records indexing in IndexCreator!
+ Finished segment seal!
+ Converting segment: /Users/host1/Desktop/test2/baseballStats_data_0 to v3 format
+ v3 segment location for segment: baseballStats_data_0 is /Users/host1/Desktop/test2/baseballStats_data_0/v3
+ Deleting files in v1 segment directory: /Users/host1/Desktop/test2/baseballStats_data_0
+ Driver, record read time : 369
+ Driver, stats collector time : 0
+ Driver, indexing time : 373
+
+Here is an example of executing a query on a Pinot table:
+
+.. code-block:: none
+
+ $ ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/bin/pinot-admin.sh PostQuery -query "select count(*) from baseballStats" ./pinot-distribution/target/apache-pinot-incubaExecuting command: PostQuery -brokerHost [broker_host] -brokerPort [broker_port] -query select count(*) from baseballStats
+ Result: {"aggregationResults":[{"function":"count_star","value":"97889"}],"exceptions":[],"numServersQueried":1,"numServersResponded":1,"numSegmentsQueried":1,"numSegmentsProcessed":1,"numSegmentsMatched":1,"numDocsScanned":97889,"numEntriesScannedInFilter":0,"numEntriesScannedPostFilter":0,"numGroupsLimitReached":false,"totalDocs":97889,"timeUsedMs":107,"segmentStatistics":[],"traceInfo":{}}
Monitoring Pinot
~~~~~~~~~~~~~~~~
diff --git a/docs/management_api.rst b/docs/management_api.rst
deleted file mode 100644
index 9468bd1..0000000
--- a/docs/management_api.rst
+++ /dev/null
@@ -1,98 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-Managing Pinot
-==============
-
-There are two ways to manage Pinot cluster, i.e. using Pinot management console and ``pinot-admin.sh`` script.
-
-Pinot Management Console
-------------------------
-
-There is a REST API which allows management of tables, tenants, segments and schemas. It can be accessed by going to
-``http://[controller_host]/help`` which offers a web UI to do these tasks, as well as document the REST API. The below
-is the screenshot of the console.
-
- .. figure:: img/pinot-console.png
-
-For example, to list all the schemas within Pinot cluster:
-
- .. figure:: img/list-schemas.png
-
-To rebalance segments of a table across servers:
-
- .. figure:: img/rebalance-table.png
-
-
-pinot-admin.sh
---------------
-
-``pinot-admin.sh`` is another way of managing Pinot cluster. This script can be generated by running
-``mvn install package -DskipTests -Pbin-dist`` in the directory in which you checked out Pinot.
-
-For example, to create a pinot segment:
-
-.. code-block:: none
-
- $ ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/bin/pinot-admin.sh CreateSegment -dataDir /Users/host1/Desktop/test/ -format CSV -outDir /Users/host1/Desktop/test2/ -tableName baseballStats -segmentName baseballStats_data -overwrite -schemaFile ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/sample_data/baseballStats_schema.json
- Executing command: CreateSegment -generatorConfigFile null -dataDir /Users/host1/Desktop/test/ -format CSV -outDir /Users/host1/Desktop/test2/ -overwrite true -tableName baseballStats -segmentName baseballStats_data -timeColumnName null -schemaFile ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/sample_data/baseballStats_schema.json -readerConfigFile null -enableStarTreeIndex false -starTreeIndexSpecFile null -hllSize 9 [...]
- Accepted files: [/Users/host1/Desktop/test/baseballStats_data.csv]
- Finished building StatsCollector!
- Collected stats for 97889 documents
- Created dictionary for INT column: homeRuns with cardinality: 67, range: 0 to 73
- Created dictionary for INT column: playerStint with cardinality: 5, range: 1 to 5
- Created dictionary for INT column: groundedIntoDoublePlays with cardinality: 35, range: 0 to 36
- Created dictionary for INT column: numberOfGames with cardinality: 165, range: 1 to 165
- Created dictionary for INT column: AtBatting with cardinality: 699, range: 0 to 716
- Created dictionary for INT column: stolenBases with cardinality: 114, range: 0 to 138
- Created dictionary for INT column: tripples with cardinality: 32, range: 0 to 36
- Created dictionary for INT column: hitsByPitch with cardinality: 41, range: 0 to 51
- Created dictionary for STRING column: teamID with cardinality: 149, max length in bytes: 3, range: ALT to WSU
- Created dictionary for INT column: numberOfGamesAsBatter with cardinality: 166, range: 0 to 165
- Created dictionary for INT column: strikeouts with cardinality: 199, range: 0 to 223
- Created dictionary for INT column: sacrificeFlies with cardinality: 20, range: 0 to 19
- Created dictionary for INT column: caughtStealing with cardinality: 36, range: 0 to 42
- Created dictionary for INT column: baseOnBalls with cardinality: 154, range: 0 to 232
- Created dictionary for STRING column: playerName with cardinality: 11976, max length in bytes: 43, range: to Zoilo Casanova
- Created dictionary for INT column: doules with cardinality: 64, range: 0 to 67
- Created dictionary for STRING column: league with cardinality: 7, max length in bytes: 2, range: AA to UA
- Created dictionary for INT column: yearID with cardinality: 143, range: 1871 to 2013
- Created dictionary for INT column: hits with cardinality: 250, range: 0 to 262
- Created dictionary for INT column: runsBattedIn with cardinality: 175, range: 0 to 191
- Created dictionary for INT column: G_old with cardinality: 166, range: 0 to 165
- Created dictionary for INT column: sacrificeHits with cardinality: 54, range: 0 to 67
- Created dictionary for INT column: intentionalWalks with cardinality: 45, range: 0 to 120
- Created dictionary for INT column: runs with cardinality: 167, range: 0 to 192
- Created dictionary for STRING column: playerID with cardinality: 18107, max length in bytes: 9, range: aardsda01 to zwilldu01
- Start building IndexCreator!
- Finished records indexing in IndexCreator!
- Finished segment seal!
- Converting segment: /Users/host1/Desktop/test2/baseballStats_data_0 to v3 format
- v3 segment location for segment: baseballStats_data_0 is /Users/host1/Desktop/test2/baseballStats_data_0/v3
- Deleting files in v1 segment directory: /Users/host1/Desktop/test2/baseballStats_data_0
- Driver, record read time : 369
- Driver, stats collector time : 0
- Driver, indexing time : 373
-
-To query a table:
-
-.. code-block:: none
-
- $ ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/bin/pinot-admin.sh PostQuery -query "select count(*) from baseballStats" ./pinot-distribution/target/apache-pinot-incubaExecuting command: PostQuery -brokerHost [broker_host] -brokerPort [broker_port] -query select count(*) from baseballStats
- Result: {"aggregationResults":[{"function":"count_star","value":"97889"}],"exceptions":[],"numServersQueried":1,"numServersResponded":1,"numSegmentsQueried":1,"numSegmentsProcessed":1,"numSegmentsMatched":1,"numDocsScanned":97889,"numEntriesScannedInFilter":0,"numEntriesScannedPostFilter":0,"numGroupsLimitReached":false,"totalDocs":97889,"timeUsedMs":107,"segmentStatistics":[],"traceInfo":{}}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org