You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by al...@apache.org on 2019/10/25 23:29:59 UTC
[kudu] branch master updated (314762d -> 27e207b)
This is an automated email from the ASF dual-hosted git repository.
alexey pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git.
from 314762d add a tool to create table
new f5578f8 [docs] Fix formatting in catalog manager design page
new 27e207b [scripts] a tool to print out HybridTime timestamps
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/design-docs/master.md | 49 +++++++++-------------
.../kudu/scripts/convert-hybrid-timestamp.py | 29 +++++++++----
2 files changed, 39 insertions(+), 39 deletions(-)
copy build-support/get-upstream-commit.sh => src/kudu/scripts/convert-hybrid-timestamp.py (59%)
mode change 100755 => 100644
[kudu] 02/02: [scripts] a tool to print out HybridTime timestamps
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
alexey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 27e207b02c28d5ac23835a96a0823078def14a8b
Author: Alexey Serbin <al...@apache.org>
AuthorDate: Thu Oct 24 10:01:27 2019 -0700
[scripts] a tool to print out HybridTime timestamps
This is Todd's script to output HybridTime timestamps in human-readable
form (works with Python2 and Python3). The script reads from standard
input until EOF/EOS and outputs the result into standard output.
The script is useful when analyzing hybrid timestamps found in Kudu
WAL files. For example:
$ echo 6435399373400084500 | python convert-hybrid-timestamp.py
2019-10-15 05:27:05.146505 L=20
Change-Id: Ifc5b0d20c6f1bca0994d00a347ff14e4ae64a14c
Reviewed-on: http://gerrit.cloudera.org:8080/14541
Tested-by: Alexey Serbin <as...@cloudera.com>
Reviewed-by: Bankim Bhavsar <ba...@cloudera.com>
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
src/kudu/scripts/convert-hybrid-timestamp.py | 40 ++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
diff --git a/src/kudu/scripts/convert-hybrid-timestamp.py b/src/kudu/scripts/convert-hybrid-timestamp.py
new file mode 100644
index 0000000..2ee09d3
--- /dev/null
+++ b/src/kudu/scripts/convert-hybrid-timestamp.py
@@ -0,0 +1,40 @@
+#!/usr/bin/env python
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#
+# An example of usage:
+#
+# $ echo 6435399373400084500 | python convert-hybrid-timestamp.py
+# 2019-10-15 05:27:05.146505 L=20
+#
+
+import time
+import re
+import sys
+
+def convert_ts(ts_match):
+ ts_i = int(ts_match.group(0))
+ phys, logical = (ts_i >> 12, ts_i & ((1 << 12) - 1))
+ phys_sec = phys / 1000000
+ phys_micros = phys % 1000000
+ f = time.strftime("%F %H:%M:%S", time.localtime(phys_sec))
+ return "%s.%06d L=%d" % (f, phys_micros, logical)
+
+for l in sys.stdin.readlines():
+ print(re.sub(r"\d{10,19}", convert_ts, l))
[kudu] 01/02: [docs] Fix formatting in catalog manager design page
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
alexey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit f5578f84691233fc9a9320337dd0d86ddbb3b637
Author: Bankim Bhavsar <ba...@cloudera.com>
AuthorDate: Fri Oct 25 14:20:46 2019 -0700
[docs] Fix formatting in catalog manager design page
Issues:
- Title incorrectly rendered
- Bullet list under Table creation not correctly rendered
https://github.com/apache/kudu/blob/master/docs/design-docs/master.md#table-creation
- Section headers not correctly marked
Tests:
- Verified with a markdown plugin on chrome
Change-Id: I4ac7df450b43d97e3dbfb3275708b3b47a0dc1e6
Reviewed-on: http://gerrit.cloudera.org:8080/14550
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
docs/design-docs/master.md | 49 ++++++++++++++++++----------------------------
1 file changed, 19 insertions(+), 30 deletions(-)
diff --git a/docs/design-docs/master.md b/docs/design-docs/master.md
index 5eb20d3..50d3358 100644
--- a/docs/design-docs/master.md
+++ b/docs/design-docs/master.md
@@ -12,9 +12,8 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-============================================================
-The Catalog Manager and System Tables
-============================================================
+# The Catalog Manager and System Tables
+
The Catalog Manager keeps track of the Kudu tables and tablets defined by the
user in the cluster.
@@ -46,31 +45,29 @@ The catalog manager maintains 3 hash-maps for looking up info in the sys table:
The TableInfo has a map [tablet-start-key] -> TabletInfo used to provide
the tablets locations to the user based on a key-range request.
-
-Table Creation
---------------
+## Table Creation
The below corresponds to the code in CatalogManager::CreateTable().
1. Client -> Master request: Create "table X" with N tablets and schema S.
2. Master: CatalogManager::CreateTable():
- a. Validate user request (e.g. ensure a valid schema).
- b. Verify that the table name is not already taken.
+ 1. Validate user request (e.g. ensure a valid schema).
+ 2. Verify that the table name is not already taken.
TODO: What about old, deleted tables?
- c. Add (in-memory) the new TableInfo (in "preparing" state).
- d. Add (in-memory) the TabletInfo based on the user-provided pre-split-keys
+ 3. Add (in-memory) the new TableInfo (in "preparing" state).
+ 4. Add (in-memory) the TabletInfo based on the user-provided pre-split-keys
field (in "preparing" state).
- e. Write the tablets info to "sys.catalog"
+ 5. Write the tablets info to "sys.catalog"
(The Master process is killed if the write fails).
- Master begins writing to disk.
- Note: If the Master crashes or restarts here or at any time previous to
this point, the table will not exist when the Master comes back online.
- f. Write the table info to "sys.catalog" with the "running" state
+ 6. Write the table info to "sys.catalog" with the "running" state
(The Master process is killed if the write fails).
- Master completes writing to disk.
- After this point, the table will exist and be re-created as necessary
at startup time after a crash or process restart.
- g. Commit the "running" state to memory, which allows clients to see the table.
+ 7. Commit the "running" state to memory, which allows clients to see the table.
3. Master -> Client response: The table has been created with some ID, i.e. "xyz"
(or, in case something went wrong, an error message).
@@ -79,8 +76,7 @@ the cluster is shut down, when it starts back up the table will still exist.
However, the tablets are not yet created (see Table Assignment, below).
-Table Deletion
---------------
+## Table Deletion
When the user sends a DeleteTable request for table T, table T is marked as
deleted by writing a "deleted" flag in the state field in T's record in the
@@ -105,8 +101,7 @@ successfully completes, the Master will send a new DeleteTablet request at the
time that the next heartbeat is received from the tablet that is to be deleted.
-Table Assignment (Tablet Creation)
-----------------------------------
+## Table Assignment (Tablet Creation)
Once a table is created, the tablets must be created on a set of replicas. In
order to do that, the master has to select the replicas and associate them to
@@ -167,8 +162,7 @@ process (see CatalogManagerBgTasksThread above), the tablet will be marked as
running and committed to disk, completing the assignment process.
-Alter Table
------------
+## Alter Table
When the user sends an alter request, which may contain changes to the schema,
table name or attributes, the Master will send a set of AlterTable() RPCs to
@@ -183,9 +177,8 @@ When the Master first comes online after being restarted, a full tablet report
will be requested from each TS, and the tablet schema version sent on the next
heartbeat will be used to determine if a given TS needs an AlterTable() call.
-============================================================
-Heartbeats and TSManager
-============================================================
+
+# Heartbeats and TSManager
Heartbeats are sent by the TS to the master. Per master.proto, a
heartbeat contains:
@@ -203,8 +196,7 @@ changed, or if the master responded to a previous heartbeat with
"needs a full tablet report" (see "Handling heartbeats" below for an
explanation of when this response will be sent).
-Handling heartbeats
--------------------
+## Handling heartbeats
Upon receiving a heartbeat from a TS, the master will:
@@ -226,8 +218,7 @@ the TS requesting a full tablet report.
5) Send a success respond to the TS.
-TSManager
----------
+## TSManager
TSManager provides in-memory storage for information sent by the
tablet server to the master (tablet servers that have been heard from,
@@ -235,8 +226,7 @@ heartbeats, tablet reports, etc...). The information is stored in a
map, where the key is the permanent uuid of a tablet server and the
value is (a pointer to) a TSDescriptor.
-IPKI: Internal Root Certificate Authority (CA) Information
-----------------------------------------------------------
+# IPKI: Internal Root Certificate Authority (CA) Information
Besides tables' metadata, the system table contains the root CA certificate
and corresponding private key when Kudu is configured to use its own IPKI
@@ -254,8 +244,7 @@ information is present in the system table. If the internal root CA information
is already present in the system table, the leader master loads that
information into memory and uses it appropriately.
-IPKI: TSK (Token Signing Keys)
-------------------------------
+## IPKI: TSK (Token Signing Keys)
The system table contains entries with TSKs used for authn/authz token signing.
The leader master generates and stores those in the system table. Upon start-up