You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by gr...@apache.org on 2020/03/24 15:11:13 UTC
[kudu] branch master updated (a33b095 -> e4596ae)
This is an automated email from the ASF dual-hosted git repository.
granthenke pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git.
from a33b095 [util] set CURLOPT_NOPROXY for EasyCurl
new 8ba5f3e [docker] Make the build image more useful for development
new e4596ae [quickstart] Add an Apache Impala quickstart guide
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docker/Dockerfile | 4 +-
docker/bootstrap-dev-env.sh | 22 +++-
docs/quickstart.adoc | 19 ++-
examples/quickstart/impala/README.adoc | 221 +++++++++++++++++++++++++++++++++
examples/quickstart/nifi/README.adoc | 11 +-
examples/quickstart/spark/README.adoc | 5 +
examples/quickstart/ycsb/README.adoc | 6 +
7 files changed, 280 insertions(+), 8 deletions(-)
create mode 100644 examples/quickstart/impala/README.adoc
[kudu] 02/02: [quickstart] Add an Apache Impala quickstart guide
Posted by gr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
granthenke pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit e4596ae406c2d0fb621d0e4b8e002c4c600ef291
Author: Grant Henke <gr...@apache.org>
AuthorDate: Tue Mar 10 17:02:51 2020 -0500
[quickstart] Add an Apache Impala quickstart guide
This patch adds a short guide to illustrate basic Impala features
when using Kudu tables. Follow on guides can provide more detailed
and complicated examples.
To enable others to test this guide I pushed an image to my personal
dockerhub repo. Use `granthenke/kudu:impala-latest` in place of
`apache/kudu:impala-latest` when walking through the guide.
I also added help sections to the existing guides and a resource
configuration section to the top level guide based on feedback
from users walking through this example.
Change-Id: I208623add96f47119d36ac975b04352854398983
Reviewed-on: http://gerrit.cloudera.org:8080/15397
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
docs/quickstart.adoc | 19 ++-
examples/quickstart/impala/README.adoc | 221 +++++++++++++++++++++++++++++++++
examples/quickstart/nifi/README.adoc | 11 +-
examples/quickstart/spark/README.adoc | 5 +
examples/quickstart/ycsb/README.adoc | 6 +
5 files changed, 257 insertions(+), 5 deletions(-)
diff --git a/docs/quickstart.adoc b/docs/quickstart.adoc
index 46e06a9..39a3286 100644
--- a/docs/quickstart.adoc
+++ b/docs/quickstart.adoc
@@ -40,6 +40,18 @@ be used for production or performance/scale testing.
Follow the Docker link:https://docs.docker.com/install/[install documentation]
to install docker in your Linux, Mac, or Windows environment.
+Configure the Docker install to have enough resources to run the quickstart guides.
+
+- link:https://docs.docker.com/docker-for-mac/#resources[Docker for Mac Resource Configuration Guide]
+
+A minimum configuration that can run all the quickstart examples comfortably is:
+
+- 4 CPUs
+- 6 GB Memory
+- 50 GB Disk
+
+NOTE: You can likely get by with a lower resource configuration, but you may lose some performance and stability.
+
You may also want to read through the Docker getting started guide, but that isn't a requirement.
== Clone the Repository
@@ -72,10 +84,10 @@ you can specify the master addresses with `localhost:7051,localhost:7151,localho
[source,bash]
----
-docker-compose -f docker/quickstart.yml up
+docker-compose -f docker/quickstart.yml up -d
----
-NOTE: You can include the `-d` flag to run the cluster in the background.
+NOTE: You can remove the `-d` flag to run the cluster in the foreground.
=== View the Web-UI
@@ -129,6 +141,9 @@ More complete walkthroughs using the quickstart Kudu cluster can be found in the
`examples/quickstart` directory. For convenience you can browse them on
link:https://github.com/apache/kudu/tree/master/examples/quickstart[Github].
+- link:https://github.com/apache/kudu/tree/master/examples/quickstart/nifi[NiFi Quickstart Guide]
+- link:https://github.com/apache/kudu/tree/master/examples/quickstart/spark[Spark Quickstart Guide]
+- link:https://github.com/apache/kudu/tree/master/examples/quickstart/impala[Impala Quickstart Guide]
== Destroying the Cluster
diff --git a/examples/quickstart/impala/README.adoc b/examples/quickstart/impala/README.adoc
new file mode 100644
index 0000000..c0f7644
--- /dev/null
+++ b/examples/quickstart/impala/README.adoc
@@ -0,0 +1,221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+= Apache Impala Quickstart
+
+Below is a brief example using Apache Impala to insert, update, delete, and query data in Apache Kudu.
+
+== Start the Kudu Quickstart Environment
+
+See the Apache Kudu
+link:https://kudu.apache.org/docs/quickstart.html[quickstart documentation]
+to setup and run the Kudu quickstart environment.
+
+== Run Apache Impala
+
+Use the following command to run the latest Apache Impala docker image:
+
+NOTE: This docker image is a single node Kudu only image published for use in this quickstart only.
+The image is running a Hive metastore backed by a Derby database along with an Impala statestore daemon,
+catalog daemon, and executor daemon. The docker run command below will expose the required RPC and HTTP ports.
+
+[source,bash]
+----
+docker run -d --name kudu-impala --network="docker_default" \
+ -p 21000:21000 -p 21050:21050 -p 25000:25000 -p 25010:25010 -p 25020:25020 \
+ --memory=4096m apache/kudu:impala-latest impala
+----
+
+You can view the running Impala instance at link:http://localhost:25000[localhost:25000]
+once it is up and running. It may take a few seconds to start.
+
+NOTE: `--network="docker_default"` is specified to connect the container the
+same network as the quickstart cluster.
+
+NOTE: You can remove the `-d` flag to run the container in the foreground.
+
+
+== Run the impala-shell
+
+Use the command below to enter the `impala-shell` in the `kudu-impala` container:
+
+[source,bash]
+----
+docker exec -it kudu-impala impala-shell
+----
+
+NOTE: If the `impala-shell` says "Could not connect", wait a few more seconds to give
+Impala time to start and then enter `connect;` in the shell to try again.
+
+== Create a Kudu Table
+
+Now that you are in an `impala-shell` that is connected to Impala you can use an
+link:https://impala.apache.org/docs/build/html/topics/impala_ddl.html[Impala DDL statement]
+to create a Kudu table.
+
+[source,bash]
+----
+CREATE TABLE my_first_table
+(
+ id BIGINT,
+ name STRING,
+ PRIMARY KEY(id)
+)
+PARTITION BY HASH PARTITIONS 4
+STORED AS KUDU;
+
+DESCRIBE my_first_table;
+----
+
+== Insert and Modify Data
+
+With `my_first_table` created you can now use
+link:https://impala.apache.org/docs/build/html/topics/impala_dml.html[Impala DML statements]
+to `INSERT`, `UPDATE`, `UPSERT`, and `DELETE` data.
+
+[source,bash]
+----
+-- Insert a row.
+INSERT INTO my_first_table VALUES (99, "sarah");
+SELECT * FROM my_first_table;
+
+-- Insert multiple rows.
+INSERT INTO my_first_table VALUES (1, "john"), (2, "jane"), (3, "jim");
+SELECT * FROM my_first_table;
+
+-- Update a row.
+UPDATE my_first_table SET name="bob" where id = 3;
+SELECT * FROM my_first_table;
+
+-- Use upsert to insert a new row and update another.
+UPSERT INTO my_first_table VALUES (3, "bobby"), (4, "grant");
+SELECT * FROM my_first_table;
+
+-- Delete a row.
+DELETE FROM my_first_table WHERE id = 99;
+SELECT * FROM my_first_table;
+
+-- Delete multiple rows.
+DELETE FROM my_first_table WHERE id < 3;
+SELECT * FROM my_first_table;
+----
+
+== Create an External Table
+
+Sometimes users want to create an Impala table that points to an existing Kudu table.
+This can be achieved by using an
+link:https://impala.apache.org/docs/build/html/topics/impala_tables.html#external_tables[external table] in Impala.
+This will create an Impala table entry that points to the existing underlying Kudu table.
+
+[source,bash]
+----
+CREATE EXTERNAL TABLE my_second_table
+STORED AS KUDU
+TBLPROPERTIES('kudu.table_name' = 'impala::default.my_first_table');
+
+DESCRIBE my_second_table;
+
+DESCRIBE EXTENDED my_second_table;
+----
+
+== Drop the Tables
+
+You can drop the tables with a simple Impala DROP TABLE statement.
+When dropping the external table the underlying Kudu table will still exist.
+But when dropping the managed table the underling Kudu data will also be dropped.
+
+[source,bash]
+----
+DROP TABLE my_second_table;
+
+DESCRIBE my_first_table;
+SELECT * FROM my_first_table;
+
+DROP TABLE my_first_table;
+----
+
+== Exit the impala-shell
+
+Use the statement below to get exit the `impala-shell` in the `kudu-impala` container:
+
+[source,bash]
+----
+exit;
+----
+
+== Shutdown Impala
+
+Once you are done with the Impala container you can shutdown in a couple of ways.
+If you ran Impala without the `-d` flag, you can use `ctrl + c` to stop the container.
+
+If you ran Impala with the `-d` flag, you can use the following to
+gracefully shutdown the container:
+
+[source,bash]
+----
+docker stop kudu-impala
+----
+
+To permanently remove the container run the following:
+
+[source,bash]
+----
+docker rm kudu-impala
+----
+
+== Next steps
+
+The above example illustrates the basics of interacting with Kudu tables in Apache Impala.
+Next explore the other quickstart guides to learn how to ingest the data using other tools.
+
+For example, the link:https://github.com/apache/kudu/tree/master/examples/quickstart/spark[Spark quickstart guide]
+and link:https://github.com/apache/kudu/tree/master/examples/quickstart/nifi[NiFi quickstart guide]
+will walk you through how to ingest and process data in Kudu. You can follow those quickstart guides
+and query the data ingested using the steps described in this quickstart.
+
+If you have already run through the Spark quickstart the following is an
+example of the code to allow you to query the `sfmta_kudu` table:
+
+[source,bash]
+----
+CREATE EXTERNAL TABLE sfmta_kudu
+STORED AS KUDU
+TBLPROPERTIES('kudu.table_name' = 'sfmta_kudu');
+
+SELECT * FROM sfmta_kudu
+ORDER BY speed
+LIMIT 5;
+----
+
+If you have already run through the NiFi quickstart the following is an
+example of the code to allow you to query the `random_user` table:
+
+[source,bash]
+----
+CREATE EXTERNAL TABLE random_user
+STORED AS KUDU
+TBLPROPERTIES('kudu.table_name' = 'random_user');
+
+SELECT count(*) FROM random_user;
+
+SELECT * FROM random_user LIMIT 5;
+----
+
+== Help
+
+If have questions, issues, or feedback on this quickstart guide, please reach out to the
+link:https://kudu.apache.org/community.html[Apache Kudu community].
diff --git a/examples/quickstart/nifi/README.adoc b/examples/quickstart/nifi/README.adoc
index 0f7c79d..de91033 100644
--- a/examples/quickstart/nifi/README.adoc
+++ b/examples/quickstart/nifi/README.adoc
@@ -31,7 +31,7 @@ Use the following command to run the latest Apache NiFi Docker image:
[source,bash]
----
-docker run --name kudu-nifi --network="docker_default" -p 8080:8080 apache/nifi:latest
+docker run -d --name kudu-nifi --network="docker_default" -p 8080:8080 apache/nifi:latest
----
You can view the running NiFi instance at link:http://localhost:8080/nifi[localhost:8080/nifi].
@@ -39,7 +39,7 @@ You can view the running NiFi instance at link:http://localhost:8080/nifi[localh
NOTE: `--network="docker_default"` is specified to connect the container the
same network as the quickstart cluster.
-NOTE: You can include the `-d` flag to run the container in the background.
+NOTE: You can remove the `-d` flag to run the container in the foreground.
== Create the Kudu table
@@ -131,7 +131,7 @@ link:https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#starting-a-comp
== Shutdown NiFi
Once you are done with the NiFi container you can shutdown in a couple of ways.
-If you ran NiFi without the `-d` flag, you can use `ctrl + c` to stop the container.
+If you ran NiFi without the `-d` flag, you can use `ctrl + c` to stop the container.
If you ran NiFi with the `-d` flag, you can use the following to
gracefully shutdown the container:
@@ -180,3 +180,8 @@ random_user.createOrReplaceTempView("random_user")
spark.sql("SELECT count(*) FROM random_user").show()
spark.sql("SELECT * FROM random_user LIMIT 5").show()
----
+
+== Help
+
+If have questions, issues, or feedback on this quickstart guide, please reach out to the
+link:https://kudu.apache.org/community.html[Apache Kudu community].
diff --git a/examples/quickstart/spark/README.adoc b/examples/quickstart/spark/README.adoc
index e833838..3ff435c 100644
--- a/examples/quickstart/spark/README.adoc
+++ b/examples/quickstart/spark/README.adoc
@@ -223,3 +223,8 @@ mutate data in a streaming fashion.
As an exercise to learn the Kudu programmatic APIs, try implementing a program
that uses the link:http://www.nextbus.com/xmlFeedDocs/NextBusXMLFeed.pdf[SFMTA XML data feed]
to ingest this same dataset in real time into the Kudu table.
+
+== Help
+
+If have questions, issues, or feedback on this quickstart guide, please reach out to the
+link:https://kudu.apache.org/community.html[Apache Kudu community].
diff --git a/examples/quickstart/ycsb/README.adoc b/examples/quickstart/ycsb/README.adoc
index 3618c6c..790d711 100644
--- a/examples/quickstart/ycsb/README.adoc
+++ b/examples/quickstart/ycsb/README.adoc
@@ -96,3 +96,9 @@ pushd YCSB
mvn -Psource-run -pl site.ycsb:kudu-binding -am clean package -DskipTests -Dkudu.version=1.12.0-SNAPSHOT
popd
----
+
+
+== Help
+
+If have questions, issues, or feedback on this quickstart guide, please reach out to the
+link:https://kudu.apache.org/community.html[Apache Kudu community].
[kudu] 01/02: [docker] Make the build image more useful for
development
Posted by gr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
granthenke pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 8ba5f3e0e8c6157bf6b731f01b0c71549192d2e6
Author: Grant Henke <gr...@apache.org>
AuthorDate: Mon Mar 23 09:23:09 2020 -0500
[docker] Make the build image more useful for development
This patch makes the docker build image more useful for
development. It contains a few small changes including:
- Fixed Sentry in thirdparty
- Added lsof package to fix HMS tests
- Replaced ntp with chrony
- Added build packages (cmake, ninja-build)
- Added vim for basic file editing
- Added ccache. A follow on change will allow the docker cache to
share the ccache between builds too.
With this patch most of the tests pass in the Docker container.
The following tests are still failing:
- hybrid_clock-test
- client-stress-test
- memory_gc-itest
- raft_consensus-itest.2
Fixes for these will be added in follow on patches.
I used the following steps to build a xenial image and run the tests:
export TARGETS="build"
./docker/docker-build.sh
docker run -it --rm apache/kudu:build-latest /bin/bash
cd /kudu
mkdir build/debug
cd build/debug
../../thirdparty/installed/common/bin/cmake \
-DCMAKE_BUILD_TYPE=debug ../..
make -j6
ctest -j6
Change-Id: I1e4b463a440a3f76fb07425aebcbda2050b9917e
Reviewed-on: http://gerrit.cloudera.org:8080/15530
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
docker/Dockerfile | 4 +++-
docker/bootstrap-dev-env.sh | 22 ++++++++++++++++++++--
2 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/docker/Dockerfile b/docker/Dockerfile
index e5ebd3f..6f2eadf 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -70,6 +70,8 @@ RUN ./bootstrap-dev-env.sh \
&& rm bootstrap-java-env.sh \
&& rm bootstrap-python-env.sh
+ENV PATH /usr/lib/ccache:/usr/lib64/ccache/:$PATH
+
# Common label arguments.
# VCS_REF is not specified to improve docker caching.
ARG DOCKERFILE
@@ -109,7 +111,7 @@ RUN build-support/enable_devtoolset.sh \
# Remove all the source files except the hive, hadoop, and sentry sources
# which are pre-built and symlinked into the installed/common/opt directory.
&& find thirdparty/src/* -maxdepth 0 -type d \
- \( ! -name 'hadoop-*' ! -name 'hive-*' ! -name 'apache-sentry-*' \) \
+ \( ! -name 'hadoop-*' ! -name 'hive-*' ! -name 'sentry-*' \) \
-prune -exec rm -rf {} \; \
# Remove all the build files except the llvm build which is symlinked into
# the clang-toolchain directory.
diff --git a/docker/bootstrap-dev-env.sh b/docker/bootstrap-dev-env.sh
index 5547d7f..4ba2c34 100755
--- a/docker/bootstrap-dev-env.sh
+++ b/docker/bootstrap-dev-env.sh
@@ -36,6 +36,7 @@ if [[ -f "/usr/bin/yum" ]]; then
yum install -y \
autoconf \
automake \
+ chrony \
chrpath \
cyrus-sasl-devel \
cyrus-sasl-gssapi \
@@ -49,8 +50,8 @@ if [[ -f "/usr/bin/yum" ]]; then
krb5-server \
krb5-workstation \
libtool \
+ lsof \
make \
- ntp \
openssl-devel \
patch \
pkgconfig \
@@ -68,6 +69,14 @@ if [[ -f "/usr/bin/yum" ]]; then
lzo-devel \
tzdata
+ # Install libraries often used for Kudu development and build performance.
+ yum install -y epel-release
+ yum install -y \
+ ccache \
+ cmake \
+ ninja-build \
+ vim
+
# Install docs build libraries.
# Note: Uncomment to include in your dev images. These are excluded to reduce image size and build time.
# yum install -y \
@@ -86,6 +95,7 @@ if [[ -f "/usr/bin/yum" ]]; then
wget ${DTLS_RPM_URL} -O ${DTLS_RPM}
yum install -y scl-utils ${DTLS_RPM}
yum install -y devtoolset-3-toolchain
+ rm -f $DTLS_RPM
fi
# Reduce the image size by cleaning up after the install.
@@ -104,6 +114,7 @@ elif [[ -f "/usr/bin/apt-get" ]]; then
apt-get install -y --no-install-recommends \
autoconf \
automake \
+ chrony \
chrpath \
curl \
flex \
@@ -121,8 +132,8 @@ elif [[ -f "/usr/bin/apt-get" ]]; then
libssl-dev \
libtool \
lsb-release \
+ lsof \
make \
- ntp \
openssl \
patch \
pkg-config \
@@ -139,6 +150,13 @@ elif [[ -f "/usr/bin/apt-get" ]]; then
liblzo2-2 \
tzdata
+ # Install libraries often used for Kudu development and build performance.
+ apt-get install -y --no-install-recommends \
+ ccache \
+ cmake \
+ ninja-build \
+ vim
+
# Install docs build libraries.
# Note: Uncomment to include in your dev images. These are excluded to reduce image size and build time.
# apt-get install -y --no-install-recommends \