You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2016/10/31 23:19:07 UTC
[1/2] incubator-impala git commit: Fix distcc instructions to help
users know they must set BUILD_FARM
Repository: incubator-impala
Updated Branches:
refs/heads/master c03044717 -> 7fc31b534
Fix distcc instructions to help users know they must set BUILD_FARM
While I'm here, remove an errant reference to IMPALA_AUX_TEST_HOME.
Change-Id: I323006110d77de05b251d9fc2578774a78e5509b
Reviewed-on: http://gerrit.cloudera.org:8080/4868
Reviewed-by: Jim Apple <jb...@cloudera.com>
Tested-by: Tim Armstrong <ta...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/5107669a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/5107669a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/5107669a
Branch: refs/heads/master
Commit: 5107669a7ff6102f14ead96b58af0f958a51e18c
Parents: c030447
Author: Jim Apple <jb...@cloudera.com>
Authored: Thu Oct 27 12:13:33 2016 -0700
Committer: Tim Armstrong <ta...@cloudera.com>
Committed: Mon Oct 31 23:18:44 2016 +0000
----------------------------------------------------------------------
bin/distcc/README.md | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/5107669a/bin/distcc/README.md
----------------------------------------------------------------------
diff --git a/bin/distcc/README.md b/bin/distcc/README.md
index 2de7d4a..d6fe8b7 100644
--- a/bin/distcc/README.md
+++ b/bin/distcc/README.md
@@ -8,16 +8,20 @@ The only requirement you should need to be aware of is, the scripts in this fold
only tested on Linux. If you are using OS X, things probably won't work out of the box.
Assuming you are using Linux, if you use the scripts in this folder, there shouldn't be
-any other requirements. The distcc program should be installed and configured
-automatically. Still, understanding what is involved could be useful.
+any other requirements other than setting up your build farm and your BUILD_FARM variable.
+
+Setting up a new distcc server is covered at the bottom of this document. Once your distcc
+servers are configured, set the environment variable BUILD_FARM on your build machine to
+to "host1/limit1,lzo host2/limit2,lzo" and so on.
+
+The rest of the setup is done for you; here is a short description of what they do:
**You shouldn't need to do any of this, this scripts do this for you.**
1. Install distcc and ccache. Most Linux distros have these packages. The scripts will
install it if you have a yum or apt-get based system. Otherwise you should install
distcc and ccache yourself through whatever package manager your system uses.
-1. Configure the remote distcc hosts. Set your environment variable BUILD_FARM to
- "host1/limit1,lzo host2/limit2,lzo" and so on.
+1. Configure the remote distcc hosts.
1. Your local compiler needs to be at the same path as it is on the remote build slaves.
That path is /opt/Impala-Toolchain/<gcc-version-folder>/bin/gcc. In other words, make
sure the Impala toolchain is available at /opt/Impala-Toolchain. That can be done
@@ -33,7 +37,7 @@ automatically. Still, understanding what is involved could be useful.
1. Source "distcc_env.sh" in this directory. The script will attempt to install distcc
if needed.
- source "$IMPALA_AUX_TEST_HOME"/distcc/distcc_env.sh
+ source "$IMPALA_HOME"/bin/distcc/distcc_env.sh
1. Run buildall.sh. The main purpose is to regenerate cmakefiles.
@@ -53,7 +57,7 @@ or
bin/make_impala.sh
```
-### Switiching back to local compilation
+### Switching back to local compilation
If you want to compile a very small change, a local build might be faster.
```
switch_compiler local
[2/2] incubator-impala git commit: IMPALA-4340: explain how to
install postgresql-9.5 or higher
Posted by ta...@apache.org.
IMPALA-4340: explain how to install postgresql-9.5 or higher
The random query generator needs to compare against PostgresQL 9.5 or
higher to take advantage of some of the more recent features, especially
as it pertains to Impala/Kudu INSERT and UPSERT queries. Developers
will need assistance setting up their development environments if they
need to use the random query generator.
This patch provides instructions on how to do so. We provide
instructions, not automation, since this will have side-effects on
developers' workstations: we can't presume to know how a developer might
want to install or configure postgres, and we haven't tested on anything
except our own development environment.
Change-Id: I1e3b510120451fcb5af97145fa47ccb4c53f00d9
Reviewed-on: http://gerrit.cloudera.org:8080/4846
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Tim Armstrong <ta...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/7fc31b53
Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/7fc31b53
Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/7fc31b53
Branch: refs/heads/master
Commit: 7fc31b534d4c5cb118c559e16556a6c1ae6ca7fc
Parents: 5107669
Author: Michael Brown <mi...@cloudera.com>
Authored: Tue Oct 25 12:36:31 2016 -0700
Committer: Tim Armstrong <ta...@cloudera.com>
Committed: Mon Oct 31 23:18:55 2016 +0000
----------------------------------------------------------------------
tests/comparison/POSTGRES.txt | 84 ++++++++++++++++++++++++++++++++++++++
tests/comparison/README | 4 ++
2 files changed, 88 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/7fc31b53/tests/comparison/POSTGRES.txt
----------------------------------------------------------------------
diff --git a/tests/comparison/POSTGRES.txt b/tests/comparison/POSTGRES.txt
new file mode 100644
index 0000000..af28240
--- /dev/null
+++ b/tests/comparison/POSTGRES.txt
@@ -0,0 +1,84 @@
+Motivation
+----------
+To test Impala/Kudu:
+ 1. INSERT with ignorable primary key violations
+ 2. "UPSERT"
+
+we need PostgresQL 9.5 or higher.
+
+https://www.postgresql.org/docs/9.5/static/sql-insert.html
+
+PostgresQL 9.5+ Installation
+----------------------------
+
+Common GNU/Linux distributions won't necessarily have PostgresQL as high
+as 9.5 available. Fortunately, PostgresQL understands this and provides
+instructions for adding sources for both Debian- and Redhat-based
+packages.
+
+https://wiki.postgresql.org/wiki/Apt
+https://yum.postgresql.org/repopackages.php
+
+What follows are instructions for getting this working on Ubuntu 14.04.
+
+Use the instructions at https://wiki.postgresql.org/wiki/Apt
+
+Note: apt pinning is possible, but if you already installed postgres
+from your standard GNU/Linux sources, you'll be forced to upgrade some
+packages like postgresql-common anyway. Resolving those problems is out
+of the scope of this document.
+
+Note: If you have postgresql-9.3 installed, your 9.3 installation will
+get updated to the latest point release version.
+
+If you then install a new PostgresQL version, you might be prompted what
+to do about /etc/postgresql-common/createcluster.conf . I elected to
+keep the local version installed.
+
+Before you start your new version of PostgresQL, you will probably want
+to make it easier to connect to the server:
+
+ $ sudo perl -pi -e 's/(?:peer|md5)/trust/g' /etc/postgresql/9.5/main/pg_hba.conf
+
+Note: By default, PostgresQL servers only bind to localhost, so the
+command above should only affect the localhost TCP server.
+
+Finally, you can 'sudo service postgresql start'. PostgresQL 9.3 is
+still listening on port 5432, but note 9.5 is also running, on 5433.
+
+PostgresQL 9.5+ Usage
+---------------------
+
+Databases aren't shared across versions. You'll need to load data, e.g.
+via:
+
+ $ tests/comparison/data_generator.py \
+ --use-postgresql \
+ --postgresql-port 5433 # <---- important \
+ [other options]
+
+... and use the random query generator in the same way:
+
+ $ tests/comparison/discrepancy_searcher.py \
+ --use-postgresql \
+ --postgresql-port 5433 # <---- important \
+ [other options]
+
+Testing
+-------
+
+The following steps have been taking to validate postgresql-9.5
+9.5.4-1.pgdg14.04+2 on Ubuntu 14.04.2 LTS:
+
+1. Migrating tpch and tpch_kudu from Impala into PostgresQL 9.5 (via
+ 'data_generator.py migrate')
+2. Loading random data into PostgresQL 9.5 (via 'data_generator.py populate')
+3. Discrepancy tests between Impala and PostgresQL 9.5, and then again
+ on 9.3, to ensure results are the same.
+4. Quick experiments to make sure ON CONFLICT DO NOTHING and ON CONFLICT
+ DO UPDATE actually work when connecting to Postgres on port 5433.
+
+Other Resources
+---------------
+https://wiki.postgresql.org/wiki/Detailed_installation_guides
+https://wiki.postgresql.org/wiki/Apt/FAQ
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/7fc31b53/tests/comparison/README
----------------------------------------------------------------------
diff --git a/tests/comparison/README b/tests/comparison/README
index 579a8cf..d3e86df 100644
--- a/tests/comparison/README
+++ b/tests/comparison/README
@@ -22,6 +22,10 @@ Requirements:
sudo apt-get install python-mysqldb
sudo apt-get install python-psycopg2 # Postgresql
+For Impala/Kudu CRUD random query generation and comparison, please see
+the supplemental POSTGRES.txt on setting up PostgresQL 9.5 or higher as
+a reference database.
+
Please see the supplemental ORACLE.txt on setting up Oracle as a reference
database.