You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by lv...@apache.org on 2018/04/25 21:47:53 UTC
[1/2] impala git commit: add impala-flake8
Repository: impala
Updated Branches:
refs/heads/master b56ac8f0a -> 65bcb1608
add impala-flake8
Add flake8 and dependencies to impala-python. The versions are
compatible with Python 2.6.6. Add the impala-flake8 entry point, similar
to impala-python.
Add setup.cfg which defines flake8 special rules and exemptions. They
are added to support 2-space indents and a max line length of 90.
Contributors writing Python can use impala-flake8 to look for formatting
mistakes. The two most common uses would be:
impala-flake8 myfile.py
or
git diff HEAD^ myfile.py | impala-flake8 --diff
In the second usage, flake8 will only examine lines changed. This allows
a contributor to fix their own code and not be penalized by flake8
violations that may already be present (though they are encouraged to
fix them if they can!)
Change-Id: Ib4ce9eca6f8b55eaec1c96e7db1ff630ac016be0
Reviewed-on: http://gerrit.cloudera.org:8080/10182
Reviewed-by: Michael Brown <mi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/2fb73f94
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/2fb73f94
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/2fb73f94
Branch: refs/heads/master
Commit: 2fb73f94b425fde488166a19f78050ddbc3d7b50
Parents: b56ac8f
Author: Michael Brown <mi...@cloudera.com>
Authored: Mon Apr 23 13:37:20 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Wed Apr 25 20:52:42 2018 +0000
----------------------------------------------------------------------
bin/impala-flake8 | 21 +++++++++++++++++++++
infra/python/deps/requirements.txt | 4 ++++
setup.cfg | 22 ++++++++++++++++++++++
3 files changed, 47 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/2fb73f94/bin/impala-flake8
----------------------------------------------------------------------
diff --git a/bin/impala-flake8 b/bin/impala-flake8
new file mode 100755
index 0000000..f1e905e
--- /dev/null
+++ b/bin/impala-flake8
@@ -0,0 +1,21 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+source "$(dirname "$0")/impala-python-common.sh"
+exec "$PY_DIR/env/bin/flake8" "$@"
http://git-wip-us.apache.org/repos/asf/impala/blob/2fb73f94/infra/python/deps/requirements.txt
----------------------------------------------------------------------
diff --git a/infra/python/deps/requirements.txt b/infra/python/deps/requirements.txt
index 06868f8..b0b7f23 100644
--- a/infra/python/deps/requirements.txt
+++ b/infra/python/deps/requirements.txt
@@ -34,6 +34,10 @@ boto3 == 1.2.3
cm-api == 10.0.0
# Already available as part of python on Linux.
readline == 6.2.4.1; sys_platform == 'darwin'
+flake8 == 2.6.2
+ mccabe == 0.3.1
+ pycodestyle == 2.0.0
+ pyflakes == 1.2.3
Flask == 0.10.1
Jinja2 == 2.8
MarkupSafe == 0.23
http://git-wip-us.apache.org/repos/asf/impala/blob/2fb73f94/setup.cfg
----------------------------------------------------------------------
diff --git a/setup.cfg b/setup.cfg
new file mode 100644
index 0000000..a3178ba
--- /dev/null
+++ b/setup.cfg
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+[flake8]
+# These are ignored in order to support 2-space indents.
+# https://pycodestyle.readthedocs.io/en/latest/intro.html#error-codes
+ignore = E111,E114
+# Change from default to conform to Apache Impala line length.
+max-line-length = 90
[2/2] impala git commit: IMPALA-6889: Fix race around terminating
processes
Posted by lv...@apache.org.
IMPALA-6889: Fix race around terminating processes
A process with empty cmdline can trip up the Process class in our test
code. When a process dies it becomes a zombie until it gets reaped by
its parent.
>From 'man proc':
/proc/[pid]/cmdline
This read-only file holds the complete command line for the
process, unless the process is a zombie. In the latter case,
there is nothing in this file: that is, a read on this file
will return 0 characters. The command-line arguments appear
in this file as a set of strings separated by null bytes
('\0'), with a further null byte after the last string.
To fix this, we take a copy of the cmdline before passing it to the
Process ctor to prevent it from changing.
I couldn't come up with a test for this.
Change-Id: Iecf630e0b71d91469650636e81f940a7bec07113
Reviewed-on: http://gerrit.cloudera.org:8080/10156
Reviewed-by: Michael Brown <mi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/65bcb160
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/65bcb160
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/65bcb160
Branch: refs/heads/master
Commit: 65bcb1608d0e24603dfd8726a5c6e59e4682717c
Parents: 2fb73f9
Author: Lars Volker <lv...@cloudera.com>
Authored: Mon Apr 23 11:22:39 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Wed Apr 25 21:19:22 2018 +0000
----------------------------------------------------------------------
tests/common/impala_cluster.py | 19 +++++++++++++------
1 file changed, 13 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/65bcb160/tests/common/impala_cluster.py
----------------------------------------------------------------------
diff --git a/tests/common/impala_cluster.py b/tests/common/impala_cluster.py
index 276c02b..315666b 100644
--- a/tests/common/impala_cluster.py
+++ b/tests/common/impala_cluster.py
@@ -109,12 +109,19 @@ class ImpalaCluster(object):
if "uid not found" in str(e):
continue
raise
- if process.name == 'impalad' and len(process.cmdline) >= 1:
- impalads.append(ImpaladProcess(process.cmdline))
- elif process.name == 'statestored' and len(process.cmdline) >= 1:
- statestored.append(StateStoreProcess(process.cmdline))
- elif process.name == 'catalogd' and len(process.cmdline) >= 1:
- catalogd = CatalogdProcess(process.cmdline)
+ # IMPALA-6889: When a process shuts down and becomes a zombie its cmdline becomes
+ # empty for a brief moment, before it gets reaped by its parent (see man proc). We
+ # copy the cmdline to prevent it from changing between the following checks and
+ # the construction of the *Process objects.
+ cmdline = process.cmdline
+ if len(cmdline) == 0:
+ continue
+ if process.name == 'impalad':
+ impalads.append(ImpaladProcess(cmdline))
+ elif process.name == 'statestored':
+ statestored.append(StateStoreProcess(cmdline))
+ elif process.name == 'catalogd':
+ catalogd = CatalogdProcess(cmdline)
except psutil.NoSuchProcess, e:
# A process from get_pid_list() no longer exists, continue.
LOG.info(e)