You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kibble.apache.org by GitBox <gi...@apache.org> on 2020/05/01 21:53:06 UTC
[GitHub] [kibble-scanners] ncg81 opened a new issue #3: Difficulty setting-up Kibble Scanners
ncg81 opened a new issue #3:
URL: https://github.com/apache/kibble-scanners/issues/3
Hello,
I've been able to set-up the **Kibble Server** and **Elastic Search** on 2 different Ubuntu Machines and a Windows 10 machine by closely following the [docs](https://apache-kibble.readthedocs.io/en/latest/setup.html), however I am experiencing a no-joy moment setting-up **kibble-scanners**.
- **Test Machine A** : `Ubuntu 20.04 LTS` , `Python 3.8.2`, `pip 20.0.2`, `gcc 9.3.0`
1. First issue, the **bttf** build fails:
```
cos@ubuntu:~/temp/kibble-scanners$ pip3 install -r requirements.txt
Collecting bttf
Using cached bttf-0.1.tar.gz (11 kB)
Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from -r requirements.txt (line 2)) (2019.11.28)
Requirement already satisfied: elasticsearch in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 3)) (7.6.0)
Requirement already satisfied: psutil in /home/cos/.local/lib/python3.8/site-packages (from -r requirements.txt (line 4)) (5.7.0)
Requirement already satisfied: python-dateutil in /usr/lib/python3/dist-packages (from -r requirements.txt (line 5)) (2.7.3)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from -r requirements.txt (line 6)) (2.22.0)
Requirement already satisfied: urllib3>=1.21.1 in /usr/lib/python3/dist-packages (from elasticsearch->-r requirements.txt (line 3)) (1.25.8)
Building wheels for collected packages: bttf
Building wheel for bttf (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"'; __file__='"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-pnzdwahd
cwd: /tmp/pip-install-z2997c34/bttf/
Complete output (37 lines):
/tmp/pip-install-z2997c34/bttf/setup.py:15: DistDeprecationWarning: Features are deprecated and will be removed in a future version. See https://github.com/pypa/setuptools/issues/65.
speedups = Feature(
/usr/lib/python3.8/distutils/core.py:108: DistDeprecationWarning: Features are deprecated and will be removed in a future version. See https://github.com/pypa/setuptools/issues/65.
_setup_distribution = dist = klass(attrs)
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/bttf
copying bttf/__init__.py -> build/lib.linux-x86_64-3.8/bttf
running egg_info
writing bttf.egg-info/PKG-INFO
writing dependency_links to bttf.egg-info/dependency_links.txt
writing top-level names to bttf.egg-info/top_level.txt
reading manifest file 'bttf.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'bttf.egg-info/SOURCES.txt'
copying bttf/_bttf.c -> build/lib.linux-x86_64-3.8/bttf
copying bttf/bytesformat.h -> build/lib.linux-x86_64-3.8/bttf
copying bttf/typehacks.h -> build/lib.linux-x86_64-3.8/bttf
running build_ext
building 'bttf._bttf' extension
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/bttf
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c bttf/_bttf.c -o build/temp.linux-x86_64-3.8/bttf/_bttf.o
In file included from bttf/_bttf.c:2:
bttf/bytesformat.h:5:1: error: static declaration of ‘_PyUnicode_FormatLong’ follows non-static declaration
5 | _PyUnicode_FormatLong(PyObject *val, int alt, int prec, int type)
| ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/python3.8/unicodeobject.h:1037,
from /usr/include/python3.8/Python.h:97,
from bttf/_bttf.c:1:
/usr/include/python3.8/cpython/unicodeobject.h:1218:23: note: previous declaration of ‘_PyUnicode_FormatLong’ was here
1218 | PyAPI_FUNC(PyObject*) _PyUnicode_FormatLong(PyObject *, int, int, int);
| ^~~~~~~~~~~~~~~~~~~~~
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for bttf
Running setup.py clean for bttf
Failed to build bttf
Installing collected packages: bttf
Running setup.py install for bttf ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"'; __file__='"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-9r9h__jy/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/cos/.local/include/python3.8/bttf
cwd: /tmp/pip-install-z2997c34/bttf/
Complete output (37 lines):
/tmp/pip-install-z2997c34/bttf/setup.py:15: DistDeprecationWarning: Features are deprecated and will be removed in a future version. See https://github.com/pypa/setuptools/issues/65.
speedups = Feature(
/usr/lib/python3.8/distutils/core.py:108: DistDeprecationWarning: Features are deprecated and will be removed in a future version. See https://github.com/pypa/setuptools/issues/65.
_setup_distribution = dist = klass(attrs)
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/bttf
copying bttf/__init__.py -> build/lib.linux-x86_64-3.8/bttf
running egg_info
writing bttf.egg-info/PKG-INFO
writing dependency_links to bttf.egg-info/dependency_links.txt
writing top-level names to bttf.egg-info/top_level.txt
reading manifest file 'bttf.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'bttf.egg-info/SOURCES.txt'
copying bttf/_bttf.c -> build/lib.linux-x86_64-3.8/bttf
copying bttf/bytesformat.h -> build/lib.linux-x86_64-3.8/bttf
copying bttf/typehacks.h -> build/lib.linux-x86_64-3.8/bttf
running build_ext
building 'bttf._bttf' extension
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/bttf
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c bttf/_bttf.c -o build/temp.linux-x86_64-3.8/bttf/_bttf.o
In file included from bttf/_bttf.c:2:
bttf/bytesformat.h:5:1: error: static declaration of ‘_PyUnicode_FormatLong’ follows non-static declaration
5 | _PyUnicode_FormatLong(PyObject *val, int alt, int prec, int type)
| ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/python3.8/unicodeobject.h:1037,
from /usr/include/python3.8/Python.h:97,
from bttf/_bttf.c:1:
/usr/include/python3.8/cpython/unicodeobject.h:1218:23: note: previous declaration of ‘_PyUnicode_FormatLong’ was here
1218 | PyAPI_FUNC(PyObject*) _PyUnicode_FormatLong(PyObject *, int, int, int);
| ^~~~~~~~~~~~~~~~~~~~~
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"'; __file__='"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-9r9h__jy/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/cos/.local/include/python3.8/bttf Check the logs for full command output.
```
2. And then:
```
cos@ubuntu:~/temp/kibble-scanners$ python3 src/kibble-scanner.py
[core]: Loaded plugins/scanners/git-sync v/0.1.2 (Sync plugin for Git repositories)
[core]: Loaded plugins/scanners/git-census v/0.1.0 (Census Scanner for Git)
[core]: Loaded plugins/scanners/git-sloc v/0.1.0 (SloC Counter for Git)
[core]: Loaded plugins/scanners/git-evolution v/0.1.0 (Git Evolution Scanner)
[core]: Loaded plugins/scanners/jira v/0.1.0 (Scanner for Atlassian JIRA)
[core]: Loaded plugins/scanners/ponymail v/0.1.0 (Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-tone v/0.1.0 (Tone/Mood Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-kpe v/0.1.0 (Key Phrase Extraction plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/pipermail v/0.1.0 (Scanner for GNU Mailman Pipermail)
[core]: Loaded plugins/scanners/github-issues v/0.1.0 (Scanner for GitHub Issues)
[core]: Loaded plugins/scanners/bugzilla v/0.1.0 (Scanner for BugZilla)
[core]: Loaded plugins/scanners/gerrit v/0.1.1 (Scanner for Gerrit Code Review)
[core]: Loaded plugins/scanners/jenkins v/0.1.0 (Scanner for Jenkins CI)
[core]: Loaded plugins/scanners/buildbot v/0.1.0 (Scanner for Buildbot)
[core]: Loaded plugins/scanners/travis v/0.1.0 (Scanner for Travis CI)
[core]: Loaded plugins/scanners/discourse v/0.1.0 (Scanner for Discourse Forums)
[core]: Kibble Scanner v/0.1.0 starting
src/kibble-scanner.py:122: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(open(CONFIG_FILE))
[core]: Loaded YAML config from conf/config.yaml
[core]: Using direct ElasticSearch broker model
[core]: Connecting to ElasticSearch database at localhost:9200...
[core]: Connected!
[core]: This is a type-less DB, expanding database names instead.
[core]: We're using ES >= 7.x, NO DOC_TYPE!
Traceback (most recent call last):
File "src/kibble-scanner.py", line 180, in <module>
main()
File "src/kibble-scanner.py", line 136, in main
for org in broker.organisations():
File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 339, in organisations
res = self.DB.search(
File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 90, in search
return self.ES.search(
File "/usr/local/lib/python3.8/dist-packages/elasticsearch/client/utils.py", line 92, in _wrapped
return func(*args, params=params, headers=headers, **kwargs)
TypeError: search() got an unexpected keyword argument '_source_include'
```
- **Test Machine B:** `Ubuntu 18.04.4 LTS`, `Python 3.6.9`, `pip 9.0.1`, `gcc 7.5.0`
1. First issue, the **bttf** build fails:
```
cos@ubuntu:~/temp/kibble-scanners$ pip3 install -r requirements.txt
Collecting bttf (from -r requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/e0/33/adb62781dcbc9124f0a58ca15d3aaa3bdb5968863e354a62bbd52ea8eb88/bttf-0.1.tar.gz
Collecting certifi (from -r requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/57/2b/26e37a4b034800c960a00c4e1b3d9ca5d7014e983e6e729e33ea2f36426c/certifi-2020.4.5.1-py2.py3-none-any.whl
Collecting elasticsearch (from -r requirements.txt (line 3))
Using cached https://files.pythonhosted.org/packages/cc/cf/7973ac58090b960857da04add0b345415bf1e1741beddf4cbe136b8ad174/elasticsearch-7.6.0-py2.py3-none-any.whl
Collecting psutil (from -r requirements.txt (line 4))
Collecting python-dateutil (from -r requirements.txt (line 5))
Using cached https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl
Collecting requests (from -r requirements.txt (line 6))
Using cached https://files.pythonhosted.org/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl
Collecting urllib3>=1.21.1 (from elasticsearch->-r requirements.txt (line 3))
Using cached https://files.pythonhosted.org/packages/e1/e5/df302e8017440f111c11cc41a6b432838672f5a70aa29227bf58149dc72f/urllib3-1.25.9-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil->-r requirements.txt (line 5))
Using cached https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl
Collecting chardet<4,>=3.0.2 (from requests->-r requirements.txt (line 6))
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<3,>=2.5 (from requests->-r requirements.txt (line 6))
Using cached https://files.pythonhosted.org/packages/89/e3/afebe61c546d18fb1709a61bee788254b40e736cff7271c7de5de2dc4128/idna-2.9-py2.py3-none-any.whl
Building wheels for collected packages: bttf
Running setup.py bdist_wheel for bttf ... error
Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-mno2wh38/bttf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp3sczfwj7pip-wheel- --python-tag cp36:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/bttf
copying bttf/__init__.py -> build/lib.linux-x86_64-3.6/bttf
running egg_info
writing bttf.egg-info/PKG-INFO
writing dependency_links to bttf.egg-info/dependency_links.txt
writing top-level names to bttf.egg-info/top_level.txt
reading manifest file 'bttf.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'bttf.egg-info/SOURCES.txt'
copying bttf/_bttf.c -> build/lib.linux-x86_64-3.6/bttf
copying bttf/bytesformat.h -> build/lib.linux-x86_64-3.6/bttf
copying bttf/typehacks.h -> build/lib.linux-x86_64-3.6/bttf
running build_ext
building 'bttf._bttf' extension
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/bttf
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c bttf/_bttf.c -o build/temp.linux-x86_64-3.6/bttf/_bttf.o
In file included from bttf/_bttf.c:2:0:
bttf/bytesformat.h:5:1: error: static declaration of ‘_PyUnicode_FormatLong’ follows non-static declaration
_PyUnicode_FormatLong(PyObject *val, int alt, int prec, int type)
^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/python3.6m/Python.h:89:0,
from bttf/_bttf.c:1:
/usr/include/python3.6m/unicodeobject.h:2307:23: note: previous declaration of ‘_PyUnicode_FormatLong’ was here
PyAPI_FUNC(PyObject*) _PyUnicode_FormatLong(PyObject *, int, int, int);
^~~~~~~~~~~~~~~~~~~~~
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Failed building wheel for bttf
Running setup.py clean for bttf
Failed to build bttf
Installing collected packages: bttf, certifi, urllib3, elasticsearch, psutil, six, python-dateutil, chardet, idna, requests
Running setup.py install for bttf ... error
Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-mno2wh38/bttf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-gge6tlz0-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/bttf
copying bttf/__init__.py -> build/lib.linux-x86_64-3.6/bttf
running egg_info
writing bttf.egg-info/PKG-INFO
writing dependency_links to bttf.egg-info/dependency_links.txt
writing top-level names to bttf.egg-info/top_level.txt
reading manifest file 'bttf.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'bttf.egg-info/SOURCES.txt'
copying bttf/_bttf.c -> build/lib.linux-x86_64-3.6/bttf
copying bttf/bytesformat.h -> build/lib.linux-x86_64-3.6/bttf
copying bttf/typehacks.h -> build/lib.linux-x86_64-3.6/bttf
running build_ext
building 'bttf._bttf' extension
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/bttf
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c bttf/_bttf.c -o build/temp.linux-x86_64-3.6/bttf/_bttf.o
In file included from bttf/_bttf.c:2:0:
bttf/bytesformat.h:5:1: error: static declaration of ‘_PyUnicode_FormatLong’ follows non-static declaration
_PyUnicode_FormatLong(PyObject *val, int alt, int prec, int type)
^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/python3.6m/Python.h:89:0,
from bttf/_bttf.c:1:
/usr/include/python3.6m/unicodeobject.h:2307:23: note: previous declaration of ‘_PyUnicode_FormatLong’ was here
PyAPI_FUNC(PyObject*) _PyUnicode_FormatLong(PyObject *, int, int, int);
^~~~~~~~~~~~~~~~~~~~~
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-mno2wh38/bttf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-gge6tlz0-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-mno2wh38/bttf/
```
2. And then:
```
cos@ubuntu:~/temp/kibble-scanners$ python3 src/kibble-scanner.py
[core]: Loaded plugins/scanners/git-sync v/0.1.2 (Sync plugin for Git repositories)
[core]: Loaded plugins/scanners/git-census v/0.1.0 (Census Scanner for Git)
[core]: Loaded plugins/scanners/git-sloc v/0.1.0 (SloC Counter for Git)
[core]: Loaded plugins/scanners/git-evolution v/0.1.0 (Git Evolution Scanner)
/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.25.9) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
[core]: Loaded plugins/scanners/jira v/0.1.0 (Scanner for Atlassian JIRA)
[core]: Loaded plugins/scanners/ponymail v/0.1.0 (Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-tone v/0.1.0 (Tone/Mood Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-kpe v/0.1.0 (Key Phrase Extraction plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/pipermail v/0.1.0 (Scanner for GNU Mailman Pipermail)
[core]: Loaded plugins/scanners/github-issues v/0.1.0 (Scanner for GitHub Issues)
[core]: Loaded plugins/scanners/bugzilla v/0.1.0 (Scanner for BugZilla)
[core]: Loaded plugins/scanners/gerrit v/0.1.1 (Scanner for Gerrit Code Review)
[core]: Loaded plugins/scanners/jenkins v/0.1.0 (Scanner for Jenkins CI)
[core]: Loaded plugins/scanners/buildbot v/0.1.0 (Scanner for Buildbot)
[core]: Loaded plugins/scanners/travis v/0.1.0 (Scanner for Travis CI)
[core]: Loaded plugins/scanners/discourse v/0.1.0 (Scanner for Discourse Forums)
[core]: Kibble Scanner v/0.1.0 starting
src/kibble-scanner.py:122: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(open(CONFIG_FILE))
[core]: Loaded YAML config from conf/config.yaml
[core]: Using direct ElasticSearch broker model
[core]: Connecting to ElasticSearch database at localhost:9200...
[core]: Connected!
[core]: This is a type-less DB, expanding database names instead.
[core]: We're using ES >= 7.x, NO DOC_TYPE!
Traceback (most recent call last):
File "src/kibble-scanner.py", line 180, in <module>
main()
File "src/kibble-scanner.py", line 136, in main
for org in broker.organisations():
File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 345, in organisations
'match_all': {}
File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 94, in search
body = body
File "/home/cos/.local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 92, in _wrapped
return func(*args, params=params, headers=headers, **kwargs)
TypeError: search() got an unexpected keyword argument '_source_include'
```
Any ideas?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble-scanners] ncg81 commented on issue #3: Difficulty setting-up Kibble Scanners
Posted by GitBox <gi...@apache.org>.
ncg81 commented on issue #3:
URL: https://github.com/apache/kibble-scanners/issues/3#issuecomment-622957506
Here's what I get now:
```
cos@ubuntu:~/temp/kibble-scanners$ python3 src/kibble-scanner.py
[core]: Loaded plugins/scanners/git-sync v/0.1.2 (Sync plugin for Git repositories)
[core]: Loaded plugins/scanners/git-census v/0.1.0 (Census Scanner for Git)
[core]: Loaded plugins/scanners/git-sloc v/0.1.0 (SloC Counter for Git)
[core]: Loaded plugins/scanners/git-evolution v/0.1.0 (Git Evolution Scanner)
[core]: Loaded plugins/scanners/jira v/0.1.0 (Scanner for Atlassian JIRA)
[core]: Loaded plugins/scanners/ponymail v/0.1.0 (Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-tone v/0.1.0 (Tone/Mood Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-kpe v/0.1.0 (Key Phrase Extraction plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/pipermail v/0.1.0 (Scanner for GNU Mailman Pipermail)
[core]: Loaded plugins/scanners/github-issues v/0.1.0 (Scanner for GitHub Issues)
[core]: Loaded plugins/scanners/bugzilla v/0.1.0 (Scanner for BugZilla)
[core]: Loaded plugins/scanners/gerrit v/0.1.1 (Scanner for Gerrit Code Review)
[core]: Loaded plugins/scanners/jenkins v/0.1.0 (Scanner for Jenkins CI)
[core]: Loaded plugins/scanners/buildbot v/0.1.0 (Scanner for Buildbot)
[core]: Loaded plugins/scanners/travis v/0.1.0 (Scanner for Travis CI)
[core]: Loaded plugins/scanners/discourse v/0.1.0 (Scanner for Discourse Forums)
[core]: Kibble Scanner v/0.1.0 starting
src/kibble-scanner.py:122: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(open(CONFIG_FILE))
[core]: Loaded YAML config from conf/config.yaml
[core]: Using direct ElasticSearch broker model
[core]: Connecting to ElasticSearch database at localhost:9200...
[core]: Connected!
[core]: This is a type-less DB, expanding database names instead.
[core]: We're using ES >= 7.x, NO DOC_TYPE!
Traceback (most recent call last):
File "src/kibble-scanner.py", line 180, in <module>
main()
File "src/kibble-scanner.py", line 129, in main
broker = plugins.brokers.kibbleES.Broker(config)
File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 318, in __init__
apidoc = es.get(index=es_config['database'], doc_type='api', id = 'current')['_source']
File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 79, in get
return self.ES.get(index = index+'_'+doc_type, id = id)
File "/home/cos/.local/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
TypeError: get() missing 1 required positional argument: 'doc_type'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble-scanners] ncg81 commented on issue #3: Difficulty setting-up Kibble Scanners
Posted by GitBox <gi...@apache.org>.
ncg81 commented on issue #3:
URL: https://github.com/apache/kibble-scanners/issues/3#issuecomment-622957760
`[core]: We're using ES >= 7.x, NO DOC_TYPE!`
`TypeError: get() missing 1 required positional argument: 'doc_type'`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble-scanners] Humbedooh commented on issue #3: Difficulty setting-up Kibble Scanners
Posted by GitBox <gi...@apache.org>.
Humbedooh commented on issue #3:
URL: https://github.com/apache/kibble-scanners/issues/3#issuecomment-622960551
Okay, so clearly an issue with the elasticsearch library then.
Please try upgrading it to >7: `pip3 install "elasticsearch>7<8"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble-scanners] ncg81 commented on issue #3: Difficulty setting-up Kibble Scanners
Posted by GitBox <gi...@apache.org>.
ncg81 commented on issue #3:
URL: https://github.com/apache/kibble-scanners/issues/3#issuecomment-622951453
It turns out I was running with elasticsearch module version `7.6.0` . After downgrading it to version `6.8.1` I'm getting `Invalid or missing API/ABI version in database kibble!`
```
cos@ubuntu:~/temp/kibble-scanners$ curl -X GET "localhost:9200/?pretty"
{
"name" : "ubuntu",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "6DoJlyXeQdqaGFQRMib71g",
"version" : {
"number" : "7.6.2",
"build_flavor" : "oss",
"build_type" : "deb",
"build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
"build_date" : "2020-03-26T06:34:37.794943Z",
"build_snapshot" : false,
"lucene_version" : "8.4.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
```
```
cos@ubuntu:/var/www/kibble/setup$ sudo python3 setup.py
...
...
...
Okay, I got all I need, setting up Kibble...
Creating index kibble_api
Creating index kibble_ci_build
Creating index kibble_ci_queue
Creating index kibble_code_commit
Creating index kibble_code_commit_unique
Creating index kibble_code_modification
Creating index kibble_evolution
Creating index kibble_file_history
Creating index kibble_forum_post
Creating index kibble_forum_topic
Creating index kibble_ghstats
Creating index kibble_im_stats
Creating index kibble_im_ops
Creating index kibble_im_msg
Creating index kibble_issue
Creating index kibble_logstats
Creating index kibble_email
Creating index kibble_mailstats
Creating index kibble_mailtop
Creating index kibble_organisation
Creating index kibble_view
Creating index kibble_publish
Creating index kibble_source
Creating index kibble_stats
Creating index kibble_social_follow
Creating index kibble_social_followers
Creating index kibble_social_follower
Creating index kibble_social_person
Creating index kibble_uisession
Creating index kibble_useraccount
Creating index kibble_message
Creating index kibble_person
Indices created! {'acknowledged': True, 'shards_acknowledged': True, 'index': 'kibble_person'}
Creating administrator account
Account created!
../api/yaml/kibble.yaml already exists! Writing to ../api/yaml/kibble.yaml.tmp instead
Writing Kibble config (../api/yaml/kibble.yaml.tmp)
All done, Kibble should...work now :)
```
```
cos@ubuntu:/var/www/kibble$ curl http://localhost:9200/_cat/indices
yellow open kibble_social_follower 6-dHZ1JhRVmJNBn8gkRVxw 5 1 0 0 1.3kb 1.3kb
yellow open kibble_file_history KTRE2H7dTS-YeZiQh9v3Lw 5 1 0 0 1.3kb 1.3kb
yellow open kibble_source THZYCPoDRCuIpN5uYsOjbA 5 1 1 0 5.9kb 5.9kb
yellow open kibble_view y7u56OaCQImYuMIiCo6jBQ 5 1 0 0 1.3kb 1.3kb
yellow open kibble_social_followers 3mqxh2PeQvOugMNmnY2cAA 5 1 0 0 1.3kb 1.3kb
yellow open kibble_mailtop R0dLPudlQfar_yX8B758tg 5 1 0 0 1.3kb 1.3kb
yellow open kibble_im_msg orKAoiVUTIGp-LRHuB9ppA 5 1 0 0 1.3kb 1.3kb
yellow open kibble_mailstats 9qasGXX3QSmTTHDNSxbxDQ 5 1 0 0 1.3kb 1.3kb
yellow open kibble_evolution 9tF5HSy9QRiIQcj3VKvx5Q 5 1 0 0 1.3kb 1.3kb
yellow open kibble_code_modification sH5BBhOZSMO5GHohVn-_gw 5 1 0 0 1.3kb 1.3kb
yellow open kibble_logstats KWre3KjYQMey0k8lwRurYg 5 1 0 0 1.3kb 1.3kb
yellow open kibble_api rkubLmLKQFaAszjxMpIj_g 5 1 1 0 4.8kb 4.8kb
yellow open kibble_code_commit P4WYrMzYQKCY-kOq3bn4IQ 5 1 0 0 1.3kb 1.3kb
yellow open kibble_im_stats QLu91uZMSRmMbe_YvlMwXg 5 1 0 0 1.3kb 1.3kb
yellow open kibble_ci_build 2DD2ul39R0uEeyX6Gt7AHQ 5 1 0 0 1.3kb 1.3kb
yellow open kibble_forum_topic -7bMi_PqRvGOsCumPj-ERw 5 1 0 0 1.3kb 1.3kb
yellow open kibble_ghstats W6Xp09CgTFualUkQShRbGA 5 1 0 0 1.3kb 1.3kb
yellow open kibble_useraccount EwbozIvrTrWST1oODz0yvA 5 1 1 0 7.6kb 7.6kb
yellow open kibble_ci_queue _QDyVycES4iV06YiRDfVdQ 5 1 0 0 1.3kb 1.3kb
yellow open kibble_social_follow v36Z9dBbQPqoQj2w_E2z4A 5 1 0 0 1.3kb 1.3kb
yellow open kibble_email Xlf_A-JDRViul8xvcHxLug 5 1 0 0 1.3kb 1.3kb
yellow open kibble_publish JHVAlU6cSLa9sjBH2AQ09A 5 1 0 0 1.3kb 1.3kb
yellow open kibble_message pZXzm3k1QvqGPW0uAvCJLA 5 1 0 0 1.3kb 1.3kb
yellow open kibble_organisation Kt7tqlvSRi-dRXfP0kDbhg 5 1 1 0 5.6kb 5.6kb
yellow open kibble_person yfuI2fGaRea7zmcAJpndoA 5 1 0 0 1.3kb 1.3kb
yellow open kibble_stats djtOMcM0QW6AAiXvpofHXA 5 1 0 0 1.3kb 1.3kb
yellow open kibble_forum_post Cbt5WkALQ86aRN5WsAtZMw 5 1 0 0 1.3kb 1.3kb
yellow open kibble_im_ops 37XncVh8QyiYPN8Tk9i-Bg 5 1 0 0 1.3kb 1.3kb
yellow open kibble_issue ou5-56zeSg-tG8pOBeRolw 5 1 0 0 1.3kb 1.3kb
yellow open kibble_social_person XCrdhLaRR86miG7p9iBzMw 5 1 0 0 1.3kb 1.3kb
yellow open kibble_uisession Bk6zyBIuR9C8QTUotolDQw 5 1 4 0 16.9kb 16.9kb
yellow open kibble_code_commit_unique TQJGIEp6T0mlqI4N91CBVA 5 1 0 0 1.3kb 1.3kb
```
And then:
```
cos@ubuntu:~/temp/kibble-scanners$ python3 src/kibble-scanner.py
[core]: Loaded plugins/scanners/git-sync v/0.1.2 (Sync plugin for Git repositories)
[core]: Loaded plugins/scanners/git-census v/0.1.0 (Census Scanner for Git)
[core]: Loaded plugins/scanners/git-sloc v/0.1.0 (SloC Counter for Git)
[core]: Loaded plugins/scanners/git-evolution v/0.1.0 (Git Evolution Scanner)
[core]: Loaded plugins/scanners/jira v/0.1.0 (Scanner for Atlassian JIRA)
[core]: Loaded plugins/scanners/ponymail v/0.1.0 (Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-tone v/0.1.0 (Tone/Mood Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-kpe v/0.1.0 (Key Phrase Extraction plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/pipermail v/0.1.0 (Scanner for GNU Mailman Pipermail)
[core]: Loaded plugins/scanners/github-issues v/0.1.0 (Scanner for GitHub Issues)
[core]: Loaded plugins/scanners/bugzilla v/0.1.0 (Scanner for BugZilla)
[core]: Loaded plugins/scanners/gerrit v/0.1.1 (Scanner for Gerrit Code Review)
[core]: Loaded plugins/scanners/jenkins v/0.1.0 (Scanner for Jenkins CI)
[core]: Loaded plugins/scanners/buildbot v/0.1.0 (Scanner for Buildbot)
[core]: Loaded plugins/scanners/travis v/0.1.0 (Scanner for Travis CI)
[core]: Loaded plugins/scanners/discourse v/0.1.0 (Scanner for Discourse Forums)
[core]: Kibble Scanner v/0.1.0 starting
src/kibble-scanner.py:122: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(open(CONFIG_FILE))
[core]: Loaded YAML config from conf/config.yaml
[core]: Using direct ElasticSearch broker model
[core]: Connecting to ElasticSearch database at localhost:9200...
[core]: Connected!
[core]: This is a type-less DB, expanding database names instead.
[core]: We're using ES >= 7.x, NO DOC_TYPE!
Invalid or missing API/ABI version in database kibble! Please ensure the database has been primed by setup.py
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble-scanners] Humbedooh commented on issue #3: Difficulty setting-up Kibble Scanners
Posted by GitBox <gi...@apache.org>.
Humbedooh commented on issue #3:
URL: https://github.com/apache/kibble-scanners/issues/3#issuecomment-622953110
What is the output of `curl http://localhost:9200/kibble_api/current?pretty` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble-scanners] Humbedooh commented on issue #3: Difficulty setting-up Kibble Scanners
Posted by GitBox <gi...@apache.org>.
Humbedooh commented on issue #3:
URL: https://github.com/apache/kibble-scanners/issues/3#issuecomment-622953893
the `/_doc` bit may be optional depending on the version of ES, not sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble-scanners] Humbedooh commented on issue #3: Difficulty setting-up Kibble Scanners
Posted by GitBox <gi...@apache.org>.
Humbedooh commented on issue #3:
URL: https://github.com/apache/kibble-scanners/issues/3#issuecomment-622956043
FWIW, the DB is the correct version, but there seems to be an issue with fetching the db document for tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble-scanners] Humbedooh edited a comment on issue #3: Difficulty setting-up Kibble Scanners
Posted by GitBox <gi...@apache.org>.
Humbedooh edited a comment on issue #3:
URL: https://github.com/apache/kibble-scanners/issues/3#issuecomment-622960551
Okay, so clearly an issue with the elasticsearch library then.
Please try upgrading it to >7: `pip3 install "elasticsearch>7<8"`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble-scanners] Humbedooh commented on issue #3: Difficulty setting-up Kibble Scanners
Posted by GitBox <gi...@apache.org>.
Humbedooh commented on issue #3:
URL: https://github.com/apache/kibble-scanners/issues/3#issuecomment-622960685
ES client 7.x should work now with yesterday's tweaks..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble-scanners] Humbedooh edited a comment on issue #3: Difficulty setting-up Kibble Scanners
Posted by GitBox <gi...@apache.org>.
Humbedooh edited a comment on issue #3:
URL: https://github.com/apache/kibble-scanners/issues/3#issuecomment-622953110
What is the output of `curl http://localhost:9200/kibble_api/_doc/current?pretty` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble-scanners] Humbedooh commented on issue #3: Difficulty setting-up Kibble Scanners
Posted by GitBox <gi...@apache.org>.
Humbedooh commented on issue #3:
URL: https://github.com/apache/kibble-scanners/issues/3#issuecomment-622586414
For the first issue, I think you can safely remove bttf from the requirements.txt, things should work without it. I'll see if we can just remove that altogether soon.
For the second issue, could you please show me the output of `pip3 show elasticsearch` ?
It sounds like the ElasticSearch pip module might be an incompatible version.
If you see a >= 7.0 version here, please try: `pip3 install "elasticsearch<7"` which will downgrade your elasticsearch client to an older version.
I'll look at whether we can dispense with the source_include altogether as well...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble-scanners] ncg81 commented on issue #3: Difficulty setting-up Kibble Scanners
Posted by GitBox <gi...@apache.org>.
ncg81 commented on issue #3:
URL: https://github.com/apache/kibble-scanners/issues/3#issuecomment-622962952
Everything seems peachy. Thank you @Humbedooh . :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble-scanners] ncg81 commented on issue #3: Difficulty setting-up Kibble Scanners
Posted by GitBox <gi...@apache.org>.
ncg81 commented on issue #3:
URL: https://github.com/apache/kibble-scanners/issues/3#issuecomment-622954464
```
cos@ubuntu:/var/www/kibble/setup$ curl http://localhost:9200/kibble_api/_doc/current?pretty
{
"_index" : "kibble_api",
"_type" : "_doc",
"_id" : "current",
"_version" : 1,
"_seq_no" : 0,
"_primary_term" : 1,
"found" : true,
"_source" : {
"apiversion" : "0.1.0",
"dbversion" : 2
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble-scanners] Humbedooh commented on issue #3: Difficulty setting-up Kibble Scanners
Posted by GitBox <gi...@apache.org>.
Humbedooh commented on issue #3:
URL: https://github.com/apache/kibble-scanners/issues/3#issuecomment-622955981
Could you try pulling in the last change in src/plugins/kibbleES.py and see what error it spits out when you run now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org