You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kibble.apache.org by GitBox <gi...@apache.org> on 2020/11/12 04:28:06 UTC
[GitHub] [kibble] skekre98 opened a new pull request #91: Kibble initial cli
skekre98 opened a new pull request #91:
URL: https://github.com/apache/kibble/pull/91
Hi, really sorry about the delay. I was able to completely migrate the setup workflow to be implemented with `click`. I am running into a minor error while running `create_es_index` which I don't believe has anything to do with the new cli implementation. Was wondering if I could get some assistance with figuring out the issue as I am a bit new to ElasticSearch 😅?
Seems I am receiving a connection error when calling elasticsearch:
```bash
elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x111424fd0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x111424fd0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known)
```
The actual cli commands and arguments are working now though 👍
```bash
$ kibble version
1.0.0dev
$ kibble setup --help
Usage: kibble setup [OPTIONS]
Options:
-u, --uri TEXT Pre-defined connection uri for ElasticSearch.
-d, --dbname TEXT Pre-defined Database prefix.
-s, --shards TEXT Predefined number of ES shards.
-r, --replicas TEXT Predefined number of replicas for ES.
-m, --mailhost TEXT Pre-defined mail server host.
-a, --autoadmin TEXT Generate generic admin account.
-k, --skiponexist TEXT Skip DB creation if DBs exist.
--help Show this message and exit.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] turbaszek commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523443244
##########
File path: kibble/__main__.py
##########
@@ -15,9 +15,83 @@
# specific language governing permissions and limitations
# under the License.
+import click
+
+from kibble.cli import setup_command
+from kibble.version import version as kv
+
+from kibble.configuration import conf
+
+
+@click.group()
+def cli():
+ """A simple command line tool for kibble"""
+
+
+@cli.command("version", short_help="displays the current kibble version")
+def version():
+ click.echo(kv)
+
+
+@cli.command("setup", short_help="starts the setup process for kibble")
+@click.option(
+ "-u",
+ "--uri",
+ default=conf.get("elasticsearch", "conn_uri"),
+ help="Connection uri for ElasticSearch.",
+)
+@click.option(
+ "-d",
+ "--dbname",
+ default=conf.get("elasticsearch", "dbname"),
+ help="Pre-defined Database prefix.",
+)
+@click.option(
+ "-s",
+ "--shards",
+ default=conf.get("elasticsearch", "shards"),
+ help="Predefined number of ES shards.",
+)
+@click.option(
+ "-r",
+ "--replicas",
+ default=conf.get("elasticsearch", "replicas"),
+ help="Predefined number of replicas for ES.",
+)
+@click.option(
+ "-m",
+ "--mailhost",
+ default=conf.get("mail", "mailhost"),
+ help="Pre-defined mail server host.",
+)
+@click.option(
+ "-a", "--autoadmin", default=False, help="Generate generic admin account."
+)
+@click.option(
+ "-k", "--skiponexist", default=True, help="Skip DB creation if DBs exist."
Review comment:
```suggestion
"-k", "--skiponexist", default=True, help="kip DB creation if DBs exist"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] turbaszek commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r522443048
##########
File path: kibble/__main__.py
##########
@@ -15,9 +15,252 @@
# specific language governing permissions and limitations
# under the License.
+import sys
+import os
+import argparse
+import logging
+import click
+from getpass import getpass
+
+import tenacity
+import bcrypt
+import json
+from elasticsearch import Elasticsearch
+
+from kibble.version import version as kv
+from kibble.configuration import conf
+
+KIBBLE_VERSION = conf.get("api", "version")
+KIBBLE_DB_VERSION = conf.get("api", "database")
+
+
+def get_user_input(msg: str, secure: bool = False):
+ value = None
+ while not value:
+ value = getpass(msg) if secure else input(msg)
+ return value
+
+
+@click.group()
+def cli():
+ """A simple command line tool for kibble"""
+
+
+@cli.command("version", short_help="displays the current kibble version")
+def version():
+ click.echo(kv)
+
+
+def create_es_index(
+ conn_uri: str,
+ dbname: str,
+ shards: int,
+ replicas: int,
+ admin_name: str,
+ admin_pass: str,
+ skiponexist: bool,
+):
+ """Creates Elasticsearch index used by Kibble"""
+
+ # elasticsearch logs lots of warnings on retries/connection failure
+ logging.getLogger("elasticsearch").setLevel(logging.ERROR)
+
+ mappings_json = os.path.join(
+ os.path.dirname(os.path.realpath(__file__)), "setup/mappings.json"
+ )
+ with open(mappings_json, "r") as f:
+ mappings = json.load(f)
+
+ es = Elasticsearch([conn_uri], max_retries=5, retry_on_timeout=True)
+ print(es.info())
+
+ es_version = es.info()["version"]["number"]
+ es6 = int(es_version.split(".")[0]) >= 6
+ es7 = int(es_version.split(".")[0]) >= 7
+
+ if not es6:
+ print(
+ f"New Kibble installations require ElasticSearch 6.x or newer! "
+ f"You appear to be running {es_version}!"
+ )
+ sys.exit(-1)
+
+ # If ES >= 7, _doc is invalid and mapping should be rooted
+ if es7:
+ mappings["mappings"] = mappings["mappings"]["_doc"]
+
+ # Check if index already exists
+ if es.indices.exists(dbname + "_api"):
+ # Skip this is DB exists and -k added
+ if skiponexist:
+ print("DB prefix exists, but --skiponexist used, skipping this step.")
+ return
+ print("Error: ElasticSearch DB prefix '%s' already exists!" % dbname)
+ sys.exit(-1)
+
+ types = [
+ "api",
+ # ci_*: CI service stats
+ "ci_build",
+ "ci_queue",
+ # code_* + evolution + file_history: git repo stats
+ "code_commit",
+ "code_commit_unique",
+ "code_modification",
+ "evolution",
+ "file_history",
+ # forum_*: forum stats (SO, Discourse, Askbot etc)
+ "forum_post",
+ "forum_topic",
+ # GitHub stats
+ "ghstats",
+ # im_*: Instant messaging stats
+ "im_stats",
+ "im_ops",
+ "im_msg",
+ "issue",
+ "logstats",
+ # email, mail*: Email statitics
+ "email",
+ "mailstats",
+ "mailtop",
+ # organisation, view, source, publish: UI Org DB
+ "organisation",
+ "view",
+ "publish",
+ "source",
+ # stats: Miscellaneous stats
+ "stats",
+ # social_*: Twitter, Mastodon, Facebook etc
+ "social_follow",
+ "social_followers",
+ "social_follower",
+ "social_person",
+ # uisession, useraccount, message: UI user DB
+ "uisession",
+ "useraccount",
+ "message",
+ # person: contributor DB
+ "person",
+ ]
+
+ for t in types:
+ iname = f"{dbname}_{t}"
+ print(f"Creating index {iname}")
+
+ settings = {"number_of_shards": shards, "number_of_replicas": replicas}
+ es.indices.create(
+ index=iname, body={"mappings": mappings["mappings"], "settings": settings}
+ )
+ print(f"Indices created!")
+ print()
+
+ salt = bcrypt.gensalt()
+ pwd = bcrypt.hashpw(admin_pass.encode("utf-8"), salt).decode("ascii")
+ print("Creating administrator account")
+ doc = {
+ "email": admin_name, # Username (email)
+ "password": pwd, # Hashed password
+ "displayName": "Administrator", # Display Name
+ "organisations": [], # Orgs user belongs to (default is none)
+ "ownerships": [], # Orgs user owns (default is none)
+ "defaultOrganisation": None, # Default org for user
+ "verified": True, # Account verified via email?
+ "userlevel": "admin", # User level (user/admin)
+ }
+ dbdoc = {
+ "apiversion": KIBBLE_VERSION, # Log current API version
+ "dbversion": KIBBLE_DB_VERSION, # Log the database revision we accept (might change!)
+ }
+ es.index(index=dbname + "_useraccount", doc_type="_doc", id=admin_name, body=doc)
+ es.index(index=dbname + "_api", doc_type="_doc", id="current", body=dbdoc)
+ print("Account created!")
+
+
+@cli.command("setup", short_help="starts the setup process for kibble")
+@click.option(
+ "-u",
+ "--uri",
+ default=conf.get("elasticsearch", "conn_uri"),
+ help="Pre-defined connection uri for ElasticSearch.",
+)
+@click.option(
+ "-d",
+ "--dbname",
+ default=conf.get("elasticsearch", "dbname"),
+ help="Pre-defined Database prefix.",
+)
+@click.option(
+ "-s",
+ "--shards",
+ default=conf.get("elasticsearch", "shards"),
+ help="Predefined number of ES shards.",
+)
+@click.option(
+ "-r",
+ "--replicas",
+ default=conf.get("elasticsearch", "replicas"),
+ help="Predefined number of replicas for ES.",
+)
+@click.option(
+ "-m",
+ "--mailhost",
+ default=conf.get("mail", "mailhost"),
+ help="Pre-defined mail server host.",
+)
+@click.option(
+ "-a", "--autoadmin", default=False, help="Generate generic admin account."
+)
+@click.option(
+ "-k", "--skiponexist", default=True, help="Skip DB creation if DBs exist."
+)
+def setup(
+ uri: str,
+ dbname: str,
+ shards: str,
+ replicas: str,
+ mailhost: str,
+ autoadmin: bool,
+ skiponexist: bool,
+):
Review comment:
In this way we will have many commands here that implements a lot of different logic. What I would propose is to move this function as well as `create_es_index` to something like `kibble/cli/setup_command.py` and then create a function `setup_command` and here do:
```py
# ... here goes all click.option
def setup(
uri: str,
dbname: str,
shards: str,
replicas: str,
mailhost: str,
autoadmin: bool,
skiponexist: bool,
):
setup_command(
uri=uri,
dbname=dbname,
shards=shards,
replicas=replicas,
mailhost=mailhost,
autoadmin=autoadmin,
skiponexist=skiponexist,
)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] turbaszek commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523422522
##########
File path: kibble/__main__.py
##########
@@ -15,9 +15,91 @@
# specific language governing permissions and limitations
# under the License.
+import click
+from getpass import getpass
+
+from kibble.cli import setup_command
+from kibble.version import version as kv
+
+from kibble.configuration import conf
+
+
+def get_user_input(msg: str, secure: bool = False):
+ value = None
+ while not value:
+ value = getpass(msg) if secure else input(msg)
+ return value
+
+
+@click.group()
+def cli():
+ """A simple command line tool for kibble"""
+
+
+@cli.command("version", short_help="displays the current kibble version")
+def version():
+ click.echo(kv)
+
+
+@cli.command("setup", short_help="starts the setup process for kibble")
+@click.option(
+ "-u",
+ "--uri",
+ default=conf.get("elasticsearch", "conn_uri"),
+ help="Pre-defined connection uri for ElasticSearch.",
Review comment:
```suggestion
help="connection uri for ElasticSearch.",
```
Let's remove this "pre-defined", in my opinion it looks weird in the `kibble setup --help`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] turbaszek commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523039825
##########
File path: kibble/cli/setup_command.py
##########
@@ -0,0 +1,197 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import sys
+import logging
+
+import click
+import tenacity
+import bcrypt
+import json
+from elasticsearch import Elasticsearch
+
+
+def create_es_index(
Review comment:
Btw if we have this logic here, we should probably delete `kibble/setup/setup.py`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] michalslowikowski00 commented on pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on pull request #91:
URL: https://github.com/apache/kibble/pull/91#issuecomment-726838070
> Hi, really sorry about the delay. I was able to completely migrate the setup workflow to be implemented with `click`. I am running into a minor error while running `create_es_index` which I don't believe has anything to do with the new cli implementation. Was wondering if I could get some assistance with figuring out the issue as I am a bit new to ElasticSearch 😅?
>
> Seems I am receiving a connection error when calling elasticsearch:
>
> ```shell
> elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x111424fd0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x111424fd0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known)
> ```
>
> The actual cli commands and arguments are working now though 👍
>
> ```shell
> $ kibble version
> 1.0.0dev
> $ kibble setup --help
> Usage: kibble setup [OPTIONS]
>
> Options:
> -u, --uri TEXT Pre-defined connection uri for ElasticSearch.
> -d, --dbname TEXT Pre-defined Database prefix.
> -s, --shards TEXT Predefined number of ES shards.
> -r, --replicas TEXT Predefined number of replicas for ES.
> -m, --mailhost TEXT Pre-defined mail server host.
> -a, --autoadmin TEXT Generate generic admin account.
> -k, --skiponexist TEXT Skip DB creation if DBs exist.
> --help Show this message and exit.
> ```
Easy, it's open source. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] michalslowikowski00 commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523033070
##########
File path: kibble/cli/setup_command.py
##########
@@ -0,0 +1,178 @@
+import os
+import sys
+import logging
+
+import click
+import tenacity
+import bcrypt
+import json
+from elasticsearch import Elasticsearch
+
+def create_es_index(
+ conn_uri: str,
+ dbname: str,
+ shards: int,
+ replicas: int,
+ admin_name: str,
+ admin_pass: str,
+ skiponexist: bool,
+):
+ """Creates Elasticsearch index used by Kibble"""
+
+ # elasticsearch logs lots of warnings on retries/connection failure
+ logging.getLogger("elasticsearch").setLevel(logging.ERROR)
+
+ mappings_json = os.path.join(
+ os.path.dirname(os.path.realpath(__file__)), "../setup/mappings.json"
+ )
+ with open(mappings_json, "r") as f:
+ mappings = json.load(f)
+
+ es = Elasticsearch([conn_uri], max_retries=5, retry_on_timeout=True)
+ print(es.info())
+
+ es_version = es.info()["version"]["number"]
+ es6 = int(es_version.split(".")[0]) >= 6
+ es7 = int(es_version.split(".")[0]) >= 7
+
+ if not es6:
+ print(
+ f"New Kibble installations require ElasticSearch 6.x or newer! "
+ f"You appear to be running {es_version}!"
+ )
+ sys.exit(-1)
+
+ # If ES >= 7, _doc is invalid and mapping should be rooted
+ if es7:
+ mappings["mappings"] = mappings["mappings"]["_doc"]
+
+ # Check if index already exists
+ if es.indices.exists(dbname + "_api"):
+ # Skip this is DB exists and -k added
+ if skiponexist:
+ print("DB prefix exists, but --skiponexist used, skipping this step.")
+ return
+ print("Error: ElasticSearch DB prefix '%s' already exists!" % dbname)
+ sys.exit(-1)
+
+ types = [
+ "api",
+ # ci_*: CI service stats
+ "ci_build",
+ "ci_queue",
+ # code_* + evolution + file_history: git repo stats
+ "code_commit",
+ "code_commit_unique",
+ "code_modification",
+ "evolution",
+ "file_history",
+ # forum_*: forum stats (SO, Discourse, Askbot etc)
+ "forum_post",
+ "forum_topic",
+ # GitHub stats
+ "ghstats",
+ # im_*: Instant messaging stats
+ "im_stats",
+ "im_ops",
+ "im_msg",
+ "issue",
+ "logstats",
+ # email, mail*: Email statitics
+ "email",
+ "mailstats",
+ "mailtop",
+ # organisation, view, source, publish: UI Org DB
+ "organisation",
+ "view",
+ "publish",
+ "source",
+ # stats: Miscellaneous stats
+ "stats",
+ # social_*: Twitter, Mastodon, Facebook etc
+ "social_follow",
+ "social_followers",
+ "social_follower",
+ "social_person",
+ # uisession, useraccount, message: UI user DB
+ "uisession",
+ "useraccount",
+ "message",
+ # person: contributor DB
+ "person",
+ ]
+
+ for t in types:
+ iname = f"{dbname}_{t}"
+ print(f"Creating index {iname}")
+
+ settings = {"number_of_shards": shards, "number_of_replicas": replicas}
+ es.indices.create(
+ index=iname, body={"mappings": mappings["mappings"], "settings": settings}
+ )
+ print(f"Indices created!")
+ print()
+
+ salt = bcrypt.gensalt()
+ pwd = bcrypt.hashpw(admin_pass.encode("utf-8"), salt).decode("ascii")
+ print("Creating administrator account")
+ doc = {
+ "email": admin_name, # Username (email)
+ "password": pwd, # Hashed password
+ "displayName": "Administrator", # Display Name
+ "organisations": [], # Orgs user belongs to (default is none)
+ "ownerships": [], # Orgs user owns (default is none)
+ "defaultOrganisation": None, # Default org for user
+ "verified": True, # Account verified via email?
+ "userlevel": "admin", # User level (user/admin)
+ }
+ dbdoc = {
+ "apiversion": KIBBLE_VERSION, # Log current API version
+ "dbversion": KIBBLE_DB_VERSION, # Log the database revision we accept (might change!)
+ }
+ es.index(index=dbname + "_useraccount", doc_type="_doc", id=admin_name, body=doc)
+ es.index(index=dbname + "_api", doc_type="_doc", id="current", body=dbdoc)
+ print("Account created!")
+
+def run(
+ uri: str,
+ dbname: str,
+ shards: str,
+ replicas: str,
+ mailhost: str,
+ autoadmin: bool,
+ skiponexist: bool,
+):
+ click.echo("Welcome to the Apache Kibble setup script!")
+
+ admin_name = "admin@kibble"
+ admin_pass = "kibbleAdmin"
+ if not autoadmin:
+ admin_name = get_user_input(
+ "Enter an email address for the administrator account: "
+ )
+ admin_pass = get_user_input(
+ "Enter a password for the administrator account: ", secure=True
+ )
+
+ # Create Elasticsearch index
+ # Retry in case ES is not yet up
+ click.echo(f"Elasticsearch: {uri}")
+ for attempt in tenacity.Retrying(
Review comment:
I am a little bit lost here.
Can you explain why for loop here is added? :)
##########
File path: kibble/cli/setup_command.py
##########
@@ -0,0 +1,178 @@
+import os
+import sys
+import logging
+
+import click
+import tenacity
+import bcrypt
+import json
+from elasticsearch import Elasticsearch
+
+def create_es_index(
+ conn_uri: str,
+ dbname: str,
+ shards: int,
+ replicas: int,
+ admin_name: str,
+ admin_pass: str,
+ skiponexist: bool,
+):
+ """Creates Elasticsearch index used by Kibble"""
+
+ # elasticsearch logs lots of warnings on retries/connection failure
+ logging.getLogger("elasticsearch").setLevel(logging.ERROR)
+
+ mappings_json = os.path.join(
+ os.path.dirname(os.path.realpath(__file__)), "../setup/mappings.json"
+ )
+ with open(mappings_json, "r") as f:
+ mappings = json.load(f)
+
+ es = Elasticsearch([conn_uri], max_retries=5, retry_on_timeout=True)
+ print(es.info())
+
+ es_version = es.info()["version"]["number"]
+ es6 = int(es_version.split(".")[0]) >= 6
+ es7 = int(es_version.split(".")[0]) >= 7
+
+ if not es6:
+ print(
+ f"New Kibble installations require ElasticSearch 6.x or newer! "
+ f"You appear to be running {es_version}!"
+ )
+ sys.exit(-1)
+
+ # If ES >= 7, _doc is invalid and mapping should be rooted
+ if es7:
+ mappings["mappings"] = mappings["mappings"]["_doc"]
+
+ # Check if index already exists
+ if es.indices.exists(dbname + "_api"):
+ # Skip this is DB exists and -k added
+ if skiponexist:
+ print("DB prefix exists, but --skiponexist used, skipping this step.")
+ return
+ print("Error: ElasticSearch DB prefix '%s' already exists!" % dbname)
+ sys.exit(-1)
+
+ types = [
+ "api",
+ # ci_*: CI service stats
+ "ci_build",
+ "ci_queue",
+ # code_* + evolution + file_history: git repo stats
+ "code_commit",
+ "code_commit_unique",
+ "code_modification",
+ "evolution",
+ "file_history",
+ # forum_*: forum stats (SO, Discourse, Askbot etc)
+ "forum_post",
+ "forum_topic",
+ # GitHub stats
+ "ghstats",
+ # im_*: Instant messaging stats
+ "im_stats",
+ "im_ops",
+ "im_msg",
+ "issue",
+ "logstats",
+ # email, mail*: Email statitics
+ "email",
+ "mailstats",
+ "mailtop",
+ # organisation, view, source, publish: UI Org DB
+ "organisation",
+ "view",
+ "publish",
+ "source",
+ # stats: Miscellaneous stats
+ "stats",
+ # social_*: Twitter, Mastodon, Facebook etc
+ "social_follow",
+ "social_followers",
+ "social_follower",
+ "social_person",
+ # uisession, useraccount, message: UI user DB
+ "uisession",
+ "useraccount",
+ "message",
+ # person: contributor DB
+ "person",
+ ]
+
+ for t in types:
+ iname = f"{dbname}_{t}"
+ print(f"Creating index {iname}")
+
+ settings = {"number_of_shards": shards, "number_of_replicas": replicas}
+ es.indices.create(
+ index=iname, body={"mappings": mappings["mappings"], "settings": settings}
+ )
+ print(f"Indices created!")
Review comment:
`print(f"Indices created! \n")`, then below print won't be necessary.
I don't know why I don't have `suggested changes`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] skekre98 commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
skekre98 commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523311266
##########
File path: kibble/cli/setup_command.py
##########
@@ -0,0 +1,197 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import sys
+import logging
+
+import click
+import tenacity
+import bcrypt
+import json
+from elasticsearch import Elasticsearch
+
+
+def create_es_index(
Review comment:
I agree, will remove it in next fix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] turbaszek merged pull request #91: Refactor setup scripts as kibble cli command
Posted by GitBox <gi...@apache.org>.
turbaszek merged pull request #91:
URL: https://github.com/apache/kibble/pull/91
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] turbaszek commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523443244
##########
File path: kibble/__main__.py
##########
@@ -15,9 +15,83 @@
# specific language governing permissions and limitations
# under the License.
+import click
+
+from kibble.cli import setup_command
+from kibble.version import version as kv
+
+from kibble.configuration import conf
+
+
+@click.group()
+def cli():
+ """A simple command line tool for kibble"""
+
+
+@cli.command("version", short_help="displays the current kibble version")
+def version():
+ click.echo(kv)
+
+
+@cli.command("setup", short_help="starts the setup process for kibble")
+@click.option(
+ "-u",
+ "--uri",
+ default=conf.get("elasticsearch", "conn_uri"),
+ help="Connection uri for ElasticSearch.",
+)
+@click.option(
+ "-d",
+ "--dbname",
+ default=conf.get("elasticsearch", "dbname"),
+ help="Pre-defined Database prefix.",
+)
+@click.option(
+ "-s",
+ "--shards",
+ default=conf.get("elasticsearch", "shards"),
+ help="Predefined number of ES shards.",
+)
+@click.option(
+ "-r",
+ "--replicas",
+ default=conf.get("elasticsearch", "replicas"),
+ help="Predefined number of replicas for ES.",
+)
+@click.option(
+ "-m",
+ "--mailhost",
+ default=conf.get("mail", "mailhost"),
+ help="Pre-defined mail server host.",
+)
+@click.option(
+ "-a", "--autoadmin", default=False, help="Generate generic admin account."
+)
+@click.option(
+ "-k", "--skiponexist", default=True, help="Skip DB creation if DBs exist."
Review comment:
```suggestion
"-k", "--skiponexist", default=True, help="skip DB creation if DBs exist"
```
should we standardise each help message to start with lower-case letter and finished without dot?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] turbaszek commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523038615
##########
File path: kibble/cli/setup_command.py
##########
@@ -0,0 +1,178 @@
+import os
+import sys
+import logging
+
+import click
+import tenacity
+import bcrypt
+import json
+from elasticsearch import Elasticsearch
+
+def create_es_index(
+ conn_uri: str,
+ dbname: str,
+ shards: int,
+ replicas: int,
+ admin_name: str,
+ admin_pass: str,
+ skiponexist: bool,
+):
+ """Creates Elasticsearch index used by Kibble"""
+
+ # elasticsearch logs lots of warnings on retries/connection failure
+ logging.getLogger("elasticsearch").setLevel(logging.ERROR)
+
+ mappings_json = os.path.join(
+ os.path.dirname(os.path.realpath(__file__)), "../setup/mappings.json"
+ )
+ with open(mappings_json, "r") as f:
+ mappings = json.load(f)
+
+ es = Elasticsearch([conn_uri], max_retries=5, retry_on_timeout=True)
+ print(es.info())
+
+ es_version = es.info()["version"]["number"]
+ es6 = int(es_version.split(".")[0]) >= 6
+ es7 = int(es_version.split(".")[0]) >= 7
+
+ if not es6:
+ print(
+ f"New Kibble installations require ElasticSearch 6.x or newer! "
+ f"You appear to be running {es_version}!"
+ )
+ sys.exit(-1)
+
+ # If ES >= 7, _doc is invalid and mapping should be rooted
+ if es7:
+ mappings["mappings"] = mappings["mappings"]["_doc"]
+
+ # Check if index already exists
+ if es.indices.exists(dbname + "_api"):
+ # Skip this is DB exists and -k added
+ if skiponexist:
+ print("DB prefix exists, but --skiponexist used, skipping this step.")
+ return
+ print("Error: ElasticSearch DB prefix '%s' already exists!" % dbname)
+ sys.exit(-1)
+
+ types = [
+ "api",
+ # ci_*: CI service stats
+ "ci_build",
+ "ci_queue",
+ # code_* + evolution + file_history: git repo stats
+ "code_commit",
+ "code_commit_unique",
+ "code_modification",
+ "evolution",
+ "file_history",
+ # forum_*: forum stats (SO, Discourse, Askbot etc)
+ "forum_post",
+ "forum_topic",
+ # GitHub stats
+ "ghstats",
+ # im_*: Instant messaging stats
+ "im_stats",
+ "im_ops",
+ "im_msg",
+ "issue",
+ "logstats",
+ # email, mail*: Email statitics
+ "email",
+ "mailstats",
+ "mailtop",
+ # organisation, view, source, publish: UI Org DB
+ "organisation",
+ "view",
+ "publish",
+ "source",
+ # stats: Miscellaneous stats
+ "stats",
+ # social_*: Twitter, Mastodon, Facebook etc
+ "social_follow",
+ "social_followers",
+ "social_follower",
+ "social_person",
+ # uisession, useraccount, message: UI user DB
+ "uisession",
+ "useraccount",
+ "message",
+ # person: contributor DB
+ "person",
+ ]
+
+ for t in types:
+ iname = f"{dbname}_{t}"
+ print(f"Creating index {iname}")
+
+ settings = {"number_of_shards": shards, "number_of_replicas": replicas}
+ es.indices.create(
+ index=iname, body={"mappings": mappings["mappings"], "settings": settings}
+ )
+ print(f"Indices created!")
+ print()
+
+ salt = bcrypt.gensalt()
+ pwd = bcrypt.hashpw(admin_pass.encode("utf-8"), salt).decode("ascii")
+ print("Creating administrator account")
+ doc = {
+ "email": admin_name, # Username (email)
+ "password": pwd, # Hashed password
+ "displayName": "Administrator", # Display Name
+ "organisations": [], # Orgs user belongs to (default is none)
+ "ownerships": [], # Orgs user owns (default is none)
+ "defaultOrganisation": None, # Default org for user
+ "verified": True, # Account verified via email?
+ "userlevel": "admin", # User level (user/admin)
+ }
+ dbdoc = {
+ "apiversion": KIBBLE_VERSION, # Log current API version
+ "dbversion": KIBBLE_DB_VERSION, # Log the database revision we accept (might change!)
+ }
+ es.index(index=dbname + "_useraccount", doc_type="_doc", id=admin_name, body=doc)
+ es.index(index=dbname + "_api", doc_type="_doc", id="current", body=dbdoc)
+ print("Account created!")
+
+def run(
+ uri: str,
+ dbname: str,
+ shards: str,
+ replicas: str,
+ mailhost: str,
+ autoadmin: bool,
+ skiponexist: bool,
+):
+ click.echo("Welcome to the Apache Kibble setup script!")
+
+ admin_name = "admin@kibble"
+ admin_pass = "kibbleAdmin"
+ if not autoadmin:
+ admin_name = get_user_input(
+ "Enter an email address for the administrator account: "
+ )
+ admin_pass = get_user_input(
+ "Enter a password for the administrator account: ", secure=True
+ )
+
+ # Create Elasticsearch index
+ # Retry in case ES is not yet up
+ click.echo(f"Elasticsearch: {uri}")
+ for attempt in tenacity.Retrying(
Review comment:
It was already there. I added it because the ES instance doesn't have to up instantly, see https://github.com/apache/kibble/pull/50#discussion_r508699630
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] turbaszek commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523443132
##########
File path: kibble/__main__.py
##########
@@ -15,9 +15,83 @@
# specific language governing permissions and limitations
# under the License.
+import click
+
+from kibble.cli import setup_command
+from kibble.version import version as kv
Review comment:
```suggestion
from kibble.version import version as kibble_version
```
How about more meaningful name?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] turbaszek commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523021900
##########
File path: kibble/__main__.py
##########
@@ -15,9 +15,102 @@
# specific language governing permissions and limitations
# under the License.
+import sys
+import os
+import argparse
+import logging
+import click
+from getpass import getpass
+
+import tenacity
+import bcrypt
+import json
+from elasticsearch import Elasticsearch
Review comment:
Do we need all those imports in this file?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] turbaszek commented on pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #91:
URL: https://github.com/apache/kibble/pull/91#issuecomment-726355050
@skekre98 have you tried using docker?
https://github.com/apache/kibble/blob/main/CONTRIBUTING.md#development-installation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] michalslowikowski00 commented on a change in pull request #91: Refactor setup scripts as kibble cli command
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523461867
##########
File path: kibble/cli/setup_command.py
##########
@@ -0,0 +1,178 @@
+import os
+import sys
+import logging
+
+import click
+import tenacity
+import bcrypt
+import json
+from elasticsearch import Elasticsearch
+
+def create_es_index(
+ conn_uri: str,
+ dbname: str,
+ shards: int,
+ replicas: int,
+ admin_name: str,
+ admin_pass: str,
+ skiponexist: bool,
+):
+ """Creates Elasticsearch index used by Kibble"""
+
+ # elasticsearch logs lots of warnings on retries/connection failure
+ logging.getLogger("elasticsearch").setLevel(logging.ERROR)
+
+ mappings_json = os.path.join(
+ os.path.dirname(os.path.realpath(__file__)), "../setup/mappings.json"
+ )
+ with open(mappings_json, "r") as f:
+ mappings = json.load(f)
+
+ es = Elasticsearch([conn_uri], max_retries=5, retry_on_timeout=True)
+ print(es.info())
+
+ es_version = es.info()["version"]["number"]
+ es6 = int(es_version.split(".")[0]) >= 6
+ es7 = int(es_version.split(".")[0]) >= 7
+
+ if not es6:
+ print(
+ f"New Kibble installations require ElasticSearch 6.x or newer! "
+ f"You appear to be running {es_version}!"
+ )
+ sys.exit(-1)
+
+ # If ES >= 7, _doc is invalid and mapping should be rooted
+ if es7:
+ mappings["mappings"] = mappings["mappings"]["_doc"]
+
+ # Check if index already exists
+ if es.indices.exists(dbname + "_api"):
+ # Skip this is DB exists and -k added
+ if skiponexist:
+ print("DB prefix exists, but --skiponexist used, skipping this step.")
+ return
+ print("Error: ElasticSearch DB prefix '%s' already exists!" % dbname)
+ sys.exit(-1)
+
+ types = [
+ "api",
+ # ci_*: CI service stats
+ "ci_build",
+ "ci_queue",
+ # code_* + evolution + file_history: git repo stats
+ "code_commit",
+ "code_commit_unique",
+ "code_modification",
+ "evolution",
+ "file_history",
+ # forum_*: forum stats (SO, Discourse, Askbot etc)
+ "forum_post",
+ "forum_topic",
+ # GitHub stats
+ "ghstats",
+ # im_*: Instant messaging stats
+ "im_stats",
+ "im_ops",
+ "im_msg",
+ "issue",
+ "logstats",
+ # email, mail*: Email statitics
+ "email",
+ "mailstats",
+ "mailtop",
+ # organisation, view, source, publish: UI Org DB
+ "organisation",
+ "view",
+ "publish",
+ "source",
+ # stats: Miscellaneous stats
+ "stats",
+ # social_*: Twitter, Mastodon, Facebook etc
+ "social_follow",
+ "social_followers",
+ "social_follower",
+ "social_person",
+ # uisession, useraccount, message: UI user DB
+ "uisession",
+ "useraccount",
+ "message",
+ # person: contributor DB
+ "person",
+ ]
+
+ for t in types:
+ iname = f"{dbname}_{t}"
+ print(f"Creating index {iname}")
+
+ settings = {"number_of_shards": shards, "number_of_replicas": replicas}
+ es.indices.create(
+ index=iname, body={"mappings": mappings["mappings"], "settings": settings}
+ )
+ print(f"Indices created!")
+ print()
+
+ salt = bcrypt.gensalt()
+ pwd = bcrypt.hashpw(admin_pass.encode("utf-8"), salt).decode("ascii")
+ print("Creating administrator account")
+ doc = {
+ "email": admin_name, # Username (email)
+ "password": pwd, # Hashed password
+ "displayName": "Administrator", # Display Name
+ "organisations": [], # Orgs user belongs to (default is none)
+ "ownerships": [], # Orgs user owns (default is none)
+ "defaultOrganisation": None, # Default org for user
+ "verified": True, # Account verified via email?
+ "userlevel": "admin", # User level (user/admin)
+ }
+ dbdoc = {
+ "apiversion": KIBBLE_VERSION, # Log current API version
+ "dbversion": KIBBLE_DB_VERSION, # Log the database revision we accept (might change!)
+ }
+ es.index(index=dbname + "_useraccount", doc_type="_doc", id=admin_name, body=doc)
+ es.index(index=dbname + "_api", doc_type="_doc", id="current", body=dbdoc)
+ print("Account created!")
+
+def run(
+ uri: str,
+ dbname: str,
+ shards: str,
+ replicas: str,
+ mailhost: str,
+ autoadmin: bool,
+ skiponexist: bool,
+):
+ click.echo("Welcome to the Apache Kibble setup script!")
+
+ admin_name = "admin@kibble"
+ admin_pass = "kibbleAdmin"
+ if not autoadmin:
+ admin_name = get_user_input(
+ "Enter an email address for the administrator account: "
+ )
+ admin_pass = get_user_input(
+ "Enter a password for the administrator account: ", secure=True
+ )
+
+ # Create Elasticsearch index
+ # Retry in case ES is not yet up
+ click.echo(f"Elasticsearch: {uri}")
+ for attempt in tenacity.Retrying(
Review comment:
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] turbaszek commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523022619
##########
File path: kibble/cli/setup_command.py
##########
@@ -0,0 +1,197 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import sys
+import logging
+
+import click
+import tenacity
+import bcrypt
+import json
+from elasticsearch import Elasticsearch
+
+
+def create_es_index(
+ conn_uri: str,
+ dbname: str,
+ shards: int,
+ replicas: int,
+ admin_name: str,
+ admin_pass: str,
+ skiponexist: bool,
+):
+ """Creates Elasticsearch index used by Kibble"""
+
+ # elasticsearch logs lots of warnings on retries/connection failure
+ logging.getLogger("elasticsearch").setLevel(logging.ERROR)
+
+ mappings_json = os.path.join(
+ os.path.dirname(os.path.realpath(__file__)), "../setup/mappings.json"
+ )
+ with open(mappings_json, "r") as f:
+ mappings = json.load(f)
+
+ es = Elasticsearch([conn_uri], max_retries=5, retry_on_timeout=True)
+ print(es.info())
+
+ es_version = es.info()["version"]["number"]
+ es6 = int(es_version.split(".")[0]) >= 6
+ es7 = int(es_version.split(".")[0]) >= 7
+
+ if not es6:
+ print(
+ f"New Kibble installations require ElasticSearch 6.x or newer! "
+ f"You appear to be running {es_version}!"
+ )
+ sys.exit(-1)
+
+ # If ES >= 7, _doc is invalid and mapping should be rooted
+ if es7:
+ mappings["mappings"] = mappings["mappings"]["_doc"]
+
+ # Check if index already exists
+ if es.indices.exists(dbname + "_api"):
+ # Skip this is DB exists and -k added
+ if skiponexist:
+ print("DB prefix exists, but --skiponexist used, skipping this step.")
+ return
+ print("Error: ElasticSearch DB prefix '%s' already exists!" % dbname)
+ sys.exit(-1)
+
+ types = [
+ "api",
+ # ci_*: CI service stats
+ "ci_build",
+ "ci_queue",
+ # code_* + evolution + file_history: git repo stats
+ "code_commit",
+ "code_commit_unique",
+ "code_modification",
+ "evolution",
+ "file_history",
+ # forum_*: forum stats (SO, Discourse, Askbot etc)
+ "forum_post",
+ "forum_topic",
+ # GitHub stats
+ "ghstats",
+ # im_*: Instant messaging stats
+ "im_stats",
+ "im_ops",
+ "im_msg",
+ "issue",
+ "logstats",
+ # email, mail*: Email statitics
+ "email",
+ "mailstats",
+ "mailtop",
+ # organisation, view, source, publish: UI Org DB
+ "organisation",
+ "view",
+ "publish",
+ "source",
+ # stats: Miscellaneous stats
+ "stats",
+ # social_*: Twitter, Mastodon, Facebook etc
+ "social_follow",
+ "social_followers",
+ "social_follower",
+ "social_person",
+ # uisession, useraccount, message: UI user DB
+ "uisession",
+ "useraccount",
+ "message",
+ # person: contributor DB
+ "person",
+ ]
+
+ for t in types:
+ iname = f"{dbname}_{t}"
+ print(f"Creating index {iname}")
+
+ settings = {"number_of_shards": shards, "number_of_replicas": replicas}
+ es.indices.create(
+ index=iname, body={"mappings": mappings["mappings"], "settings": settings}
+ )
+ print(f"Indices created!")
+ print()
+
+ salt = bcrypt.gensalt()
+ pwd = bcrypt.hashpw(admin_pass.encode("utf-8"), salt).decode("ascii")
+ print("Creating administrator account")
+ doc = {
+ "email": admin_name, # Username (email)
+ "password": pwd, # Hashed password
+ "displayName": "Administrator", # Display Name
+ "organisations": [], # Orgs user belongs to (default is none)
+ "ownerships": [], # Orgs user owns (default is none)
+ "defaultOrganisation": None, # Default org for user
+ "verified": True, # Account verified via email?
+ "userlevel": "admin", # User level (user/admin)
+ }
+ dbdoc = {
+ "apiversion": KIBBLE_VERSION, # Log current API version
+ "dbversion": KIBBLE_DB_VERSION, # Log the database revision we accept (might change!)
+ }
+ es.index(index=dbname + "_useraccount", doc_type="_doc", id=admin_name, body=doc)
+ es.index(index=dbname + "_api", doc_type="_doc", id="current", body=dbdoc)
+ print("Account created!")
+
+
+def run(
Review comment:
```suggestion
def do_setup(
```
How about this name?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] turbaszek commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523021516
##########
File path: docker-compose-dev.yaml
##########
@@ -7,7 +7,7 @@ services:
build:
context: .
dockerfile: Dockerfile.dev
- command: bash -c "python kibble/setup/setup.py --autoadmin --skiponexist"
+ command: bash -c "python kibble/__main__.py setup --autoadmin --skiponexist"
Review comment:
```suggestion
command: bash -c "kibble setup --autoadmin --skiponexist"
```
Should work too 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] michalslowikowski00 commented on pull request #91: Refactor setup scripts as kibble cli command
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on pull request #91:
URL: https://github.com/apache/kibble/pull/91#issuecomment-727263770
Great job @skekre98. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] skekre98 commented on pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
skekre98 commented on pull request #91:
URL: https://github.com/apache/kibble/pull/91#issuecomment-726482634
> @skekre98 have you tried using docker?
> https://github.com/apache/kibble/blob/main/CONTRIBUTING.md#development-installation
Got it working now, still getting used to the repository 😅 . I've also made the change to the docker compose file to use the new setup workflow with the cli.
```bash
$ docker-compose -f docker-compose-dev.yaml up setup
kibble_elasticsearch_1 is up-to-date
Recreating kibble_setup_1 ...
Recreating kibble_setup_1 ... done
Attaching to kibble_setup_1
setup_1 | Welcome to the Apache Kibble setup script!
setup_1 | Elasticsearch: elasticsearch:9200
setup_1 | Trying to create ES index...
setup_1 | {'name': 'es01', 'cluster_name': 'kibble', 'cluster_uuid': '8-2XRavIQiaZFKLh7n0u_w', 'version': {'number': '7.9.2', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'd34da0ea4a966c4e49417f2da2f244e3e97b4e6e', 'build_date': '2020-09-23T00:45:33.626720Z', 'build_snapshot': False, 'lucene_version': '8.6.2', 'minimum_wire_compatibility_version': '6.8.0', 'minimum_index_compatibility_version': '6.0.0-beta1'}, 'tagline': 'You Know, for Search'}
setup_1 | DB prefix exists, but --skiponexist used, skipping this step.
setup_1 |
setup_1 | All done, Kibble should...work now :)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] turbaszek commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523411209
##########
File path: kibble/cli/setup_command.py
##########
@@ -212,69 +157,46 @@ def create_es_index(
print("Account created!")
-def get_user_input(msg: str, secure: bool = False):
- value = None
- while not value:
- value = getpass(msg) if secure else input(msg)
- return value
-
-
-def print_configuration(args):
- print(
- "Configuring Apache Kibble elasticsearch instance with the following arguments:"
- )
- print(f"- conn_uri: {args.conn_uri}")
- print(f"- dbname: {args.dbname}")
- print(f"- shards: {int(args.shards)}")
- print(f"- replicas: {int(args.replicas)}")
- print()
-
-
-def main():
- """
- The main Kibble setup logic. Using users input we create:
- - Elasticsearch indexes used by Apache Kibble app
- - Configuration yaml file
- """
- parser = get_parser()
- args = parser.parse_args()
-
- print("Welcome to the Apache Kibble setup script!")
- print_configuration(args)
+def do_setup(
+ uri: str,
+ dbname: str,
+ shards: str,
+ replicas: str,
+ mailhost: str,
+ autoadmin: bool,
+ skiponexist: bool,
+):
+ click.echo("Welcome to the Apache Kibble setup script!")
admin_name = "admin@kibble"
admin_pass = "kibbleAdmin"
- if not args.autoadmin:
+ if not autoadmin:
admin_name = get_user_input(
- "Enter an email address for the administrator account:"
+ "Enter an email address for the administrator account: "
Review comment:
Does it work? The `get_user_input` is undefined in this module as it was left in `__main__.py`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] skekre98 commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
skekre98 commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523121926
##########
File path: kibble/cli/setup_command.py
##########
@@ -0,0 +1,178 @@
+import os
+import sys
+import logging
+
+import click
+import tenacity
+import bcrypt
+import json
+from elasticsearch import Elasticsearch
+
+def create_es_index(
+ conn_uri: str,
+ dbname: str,
+ shards: int,
+ replicas: int,
+ admin_name: str,
+ admin_pass: str,
+ skiponexist: bool,
+):
+ """Creates Elasticsearch index used by Kibble"""
+
+ # elasticsearch logs lots of warnings on retries/connection failure
+ logging.getLogger("elasticsearch").setLevel(logging.ERROR)
+
+ mappings_json = os.path.join(
+ os.path.dirname(os.path.realpath(__file__)), "../setup/mappings.json"
+ )
+ with open(mappings_json, "r") as f:
+ mappings = json.load(f)
+
+ es = Elasticsearch([conn_uri], max_retries=5, retry_on_timeout=True)
+ print(es.info())
+
+ es_version = es.info()["version"]["number"]
+ es6 = int(es_version.split(".")[0]) >= 6
+ es7 = int(es_version.split(".")[0]) >= 7
+
+ if not es6:
+ print(
+ f"New Kibble installations require ElasticSearch 6.x or newer! "
+ f"You appear to be running {es_version}!"
+ )
+ sys.exit(-1)
+
+ # If ES >= 7, _doc is invalid and mapping should be rooted
+ if es7:
+ mappings["mappings"] = mappings["mappings"]["_doc"]
+
+ # Check if index already exists
+ if es.indices.exists(dbname + "_api"):
+ # Skip this is DB exists and -k added
+ if skiponexist:
+ print("DB prefix exists, but --skiponexist used, skipping this step.")
+ return
+ print("Error: ElasticSearch DB prefix '%s' already exists!" % dbname)
+ sys.exit(-1)
+
+ types = [
+ "api",
+ # ci_*: CI service stats
+ "ci_build",
+ "ci_queue",
+ # code_* + evolution + file_history: git repo stats
+ "code_commit",
+ "code_commit_unique",
+ "code_modification",
+ "evolution",
+ "file_history",
+ # forum_*: forum stats (SO, Discourse, Askbot etc)
+ "forum_post",
+ "forum_topic",
+ # GitHub stats
+ "ghstats",
+ # im_*: Instant messaging stats
+ "im_stats",
+ "im_ops",
+ "im_msg",
+ "issue",
+ "logstats",
+ # email, mail*: Email statitics
+ "email",
+ "mailstats",
+ "mailtop",
+ # organisation, view, source, publish: UI Org DB
+ "organisation",
+ "view",
+ "publish",
+ "source",
+ # stats: Miscellaneous stats
+ "stats",
+ # social_*: Twitter, Mastodon, Facebook etc
+ "social_follow",
+ "social_followers",
+ "social_follower",
+ "social_person",
+ # uisession, useraccount, message: UI user DB
+ "uisession",
+ "useraccount",
+ "message",
+ # person: contributor DB
+ "person",
+ ]
+
+ for t in types:
+ iname = f"{dbname}_{t}"
+ print(f"Creating index {iname}")
+
+ settings = {"number_of_shards": shards, "number_of_replicas": replicas}
+ es.indices.create(
+ index=iname, body={"mappings": mappings["mappings"], "settings": settings}
+ )
+ print(f"Indices created!")
Review comment:
Oh good catch. Definitely an unnecessary print.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] turbaszek commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523443214
##########
File path: kibble/__main__.py
##########
@@ -15,9 +15,83 @@
# specific language governing permissions and limitations
# under the License.
+import click
+
+from kibble.cli import setup_command
+from kibble.version import version as kv
+
+from kibble.configuration import conf
+
+
+@click.group()
+def cli():
+ """A simple command line tool for kibble"""
+
+
+@cli.command("version", short_help="displays the current kibble version")
+def version():
+ click.echo(kv)
+
+
+@cli.command("setup", short_help="starts the setup process for kibble")
+@click.option(
+ "-u",
+ "--uri",
+ default=conf.get("elasticsearch", "conn_uri"),
+ help="Connection uri for ElasticSearch.",
+)
+@click.option(
+ "-d",
+ "--dbname",
+ default=conf.get("elasticsearch", "dbname"),
+ help="Pre-defined Database prefix.",
+)
+@click.option(
+ "-s",
+ "--shards",
+ default=conf.get("elasticsearch", "shards"),
+ help="Predefined number of ES shards.",
+)
+@click.option(
+ "-r",
+ "--replicas",
+ default=conf.get("elasticsearch", "replicas"),
+ help="Predefined number of replicas for ES.",
+)
+@click.option(
+ "-m",
+ "--mailhost",
+ default=conf.get("mail", "mailhost"),
+ help="Pre-defined mail server host.",
Review comment:
```suggestion
help="mail server host",
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] turbaszek commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523443169
##########
File path: kibble/__main__.py
##########
@@ -15,9 +15,83 @@
# specific language governing permissions and limitations
# under the License.
+import click
+
+from kibble.cli import setup_command
+from kibble.version import version as kv
+
+from kibble.configuration import conf
+
+
+@click.group()
+def cli():
+ """A simple command line tool for kibble"""
+
+
+@cli.command("version", short_help="displays the current kibble version")
+def version():
+ click.echo(kv)
+
+
+@cli.command("setup", short_help="starts the setup process for kibble")
+@click.option(
+ "-u",
+ "--uri",
+ default=conf.get("elasticsearch", "conn_uri"),
+ help="Connection uri for ElasticSearch.",
+)
+@click.option(
+ "-d",
+ "--dbname",
+ default=conf.get("elasticsearch", "dbname"),
+ help="Pre-defined Database prefix.",
Review comment:
```suggestion
help="elasticsearch database prefix.",
```
##########
File path: kibble/__main__.py
##########
@@ -15,9 +15,83 @@
# specific language governing permissions and limitations
# under the License.
+import click
+
+from kibble.cli import setup_command
+from kibble.version import version as kv
+
+from kibble.configuration import conf
+
+
+@click.group()
+def cli():
+ """A simple command line tool for kibble"""
+
+
+@cli.command("version", short_help="displays the current kibble version")
+def version():
+ click.echo(kv)
+
+
+@cli.command("setup", short_help="starts the setup process for kibble")
+@click.option(
+ "-u",
+ "--uri",
+ default=conf.get("elasticsearch", "conn_uri"),
+ help="Connection uri for ElasticSearch.",
+)
+@click.option(
+ "-d",
+ "--dbname",
+ default=conf.get("elasticsearch", "dbname"),
+ help="Pre-defined Database prefix.",
+)
+@click.option(
+ "-s",
+ "--shards",
+ default=conf.get("elasticsearch", "shards"),
+ help="Predefined number of ES shards.",
Review comment:
```suggestion
help="number of ES shards.",
```
##########
File path: kibble/__main__.py
##########
@@ -15,9 +15,83 @@
# specific language governing permissions and limitations
# under the License.
+import click
+
+from kibble.cli import setup_command
+from kibble.version import version as kv
+
+from kibble.configuration import conf
+
+
+@click.group()
+def cli():
+ """A simple command line tool for kibble"""
+
+
+@cli.command("version", short_help="displays the current kibble version")
+def version():
+ click.echo(kv)
+
+
+@cli.command("setup", short_help="starts the setup process for kibble")
+@click.option(
+ "-u",
+ "--uri",
+ default=conf.get("elasticsearch", "conn_uri"),
+ help="Connection uri for ElasticSearch.",
+)
+@click.option(
+ "-d",
+ "--dbname",
+ default=conf.get("elasticsearch", "dbname"),
+ help="Pre-defined Database prefix.",
+)
+@click.option(
+ "-s",
+ "--shards",
+ default=conf.get("elasticsearch", "shards"),
+ help="Predefined number of ES shards.",
+)
+@click.option(
+ "-r",
+ "--replicas",
+ default=conf.get("elasticsearch", "replicas"),
+ help="Predefined number of replicas for ES.",
Review comment:
```suggestion
help="number of replicas for ES.",
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] skekre98 commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
skekre98 commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r522551736
##########
File path: kibble/__main__.py
##########
@@ -15,9 +15,252 @@
# specific language governing permissions and limitations
# under the License.
+import sys
+import os
+import argparse
+import logging
+import click
+from getpass import getpass
+
+import tenacity
+import bcrypt
+import json
+from elasticsearch import Elasticsearch
+
+from kibble.version import version as kv
+from kibble.configuration import conf
+
+KIBBLE_VERSION = conf.get("api", "version")
+KIBBLE_DB_VERSION = conf.get("api", "database")
+
+
+def get_user_input(msg: str, secure: bool = False):
+ value = None
+ while not value:
+ value = getpass(msg) if secure else input(msg)
+ return value
+
+
+@click.group()
+def cli():
+ """A simple command line tool for kibble"""
+
+
+@cli.command("version", short_help="displays the current kibble version")
+def version():
+ click.echo(kv)
+
+
+def create_es_index(
+ conn_uri: str,
+ dbname: str,
+ shards: int,
+ replicas: int,
+ admin_name: str,
+ admin_pass: str,
+ skiponexist: bool,
+):
+ """Creates Elasticsearch index used by Kibble"""
+
+ # elasticsearch logs lots of warnings on retries/connection failure
+ logging.getLogger("elasticsearch").setLevel(logging.ERROR)
+
+ mappings_json = os.path.join(
+ os.path.dirname(os.path.realpath(__file__)), "setup/mappings.json"
+ )
+ with open(mappings_json, "r") as f:
+ mappings = json.load(f)
+
+ es = Elasticsearch([conn_uri], max_retries=5, retry_on_timeout=True)
+ print(es.info())
+
+ es_version = es.info()["version"]["number"]
+ es6 = int(es_version.split(".")[0]) >= 6
+ es7 = int(es_version.split(".")[0]) >= 7
+
+ if not es6:
+ print(
+ f"New Kibble installations require ElasticSearch 6.x or newer! "
+ f"You appear to be running {es_version}!"
+ )
+ sys.exit(-1)
+
+ # If ES >= 7, _doc is invalid and mapping should be rooted
+ if es7:
+ mappings["mappings"] = mappings["mappings"]["_doc"]
+
+ # Check if index already exists
+ if es.indices.exists(dbname + "_api"):
+ # Skip this is DB exists and -k added
+ if skiponexist:
+ print("DB prefix exists, but --skiponexist used, skipping this step.")
+ return
+ print("Error: ElasticSearch DB prefix '%s' already exists!" % dbname)
+ sys.exit(-1)
+
+ types = [
+ "api",
+ # ci_*: CI service stats
+ "ci_build",
+ "ci_queue",
+ # code_* + evolution + file_history: git repo stats
+ "code_commit",
+ "code_commit_unique",
+ "code_modification",
+ "evolution",
+ "file_history",
+ # forum_*: forum stats (SO, Discourse, Askbot etc)
+ "forum_post",
+ "forum_topic",
+ # GitHub stats
+ "ghstats",
+ # im_*: Instant messaging stats
+ "im_stats",
+ "im_ops",
+ "im_msg",
+ "issue",
+ "logstats",
+ # email, mail*: Email statitics
+ "email",
+ "mailstats",
+ "mailtop",
+ # organisation, view, source, publish: UI Org DB
+ "organisation",
+ "view",
+ "publish",
+ "source",
+ # stats: Miscellaneous stats
+ "stats",
+ # social_*: Twitter, Mastodon, Facebook etc
+ "social_follow",
+ "social_followers",
+ "social_follower",
+ "social_person",
+ # uisession, useraccount, message: UI user DB
+ "uisession",
+ "useraccount",
+ "message",
+ # person: contributor DB
+ "person",
+ ]
+
+ for t in types:
+ iname = f"{dbname}_{t}"
+ print(f"Creating index {iname}")
+
+ settings = {"number_of_shards": shards, "number_of_replicas": replicas}
+ es.indices.create(
+ index=iname, body={"mappings": mappings["mappings"], "settings": settings}
+ )
+ print(f"Indices created!")
+ print()
+
+ salt = bcrypt.gensalt()
+ pwd = bcrypt.hashpw(admin_pass.encode("utf-8"), salt).decode("ascii")
+ print("Creating administrator account")
+ doc = {
+ "email": admin_name, # Username (email)
+ "password": pwd, # Hashed password
+ "displayName": "Administrator", # Display Name
+ "organisations": [], # Orgs user belongs to (default is none)
+ "ownerships": [], # Orgs user owns (default is none)
+ "defaultOrganisation": None, # Default org for user
+ "verified": True, # Account verified via email?
+ "userlevel": "admin", # User level (user/admin)
+ }
+ dbdoc = {
+ "apiversion": KIBBLE_VERSION, # Log current API version
+ "dbversion": KIBBLE_DB_VERSION, # Log the database revision we accept (might change!)
+ }
+ es.index(index=dbname + "_useraccount", doc_type="_doc", id=admin_name, body=doc)
+ es.index(index=dbname + "_api", doc_type="_doc", id="current", body=dbdoc)
+ print("Account created!")
+
+
+@cli.command("setup", short_help="starts the setup process for kibble")
+@click.option(
+ "-u",
+ "--uri",
+ default=conf.get("elasticsearch", "conn_uri"),
+ help="Pre-defined connection uri for ElasticSearch.",
+)
+@click.option(
+ "-d",
+ "--dbname",
+ default=conf.get("elasticsearch", "dbname"),
+ help="Pre-defined Database prefix.",
+)
+@click.option(
+ "-s",
+ "--shards",
+ default=conf.get("elasticsearch", "shards"),
+ help="Predefined number of ES shards.",
+)
+@click.option(
+ "-r",
+ "--replicas",
+ default=conf.get("elasticsearch", "replicas"),
+ help="Predefined number of replicas for ES.",
+)
+@click.option(
+ "-m",
+ "--mailhost",
+ default=conf.get("mail", "mailhost"),
+ help="Pre-defined mail server host.",
+)
+@click.option(
+ "-a", "--autoadmin", default=False, help="Generate generic admin account."
+)
+@click.option(
+ "-k", "--skiponexist", default=True, help="Skip DB creation if DBs exist."
+)
+def setup(
+ uri: str,
+ dbname: str,
+ shards: str,
+ replicas: str,
+ mailhost: str,
+ autoadmin: bool,
+ skiponexist: bool,
+):
Review comment:
That definitely sounds a lot cleaner. Will refactor current implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kibble] skekre98 commented on a change in pull request #91: Kibble initial cli
Posted by GitBox <gi...@apache.org>.
skekre98 commented on a change in pull request #91:
URL: https://github.com/apache/kibble/pull/91#discussion_r523441728
##########
File path: kibble/cli/setup_command.py
##########
@@ -212,69 +157,46 @@ def create_es_index(
print("Account created!")
-def get_user_input(msg: str, secure: bool = False):
- value = None
- while not value:
- value = getpass(msg) if secure else input(msg)
- return value
-
-
-def print_configuration(args):
- print(
- "Configuring Apache Kibble elasticsearch instance with the following arguments:"
- )
- print(f"- conn_uri: {args.conn_uri}")
- print(f"- dbname: {args.dbname}")
- print(f"- shards: {int(args.shards)}")
- print(f"- replicas: {int(args.replicas)}")
- print()
-
-
-def main():
- """
- The main Kibble setup logic. Using users input we create:
- - Elasticsearch indexes used by Apache Kibble app
- - Configuration yaml file
- """
- parser = get_parser()
- args = parser.parse_args()
-
- print("Welcome to the Apache Kibble setup script!")
- print_configuration(args)
+def do_setup(
+ uri: str,
+ dbname: str,
+ shards: str,
+ replicas: str,
+ mailhost: str,
+ autoadmin: bool,
+ skiponexist: bool,
+):
+ click.echo("Welcome to the Apache Kibble setup script!")
admin_name = "admin@kibble"
admin_pass = "kibbleAdmin"
- if not args.autoadmin:
+ if not autoadmin:
admin_name = get_user_input(
- "Enter an email address for the administrator account:"
+ "Enter an email address for the administrator account: "
Review comment:
Oh good catch. Will move it over 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org