You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@trafficcontrol.apache.org by oc...@apache.org on 2021/03/28 01:18:01 UTC
[trafficcontrol] branch master updated: Use the Python Postinstall
implementation by default (#5681)
This is an automated email from the ASF dual-hosted git repository.
ocket8888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/trafficcontrol.git
The following commit(s) were added to refs/heads/master by this push:
new 9045938 Use the Python Postinstall implementation by default (#5681)
9045938 is described below
commit 9045938057fa19d15da2d0848a8e344fab9174e6
Author: Zach Hoffman <zr...@apache.org>
AuthorDate: Sat Mar 27 19:17:20 2021 -0600
Use the Python Postinstall implementation by default (#5681)
* Use hashlib.scrypt() if running from Python 3
* Popen.stdout.strip() -> Popen.stdout.read().strip()
* Use Popen.communicate() instead of Popen.wait() to avoid stdout/stderr deadlock
* Do not cast port to int
* Rename _postinstall to _postinstall.pl
* Rename postinstall.py to _postinstall
* Sort input.json
* Bring input.json up-to-date with postinstall
* Initialize, migrate and seed the database using the postinstall script
instead of using db/admin directly
* Docker layer optimization: Install Traffic Ops dependencies before the
Traffic Ops RPM
* Use envsubst instead of an unquoted here document
* Add traffic_ops_data and GeoLite2-City.mmdb.gz before installing the
Traffic Ops RPM
* Deprecate non-boolean "hidden" property values in input.json
* Fix issue creation link
* Support loading _postinstall module with no file extension from tests
* GHA Workflow to run the Postinstall unit tests
* Test using Python 3.6, not Python 3.9
---
.github/workflows/postinstall.tests.yml | 53 +
CHANGELOG.md | 4 +-
docs/source/admin/traffic_ops.rst | 14 +-
docs/source/development/traffic_ops.rst | 4 +-
infrastructure/cdn-in-a-box/traffic_ops/Dockerfile | 48 +-
infrastructure/cdn-in-a-box/traffic_ops/config.sh | 183 +-
infrastructure/cdn-in-a-box/traffic_ops/run-go.sh | 2 -
infrastructure/cdn-in-a-box/variables.env | 1 -
traffic_ops/install/bin/_postinstall | 2438 ++++++++++++--------
.../install/bin/{_postinstall => _postinstall.pl} | 0
traffic_ops/install/bin/input.json | 196 +-
traffic_ops/install/bin/postinstall | 17 +-
traffic_ops/install/bin/postinstall.py | 1522 ------------
traffic_ops/install/bin/postinstall.test.sh | 20 +-
14 files changed, 1797 insertions(+), 2705 deletions(-)
diff --git a/.github/workflows/postinstall.tests.yml b/.github/workflows/postinstall.tests.yml
new file mode 100644
index 0000000..2428d34
--- /dev/null
+++ b/.github/workflows/postinstall.tests.yml
@@ -0,0 +1,53 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Postinstall Unit Tests
+
+on:
+ push:
+ paths:
+ - .github/workflows/postinstall.tests.yml
+ - traffic_ops/install/bin/_postinstall
+ - traffic_ops/install/bin/postinstall.test.sh
+ create:
+ pull_request:
+ paths:
+ - .github/workflows/postinstall.tests.yml
+ - traffic_ops/install/bin/_postinstall
+ - traffic_ops/install/bin/postinstall.test.sh
+ types: [ opened, reopened, ready_for_review, synchronize ]
+
+jobs:
+ postinstall-tests:
+ if: github.event.pull_request.draft == false
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout repo
+ uses: actions/checkout@master
+ if: ${{ (github.repository_owner == 'apache' && github.ref == 'refs/heads/master' ) || github.event_name != 'schedule' }}
+ id: checkout
+ - name: Install Python 3.6
+ uses: actions/setup-python@v2
+ if: ${{ steps.checkout.outcome == 'success' }}
+ with: { python-version: 3.6 }
+ - name: Install Python 2.7
+ uses: actions/setup-python@v2
+ if: ${{ steps.checkout.outcome == 'success' }}
+ with: { python-version: 2.7 }
+ - name: Run Postinstall Tests
+ if: ${{ steps.checkout.outcome == 'success' }}
+ run: traffic_ops/install/bin/postinstall.test.sh
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e5aa937..38ebcc6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,7 +5,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
## [unreleased]
### Added
-- Python client: [#5611] Added server_detail endpoint
+- Python client: [#5611](https://github.com/apache/trafficcontrol/pull/5611) Added server_detail endpoint
+- Ported the Postinstall script to Python. The Perl version has been moved to `install/bin/_postinstall.pl` and has been deprecated, pending removal in a future release.
+- CDN-in-a-Box: Generate config files using the Postinstall script
- Apache Traffic Server: [#5627](https://github.com/apache/trafficcontrol/pull/5627) - Added the creation of Centos8 RPMs for Apache Traffic Server
- Traffic Ops/Traffic Portal: [#5479](https://github.com/apache/trafficcontrol/issues/5479) - Added the ability to change a server capability name
- Traffic Ops: [#3577](https://github.com/apache/trafficcontrol/issues/3577) - Added a query param (server host_name or ID) for servercheck API
diff --git a/docs/source/admin/traffic_ops.rst b/docs/source/admin/traffic_ops.rst
index 4f01e08..fc865a9 100644
--- a/docs/source/admin/traffic_ops.rst
+++ b/docs/source/admin/traffic_ops.rst
@@ -105,7 +105,7 @@ Guide
Password:
to-#
-#. Now, run the following command as the root user (or with :manpage:`sudo(8)`): :file:`/opt/traffic_ops/install/bin/postinstall`. The :program:`postinstall` script will first get all required Perl packages from :abbr:`CPAN (The Comprehensive Perl Archive Network)`. This may take a while, expect up to 30 minutes on the first install. If there are any prompts in this phase, please just answer with the defaults (some :abbr:`CPAN (The Comprehensive Perl Archive Network)` installs can prompt [...]
+#. Now, run the following command as the root user (or with :manpage:`sudo(8)`): :file:`/opt/traffic_ops/install/bin/postinstall`. Some additional files will be installed, and then it will proceed with the next phase of the install, where it will ask you about the local environment for your CDN. Please make sure you remember all your answers and verify that the database answers match the information previously used to create the database.
.. code-block:: console
:caption: Example Output
@@ -214,7 +214,17 @@ Guide
| Password for the admin user | The password for the administrative Traffic Ops user. |
+----------------------------------------------------+------------------------------------------------------------------------------------------------+
-.. note:: A Python postinstall script also exists, and, unlike the Perl postinstall script, has no dependencies besides the interpreter itself. To use it, run ``/opt/traffic_ops/install/bin/postinstall.py`` with the same arguments you would have passed to ``/opt/traffic_ops/install/bin/postinstall`` (runs under either Python 3 or Python 2).
+.. deprecated:: ATCv6
+ The postinstall script is now written in Python. If you run into issues with the postinstall script, you are encouraged to file an issue at https://github.com/apache/trafficcontrol/issues/new/choose. The original Perl postinstall script is deprecated and will be removed in a future ATC release. To use the deprecated version anyway, run ``/opt/traffic_ops/install/bin/_postinstall.pl`` directly instead of ``/opt/traffic_ops/install/bin/postinstall``.
+
+The postinstall script can also be run non-interactively using :atc-file:`traffic_ops/install/bin/input.json`. To use it, first change the values to match your environment, then pass it to the ``postinstall`` script:
+ .. code-block:: console
+ :caption: Postinstall in Automatic (-a) mode
+
+ /opt/traffic_ops/install/bin/postinstall -a --cfile /opt/traffic_ops/install/bin/input.json
+
+.. deprecated:: ATCv6
+ Once the Perl script is removed, the values in ``input.json`` for the ``"hidden"`` properties will be changed from ``"1"`` and ``"0"`` to ``true`` and ``false``.
.. _to-upgrading:
diff --git a/docs/source/development/traffic_ops.rst b/docs/source/development/traffic_ops.rst
index 19453b4..ca86f93 100644
--- a/docs/source/development/traffic_ops.rst
+++ b/docs/source/development/traffic_ops.rst
@@ -28,7 +28,7 @@ Software Requirements
=====================
Traffic Ops is only supported on CentOS 7+ systems (although many developers do use Mac OS with some success). Here are the requirements:
-- `Goose <https://bitbucket.org/liamstask/goose/>`_ (although the ``postinstall`` Perl script will install this if desired)
+- `Goose <https://bitbucket.org/liamstask/goose/>`_ (although the ``postinstall`` script will install this)
- `PostgreSQL 13.2 <https://www.postgresql.org/download/>`_ - the machine where Traffic Ops is running must have the client tool set (e.g. :manpage:`psql(1)`), but the actual database can be run anywhere so long as it is accessible.
.. note:: Prior to version 13.2, Traffic Ops used version 9.6. For upgrading an existing Mac OS Homebrew-based PostgreSQL instance, you can use `Homebrew <https://brew.sh/>`_ to easily upgrade from 9.6 to 13.2:
@@ -266,7 +266,7 @@ To install the Traffic Ops Developer environment:
#. Use the ``reset`` and ``upgrade`` :option:`command`\ s of :program:`admin` (see :ref:`database-management` for usage) to set up the ``traffic_ops`` database(s).
-#. Run the :atc-file:`traffic_ops/install/bin/postinstall` script, it will prompt for information like the default user login credentials
+#. Run the :atc-file:`traffic_ops/install/bin/postinstall` script, it will prompt for information like the default user login credentials.
#. To run Traffic Ops, follow the instructions in :ref:`to-running`.
Test Cases
diff --git a/infrastructure/cdn-in-a-box/traffic_ops/Dockerfile b/infrastructure/cdn-in-a-box/traffic_ops/Dockerfile
index eec5d45..2cca7bb 100644
--- a/infrastructure/cdn-in-a-box/traffic_ops/Dockerfile
+++ b/infrastructure/cdn-in-a-box/traffic_ops/Dockerfile
@@ -35,38 +35,58 @@ RUN set -o nounset -o errexit && \
if [[ "${RHEL_VERSION%%.*}" -eq 7 ]]; then \
use_repo=''; \
enable_repo=''; \
+ llvm_version=5.0; \
# needed for llvm-toolset-7-clang, which is needed for postgresql13-devel-13.2-1PGDG, required by TO rpm
dnf -y install gcc centos-release-scl-rh; \
else \
use_repo='--repo=pgdg13'; \
enable_repo='--enablerepo=powertools'; \
+ llvm_version=''; \
fi; \
dnf -y install "https://download.postgresql.org/pub/repos/yum/reporpms/EL-${RHEL_VERSION%%.*}-x86_64/pgdg-redhat-repo-latest.noarch.rpm"; \
- # libicu required by postgresql13
- dnf -y install libicu; \
- dnf -y $use_repo -- install postgresql13; \
dnf -y install epel-release; \
- dnf -y $enable_repo install \
+ dnf -y install \
+ # libicu is required by postgresql13
+ libicu \
+ # libicu-devel, clang-devel, and llvm-devel are required by postgresql13-devel
+ libicu-devel clang-devel llvm${llvm_version}-devel; \
+ dnf -y $use_repo -- install postgresql13 postgresql13-devel; \
+ dnf -y $enable_repo install \
bind-utils \
gettext \
git \
- golang \
# ip commands is used in set-to-ips-from-dns.sh
iproute \
isomd5sum \
jq \
- libidn-devel \
- libpcap-devel \
- mkisofs \
net-tools \
nmap-ncat \
openssl \
+ # rsync is used to copy certs in "Shared SSL certificate generation" step
+ rsync \
+
+ # Traffic Ops dependencies (Not all needed for CDN in a Box, but all
+ # required by the Traffic Ops RPM)
+ cpanminus \
+ expat-devel \
+ gcc-c++ \
+ golang \
+ libcap \
+ libcurl-devel \
+ libidn-devel \
+ libpcap-devel \
+ mkisofs \
+ openssl-devel \
+ perl-core \
perl-Crypt-ScryptKDF \
+ perl-DBD-Pg \
+ perl-DBI \
perl-Digest-SHA1 \
- perl-JSON-PP \
- python3 \
- # rsync is used to copy certs in "Shared SSL certificate generation" step
- rsync; \
+ perl-JSON \
+ perl-libwww-perl \
+ perl-TermReadKey \
+ perl-Test-CPAN-Meta \
+ perl-WWW-Curl; \
dnf clean all
EXPOSE 443
@@ -75,7 +95,7 @@ ADD traffic_router/core/src/test/resources/geo/GeoLite2-City.mmdb.gz /opt/traffi
WORKDIR /opt/traffic_ops/app
ADD traffic_ops/install/bin/install_goose.sh ./
-RUN ./install_goose.sh && rm ./install_goose.sh && dnf -y remove git && dnf clean all
+RUN ./install_goose.sh && rm ./install_goose.sh
ADD infrastructure/cdn-in-a-box/traffic_ops_data /traffic_ops_data
@@ -84,7 +104,7 @@ ADD infrastructure/cdn-in-a-box/traffic_ops_data /traffic_ops_data
ARG TRAFFIC_OPS_RPM=infrastructure/cdn-in-a-box/traffic_ops/traffic_ops.rpm
COPY $TRAFFIC_OPS_RPM /traffic_ops.rpm
-RUN yum -y install /traffic_ops.rpm && \
+RUN rpm -Uvh /traffic_ops.rpm && \
rm /traffic_ops.rpm
COPY infrastructure/cdn-in-a-box/enroller/server_template.json \
diff --git a/infrastructure/cdn-in-a-box/traffic_ops/config.sh b/infrastructure/cdn-in-a-box/traffic_ops/config.sh
index cb4898b..264763b 100755
--- a/infrastructure/cdn-in-a-box/traffic_ops/config.sh
+++ b/infrastructure/cdn-in-a-box/traffic_ops/config.sh
@@ -31,15 +31,12 @@
# ADMIN_PASS
# TO_HOST
# TO_PORT
-# TO_PERL_HOST
-# TO_PERL_PORT
# TP_HOST
#
# Check that env vars are set
envvars=( DB_SERVER DB_PORT DB_ROOT_PASS DB_USER DB_USER_PASS ADMIN_USER ADMIN_PASS DOMAIN TO_HOST TO_PORT TP_HOST)
-for v in $envvars
-do
- if [[ -z "${!v}" ]]; then echo "$v is unset"; exit 1; fi
+for v in $envvars; do
+ if [[ -z "${!v}" ]]; then echo "$v is unset"; exit 1; fi
done
until [[ -f "$X509_CA_ENV_FILE" ]]; do
@@ -49,8 +46,7 @@ done
# these expected to be stored in $X509_CA_ENV_FILE, but a race condition could render the contents
# blank until it gets sync'd. Ensure vars defined before writing cdn.conf.
-until [[ -v X509_GENERATION_COMPLETE && -n "$X509_GENERATION_COMPLETE" ]]
-do
+until [[ -v X509_GENERATION_COMPLETE && -n "$X509_GENERATION_COMPLETE" ]]; do
echo "Waiting on X509 vars to be defined"
sleep 1
source "$X509_CA_ENV_FILE"
@@ -70,24 +66,20 @@ if [[ "$TO_DEBUG_ENABLE" == true ]]; then
DEBUGGING_TIMEOUT=$(( 60 * 60 * 24 )); # Timing out debugging after 1 day seems fair
fi;
-cat <<-EOF >/opt/traffic_ops/app/conf/cdn.conf
+
+echo "$(<postinstall.json envsubst)" >postinstall.json
+
+cdn_conf=/opt/traffic_ops/app/conf/cdn.conf
+>"$cdn_conf" echo "$(jq -s '.[0] * .[1]' "$cdn_conf" <(cat <<-EOF
{
"hypnotoad" : {
"listen" : [
"https://[::]?cert=$crt&key=$key&verify=0x00&ciphers=AES128-GCM-SHA256:HIGH:!RC4:!MD5:!aNULL:!EDH:!ED"
- ],
- "user" : "trafops",
- "group" : "trafops",
- "heartbeat_timeout" : 20,
- "pid_file" : "/var/run/traffic_ops.pid",
- "workers" : 12
+ ]
},
"use_ims": true,
"traffic_ops_golang" : {
- "insecure": true,
- "port" : "$TO_PORT",
"proxy_timeout" : ${DEBUGGING_TIMEOUT:-60},
- "proxy_keep_alive" : 60,
"proxy_tls_timeout" : ${DEBUGGING_TIMEOUT:-60},
"proxy_read_header_timeout" : ${DEBUGGING_TIMEOUT:-60},
"read_timeout" : ${DEBUGGING_TIMEOUT:-60},
@@ -100,108 +92,103 @@ cat <<-EOF >/opt/traffic_ops/app/conf/cdn.conf
"log_location_info": "$TO_LOG_INFO",
"log_location_debug": "$TO_LOG_DEBUG",
"log_location_event": "$TO_LOG_EVENT",
- "max_db_connections": 20,
"db_conn_max_lifetime_seconds": ${DEBUGGING_TIMEOUT:-60},
- "db_query_timeout_seconds": ${DEBUGGING_TIMEOUT:-20},
- "backend_max_connections": {
- "mojolicious": 4
- },
- "whitelisted_oauth_urls": [],
- "oauth_client_secret": "",
- "routing_blacklist": {
- "ignore_unknown_routes": false,
- "disabled_routes": []
- },
- "supported_ds_metrics": [ "kbps", "tps_total", "tps_2xx", "tps_3xx", "tps_4xx", "tps_5xx" ]
- },
- "cors" : {
- "access_control_allow_origin" : "*"
+ "db_query_timeout_seconds": ${DEBUGGING_TIMEOUT:-20}
},
"to" : {
- "base_url" : "https://$TO_FQDN",
- "email_from" : "no-reply@$INFRA_SUBDOMAIN.$TLD_DOMAIN",
- "no_account_found_msg" : "A Traffic Ops user account is required for access. Please contact your Traffic Ops user administrator."
+ "email_from" : "no-reply@$INFRA_SUBDOMAIN.$TLD_DOMAIN"
},
"portal" : {
"base_url" : "https://$TP_HOST.$INFRA_SUBDOMAIN.$TLD_DOMAIN/#!/",
- "email_from" : "no-reply@$INFRA_SUBDOMAIN.$TLD_DOMAIN",
- "pass_reset_path" : "user",
- "user_register_path" : "user"
- },
- "secrets" : [
- "$TO_SECRET"
- ],
- "geniso" : {
- "iso_root_path" : "/opt/traffic_ops/app/public"
+ "email_from" : "no-reply@$INFRA_SUBDOMAIN.$TLD_DOMAIN"
},
- "inactivity_timeout" : 60,
"smtp" : {
"enabled" : true,
- "user" : "",
- "password" : "",
"address" : "${SMTP_FQDN}:${SMTP_PORT}"
},
"InfluxEnabled": true,
"influxdb_conf_path": "/opt/traffic_ops/app/conf/production/influx.conf",
"lets_encrypt" : {
- "user_email" : "",
- "send_expiration_email": false,
- "convert_self_signed": false,
- "renew_days_before_expiration": 30,
"environment": "staging"
- },
- "acme_renewal": {
- "summary_email": "",
- "renew_days_before_expiration": 30
- },
- "acme_accounts": [
- {
- "acme_provider" : "",
- "user_email" : "",
- "acme_url" : "",
- "kid" : "",
- "hmac_encoded" : ""
- }
- ]
+ }
}
EOF
+))"
-cat <<-EOF >/opt/traffic_ops/app/conf/production/database.conf
+<<RIAK_CONF cat >/opt/traffic_ops/app/conf/production/riak.conf
{
- "description": "Local PostgreSQL database on port 5432",
- "dbname": "$DB_NAME",
- "hostname": "$DB_FQDN",
- "user": "$DB_USER",
- "password": "$DB_USER_PASS",
- "port": "$DB_PORT",
- "ssl": false,
- "type": "Pg"
-}
-EOF
-
-cat <<-EOF >/opt/traffic_ops/app/db/dbconf.yml
-version: "1.0"
-name: dbconf.yml
-
-production:
- driver: postgres
- open: host=$DB_FQDN port=$DB_PORT user=$DB_USER password=$DB_USER_PASS dbname=$DB_NAME sslmode=disable
-test:
- driver: postgres
- open: host=$DB_FQDN port=$DB_PORT user=$DB_USER password=$DB_USER_PASS dbname=to_test sslmode=disable
-EOF
-
-cat <<-EOF >/opt/traffic_ops/app/conf/production/riak.conf
-{ "user": "$TV_RIAK_USER",
+ "MaxTLSVersion": "1.1",
"password": "$TV_RIAK_PASSWORD",
- "MaxTLSVersion": "1.1"
+ "user": "$TV_RIAK_USER"
}
-EOF
+RIAK_CONF
-cat <<-EOF >/opt/traffic_ops/app/conf/production/influx.conf
+<<INFLUX_CONF cat >/opt/traffic_ops/app/conf/production/influx.conf
{
- "user": "$INFLUXDB_ADMIN_USER",
- "password": "$INFLUXDB_ADMIN_PASSWORD",
- "secure": false
+ "password": "$INFLUXDB_ADMIN_PASSWORD",
+ "secure": false,
+ "user": "$INFLUXDB_ADMIN_USER"
}
-EOF
+INFLUX_CONF
+
+install_bin=/opt/traffic_ops/install/bin
+input_json="${install_bin}/input.json"
+echo "$(jq "$(<<'JQ_FILTER' envsubst
+ ."/opt/traffic_ops/app/conf/cdn.conf"[] |= (
+ (select(.config_var == "base_url") |= with_entries(if .key | test("^[A-Z]") then .value =
+ "${TO_URL}"
+ else . end))
+ ) |
+ ."/opt/traffic_ops/app/conf/production/database.conf"[] |= (
+ (select(.config_var == "dbname") |= with_entries(if .key | test("^[A-Z]") then .value =
+ "${DB_NAME}"
+ else . end)) |
+ (select(.config_var == "hostname") |= with_entries(if .key | test("^[A-Z]") then .value =
+ "${DB_FQDN}"
+ else . end)) |
+ (select(.config_var == "user") |= with_entries(if .key | test("^[A-Z]") then .value =
+ "${DB_USER}"
+ else . end)) |
+ (select(.config_var == "password") |= with_entries(if .key | test("^[A-Z]") then .value =
+ "${DB_USER_PASS}"
+ else . end))
+ ) |
+ ."/opt/traffic_ops/app/db/dbconf.yml"[] |= (
+ (select(.config_var == "pgUser") |= with_entries(if .key | test("^[A-Z]") then .value =
+ "${DB_USER}"
+ else . end)) |
+ (select(.config_var == "pgPassword") |= with_entries(if .key | test("^[A-Z]") then .value =
+ "${DB_USER_PASS}"
+ else . end))
+ ) |
+ ."/opt/traffic_ops/install/data/json/openssl_configuration.json"[] |= (
+ (select(.config_var == "genCert") |= with_entries(if .key | test("^[A-Z]") then .value =
+ "no"
+ else . end)) |
+ (select(.config_var == "pgPassword") |= with_entries(if .key | test("^[A-Z]") then .value =
+ "${DB_USER_PASS}"
+ else . end))
+ ) |
+ ."/opt/traffic_ops/install/data/json/profiles.json"[] |= (
+ (select(.config_var == "tm.url") |= with_entries(if .key | test("^[A-Z]") then .value =
+ "${TO_URL}"
+ else . end)) |
+ (select(.config_var == "cdn_name") |= with_entries(if .key | test("^[A-Z]") then .value =
+ "${CDN_NAME}"
+ else . end)) |
+ (select(.config_var == "dns_subdomain") |= with_entries(if .key | test("^[A-Z]") then .value =
+ "${CDN_SUBDOMAIN}.${TLD_DOMAIN}"
+ else . end))
+ ) |
+ ."/opt/traffic_ops/install/data/json/users.json"[] |= (
+ (select(.config_var == "tmAdminUser") |= with_entries(if .key | test("^[A-Z]") then .value =
+ "${TO_ADMIN_USER}"
+ else . end)) |
+ (select(.config_var == "tmAdminPw") |= with_entries(if .key | test("^[A-Z]") then .value =
+ "${TO_ADMIN_PASSWORD}"
+ else . end))
+ )
+JQ_FILTER
+)" "$input_json")" >"$input_json"
+
+"${install_bin}/postinstall" -a --cfile "$input_json" -n --no-restart-to
diff --git a/infrastructure/cdn-in-a-box/traffic_ops/run-go.sh b/infrastructure/cdn-in-a-box/traffic_ops/run-go.sh
index 237d8b9..0a89b84 100755
--- a/infrastructure/cdn-in-a-box/traffic_ops/run-go.sh
+++ b/infrastructure/cdn-in-a-box/traffic_ops/run-go.sh
@@ -113,8 +113,6 @@ while ! $pg_isready -h "$DB_SERVER" -p "$DB_PORT" -d "$DB_NAME"; do
done
cd /opt/traffic_ops/app;
-./db/admin --env=production reset;
-./db/admin --env=production upgrade;
# Add admin user -- all other users should be created using the API
/adduser.pl "$TO_ADMIN_USER" "$TO_ADMIN_PASSWORD" "admin" "root" | psql -v ON_ERROR_STOP=1 -U "$DB_USER" -h "$DB_SERVER" -d "$DB_NAME";
diff --git a/infrastructure/cdn-in-a-box/variables.env b/infrastructure/cdn-in-a-box/variables.env
index d617031..c9777b5 100644
--- a/infrastructure/cdn-in-a-box/variables.env
+++ b/infrastructure/cdn-in-a-box/variables.env
@@ -83,7 +83,6 @@ TS_DEBUG_ENABLE=false
TO_EMAIL=cdnadmin@example.com
TO_HOST=trafficops
TO_PORT=443
-TO_SECRET=blahblah
TO_LOG_ERROR=/var/log/traffic_ops/error.log
TO_LOG_WARNING=/var/log/traffic_ops/warning.log
TO_LOG_INFO=/var/log/traffic_ops/info.log
diff --git a/traffic_ops/install/bin/_postinstall b/traffic_ops/install/bin/_postinstall
index e56498d..ae1213b 100755
--- a/traffic_ops/install/bin/_postinstall
+++ b/traffic_ops/install/bin/_postinstall
@@ -1,5 +1,5 @@
-#!/usr/bin/perl
-
+#!/usr/bin/env bash
+"exec" "bash" "-c" "PATH+=:/usr/libexec/; exec \$(type -p python38 python3.8 python36 python3.8 python3 python python27 python2.7 python2 platform-python | head -n1) \"$0\" $*"
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -13,937 +13,1513 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-
-use lib qw(/opt/traffic_ops/install/lib /opt/traffic_ops/app/lib /opt/traffic_ops/app/local/lib/perl5);
-
-$ENV{PERL5LIB} = "/opt/traffic_ops/install/lib:/opt/traffic_ops/app/lib:/opt/traffic_ops/app/local/lib/perl5:$ENV{PERL5LIB}";
-$ENV{PATH} = "/usr/bin:/opt/traffic_ops/go/bin:/usr/local/go/bin:/opt/traffic_ops/install/bin:$ENV{PATH}";
-$ENV{GOPATH} = "/opt/traffic_ops/go";
-
-use strict;
-use warnings;
-
-use DBI;
-use POSIX;
-use File::Basename qw{dirname};
-use File::Path qw{make_path};
-use Crypt::ScryptKDF qw(scrypt_hash);
-use Data::Dumper qw(Dumper);
-use Scalar::Util qw(looks_like_number);
-use Getopt::Long;
-
-use InstallUtils qw{ :all };
-use GenerateCert qw{ :all };
-use Database qw{ connect };
-
-# paths of the output configuration files
-my $databaseConfFile = "/opt/traffic_ops/app/conf/production/database.conf";
-my $dbConfFile = "/opt/traffic_ops/app/db/dbconf.yml";
-my $cdnConfFile = "/opt/traffic_ops/app/conf/cdn.conf";
-my $ldapConfFile = "/opt/traffic_ops/app/conf/ldap.conf";
-my $usersConfFile = "/opt/traffic_ops/install/data/json/users.json";
-my $profilesConfFile = "/opt/traffic_ops/install/data/profiles/";
-my $opensslConfFile = "/opt/traffic_ops/install/data/json/openssl_configuration.json";
-my $paramConfFile = "/opt/traffic_ops/install/data/json/profiles.json";
-
-my $custom_profile_dir = $profilesConfFile . "custom";
-
-# stores parameters for traffic ops config
-my $parameters;
-
-# location of traffic ops profiles
-my $profileDir = "/opt/traffic_ops/install/data/profiles/";
-my $post_install_cfg = "/opt/traffic_ops/install/data/json/post_install.json";
-
-# log file for the installer
-my $logFile = "/var/log/traffic_ops/postinstall.log";
-
-# debug mode
-my $debug = 1;
-
-# log file for cpan output
-my $cpanLogFile = "/var/log/traffic_ops/cpan.log";
-
-# maximum size the uncompressed log file should be before rotating it - rotating it copies the current log
-# file to the same name appended with .bkp replacing the old backup if any is there
-my $maxLogSize = 10000000; #bytes
-
-# whether to create a config file with default values
-my $dumpDefaults;
-
-# configuration file output with answers which can be used as input to postinstall
-my $outputConfigFile = "/opt/traffic_ops/install/bin/configuration_file.json";
-
-my $inputFile = "";
-my $automatic = 0;
-my %defaultInputs;
-
-# given a var to the hash of config_var and question, will return the question
-sub getConfigQuestion {
- my $var = shift;
- foreach my $key ( keys %{ $var } ) {
- if ( $key ne "hidden" && $key ne "config_var" ) {
- return $key;
- }
- }
-}
-
-# question: The question given in the config file
-# config_answer: The answer given in the config file - if no config file given will be defaultInput
-# hidden: Whether or not the answer should be hidden from the terminal and logs, ex. passwords
-#
-# Determines if the script is being run in complete interactive mode and prompts user - otherwise
-# returns answer to question in config or defaults
-
-sub getField {
- my $question = shift;
- my $config_answer = shift;
- my $hidden = shift;
-
- # if there is no config file and not in automatic mode prompt for all questions with default answers
- if ( !$inputFile && !$automatic ) {
-
- # if hidden then dont show password in terminal
- if ($hidden) {
- return InstallUtils::promptPasswordVerify($question);
- }
- else {
- return InstallUtils::promptUser( $question, $config_answer );
- }
- }
-
- return $config_answer;
-}
-
-# userInput: The entire input config file which is either user input or the defaults
-# fileName: The name of the output config file given by the input config file
-#
-# Loops through an input config file and determines answers to each question using getField
-# and returns the hash of answers
-
-sub getConfig {
- my %userInput = %{$_[0]}; shift;
- my $fileName = shift;
-
- my %config;
-
- if ( !defined $userInput{$fileName} ) {
- InstallUtils::logger( "No $fileName found in config", "error" );
- }
-
- InstallUtils::logger( "===========$fileName===========", "info" );
-
- foreach my $var ( @{ $userInput{$fileName} } ) {
- my $question = getConfigQuestion($var);
- my $hidden = $var->{"hidden"} if ( exists $var->{"hidden"} );
- my $answer = $config{ $var->{"config_var"} } = getField( $question, $var->{$question}, $hidden );
-
- $config{ $var->{"config_var"} } = $answer;
- if ( !$hidden ) {
- InstallUtils::logger( "$question: $answer", "info" );
- }
- }
- return %config;
-}
-
-# userInput: The entire input config file which is either user input or the defaults
-# dbFileName: The filename of the output config file for the database
-# toDBFileName: The filename of the output config file for the Traffic Ops database
-#
-# Generates a config file for the database based on the questions and answers in the input config file
-
-sub generateDbConf {
- my %userInput = %{$_[0]}; shift;
- my $dbFileName = shift;
- my $toDBFileName = shift;
-
- my %dbconf = getConfig( \%userInput, $dbFileName );
- $dbconf{"description"} = "$dbconf{type} database on $dbconf{hostname}:$dbconf{port}";
- make_path( dirname($dbFileName), { mode => 0755 } );
- InstallUtils::writeJson( $dbFileName, \%dbconf );
- InstallUtils::logger( "Database configuration has been saved", "info" );
-
- # broken out into separate file/config area
- my %todbconf = getConfig( \%userInput, $toDBFileName );
-
- # Check if the Postgres db is used and set the driver to be "postgres"
- my $dbDriver = $dbconf{type};
- if ( $dbconf{type} eq "Pg" ) {
- $dbDriver = "postgres";
- }
-
- # No YAML library installed, but this is a simple file..
- open( my $fh, '>', $toDBFileName ) or errorOut("Can't write to $toDBFileName!");
- print $fh "production:\n";
- print $fh " driver: $dbDriver\n";
- print $fh " open: host=$dbconf{hostname} port=$dbconf{port} user=$dbconf{user} password=$dbconf{password} dbname=$dbconf{dbname} sslmode=disable\n";
- close $fh;
-
- return \%todbconf;
-}
-
-# userInput: The entire input config file which is either user input or the defaults
-# fileName: The filename of the output config file
-#
-# Generates a config file for the CDN
-
-sub generateCdnConf {
- my %userInput = %{$_[0]}; shift;
- my $fileName = shift;
-
- my %cdnConfiguration = getConfig( \%userInput, $fileName );
-
- # First, read existing one -- already loaded with a bunch of stuff
- my $cdnConf;
- if ( -f $fileName ) {
- $cdnConf = InstallUtils::readJson($fileName) or errorOut("Error loading $fileName: $@");
- }
- if ( lc $cdnConfiguration{genSecret} =~ /^y(?:es)?/ ) {
- my @secrets;
- my $newSecret = InstallUtils::randomWord();
-
- if (defined($cdnConf->{secrets})) {
- @secrets = @{ $cdnConf->{secrets} };
- $cdnConf->{secrets} = \@secrets;
- InstallUtils::logger( "Secrets found in cdn.conf file", "debug" );
- } else {
- $cdnConf->{secrets} = \@secrets;
- InstallUtils::logger( "No secrets found in cdn.conf file", "debug" );
- }
- unshift @secrets, InstallUtils::randomWord();
- if ( $cdnConfiguration{keepSecrets} > 0 && $#secrets > $cdnConfiguration{keepSecrets} - 1 ) {
-
- # Shorten the array to requested length
- $#secrets = $cdnConfiguration{keepSecrets} - 1;
- }
- }
- if (exists $cdnConfiguration{base_url}) {
- $cdnConf->{to}{base_url} = $cdnConfiguration{base_url};
- }
- if (exists $cdnConfiguration{port}) {
- $cdnConf->{"traffic_ops_golang"}{port} = $cdnConfiguration{port};
- }
- $cdnConf->{"traffic_ops_golang"}{"log_location_error"} = "/var/log/traffic_ops/error.log";
- $cdnConf->{"traffic_ops_golang"}{"log_location_event"} = "/var/log/traffic_ops/access.log";
-
- $cdnConf->{hypnotoad}{workers} = $cdnConfiguration{workers};
- #InstallUtils::logger("cdnConf: " . Dumper($cdnConf), "info" );
- InstallUtils::writeJson( $fileName, $cdnConf );
- InstallUtils::logger( "CDN configuration has been saved", "info" );
-}
-
-sub hash_pass {
- my $pass = shift;
- return scrypt_hash($pass, \64, 16384, 8, 1, 64);
-}
-
-# userInput: The entire input config file which is either user input or the defaults
-# fileName: The filename of the output config file
-#
-# Generates an LDAP config file
-
-sub generateLdapConf {
- my %userInput = %{$_[0]}; shift;
- my $fileName = shift;
- my %ldapInput = %{@{$userInput{$fileName}}[0]};
- my $useLdap = $ldapInput{"Do you want to set up LDAP?"};
-
- if ( !lc $useLdap =~ /^y(?:es)?/ ) {
- InstallUtils::logger( "Not setting up ldap", "info" );
- return;
- }
-
- my %ldapConf = getConfig( \%userInput, $fileName );
- # convert any deprecated keys to the correct key name
- my %keys_converted = ( password => 'admin_pass', hostname => 'host' );
- for my $key (keys %ldapConf) {
- if ( exists $keys_converted{$key} ) {
- $ldapConf{ $keys_converted{$key} } = delete $ldapConf{$key};
- }
- }
-
- my @requiredKeys = qw{ host admin_dn admin_pass search_base search_query insecure ldap_timeout_secs };
- for my $k (@requiredKeys) {
- if (! exists $ldapConf{$k} ) {
- errorOut("$k is a required key in $fileName");
- }
- }
-
- delete $ldapConf{setupLdap};
-
- # do a very loose check of form -- 'host' must be hostname:port
- if ( $ldapConf{ host } !~ /^\S+:\d+$/ ) {
- errorOut("host in $fileName must be of form 'hostname:port'");
- }
-
- make_path( dirname($fileName), { mode => 0755 } );
- InstallUtils::writeJson( $fileName, \%ldapConf );
-}
-
-sub generateUsersConf {
- my %userInput = %{$_[0]}; shift;
- my $fileName = shift;
-
- my %user = ();
- my %config = getConfig( \%userInput, $fileName );
-
- $user{username} = $config{tmAdminUser};
- $user{password} = hash_pass( $config{tmAdminPw} );
-
- InstallUtils::writeJson( $fileName, \%user );
- $user{password} = $config{tmAdminPw};
- return \%user;
-}
-
-sub generateProfilesDir {
- my %userInput = %{$_[0]}; shift;
- my $fileName = shift;
-
- my $userIn = $userInput{$fileName};
-}
-
-sub generateOpenSSLConf {
- my %userInput = %{$_[0]}; shift;
- my $fileName = shift;
-
- my %config = getConfig( \%userInput, $fileName );
- return \%config;
-}
-
-sub generateParamConf {
- my %userInput = %{$_[0]}; shift;
- my $fileName = shift;
-
- my %config = getConfig( \%userInput, $fileName );
- InstallUtils::writeJson( $fileName, \%config );
- return \%config;
-}
-
-# check default values for missing config_var parameter
-sub sanityCheckDefaults {
- foreach my $file ( ( keys %defaultInputs ) ) {
- foreach my $defaultValue ( @{ $defaultInputs{$file} } ) {
- my $question = getConfigQuestion(\%$defaultValue);
-
- my %defaultValueHash = %$defaultValue;
- if ( !defined $defaultValueHash{"config_var"}
- || $defaultValueHash{"config_var"} eq "" )
- {
- errorOut("Question '$question' in file '$file' has no config_var");
- }
- }
- }
-}
-
-# userInput: The entire input config file which is either user input or the defaults
-#
-# Checks the input config file against the default inputs. If there is a question located in the default inputs which
-# is not located in the input config file it will output a warning message.
-
-sub sanityCheckConfig {
- my %userInput = %{$_[0]}; shift;
- my $diffs = 0;
-
- foreach my $file ( ( keys %defaultInputs ) ) {
- if ( !defined $userInput{$file} ) {
- InstallUtils::logger( "File '$file' found in defaults but not config file", "warn" );
- @{$userInput{$file}} = [];
- }
-
- foreach my $defaultValue ( @{ $defaultInputs{$file} } ) {
-
- my $found = 0;
- foreach my $configValue ( @{ $userInput{$file} } ) {
- if ( $defaultValue->{"config_var"} eq $configValue->{"config_var"} ) {
- $found = 1;
- }
- }
-
- # if the question is not found in the config file add it from defaults
- if ( !$found ) {
- my $question = getConfigQuestion($defaultValue);
- InstallUtils::logger( "Question '$question' found in defaults but not in '$file'", "warn" );
-
- my %temp;
- my $answer;
- my $hidden = exists $defaultValue->{"hidden"} && $defaultValue->{"hidden"} ? 1 : 0;
-
- # in automatic mode add the missing question with default answer
- if ($automatic) {
- $answer = $defaultValue->{$question};
- InstallUtils::logger( "Adding question '$question' with default answer " . ( $hidden ? "" : "'$answer'" ), "info" );
- }
-
- # in interactive mode prompt the user for answer to missing question
- else {
- InstallUtils::logger( "Prompting user for answer", "info" );
- if ($hidden) {
- $answer = InstallUtils::promptPasswordVerify($question);
- }
- else {
- $answer = InstallUtils::promptUser( $question, $defaultValue->{$question} );
- }
- }
-
- %temp = (
- "config_var" => $defaultValue->{"config_var"},
- $question => $answer
- );
-
- if ($hidden) {
- $temp{"hidden"} .= "true";
- }
-
- push @{ $userInput{$file} }, \%temp;
-
- $diffs++;
- }
- }
- }
-
- InstallUtils::logger( "File sanity check complete - found $diffs difference" . ( $diffs == 1 ? "" : "s" ), "info" );
-}
-
-# A function which returns the default inputs data structure. These questions and answers will be used if there is no
-# user input config file or if there are questions in the input config file which do not have answers
-
-sub getDefaults {
- return (
- $databaseConfFile => [
- {
- "Database type" => "Pg",
- "config_var" => "type"
- },
- {
- "Database name" => "traffic_ops",
- "config_var" => "dbname"
- },
- {
- "Database server hostname IP or FQDN" => "localhost",
- "config_var" => "hostname"
- },
- {
- "Database port number" => "5432",
- "config_var" => "port"
- },
- {
- "Traffic Ops database user" => "traffic_ops",
- "config_var" => "user"
- },
- {
- "Password for Traffic Ops database user" => "",
- "config_var" => "password",
- "hidden" => "true"
- }
- ],
- $dbConfFile => [
- {
- "Database server root (admin) user" => "postgres",
- "config_var" => "pgUser"
- },
- {
- "Password for database server admin" => "",
- "config_var" => "pgPassword",
- "hidden" => "true"
- },
- {
- "Download Maxmind Database?" => "yes",
- "config_var" => "maxmind"
- }
- ],
- $cdnConfFile => [
- {
- "Generate a new secret?" => "yes",
- "config_var" => "genSecret"
- },
- {
- "Number of secrets to keep?" => "1",
- "config_var" => "keepSecrets"
- },
- {
- "Port to serve on?" => "443",
- "config_var" => "port"
- },
- {
- "Number of workers?" => "12",
- "config_var" => "workers"
- },
- {
- "Traffic Ops url?" => "http://localhost:3000",
- "config_var" => "base_url"
- },
- {
- "ldap.conf location? (default is /opt/traffic_ops/app/conf/ldap.conf)" => "",
- "config_var" => "ldap_conf_location"
- }
- ],
- $ldapConfFile => [
- {
- "Do you want to set up LDAP?" => "no",
- "config_var" => "setupLdap"
- },
- {
- "LDAP server hostname" => "",
- "config_var" => "host"
- },
- {
- "LDAP Admin DN" => "",
- "config_var" => "admin_dn"
- },
- {
- "LDAP Admin Password" => "",
- "config_var" => "admin_pass",
- "hidden" => "true"
- },
- {
- "LDAP Search Base" => "",
- "config_var" => "search_base"
- },
- {
- "LDAP Search Query" => "",
- "config_var" => "search_query"
- },
- {
- "LDAP Skip TLS verify" => "",
- "config_var" => "insecure"
- },
- {
- "LDAP Timeout Seconds" => "",
- "config_var" => "ldap_timeout_secs"
- }
- ],
- $usersConfFile => [
- {
- "Administration username for Traffic Ops" => "admin",
- "config_var" => "tmAdminUser"
- },
- {
- "Password for the admin user" => "",
- "config_var" => "tmAdminPw",
- "hidden" => "true"
- }
- ],
- $profilesConfFile => [
- {
- "Add custom profiles?" => "no",
- "config_var" => "custom_profiles"
- }
- ],
- $opensslConfFile => [
- {
- "Do you want to generate a certificate?" => "yes",
- "config_var" => "genCert"
- },
- {
- "Country Name (2 letter code)" => "",
- "config_var" => "country"
- },
- {
- "State or Province Name (full name)" => "",
- "config_var" => "state"
- },
- {
- "Locality Name (eg, city)" => "",
- "config_var" => "locality"
- },
- {
- "Organization Name (eg, company)" => "",
- "config_var" => "company"
- },
- {
- "Organizational Unit Name (eg, section)" => "",
- "config_var" => "org_unit"
- },
- {
- "Common Name (eg, your name or your server's hostname)" => "",
- "config_var" => "common_name"
- },
- {
- "RSA Passphrase" => "CHANGEME!!",
- "config_var" => "rsaPassword",
- "hidden" => "true"
- }
- ],
- $paramConfFile => [
- {
- "Traffic Ops url" => "https://localhost",
- "config_var" => "tm.url"
- },
- {
- "Human-readable CDN Name. (No whitespace, please)" => "kabletown_cdn",
- "config_var" => "cdn_name"
- },
- {
- "DNS sub-domain for which your CDN is authoritative" => "cdn1.kabletown.net",
- "config_var" => "dns_subdomain"
- }
- ],
- );
-}
-
-# carried over from old postinstall
-#
-# todbconf: The database configuration to be used
-# opensslconf: The openssl configuration if any
-
-sub setupDatabaseData {
- my $dbh = shift;
- my $adminconf = shift;
- my $paramconf = shift;
- InstallUtils::logger( "paramconf " . Dumper($paramconf), "info" );
-
- my $result;
-
- my $q = <<"QUERY";
- select exists(select 1 from pg_tables where schemaname = 'public' and tablename = 'tm_user')
-QUERY
-
- my $stmt = $dbh->prepare($q);
- $stmt->execute();
-
- InstallUtils::logger( "Setting up the database data", "info" );
- my $tables_found;
- while ( my $row = $stmt->fetch() ) {
- $tables_found = $row->[0];
- }
- if ($tables_found) {
- InstallUtils::logger( "Found existing tables skipping table creation", "info" );
- } else {
- invoke_db_admin_pl("load_schema");
- }
- invoke_db_admin_pl("migrate");
- invoke_db_admin_pl("seed");
- invoke_db_admin_pl("patch");
-
- # Skip the insert if the admin 'username' is already there.
- my $hashed_passwd = hash_pass( $adminconf->{"password"} );
- my $insert_admin = <<"ADMIN";
- insert into tm_user (username, tenant_id, role, local_passwd, confirm_local_passwd)
- values ('$adminconf->{"username"}',
- (select id from tenant where name = 'root'),
- (select id from role where name = 'admin'),
- '$hashed_passwd',
- '$hashed_passwd' )
- ON CONFLICT (username) DO NOTHING;
-ADMIN
- $dbh->do($insert_admin);
-
- insert_cdn($dbh, $paramconf);
- insert_parameters($dbh, $paramconf);
- insert_profiles($dbh, $paramconf);
-
-
-}
-
-sub invoke_db_admin_pl {
- my $action = shift;
-
- chdir("/opt/traffic_ops/app");
- my $result = InstallUtils::execCommand( "db/admin", "--env=production", $action );
-
- if ( $result != 0 ) {
- errorOut("Database $action failed");
- }
- else {
- InstallUtils::logger( "Database $action succeeded", "info" );
- }
-
- return $result;
-}
-
-sub setupMaxMind {
- my $setupMaxmind = shift;
-
- my $result;
-
- if ( $setupMaxmind =~ /^y(?:es)?/ ) {
- InstallUtils::logger( "Downloading Maxmind data", "info" );
- chdir("/opt/traffic_ops/app/public/routing");
- $result = InstallUtils::execCommand("/usr/bin/wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz");
- if ( $result != 0 ) {
- InstallUtils::logger( "Failed to download MaxMind data", "error" );
- }
-
- $result = InstallUtils::execCommand("/usr/bin/wget https://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz");
- if ( $result != 0 ) {
- InstallUtils::logger( "Failed to download MaxMind data", "error" );
- }
- }
- else {
- InstallUtils::logger("Not downloading Maxmind data");
- }
-
+# There's a bug in asteroid with Python 3.9's NamedTuple being
+# recognized for the dynamically generated class that it is. Should be fixed
+# with the next release, but 'til then...
+#pylint:disable=inherit-non-class
+from __future__ import print_function
+
+"""
+This script is meant as a drop-in replacement for the old _postinstall Perl script.
+
+It does, however, offer several more command-line flags not present in the original, to aid in
+testing.
+
+-a, --automatic If there are questions in the config file which do not have answers,
+ the script will look to the defaults for the answer. If the answer is
+ not in the defaults the script will exit.
+--cfile [FILE] An input config file used to ask and answer questions.
+--debug Enables verbose logging output.
+--defaults [FILE] Writes out a configuration file with defaults which can be used as
+ input. If no FILE is given, writes to stdout.
+-n, --no-root Enable running as a non-root user (may cause failure).
+-r DIR, --root-directory DIR Set the directory to be treated as the system's root directory (e.g.
+ for testing). Default: /
+-u USER, --ops-user USER Specify a username to own Traffic Ops files and processes.
+ Default: trafops
+-g GROUP, --ops-group GROUP Specify the group to own Traffic Ops files and processes.
+ Default: trafops
+--no-restart-to Skip restarting Traffic Ops after configuration and database changes
+ are applied.
+--no-database Skip all database operations.
+
+>>> [c for c in [[a for a in b if not a.config_var] for b in DEFAULTS.values()] if c]
+[]
+"""
+import argparse
+import base64
+import errno
+import getpass
+import grp
+import hashlib
+import json
+import logging
+import os
+import pwd
+import random
+import re
+import shutil
+import stat
+import string
+import subprocess
+import sys
+
+from collections import namedtuple
+from struct import unpack, pack
+
+# Paths for output configuration files
+DATABASE_CONF_FILE = "/opt/traffic_ops/app/conf/production/database.conf"
+DB_CONF_FILE = "/opt/traffic_ops/app/db/dbconf.yml"
+CDN_CONF_FILE = "/opt/traffic_ops/app/conf/cdn.conf"
+LDAP_CONF_FILE = "/opt/traffic_ops/app/conf/ldap.conf"
+USERS_CONF_FILE = "/opt/traffic_ops/install/data/json/users.json"
+PROFILES_CONF_FILE = "/opt/traffic_ops/install/data/profiles/"
+OPENSSL_CONF_FILE = "/opt/traffic_ops/install/data/json/openssl_configuration.json"
+PARAM_CONF_FILE = "/opt/traffic_ops/install/data/json/profiles.json"
+
+
+POST_INSTALL_CFG = "/opt/traffic_ops/install/data/json/post_install.json"
+
+# Log file for the installer
+# TODO: determine if logging to a file should be directly supported.
+# LOG_FILE = "/var/log/traffic_ops/postinstall.log"
+
+# Log file for CPAN output
+# TODO: The Perl used to "rotate" this file on every run, for some reason. Should we?
+# CPAN_LOG_FILE = "/var/log/traffic_ops/cpan.log"
+
+# Configuration file output with answers which can be used as input to postinstall
+# TODO: Perl used to always write its defaults out to this file when requested.
+# Python, instead, outputs to stdout. This is breaking, but more flexible. Change it?
+# OUTPUT_CONFIG_FILE = "/opt/traffic_ops/install/bin/configuration_file.json"
+
+if sys.version_info.major >= 3:
+ # Accepting a string for json.dump()'s `indent` keyword argument is a Python 3 feature
+ indent = "\t" # type: str
+else:
+ indent = 4 # type: int
+ str = unicode # type: type[unicode]
+
+class Question(object):
+ """
+ Question represents a single question to be asked of the user, to determine a configuration
+ value.
+
+ >>> Question("question", "answer", "var")
+ Question(question='question', default='answer', config_var='var', hidden=False)
+ """
+
+ def __init__(self, question, default, config_var, hidden = False): # type: (str, str, str, bool) -> None
+ self.question = question
+ self.default = default
+ self.config_var = config_var
+ self.hidden = hidden
+
+ def __str__(self): # type: () -> str
+ if self.default:
+ return "{question} [{default}]: ".format(question=self.question, default=self.default)
+ return "{question}: ".format(question=self.question)
+
+ def __repr__(self): # type: () -> str
+ qstn = self.question
+ ans = self.default
+ cfgvr = self.config_var
+ hddn = self.hidden
+ return "Question(question='{qstn}', default='{ans}', config_var='{cfgvr}', hidden={hddn})".format(qstn=qstn, ans=ans, cfgvr=cfgvr, hddn=hddn)
+
+ def ask(self): # type: () -> str
+ """
+ Asks the user the Question interactively.
+
+ If 'hidden' is true, output will not be echoed.
+ """
+ if self.hidden:
+ while True:
+ passwd = getpass.getpass(str(self))
+ if not passwd:
+ continue
+ if passwd == getpass.getpass("Re-Enter {question}: ".format(question=self.question)):
+ return passwd
+ print("Error: passwords do not match, try again")
+ ipt = input(self)
+ return ipt if ipt else self.default
+
+ def to_json(self): # type: () -> str
+ """
+ Converts a question to JSON encoding.
+
+ >>> Question("Do the thing?", "yes", "cfg_var", True).to_json()
+ '{"Do the thing?": "yes", "config_var": "cfg_var", "hidden": true}'
+ >>> Question("Do the other thing?", "no", "other cfg_var").to_json()
+ '{"Do the other thing?": "no", "config_var": "other cfg_var"}'
+ """
+ qstn = self.question
+ ans = self.default
+ cfgvr = self.config_var
+ if self.hidden:
+ return '{{"{qstn}": "{ans}", "config_var": "{cfgvr}", "hidden": true}}'.format(qstn=qstn, ans=ans, cfgvr=cfgvr)
+ return '{{"{qstn}": "{ans}", "config_var": "{cfgvr}"}}'.format(qstn=qstn, ans=ans, cfgvr=cfgvr)
+
+ def serialize(self): # type: () -> object
+ """Returns a serializable dictionary, suitable for converting to JSON."""
+ return {self.question: self.default, "config_var": self.config_var, "hidden": self.hidden}
+
+class User(namedtuple('User', ['username', 'password'])):
+ """Users represents a user that will be inserted into the Traffic Ops database.
+
+ Attributes
+ ----------
+ self.username: str
+ The user's username.
+ self.password: str
+ The user's password - IN PLAINTEXT.
+ """
+
+class SSLConfig:
+ """SSLConfig bundles the options for generating new (self-signed) SSL certificates"""
+
+ def __init__(self, gen_cert, cfg_map): # type: (bool, dict[str, str]) -> None
+
+ self.gen_cert = gen_cert
+ self.rsa_password = cfg_map["rsaPassword"]
+ self.params = "/C={country}/ST={state}/L={locality}/O={company}/OU={org_unit}/CN={common_name}/"
+ self.params = self.params.format(**cfg_map)
+
+class CDNConfig(namedtuple('CDNConfig', ['gen_secret', 'num_secrets', 'port', 'num_workers', 'url', 'ldap_conf_location'])):
+ """CDNConfig holds all of the options needed to format a cdn.conf file."""
+
+ def generate_secret(self, conf):
+ """
+ Generates new secrets - if configured to do so - and adds them to the passed cdn.conf
+ configuration.
+ """
+ if not self.gen_secret:
+ return
+
+ if isinstance(conf, dict) and "secrets" in conf and isinstance(conf["secrets"], list):
+ logging.debug("Secrets found in cdn.conf file")
+ else:
+ conf["secrets"] = []
+ logging.debug("No secrets found in cdn.conf file")
+
+ conf["secrets"].insert(0, random_word())
+
+ if self.num_secrets and len(conf["secrets"]) > self.num_secrets:
+ conf["secrets"] = conf["secrets"][:self.num_secrets - 1]
+
+ def insert_url(self, conf):
+ """
+ Inserts the configured URL - if it is not an empty string - into the passed cdn.conf
+ configuration, in to.base_url.
+ """
+ if not self.url:
+ return
+
+ if "to" not in conf or not isinstance(conf["to"], dict):
+ conf["to"] = {}
+ conf["to"]["base_url"] = self.url
+
+# The default question/answer set
+DEFAULTS = {
+ DATABASE_CONF_FILE: [
+ Question("Database type", "Pg", "type"),
+ Question("Database name", "traffic_ops", "dbname"),
+ Question("Database server hostname IP or FQDN", "localhost", "hostname"),
+ Question("Database port number", "5432", "port"),
+ Question("Traffic Ops database user", "traffic_ops", "user"),
+ Question("Password for Traffic Ops database user", "", "password", hidden=True)
+ ],
+ DB_CONF_FILE: [
+ Question("Database server root (admin) user", "postgres", "pgUser"),
+ Question("Password for database server admin", "", "pgPassword", hidden=True),
+ Question("Download Maxmind Database?", "yes", "maxmind")
+ ],
+ CDN_CONF_FILE: [
+ Question("Generate a new secret?", "yes", "genSecret"),
+ Question("Number of secrets to keep?", "1", "keepSecrets"),
+ Question("Port to serve on?", "443", "port"),
+ Question("Number of workers?", "12", "workers"),
+ Question("Traffic Ops url?", "http://localhost:3000", "base_url"),
+ Question("ldap.conf location?", "/opt/traffic_ops/app/conf/ldap.conf", "ldap_conf_location")
+ ],
+ LDAP_CONF_FILE:[
+ Question("Do you want to set up LDAP?", "no", "setupLdap"),
+ Question("LDAP server hostname", "", "host"),
+ Question("LDAP Admin DN", "", "admin_dn"),
+ Question("LDAP Admin Password", "", "admin_pass", hidden=True),
+ Question("LDAP Search Base", "", "search_base"),
+ Question("LDAP Search Query", "", "search_query"),
+ Question("LDAP Skip TLS verify", "", "insecure"),
+ Question("LDAP Timeout Seconds", "", "ldap_timeout_secs")
+ ],
+ USERS_CONF_FILE: [
+ Question("Administration username for Traffic Ops", "admin", "tmAdminUser"),
+ Question("Password for the admin user", "", "tmAdminPw", hidden=True)
+ ],
+ PROFILES_CONF_FILE: [
+ Question("Add custom profiles?", "no", "custom_profiles")
+ ],
+ OPENSSL_CONF_FILE: [
+ Question("Do you want to generate a certificate?", "yes", "genCert"),
+ Question("Country Name (2 letter code)", "", "country"),
+ Question("State or Province Name (full name)", "", "state"),
+ Question("Locality Name (eg, city)", "", "locality"),
+ Question("Organization Name (eg, company)", "", "company"),
+ Question("Organizational Unit Name (eg, section)", "", "org_unit"),
+ Question("Common Name (eg, your name or your server's hostname)", "", "common_name"),
+ Question("RSA Passphrase", "CHANGEME!!", "rsaPassword", hidden=True)
+ ],
+ PARAM_CONF_FILE: [
+ Question("Traffic Ops url", "https://localhost", "tm.url"),
+ Question("Human-readable CDN Name. (No whitespace, please)", "kabletown_cdn", "cdn_name"),
+ Question(
+ "DNS sub-domain for which your CDN is authoritative",
+ "cdn1.kabletown.net",
+ "dns_subdomain"
+ )
+ ]
}
-sub setupCertificates {
- my $opensslconf = shift;
-
- my $result;
-
- if ( lc $opensslconf->{"genCert"} =~ /^y(?:es)?/ ) {
- if ( -x "/usr/bin/openssl" ) {
- InstallUtils::logger( "Installing SSL Certificates", "info" );
- $result = GenerateCert::createCert($opensslconf);
-
- if ( $result != 0 ) {
- errorOut("SSL Certificate Installation failed");
- }
- else {
- InstallUtils::logger( "SSL Certificates have been installed", "info" );
- }
- }
- else {
- InstallUtils::logger( "Unable to install SSL certificates as openssl is not installed", "error" );
- InstallUtils::logger( "Install openssl and then run /opt/traffic_ops/install/bin/generateCert to install SSL certificates", "error" );
- exit 4;
- }
- }
- else {
- InstallUtils::logger( "Not generating openssl certification", "info" );
- }
-}
-
-#------------------------------------
-sub insert_cdn {
-
- my $dbh = shift;
- my $paramconf = shift;
-
- InstallUtils::logger( "=========== Setting up cdn", "info" );
-
- # Enable multiple inserts into one commit
- $dbh->{pg_server_prepare} = 0;
-
- my $cdn_name = $paramconf->{"cdn_name"};
- my $dns_subdomain = $paramconf->{"dns_subdomain"};
-
- my $insert_stmt = <<INSERTS;
-
- -- global parameters
- insert into cdn (name, domain_name, dnssec_enabled)
- values ('$cdn_name', '$dns_subdomain', false)
- ON CONFLICT (name) DO NOTHING;
-
-INSERTS
- doInsert($dbh, $insert_stmt);
-}
-
-#------------------------------------
-sub insert_parameters {
- my $dbh = shift;
- my $paramconf = shift;
-
- InstallUtils::logger( "=========== Setting up parameters", "info" );
-
- # Enable multiple inserts into one commit
- $dbh->{pg_server_prepare} = 0;
-
- my $tm_url = $paramconf->{"tm.url"};
-
- my $insert_stmt = <<INSERTS;
- -- global parameters
- insert into parameter (name, config_file, value)
- values ('tm.url', 'global', '$tm_url')
- ON CONFLICT (name, config_file, value) DO NOTHING;
-
- insert into parameter (name, config_file, value)
- values ('tm.infourl', 'global', '$tm_url/doc')
- ON CONFLICT (name, config_file, value) DO NOTHING;
-
- -- CRConfig.json parameters
- insert into parameter (name, config_file, value)
- values ('geolocation.polling.url', 'CRConfig.json', '$tm_url/routing/GeoLite2-City.mmdb.gz')
- ON CONFLICT (name, config_file, value) DO NOTHING;
-
- insert into parameter (name, config_file, value)
- values ('geolocation6.polling.url', 'CRConfig.json', '$tm_url/routing/GeoLiteCityv6.dat.gz')
- ON CONFLICT (name, config_file, value) DO NOTHING;
-
-INSERTS
- doInsert($dbh, $insert_stmt);
-}
-
-#------------------------------------
-sub insert_profiles {
- my $dbh = shift;
- my $paramconf = shift;
-
- InstallUtils::logger( "\n=========== Setting up profiles", "info" );
- my $tm_url = $paramconf->{"tm.url"};
-
- my $insert_stmt = <<INSERTS;
-
- -- global parameters
- insert into profile (name, description, type, cdn)
- values ('GLOBAL', 'Global Traffic Ops profile, DO NOT DELETE', 'UNK_PROFILE', (SELECT id FROM cdn WHERE name='ALL'))
- ON CONFLICT (name) DO NOTHING;
-
- insert into profile_parameter (profile, parameter)
- values ( (select id from profile where name = 'GLOBAL'), (select id from parameter where name = 'tm.url' and config_file = 'global' and value = '$tm_url') )
- ON CONFLICT (profile, parameter) DO NOTHING;
-
- insert into profile_parameter (profile, parameter)
- values ( (select id from profile where name = 'GLOBAL'), (select id from parameter where name = 'tm.infourl' and config_file = 'global' and value = '$tm_url/doc') )
- ON CONFLICT (profile, parameter) DO NOTHING;
-
- insert into profile_parameter (profile, parameter)
- values ( (select id from profile where name = 'GLOBAL'), (select id from parameter where name = 'geolocation.polling.url' and config_file = 'CRConfig.json' and value = '$tm_url/routing/GeoLite2-City.mmdb.gz') )
- ON CONFLICT (profile, parameter) DO NOTHING;
-
- insert into profile_parameter (profile, parameter)
- values ( (select id from profile where name = 'GLOBAL'), (select id from parameter where name = 'geolocation6.polling.url' and config_file = 'CRConfig.json' and value = '$tm_url/routing/GeoLiteCityv6.dat.gz') )
- ON CONFLICT (profile, parameter) DO NOTHING;
-
-INSERTS
- doInsert($dbh, $insert_stmt);
-}
-
-#------------------------------------
-sub doInsert {
- my $dbh = shift;
- my $insert_stmt = shift;
+class ConfigEncoder(json.JSONEncoder):
+ """
+ ConfigEncoder encodes a dictionary of filenames to configuration question lists as JSON.
- InstallUtils::logger( "\n" . $insert_stmt, "info" );
- my $stmt = $dbh->prepare($insert_stmt);
- $stmt->execute();
-}
-
-
-
-# -cfile - Input File: The input config file used to ask and answer questions
-# -a - Automatic mode: If there are questions in the config file which do not have answers, the script
-# will look to the defaults for the answer. If the answer is not in the defaults
-# the script will exit
-# -defaults - Defaults: Writes out a configuration file with defaults which can be used as input
-# -debug - Debug Mode: More output to the terminal
-# -h - Help: Basic command line help menu
-
-sub main {
- my $help = 0;
-
- # help string
- my $usageString = "Usage: postinstall [-a] [-debug] [-defaults[=<outfile]] [-r] -cfile=[config_file]\n";
-
- GetOptions(
- "cfile=s" => \$inputFile,
- "automatic" => \$automatic,
- "defaults:s" => \$dumpDefaults,
- "debug" => \$debug,
- "help" => \$help
- ) or die($usageString);
-
- # stores the default questions and answers
- %defaultInputs = getDefaults();
-
- if ($help) {
- print $usageString;
- return;
- }
-
- # check if the user running postinstall is root
- if ( $> != 0 ) {
- errorOut("You must run this script as the root user");
- }
-
- InstallUtils::initLogger( $debug, $logFile );
-
- print("unzipping log\n");
- if ( -f "$logFile.gz" ) {
- InstallUtils::execCommand( "/bin/gunzip", "-f", "$logFile.gz" );
- }
-
- InstallUtils::logger( "Starting postinstall", "info" );
-
- InstallUtils::logger( "Debug is on", "info" );
-
- if ($automatic) {
- InstallUtils::logger( "Running in automatic mode", "info" );
- }
-
- if (defined $dumpDefaults) {
- # -defaults flag provided.
- if ($dumpDefaults ne "") {
- # -defaults=<filename> -- if -defaults without a file name, use the default.
- # dumpDefaults with value -- use that as output file name
- $outputConfigFile = $dumpDefaults;
- }
- InstallUtils::logger( "Writing default configuration to $outputConfigFile", "info" );
- InstallUtils::writeJson( $outputConfigFile, %defaultInputs );
- return;
- }
-
- InstallUtils::rotateLog($cpanLogFile);
-
- if ( -s $logFile > $maxLogSize ) {
- InstallUtils::logger( "Postinstall log above max size of $maxLogSize bytes - rotating", "info" );
- rotateLog($logFile);
- }
-
- # used to store the questions and answers provided by the user
- my %userInput;
-
- # if no input file provided use the defaults
- if ( $inputFile eq "" ) {
- InstallUtils::logger( "No input file given - using defaults", "info" );
- %userInput = %defaultInputs;
- }
- else {
- InstallUtils::logger( "Using input file $inputFile", "info" );
-
- # check if the input file exists
- errorOut("File '$inputFile' not found") if ( !-f $inputFile );
-
- # read and store the input file
- %userInput = %{InstallUtils::readJson($inputFile)};
- }
-
- # sanity check the defaults if running them automatically
- sanityCheckDefaults();
-
- # check the input config file against the defaults to check for missing questions
- sanityCheckConfig(\%userInput) if ( $inputFile ne "" );
-
- chdir("/opt/traffic_ops/install/bin");
-
- # The generator functions handle checking input/default/automatic mode
- # todbconf will be used later when setting up the database
- my $todbconf = generateDbConf( \%userInput, $databaseConfFile, $dbConfFile );
- generateLdapConf( \%userInput, $ldapConfFile );
- my $adminconf = generateUsersConf( \%userInput, $usersConfFile );
- my $custom_profile = generateProfilesDir( \%userInput, $profilesConfFile );
- my $opensslconf = generateOpenSSLConf( \%userInput, $opensslConfFile );
- my $paramconf = generateParamConf( \%userInput, $paramConfFile );
-
- if ( !-f $post_install_cfg ) {
- InstallUtils::writeJson( $post_install_cfg, {} );
- }
-
- setupMaxMind( $todbconf->{"maxmind"} );
- setupCertificates( $opensslconf );
- generateCdnConf( \%userInput, $cdnConfFile );
-
- my $dbh = Database::connect($databaseConfFile, $todbconf);
- if (!$dbh) {
- InstallUtils::logger("Can't connect to the database. Use the script `/opt/traffic_ops/install/bin/todb_bootstrap.sh` on the db server to create it and run `postinstall` again.", "error");
- exit(-1);
- }
-
- setupDatabaseData( $dbh, $adminconf, $paramconf );
-
- InstallUtils::logger("Starting Traffic Ops", "info" );
- InstallUtils::execCommand("/sbin/service traffic_ops restart");
-
- InstallUtils::logger("Waiting for Traffic Ops to restart", "info" );
-
- InstallUtils::logger("Success! Postinstall complete.");
-
- #InstallUtils::logger("Zipping up $logFile to $logFile.gz");
- #InstallUtils::execCommand( "/bin/gzip", "$logFile" );
-
- # Success!
- $dbh->disconnect();
-}
+ >>> ConfigEncoder().encode({'/test/file':[Question('question', 'default', 'cfg_var', True)]})
+ '{"/test/file": [{"question": "default", "config_var": "cfg_var", "hidden": true}]}'
+ """
-main;
+ # The linter is just wrong about this
+ def default(self, o): # type: (object) -> object
+ """
+ Returns a serializable representation of 'o'.
+
+ Specifically, it does this by attempting to convert a dictionary of filenames to Question
+ lists to a dictionary of filenames to lists of dictionaries of strings to strings, falling
+ back on default encoding if the proper typing is not found.
+ """
+ if isinstance(o, Question):
+ return o.serialize()
+
+ return json.JSONEncoder.default(self, o)
-# vi:syntax=perl
+def get_config(questions, fname, automatic = False): # type: (list[Question], str, bool) -> dict[str, str]
+ """Asks all provided questions, or uses their defaults in automatic mode"""
+
+ logging.info("===========%s===========", fname)
+
+ config = {}
+
+ for question in questions:
+ answer = question.default if automatic else question.ask()
+
+ config[question.config_var] = answer
+
+ return config
+
+def generate_db_conf(qstns, fname, automatic, root): # (list[Question], str, bool, str) -> dict
+ """
+ Generates the database.conf file and returns a map of its configuration.
+
+ Also writes the configuration file to the file 'fname' under the directory 'root'.
+ """
+ db_conf = get_config(qstns, fname, automatic)
+ typ = db_conf.get("type", "UNKNOWN")
+ hostname = db_conf.get("hostname", "UNKNOWN")
+ port = db_conf.get("port", "UNKNOWN")
+
+ db_conf["description"] = "{typ} database on {hostname}:{port}".format(typ=typ, hostname=hostname, port=port)
+
+ path = os.path.join(root, fname.lstrip('/'))
+ with open(path, 'w+') as conf_file:
+ json.dump(db_conf, conf_file, indent=indent)
+ print(file=conf_file)
+
+ logging.info("Database configuration has been saved")
+
+ return db_conf
+
+def generate_todb_conf(qstns, fname, auto, root, conf): # (list, str, bool, str, dict) -> dict
+ """
+ Generates the dbconf.yml file and returns a map of its configuration.
+
+ Also writes the configuration file to the file 'fname' under the directory 'root'.
+ """
+ todbconf = get_config(qstns, fname, auto)
+
+ driver = "postgres"
+ if "type" not in conf:
+ logging.warning("Driver type not found in todb config; using 'postgres'")
+ else:
+ driver = "postgres" if conf["type"] == "Pg" else conf["type"]
+
+ path = os.path.join(root, fname.lstrip('/'))
+ hostname = conf.get('hostname', 'UNKNOWN')
+ port = conf.get('port', 'UNKNOWN')
+ user = conf.get('user', 'UNKNOWN')
+ password = conf.get('password', 'UNKNOWN')
+ dbname = conf.get('dbname', 'UNKNOWN')
+
+ open_line = "host={hostname} port={port} user={user} password={password} dbname={dbname}".format(hostname=hostname, port=port, user=user, password=password, dbname=dbname)
+ with open(path, 'w+') as conf_file:
+ print("production:", file=conf_file)
+ print(" driver:", driver, file=conf_file)
+ print(" open: {open_line} sslmode=disable".format(open_line=open_line), file=conf_file)
+
+ return todbconf
+
+def generate_ldap_conf(questions, fname, automatic, root): # type: (list[Question], str, bool, str) -> None
+ """
+ Generates the ldap.conf file by asking the questions or using default answers in auto mode.
+
+ Also writes the configuration to the file 'fname' under the directory 'root'
+ """
+ use_ldap_question = [q for q in questions if q.question == "Do you want to set up LDAP?"]
+ if not use_ldap_question:
+ logging.warning("Couldn't find question asking if LDAP should be set up, using default: no")
+ return
+ use_ldap = use_ldap_question[0].default if automatic else use_ldap_question[0].ask()
+
+ if use_ldap.lower() not in {'y', 'yes'}:
+ logging.info("Not setting up ldap")
+ return
+
+ ldap_conf = get_config([q for q in questions if q is not use_ldap_question[0]], fname, automatic)
+ keys = (
+ 'host',
+ 'admin_dn',
+ 'admin_pass',
+ 'search_base',
+ 'search_query',
+ 'insecure',
+ 'ldap_timeout_secs'
+ )
+
+ for key in keys:
+ if key not in ldap_conf:
+ raise ValueError("{key} is a required key in {fname}".format(key=key, fname=fname))
+
+ if not re.match(r"^\S+:\d+$", ldap_conf["host"]):
+ raise ValueError("host in {fname} must be of form 'hostname:port'".format(fname=fname))
+
+ path = os.path.join(root, fname.lstrip('/'))
+ try:
+ os.makedirs(os.path.dirname(path))
+ except OSError as e:
+ if e.errno == errno.EEXIST:
+ pass
+ with open(path, 'w+') as conf_file:
+ json.dump(ldap_conf, conf_file, indent=indent)
+ print(file=conf_file)
+
+def hash_pass(passwd): # type: (str) -> str
+ """
+ Generates a Scrypt-based hash of the given password in a Perl-compatible format.
+ It's hard-coded - like the Perl - to use 64 random bytes for the salt, n=16384,
+ r=8, p=1 and dklen=64.
+ """
+ n = 2 ** 14
+ r_val = 8
+ p_val = 1
+ dklen = 64
+ salt = os.urandom(dklen)
+ if sys.version_info.major >= 3:
+ hashed = hashlib.scrypt(passwd.encode(), salt=salt, n=n, r=r_val, p=p_val, dklen=dklen)
+ else:
+ hashed = Scrypt(password=passwd.encode(), salt=salt, cost_factor=n, block_size_factor=r_val, parallelization_factor=p_val, key_length=dklen).derive()
+ hashed_b64 = base64.standard_b64encode(hashed).decode()
+ salt_b64 = base64.standard_b64encode(salt).decode()
+
+ return "SCRYPT:{n}:{r_val}:{p_val}:{salt_b64}:{hashed_b64}".format(n=n, r_val=r_val, p_val=p_val, salt_b64=salt_b64, hashed_b64=hashed_b64)
+
+
+class Scrypt:
+ def __init__(self, password, salt, cost_factor, block_size_factor, parallelization_factor, key_length): # type: (bytes, bytes, int, int, int, int) -> None
+ self.password = password # type: bytes
+ self.salt = salt # type: bytes
+ self.cost_factor = cost_factor # type: int
+ self.block_size_factor = block_size_factor # type: int
+ self.parallelization_factor = parallelization_factor # type: int
+ self.key_length = key_length
+ self.block_unit = 32 * self.block_size_factor # 1 block unit = 32 * block_size_factor 32-bit ints
+
+ def derive(self): # type: () -> bytes
+ salt_length = 2 ** 7 * self.block_size_factor * self.parallelization_factor # type: int
+ pack_format = '<' + 'L' * int(salt_length / 4) # `<` means `little-endian` and `L` means `unsigned long`
+ salt = hashlib.pbkdf2_hmac('sha256', password=self.password, salt=self.salt, iterations=1, dklen=salt_length) # type: bytes
+ block = list(unpack(pack_format, salt)) # type: list[int]
+ block = self.ROMix(block)
+ salt = pack(pack_format, *block)
+ key = hashlib.pbkdf2_hmac('sha256', password=self.password, salt=salt, iterations=1, dklen=self.key_length) # type: bytes
+ return key
+
+ def ROMix(self, block): # type: (list[int]) -> list[int]
+ xored_block = [0] * len(block) # type: list[int]
+ variations = [list()] * self.cost_factor # type: list[list[int]]
+ variations[0] = block
+ index = 1
+ while index < self.cost_factor:
+ variations[index] = self.block_mix(variations[index - 1])
+ index += 1
+ block = self.block_mix(variations[-1])
+ for unused in variations:
+ variation_index = block[self.block_unit - 16] % self.cost_factor # type: int
+ variation = variations[variation_index]
+ for index, unused in enumerate(xored_block):
+ xored_block[index] = block[index] ^ variation[index]
+ block = self.block_mix(xored_block)
+ return block
+
+ def block_mix(self, previous_block): # type: (list[int]) -> list[int]
+ block = previous_block[:] # type: list[int]
+ X_length = 16 # X is the list of numbers within `block` that we mix
+ copy_index = self.block_unit - X_length
+ X = previous_block[copy_index:copy_index + X_length] # type: list[int]
+ octet_index = 0 # type: int
+ block_xor_index = 0
+ while octet_index < 2 * self.block_size_factor:
+ for index, unused in enumerate(X):
+ X[index] ^= previous_block[block_xor_index + index]
+ block_xor_index += X_length
+ self.salsa20(X)
+ block_offset = (int(octet_index / 2) + octet_index % 2 * self.block_size_factor) * X_length
+ block[block_offset:block_offset + X_length] = X
+ octet_index += 1
+ return block
+
+ def salsa20(self, block): # type: (list[int]) -> None
+ X = block[:] # make a copy (list.copy() is Python 3-only)
+ for i in range(0, 4):
+ # These bit shifting operations could be condensed into a single line of list comprehensions,
+ # but there is a >3x performance benefit from writing it out explicitly.
+ bits = X[0] + X[12] & 0xffffffff
+ X[4] ^= bits << 7 | bits >> 32 - 7
+ bits = X[4] + X[0] & 0xffffffff
+ X[8] ^= bits << 9 | bits >> 32 - 9
+ bits = X[8] + X[4] & 0xffffffff
+ X[12] ^= bits << 13 | bits >> 32 - 13
+ bits = X[12] + X[8] & 0xffffffff
+ X[0] ^= bits << 18 | bits >> 32 - 18
+ bits = X[5] + X[1] & 0xffffffff
+ X[9] ^= bits << 7 | bits >> 32 - 7
+ bits = X[9] + X[5] & 0xffffffff
+ X[13] ^= bits << 9 | bits >> 32 - 9
+ bits = X[13] + X[9] & 0xffffffff
+ X[1] ^= bits << 13 | bits >> 32 - 13
+ bits = X[1] + X[13] & 0xffffffff
+ X[5] ^= bits << 18 | bits >> 32 - 18
+ bits = X[10] + X[6] & 0xffffffff
+ X[14] ^= bits << 7 | bits >> 32 - 7
+ bits = X[14] + X[10] & 0xffffffff
+ X[2] ^= bits << 9 | bits >> 32 - 9
+ bits = X[2] + X[14] & 0xffffffff
+ X[6] ^= bits << 13 | bits >> 32 - 13
+ bits = X[6] + X[2] & 0xffffffff
+ X[10] ^= bits << 18 | bits >> 32 - 18
+ bits = X[15] + X[11] & 0xffffffff
+ X[3] ^= bits << 7 | bits >> 32 - 7
+ bits = X[3] + X[15] & 0xffffffff
+ X[7] ^= bits << 9 | bits >> 32 - 9
+ bits = X[7] + X[3] & 0xffffffff
+ X[11] ^= bits << 13 | bits >> 32 - 13
+ bits = X[11] + X[7] & 0xffffffff
+ X[15] ^= bits << 18 | bits >> 32 - 18
+ bits = X[0] + X[3] & 0xffffffff
+ X[1] ^= bits << 7 | bits >> 32 - 7
+ bits = X[1] + X[0] & 0xffffffff
+ X[2] ^= bits << 9 | bits >> 32 - 9
+ bits = X[2] + X[1] & 0xffffffff
+ X[3] ^= bits << 13 | bits >> 32 - 13
+ bits = X[3] + X[2] & 0xffffffff
+ X[0] ^= bits << 18 | bits >> 32 - 18
+ bits = X[5] + X[4] & 0xffffffff
+ X[6] ^= bits << 7 | bits >> 32 - 7
+ bits = X[6] + X[5] & 0xffffffff
+ X[7] ^= bits << 9 | bits >> 32 - 9
+ bits = X[7] + X[6] & 0xffffffff
+ X[4] ^= bits << 13 | bits >> 32 - 13
+ bits = X[4] + X[7] & 0xffffffff
+ X[5] ^= bits << 18 | bits >> 32 - 18
+ bits = X[10] + X[9] & 0xffffffff
+ X[11] ^= bits << 7 | bits >> 32 - 7
+ bits = X[11] + X[10] & 0xffffffff
+ X[8] ^= bits << 9 | bits >> 32 - 9
+ bits = X[8] + X[11] & 0xffffffff
+ X[9] ^= bits << 13 | bits >> 32 - 13
+ bits = X[9] + X[8] & 0xffffffff
+ X[10] ^= bits << 18 | bits >> 32 - 18
+ bits = X[15] + X[14] & 0xffffffff
+ X[12] ^= bits << 7 | bits >> 32 - 7
+ bits = X[12] + X[15] & 0xffffffff
+ X[13] ^= bits << 9 | bits >> 32 - 9
+ bits = X[13] + X[12] & 0xffffffff
+ X[14] ^= bits << 13 | bits >> 32 - 13
+ bits = X[14] + X[13] & 0xffffffff
+ X[15] ^= bits << 18 | bits >> 32 - 18
+
+ for index in range(0, 16):
+ block[index] = block[index] + X[index] & 0xffffffff
+
+
+def generate_users_conf(qstns, fname, auto, root): # type: (list[Question], str, bool, str) -> User
+ """
+ Generates a users.json file from the given questions and returns a User containing the same
+ information.
+ """
+ config = get_config(qstns, fname, auto)
+
+ if "tmAdminUser" not in config or "tmAdminPw" not in config:
+ raise ValueError("{fname} must include 'tmAdminUser' and 'tmAdminPw'".format(fname=fname))
+
+ hashed_pass = hash_pass(config["tmAdminPw"])
+
+ path = os.path.join(root, fname.lstrip('/'))
+ with open(path, 'w+') as conf_file:
+ json.dump({"username": config["tmAdminUser"], "password": hashed_pass}, conf_file, indent=indent)
+ print(file=conf_file)
+
+ return User(config["tmAdminUser"], config["tmAdminPw"])
+
+def generate_profiles_dir(questions): # type: (list[Question]) -> None
+ """
+ I truly have no idea what's going on here. This is what the Perl did, so I
+ copied it. It does nothing. Literally nothing.
+ """
+ #pylint:disable=unused-variable
+ user_in = questions
+ #pylint:enable=unused-variable
+
+def generate_openssl_conf(questions, fname, auto): # type: (list[Question], str, bool) -> SSLConfig
+ """
+ Constructs an SSLConfig by asking the passed questions, or using their default answers if in
+ auto mode.
+ """
+ cfg_map = get_config(questions, fname, auto)
+ if "genCert" not in cfg_map:
+ raise ValueError("missing 'genCert' key")
+
+ gen_cert = cfg_map["genCert"].lower() in {"y", "yes"}
+
+ return SSLConfig(gen_cert, cfg_map)
+
+def generate_param_conf(qstns, fname, auto, root): # type: (list[Question], str, bool, str) -> dict
+ """
+ Generates a profiles.json by asking the passed questions, or using their default answers in auto
+ mode.
+
+ Also writes the file to 'fname' in the directory 'root'.
+ """
+ conf = get_config(qstns, fname, auto)
+
+ path = os.path.join(root, fname.lstrip('/'))
+ with open(path, 'w+') as conf_file:
+ json.dump(conf, conf_file, indent=indent)
+ print(file=conf_file)
+
+ return conf
+
+def sanity_check_config(cfg, automatic): # type: (dict[str, list[Question]], bool) -> int
+ """
+ Checks a user-input configuration file, and outputs the number of files in the
+ default question set that did not appear in the input.
+
+ :param cfg: The user's parsed input questions.
+ :param automatic: If :keyword:`True` all missing questions will use their default answers.
+ Otherwise, the user will be prompted for answers.
+ """
+ diffs = 0
+
+ for fname, file in DEFAULTS.items():
+ if fname not in cfg:
+ logging.warning("File '%s' found in defaults but not config file", fname)
+ cfg[fname] = []
+
+ for default_value in file:
+ for config_value in cfg[fname]:
+ if default_value.config_var == config_value.config_var:
+ break
+ else:
+ question = default_value.question
+ answer = default_value.default
+
+ if not automatic:
+ logging.info("Prompting user for answer")
+ if default_value.hidden:
+ answer = default_value.ask()
+ elif default_value.hidden:
+ logging.info("Adding question '%s' with default answer", question)
+ else:
+ logging.info("Adding question '%s' with default answer %s", question, answer)
+
+ # The Perl here would ask questions, but those would just get asked later
+ # anyway, so I'm not sure why.
+ cfg[fname].append(Question(question, answer, default_value.config_var, default_value.hidden))
+ diffs += 1
+
+ return diffs
+
+def unmarshal_config(dct): # type: (dict) -> dict[str, list[Question]]
+ """
+ Reads in a raw parsed configuration file and returns the resulting configuration.
+
+ >>> unmarshal_config({"test": [{"Do the thing?": "yes", "config_var": "thing"}]})
+ {'test': [Question(question='Do the thing?', default='yes', config_var='thing', hidden=False)]}
+ >>> unmarshal_config({"test": [{"foo": "", "config_var": "bar", "hidden": True}]})
+ {'test': [Question(question='foo', default='', config_var='bar', hidden=True)]}
+ """
+ ret = {}
+ for file, questions in dct.items():
+ if not isinstance(questions, list):
+ raise ValueError("file '{file}' has malformed questions".format(file=file))
+
+ qstns = []
+ for qstn in questions:
+ if not isinstance(qstn, dict):
+ raise ValueError("file '{file}' has a malformed question ({qstn})".format(file=file, qstn=qstn))
+ try:
+ question = next(key for key in qstn.keys() if key not in ("hidden", "config_var"))
+ except StopIteration:
+ raise ValueError("question in '{file}' has no question/answer properties ({qstn})".format(file=file, qstn=qstn))
+
+ answer = qstn[question]
+ if not isinstance(question, str) or not isinstance(answer, str):
+ errstr = "question in '{file}' has malformed question/answer property ({question}: {answer})".format(file=file, question=question, answer=answer)
+ raise ValueError(errstr)
+
+ del qstn[question]
+ hidden = False
+ if "hidden" in qstn:
+ hidden = bool(qstn["hidden"])
+ del qstn["hidden"]
+
+ if "config_var" not in qstn:
+ raise ValueError("question in '{file}' has no 'config_var' property".format(file=file))
+ cfg_var = qstn["config_var"]
+ if not isinstance(cfg_var, str):
+ raise ValueError("question in '{file}' has malformed 'config_var' property ({cfg_var})".format(file=file, cfg_var=cfg_var))
+ del qstn["config_var"]
+
+ if qstn:
+ logging.warning("Found unknown extra properties in question in '%s' (%r)", file, qstn.keys())
+
+ qstns.append(Question(question, answer, cfg_var, hidden=hidden))
+ ret[file] = qstns
+
+ return ret
+
+def setup_maxmind(maxmind_answer, root): # type: (str, str) -> None
+ """
+ If 'maxmind_answer' is a truthy response ('y' or 'yes' (case-insensitive), sets up a Maxmind
+ database using `wget`.
+ """
+ if maxmind_answer.lower() not in {'y', 'yes'}:
+ logging.info("Not downloading Maxmind data")
+ return
+
+ os.chdir(os.path.join(root, 'opt/traffic_ops/app/public/routing'))
+
+ def failed_download(e, ip_version): # type: (Exception, int) -> None
+ logging.error("Failed to download MaxMind data")
+ logging.debug("(ipv%d) Exception: %s", ip_version, e)
+
+ wget = "/usr/bin/wget"
+ cmd = [wget, "https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz"]
+ # Perl ignored errors downloading the databases, so we do too
+ try:
+ subprocess.check_call(
+ cmd,
+ stderr=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ universal_newlines=True
+ )
+ except subprocess.CalledProcessError as e:
+ failed_download(e, 4)
+ except subprocess.SubprocessError as e:
+ failed_download(e, 4)
+
+ cmd[1] = (
+ "https://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz"
+ )
+ try:
+ subprocess.check_call(
+ cmd,
+ stderr=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ universal_newlines=True
+ )
+ except subprocess.CalledProcessError as e:
+ failed_download(e, 6)
+ except subprocess.SubprocessError as e:
+ failed_download(e, 6)
+
+def exec_openssl(description, *cmd_args): # type: (str, ...) -> bool
+ """
+ Executes openssl with the supplied command-line arguments.
+
+ :param description: Describes the operation taking place for logging purposes.
+ :returns: Whether or not the execution succeeded, success being defined by an exit code of zero
+ """
+ logging.info(description)
+
+ cmd = ("/usr/bin/openssl",) + cmd_args
+
+ while True:
+ proc = subprocess.Popen(
+ cmd,
+ stderr=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ universal_newlines=True,
+ )
+ proc.wait()
+ if proc.returncode == 0:
+ return True
+
+ logging.debug("openssl exec failed with code %s; stderr: %s", proc.returncode, proc.stderr)
+ while True:
+ ans = input("{description} failed. Try again (y/n) [y]: ".format(description=description))
+ if not ans or ans.lower().startswith('n'):
+ return False
+ if ans.lower().startswith('y'):
+ break
+
+def setup_certificates(conf, root, ops_user, ops_group): # type: (SSLConfig, str, str, str) -> int
+ """
+ Generates self-signed SSL certificates from the given configuration.
+ :returns: For whatever reason this subroutine needs to dictate the return code of the script, so that's what it returns.
+ """
+ if not conf.gen_cert:
+ logging.info("Not generating openssl certification")
+ return 0
+
+ if not os.path.isfile('/usr/bin/openssl') or not os.access('/usr/bin/openssl', os.X_OK):
+ logging.error("Unable to install SSL certificates as openssl is not installed")
+ cmd = os.path.join(root, "opt/traffic_ops/install/bin/generateCert")
+ logging.error("Install openssl and then run %s to install SSL certificates", cmd)
+ return 4
+
+ logging.info("Installing SSL Certificates")
+ logging.info("\n\tWe're now running a script to generate a self signed X509 SSL certificate")
+ logging.info("Postinstall SSL Certificate Creation")
+
+ # Perl logs this before actually generating a key. So we do too.
+ logging.info("The server key has been generated")
+
+ args = (
+ "genrsa",
+ "-des3",
+ "-out",
+ "server.key",
+ "-passout",
+ "pass:{rsa_password}".format(rsa_password=conf.rsa_password),
+ "1024"
+ )
+ if not exec_openssl("Generating an RSA Private Server Key", *args):
+ return 1
+
+ args = (
+ "req",
+ "-new",
+ "-key",
+ "server.key",
+ "-out",
+ "server.csr",
+ "-passin",
+ "pass:{rsa_password}".format(rsa_password=conf.rsa_password),
+ "-subj",
+ conf.params
+ )
+ if not exec_openssl("Creating a Certificate Signing Request (CSR)", *args):
+ return 1
+
+ logging.info("The Certificate Signing Request has been generated")
+ os.rename("server.key", "server.key.orig")
+
+ args = (
+ "rsa",
+ "-in",
+ "server.key.orig",
+ "-out",
+ "server.key",
+ "-passin",
+ "pass:{rsa_password}".format(rsa_password=conf.rsa_password)
+ )
+ if not exec_openssl("Removing the pass phrase from the server key", *args):
+ return 1
+
+ logging.info("The pass phrase has been removed from the server key")
+
+ args = (
+ "x509",
+ "-req",
+ "-days",
+ "365",
+ "-in",
+ "server.csr",
+ "-signkey",
+ "server.key",
+ "-out",
+ "server.crt"
+ )
+ if not exec_openssl("Generating a Self-signed certificate", *args):
+ return 1
+
+ logging.info("A server key and self signed certificate has been generated")
+ logging.info("Installing a server key and certificate")
+
+ keypath = os.path.join(root, 'etc/pki/tls/private/localhost.key')
+ shutil.copy("server.key", keypath)
+ os.chmod(keypath, stat.S_IRUSR | stat.S_IWUSR)
+ os.chown(keypath, pwd.getpwnam(ops_user).pw_uid, grp.getgrnam(ops_group).gr_gid)
+
+ logging.info("The private key has been installed")
+ logging.info("Installing self signed certificate")
+
+ certpath = os.path.join(root, 'etc/pki/tls/certs/localhost.crt')
+ shutil.copy("server.crt", certpath)
+ os.chmod(certpath, stat.S_IRUSR | stat.S_IWUSR)
+ os.chown(certpath, pwd.getpwnam(ops_user).pw_uid, grp.getgrnam(ops_group).gr_gid)
+
+ logging.info("Saving the self signed csr")
+
+ csrpath = os.path.join(root, 'etc/pki/tls/certs/localhost.csr')
+ shutil.copy("server.csr", csrpath)
+ os.chmod(csrpath, stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IWGRP | stat.S_IROTH)
+ os.chown(csrpath, pwd.getpwnam(ops_user).pw_uid, grp.getgrnam(ops_group).gr_gid)
+
+ log_msg = """
+ The self signed certificate has now been installed.
+
+ You may obtain a certificate signed by a Certificate Authority using the
+ server.csr file saved in the current directory. Once you have obtained
+ a signed certificate, copy it to %s and
+ restart Traffic Ops."""
+ logging.info(log_msg, certpath)
+
+ cdn_conf_path = os.path.join(root, "opt/traffic_ops/app/conf/cdn.conf")
+
+ try:
+ with open(cdn_conf_path) as conf_file:
+ cdn_conf = json.load(conf_file)
+ except (OSError, ValueError) as e:
+ exception = OSError("reading {cdn_conf_path}: {e}".format(cdn_conf_path=cdn_conf_path, e=e))
+ exception.__cause__ = e
+ raise exception
+
+ if (
+ not isinstance(cdn_conf, dict) or
+ "hypnotoad" not in cdn_conf or
+ not isinstance(cdn_conf["hypnotoad"], dict)
+ ):
+ logging.critical("Malformed %s; improper object and/or missing 'hypnotoad' key", cdn_conf_path)
+ return 1
+
+ hypnotoad = cdn_conf["hypnotoad"]
+ if (
+ "listen" not in hypnotoad or
+ not isinstance(hypnotoad["listen"], list) or
+ not hypnotoad["listen"] or
+ not isinstance(hypnotoad["listen"][0], str)
+ ):
+ log_msg = """ The "listen" portion of %s is missing from %s
+ Please ensure it contains the same structure as the one originally installed"""
+ logging.error(log_msg, cdn_conf_path, cdn_conf_path)
+ return 1
+
+ listen = hypnotoad["listen"][0]
+
+ if "cert={certpath}".format(certpath=certpath) not in listen or "key={keypath}".format(keypath=keypath) not in listen:
+ log_msg = """ The "listen" portion of %s is:
+ %s
+ and does not reference the same "cert=" and "key=" values as are created here.
+ Please modify %s to add the following as parameters:
+ ?cert=%s&key=%s"""
+ logging.error(log_msg, cdn_conf_path, listen, cdn_conf_path, certpath, keypath)
+ return 1
+
+ return 0
+
+def random_word(length = 12): # type: (int) -> str
+ """
+ Returns a randomly generated string 'length' characters long containing only word
+ characters ([a-zA-Z0-9_]).
+ """
+ word_chars = string.ascii_letters + string.digits + '_'
+ return ''.join(random.choice(word_chars) for _ in range(length))
+
+def generate_cdn_conf(questions, fname, automatic, root): # type: (list[Question], str, bool, str) -> None
+ """
+ Generates some properties of a cdn.conf file based on the passed questions.
+
+ This modifies or writes the file 'fname' under the directory 'root'.
+ """
+ cdn_conf = get_config(questions, fname, automatic)
+
+ if "genSecret" not in cdn_conf:
+ raise ValueError("missing 'genSecret' config_var")
+
+ gen_secret = cdn_conf["genSecret"].lower() in {'y', 'yes'}
+
+ try:
+ num_secrets = int(cdn_conf["keepSecrets"])
+ except KeyError as e:
+ exception = ValueError("missing 'keepSecrets' config_var")
+ exception.__cause__ = e
+ raise exception
+ except ValueError as e:
+ exception = ValueError("invalid 'keepSecrets' config_var value: {e}".format(e=e))
+ exception.__cause__ = e
+ raise exception
+
+ try:
+ port = cdn_conf["port"] # type: str
+ except KeyError as e:
+ exception = ValueError("missing 'port' config_var")
+ exception.__cause__ = e
+ raise exception
+ except ValueError as e:
+ exception = ValueError("invalid 'port' config_var value: {e}".format(e=e))
+ exception.__cause__ = e
+ raise exception
+
+ try:
+ workers = int(cdn_conf["workers"])
+ except KeyError as e:
+ exception = ValueError("missing 'workers' config_var")
+ exception.__cause__ = e
+ raise exception
+ except ValueError as e:
+ exception = ValueError("invalid 'workers' config_var value: {e}".format(e=e))
+ exception.__cause__ = e
+ raise exception
+
+ try:
+ url = cdn_conf["base_url"]
+ except KeyError as e:
+ exception = ValueError("missing 'base_url' config_var")
+ exception.__cause__ = e
+ raise exception
+
+ try:
+ ldap_loc = cdn_conf["ldap_conf_location"]
+ except KeyError as e:
+ exception = ValueError("missing 'ldap_conf_location' config_var")
+ exception.__cause__ = e
+ raise exception
+
+ conf = CDNConfig(gen_secret, num_secrets, port, workers, url, ldap_loc)
+
+ path = os.path.join(root, fname.lstrip('/'))
+ existing_conf = {}
+ if os.path.isfile(path):
+ with open(path) as conf_file:
+ try:
+ existing_conf = json.load(conf_file)
+ except ValueError as e:
+ exception = ValueError("invalid existing cdn.config at {path}: {e}".format(path=path, e=e))
+ exception.__cause__ = e
+ raise exception
+
+ if not isinstance(existing_conf, dict):
+ logging.warning("Existing cdn.conf (at '%s') is not an object - overwriting", path)
+ existing_conf = {}
+
+ conf.generate_secret(existing_conf)
+ conf.insert_url(existing_conf)
+
+ if (
+ "traffic_ops_golang" not in existing_conf or
+ not isinstance(existing_conf["traffic_ops_golang"], dict)
+ ):
+ existing_conf["traffic_ops_golang"] = {}
+
+ existing_conf["traffic_ops_golang"]["port"] = conf.port
+ err_log = os.path.join(root, "var/log/traffic_ops/error.log")
+ existing_conf["traffic_ops_golang"]["log_location_error"] = err_log
+ access_log = os.path.join(root, "var/log/traffic_ops/access.log")
+ existing_conf["traffic_ops_golang"]["log_location_event"] = access_log
+
+ if "hypnotoad" not in existing_conf or not isinstance(existing_conf["hypnotoad"], dict):
+ existing_conf["hypnotoad"]["workers"] = conf.num_workers
+
+ with open(path, "w+") as conf_file:
+ json.dump(existing_conf, conf_file, indent=indent)
+ print(file=conf_file)
+ logging.info("CDN configuration has been saved")
+
+def db_connection_string(dbconf): # type: (dict) -> str
+ """
+ Constructs a database connection string from the passed configuration object.
+ """
+ user = dbconf["user"]
+ password = dbconf["password"]
+ db_name = "traffic_ops" if dbconf["type"] == "Pg" else dbconf["type"]
+ hostname = dbconf["hostname"]
+ port = dbconf["port"]
+ return "postgresql://{user}:{password}@{hostname}:{port}/{db_name}".format(user=user, password=password, hostname=hostname, port=port, db_name=db_name)
+
+def exec_psql(conn_str, query): # type: (str, str) -> str
+ """
+ Executes SQL queries by forking and exec-ing '/usr/bin/psql'.
+
+ :param conn_str: A "connection string" that defines the postgresql resource in the format
+ {schema}://{user}:{password}@{host or IP}:{port}/{database}
+ :param query: The query to be run. It can actually be a script containing multiple queries.
+ :returns: The comma-separated columns of each line-delimited row of the results of the query.
+ """
+ cmd = ["/usr/bin/psql", "--tuples-only", "-d", conn_str, "-c", query]
+ proc = subprocess.Popen(
+ cmd,
+ stderr=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ universal_newlines=True,
+ )
+ proc.wait()
+ if proc.returncode != 0:
+ logging.debug("psql exec failed; stderr: %s\n\tstdout: %s", proc.stderr, proc.stdout)
+ raise OSError("failed to execute database query")
+ if sys.version_info.major >= 3:
+ return proc.stdout.read().strip()
+ else:
+ return string.strip(proc.stdout.read())
+
+def invoke_db_admin_pl(action, root): # type: (str, str) -> None
+ """
+ Exectues admin with the given action, and looks for it from the given root directory.
+ """
+ path = os.path.join(root, "opt/traffic_ops/app")
+ # This is a workaround for admin using hard-coded relative paths. That
+ # should be fixed at some point, IMO, but for now this works.
+ os.chdir(path)
+ cmd = [os.path.join(path, "db/admin"), "--env=production", action]
+ proc = subprocess.Popen(
+ cmd,
+ stderr=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ universal_newlines=True,
+ )
+ output = proc.communicate() # type: str
+ if proc.returncode != 0:
+ logging.debug("admin exec failed; stderr: %s\n\tstdout: %s", output[1], output[0])
+ raise OSError("Database {action} failed".format(action=action))
+ logging.info("Database %s succeeded", action)
+
+def setup_database_data(conn_str, user, param_conf, root): # type: (str, User, dict, str) -> None
+ """
+ Sets up all necessary initial database data using `/usr/bin/sql`
+ """
+ logging.info("paramconf %s", param_conf)
+ logging.info("Setting up the database data")
+
+ tables_found_query = '''
+ SELECT EXISTS(
+ SELECT 1
+ FROM pg_tables
+ WHERE schemaname = 'public'
+ AND tablename = 'tm_user'
+ );'''
+ if exec_psql(conn_str, tables_found_query) == "t":
+ logging.info("Found existing tables skipping table creation")
+ else:
+ invoke_db_admin_pl("load_schema", root)
+
+ invoke_db_admin_pl("migrate", root)
+ invoke_db_admin_pl("seed", root)
+ invoke_db_admin_pl("patch", root)
+
+ hashed_pass = hash_pass(user.password)
+ insert_admin_query = '''
+ INSERT INTO tm_user (username, tenant_id, role, local_passwd, confirm_local_passwd)
+ VALUES (
+ '{}',
+ (SELECT id FROM tenant WHERE name = 'root'),
+ (SELECT id FROM role WHERE name = 'admin'),
+ '{hashed_pass}',
+ '{hashed_pass}'
+ )
+ ON CONFLICT (username) DO NOTHING;
+ '''.format(user.username, hashed_pass=hashed_pass)
+ _ = exec_psql(conn_str, insert_admin_query)
+
+ logging.info("=========== Setting up cdn")
+ insert_cdn_query = "\n\t-- global parameters" + '''
+ INSERT INTO cdn (name, domain_name, dnssec_enabled)
+ VALUES ('{cdn_name}', '{dns_subdomain}', false)
+ ON CONFLICT DO NOTHING;
+ '''.format(**param_conf)
+ logging.info("\n%s", insert_cdn_query)
+ _ = exec_psql(conn_str, insert_cdn_query)
+
+ tm_url = param_conf["tm.url"]
+
+ logging.info("=========== Setting up parameters")
+ insert_parameters_query = "\n\t-- global parameters" + '''
+ INSERT INTO parameter (name, config_file, value)
+ VALUES ('tm.url', 'global', '{tm_url}'),
+ ('tm.infourl', 'global', '{tm_url}/doc'),
+ -- CRConfic.json parameters
+ ('geolocation.polling.url', 'CRConfig.json', '{tm_url}/routing/GeoLite2-City.mmdb.gz'),
+ ('geolocation6.polling.url', 'CRConfig.json', '{tm_url}/routing/GeoLiteCityv6.dat.jz')
+ ON CONFLICT (name, config_file, value) DO NOTHING;
+ '''.format(tm_url=tm_url)
+ logging.info("\n%s", insert_parameters_query)
+ _ = exec_psql(conn_str, insert_parameters_query)
+
+ logging.info("\n=========== Setting up profiles")
+ insert_profiles_query = "\n\t-- global parameters" + '''
+ INSERT INTO profile (name, description, type, cdn)
+ VALUES ('GLOBAL' 'Global Traffic Ops profile, DO NOT DELETE', 'UNK_PROFILE', (SELECT id FROM cdn WHERE name='ALL'))
+ ON CONFLICT DO NOTHING;
+
+ INSERT INTO profile_parameter (profile, parameter)
+ VALUES
+ (
+ (SELECT id FROM profile WHERE name = 'GLOBAL'),
+ (
+ SELECT id
+ FROM parameter
+ WHERE name = 'tm.url'
+ AND config_file = 'global'
+ AND value = '{tm_url}'
+ )
+ ),
+ (
+ (SELECT id FROM profile WHERE name = 'GLOBAL'),
+ (
+ SELECT id
+ FROM parameter
+ WHERE name = 'tm.infourl'
+ AND config_file = 'global'
+ AND value = '{tm_url}/doc'
+ )
+ ),
+ (
+ (SELECT id FROM profile WHERE name = 'GLOBAL'),
+ (
+ SELECT id
+ FROM parameter
+ WHERE name = 'geolocation.polling.url'
+ AND config_file = 'CRConfig.json'
+ AND value = '{tm_url}/routing/GeoLite2-City.mmdb.gz'
+ )
+ ),
+ (
+ (SELECT id FROM profile WHERE name = 'GLOBAL'),
+ (
+ SELECT id
+ FROM parameter
+ WHERE name = 'geolocation6.polling.url'
+ AND config_file = 'CRConfig.json'
+ AND value = '{tm_url}/routing/GeoLiteCityv6.mmdb.gz'
+ )
+ )
+ ON CONFLICT (profile, parameter) DO NOTHING;
+ '''.format(tm_url=tm_url)
+ logging.info("\n%s", insert_profiles_query)
+ _ = exec_psql(conn_str, insert_cdn_query)
+
+def main(
+automatic, # type: bool
+debug, # type: bool
+defaults, # type: str
+cfile, # type: str
+root_dir, # type: str
+ops_user, # type: str
+ops_group, # type: str
+no_restart_to, # type: bool
+no_database, # type: bool
+):
+ """
+ Runs the main routine given the parsed arguments as input.
+ :rtype: int
+ """
+ if debug:
+ logging.getLogger().setLevel(logging.DEBUG)
+ else:
+ logging.getLogger().setLevel(logging.INFO)
+
+ # At this point, the Perl script... unzipped its own logfile?
+
+ logging.info("Starting postinstall")
+ # The Perl printed this whether or not the logger was actually at the debug level
+ # so we do too
+ logging.info("Debug is on")
+
+ if automatic:
+ logging.info("Running in automatic mode")
+
+ if defaults is not None:
+ try:
+ if defaults:
+ try:
+ with open(defaults, "w") as dump_file:
+ json.dump(DEFAULTS, dump_file, indent=indent)
+ except OSError as e:
+ logging.critical("Writing output: %s", e)
+ return 1
+ else:
+ json.dump(DEFAULTS, sys.stdout, cls=ConfigEncoder, indent=indent)
+ print()
+ except ValueError as e:
+ logging.critical("Converting defaults to JSON: %s", e)
+ return 1
+ return 0
+
+ if not cfile:
+ logging.info("No input file given - using defaults")
+ user_input = DEFAULTS
+ else:
+ logging.info("Using input file %s", cfile)
+ try:
+ with open(cfile) as conf_file:
+ user_input = unmarshal_config(json.load(conf_file))
+ diffs = sanity_check_config(user_input, automatic)
+ logging.info(
+ "File sanity check complete - found %s difference%s",
+ diffs,
+ '' if diffs == 1 else 's'
+ )
+ except (OSError, ValueError) as e:
+ logging.critical("Reading in input file '%s': %s", cfile, e)
+ return 1
+
+ try:
+ dbconf = generate_db_conf(user_input[DATABASE_CONF_FILE], DATABASE_CONF_FILE, automatic, root_dir)
+ todbconf = generate_todb_conf(user_input[DB_CONF_FILE], DB_CONF_FILE, automatic, root_dir, dbconf)
+ generate_ldap_conf(user_input[LDAP_CONF_FILE], LDAP_CONF_FILE, automatic, root_dir)
+ admin_conf = generate_users_conf(
+ user_input[USERS_CONF_FILE],
+ USERS_CONF_FILE,
+ automatic,
+ root_dir
+ )
+ generate_profiles_dir(user_input[PROFILES_CONF_FILE])
+ opensslconf = generate_openssl_conf(user_input[OPENSSL_CONF_FILE], OPENSSL_CONF_FILE, automatic)
+ paramconf = generate_param_conf(user_input[PARAM_CONF_FILE], PARAM_CONF_FILE, automatic, root_dir)
+ postinstall_cfg = os.path.join(root_dir, POST_INSTALL_CFG.lstrip('/'))
+ if not os.path.isfile(postinstall_cfg):
+ with open(postinstall_cfg, 'w+') as conf_file:
+ print("{}", file=conf_file)
+ except OSError as e:
+ logging.critical("Writing configuration: %s", e)
+ return 1
+ except ValueError as e:
+ logging.critical("Generating configuration: %s", e)
+ return 1
+
+ try:
+ setup_maxmind(todbconf.get("maxmind", "no"), root_dir)
+ except OSError as e:
+ logging.critical("Setting up MaxMind: %s", e)
+ return 1
+
+ try:
+ cert_code = setup_certificates(opensslconf, root_dir, ops_user, ops_group)
+ if cert_code:
+ return cert_code
+ except OSError as e:
+ logging.critical("Setting up SSL Certificates: %s", e)
+ return 1
+
+ try:
+ generate_cdn_conf(user_input[CDN_CONF_FILE], CDN_CONF_FILE, automatic, root_dir)
+ except OSError as e:
+ logging.critical("Generating cdn.conf: %s", e)
+ return 1
+
+ if not no_database:
+ try:
+ conn_str = db_connection_string(dbconf)
+ except KeyError as e:
+ logging.error("Missing database connection variable: %s", e)
+ logging.error(
+ "Can't connect to the database. " \
+ "Use the script `/opt/traffic_ops/install/bin/todb_bootstrap.sh` " \
+ "on the db server to create it and run `postinstall` again."
+ )
+ return 1
+
+ if not os.path.isfile("/usr/bin/psql") or not os.access("/usr/bin/psql", os.X_OK):
+ logging.critical("psql is not installed, please install it to continue with database setup")
+ return 1
+
+ def db_connect_failed():
+ logging.error("Failed to set up database: %s", e)
+ logging.error(
+ "Can't connect to the database. "
+ "Use the script `/opt/traffic_ops/install/bin/todb_bootstrap.sh` "
+ "on the db server to create it and run `postinstall` again."
+ )
+
+ try:
+ setup_database_data(conn_str, admin_conf, paramconf, root_dir)
+ except (OSError, subprocess.CalledProcessError)as e:
+ db_connect_failed()
+ return 1
+ except subprocess.SubprocessError as e:
+ db_connect_failed()
+ return 1
+
+
+ if not no_restart_to:
+ logging.info("Starting Traffic Ops")
+ try:
+ cmd = ["/sbin/service", "traffic_ops", "restart"]
+ proc = subprocess.Popen(
+ cmd,
+ stderr=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ universal_newlines=True,
+ )
+ if proc.wait():
+ raise subprocess.CalledProcessError(proc.returncode, cmd)
+ except subprocess.CalledProcessError as e:
+ logging.critical("Failed to restart Traffic Ops, return code %s: %s", e.returncode, e)
+ logging.debug("stderr: %s\n\tstdout: %s", proc.stderr, proc.stdout)
+ return 1
+ except OSError as e:
+ logging.critical("Failed to restart Traffic Ops: unknown error occurred: %s", e)
+ return 1
+ # Perl didn't actually do any "waiting" before reporting success, so
+ # neither do we
+ logging.info("Waiting for Traffic Ops to restart")
+ else:
+ logging.info("Skipping Traffic Ops restart")
+ logging.info("Success! Postinstall complete.")
+
+ return 0
+
+if __name__ == '__main__':
+ PARSER = argparse.ArgumentParser()
+ PARSER.add_argument(
+ "-a",
+ "--automatic",
+ help="If there are questions in the config file which do not have answers, the script " +
+ "will look to the defaults for the answer. If the answer is not in the defaults the " +
+ "script will exit",
+ action="store_true"
+ )
+ PARSER.add_argument(
+ "--cfile",
+ help="An input config file used to ask and answer questions",
+ type=str,
+ default=None
+ )
+ PARSER.add_argument(
+ "-cfile",
+ help=argparse.SUPPRESS,
+ type=str,
+ default=None,
+ dest="legacy_cfile"
+ )
+ PARSER.add_argument("--debug", help="Enables verbose output", action="store_true")
+ PARSER.add_argument("-debug", help=argparse.SUPPRESS, dest="legacy_debug", action="store_true")
+ PARSER.add_argument(
+ "--defaults",
+ help="Writes out a configuration file with defaults which can be used as input",
+ type=str,
+ nargs="?",
+ default=None,
+ const=""
+ )
+ PARSER.add_argument(
+ "-defaults",
+ help=argparse.SUPPRESS,
+ type=str,
+ nargs="?",
+ default=None,
+ const="",
+ dest="legacy_defaults"
+ )
+ PARSER.add_argument(
+ "-n",
+ "--no-root",
+ help="Enable running as a non-root user (may cause failure)",
+ action="store_true"
+ )
+ PARSER.add_argument(
+ "-r",
+ "--root-directory",
+ help="Set the directory to be treated as the system's root directory (e.g. for testing)",
+ type=str,
+ default="/"
+ )
+ PARSER.add_argument(
+ "-u",
+ "--ops-user",
+ help="Specify a username to own Traffic Ops files and processes",
+ type=str,
+ default="trafops"
+ )
+ PARSER.add_argument(
+ "-g",
+ "--ops-group",
+ help="Specify the group to own Traffic Ops files and processes",
+ type=str,
+ default="trafops"
+ )
+ PARSER.add_argument(
+ "--no-restart-to",
+ help="Skip restarting Traffic Ops after configuration and database changes are applied",
+ action="store_true"
+ )
+ PARSER.add_argument("--no-database", help="Skip all database operations", action="store_true")
+
+ ARGS = PARSER.parse_args()
+
+ USED_LEGACY_ARGS = False
+ DEFAULTS_ARG = None
+ if ARGS.legacy_defaults:
+ if ARGS.defaults:
+ logging.error("cannot specify both '--defaults' and '-defaults'")
+ sys.exit(1)
+ USED_LEGACY_ARGS = True
+ DEFAULTS_ARG = ARGS.legacy_defaults
+ else:
+ DEFAULTS_ARG = ARGS.defaults
+
+ DEBUG = False
+ if ARGS.legacy_debug:
+ if ARGS.debug:
+ logging.error("cannot specify both '--debug' and '-debug'")
+ sys.exit(1)
+ USED_LEGACY_ARGS = True
+ DEBUG = ARGS.legacy_debug
+ else:
+ DEBUG = ARGS.debug
+
+ CFILE = None
+ if ARGS.legacy_cfile:
+ if ARGS.cfile:
+ logging.error("cannot specify both '--cfile' and '-cfile'")
+ sys.exit(1)
+ USED_LEGACY_ARGS = True
+ CFILE = ARGS.legacy_cfile
+ else:
+ CFILE = ARGS.cfile
+
+ if not ARGS.no_root and os.getuid() != 0:
+ logging.error("You must run this script as the root user")
+ logging.shutdown()
+ sys.exit(1)
+
+ if USED_LEGACY_ARGS:
+ logging.warning(
+ "passing long options with a single '-' is deprecated, please use '--' in the future"
+ )
+
+ try:
+ EXIT_CODE = main(
+ ARGS.automatic,
+ DEBUG,
+ DEFAULTS_ARG,
+ CFILE,
+ os.path.abspath(ARGS.root_directory),
+ ARGS.ops_user,
+ ARGS.ops_group,
+ ARGS.no_restart_to,
+ ARGS.no_database
+ )
+ sys.exit(EXIT_CODE)
+ except KeyboardInterrupt:
+ sys.exit(1)
+ finally:
+ logging.shutdown()
diff --git a/traffic_ops/install/bin/_postinstall b/traffic_ops/install/bin/_postinstall.pl
similarity index 100%
copy from traffic_ops/install/bin/_postinstall
copy to traffic_ops/install/bin/_postinstall.pl
diff --git a/traffic_ops/install/bin/input.json b/traffic_ops/install/bin/input.json
index 871b139..c35fdae 100644
--- a/traffic_ops/install/bin/input.json
+++ b/traffic_ops/install/bin/input.json
@@ -1,179 +1,151 @@
{
- "/opt/traffic_ops/app/conf/production/database.conf":[
+ "/opt/traffic_ops/app/conf/cdn.conf": [
{
- "Database type":"Pg",
- "config_var":"type"
+ "Generate a new secret?": "yes",
+ "config_var": "genSecret"
},
{
- "Database name":"traffic_ops_db",
- "config_var":"dbname"
+ "Number of secrets to keep?": "10",
+ "config_var": "keepSecrets"
},
{
- "Database server hostname IP or FQDN":"localhost",
- "config_var":"hostname"
+ "Number of workers?": "12",
+ "config_var": "workers"
},
{
- "Database port number":"5432",
- "config_var":"port"
- },
- {
- "Traffic Ops database user":"traffic_ops",
- "config_var":"user"
- },
- {
- "Traffic Ops database password":"default",
- "config_var":"password",
- "hidden":"1"
+ "Traffic Ops url?": "https://[::]",
+ "config_var": "base_url"
}
],
- "/opt/traffic_ops/app/db/dbconf.yml":[
+ "/opt/traffic_ops/app/conf/ldap.conf": [
{
- "Database server root (admin) user":"root",
- "config_var":"dbAdminUser"
+ "Do you want to set up LDAP?": "no",
+ "config_var": "setupLdap"
},
{
- "Database server admin password":"default",
- "config_var":"dbAdminPw",
- "hidden":"1"
+ "LDAP server hostname": "",
+ "config_var": "hostname"
},
{
- "Download Maxmind Database?":"yes",
- "config_var":"maxmind"
- }
- ],
- "/opt/traffic_ops/app/conf/cdn.conf":[
- {
- "Generate a new secret?":"yes",
- "config_var":"genSecret"
+ "LDAP Admin DN": "",
+ "config_var": "admin_dn"
},
{
- "Number of secrets to keep?":"10",
- "config_var":"keepSecrets"
+ "LDAP Admin Password": "",
+ "config_var": "password",
+ "hidden": "1"
},
{
- "Port to serve on?":"443",
- "config_var":"port"
+ "LDAP Search Base": "",
+ "config_var": "search_base"
}
],
- "/opt/traffic_ops/app/conf/ldap.conf":[
+ "/opt/traffic_ops/app/conf/production/database.conf": [
{
- "Do you want to set up LDAP?":"no",
- "config_var":"setupLdap"
+ "Database type": "Pg",
+ "config_var": "type"
},
{
- "LDAP server hostname":"",
- "config_var":"hostname"
+ "Database name": "traffic_ops_db",
+ "config_var": "dbname"
},
{
- "LDAP Admin DN":"",
- "config_var":"admin_dn"
+ "Database server hostname IP or FQDN": "localhost",
+ "config_var": "hostname"
},
{
- "LDAP Admin Password":"",
- "config_var":"password",
- "hidden":"1"
+ "Database port number": "5432",
+ "config_var": "port"
},
{
- "LDAP Search Base":"",
- "config_var":"search_base"
+ "Traffic Ops database user": "dbuser",
+ "config_var": "user"
+ },
+ {
+ "Traffic Ops database password": "dbpass",
+ "config_var": "password",
+ "hidden": "1"
}
],
- "/opt/traffic_ops/install/data/json/users.json":[
+ "/opt/traffic_ops/app/db/dbconf.yml": [
{
- "Administration username for Traffic Ops":"root",
- "config_var":"tmAdminUser"
+ "Database server root (admin) username": "dbuser",
+ "config_var": "pgUser"
},
{
- "Password for the admin user":"default",
- "config_var":"tmAdminPw",
- "hidden":"1"
- }
- ],
- "/opt/traffic_ops/install/data/profiles/":[
+ "Database server admin password": "dbpass",
+ "config_var": "pgPassword",
+ "hidden": "1"
+ },
{
- "Add custom profiles?":"no",
- "config_var":"custom_profiles"
+ "Download Maxmind Database?": "no",
+ "config_var": "maxmind"
}
],
- "/opt/traffic_ops/install/data/json/openssl_configuration.json":[
+ "/opt/traffic_ops/install/data/json/openssl_configuration.json": [
{
- "Do you want to generate a certificate?":"yes",
- "config_var":"genCert"
+ "Do you want to generate a certificate?": "yes",
+ "config_var": "genCert"
},
{
- "Country Name (2 letter code)":"XX",
- "config_var":"country"
+ "Country Name (2 letter code)": "XX",
+ "config_var": "country"
},
{
- "State or Province Name (full name)":"Default State",
- "config_var":"state"
+ "State or Province Name (full name)": "Default State",
+ "config_var": "state"
},
{
- "Locality Name (eg, city)":"Default City",
- "config_var":"locality"
+ "Locality Name (eg, city)": "Default City",
+ "config_var": "locality"
},
{
- "Organization Name (eg, company)":"Default Company Ltd",
- "config_var":"company"
+ "Organization Name (eg, company)": "Default Company Ltd",
+ "config_var": "company"
},
{
- "Organizational Unit Name (eg, section)":"",
- "config_var":"org_unit"
+ "Organizational Unit Name (eg, section)": "",
+ "config_var": "org_unit"
},
{
- "Common Name (eg, your name or your server's hostname)":"example.com",
- "config_var":"common_name"
+ "Common Name (eg, your name or your server's hostname)": "example.com",
+ "config_var": "common_name"
},
{
- "RSA Passphrase":"password",
- "config_var":"rsaPassword",
- "hidden":"1"
+ "RSA Passphrase": "password",
+ "config_var": "rsaPassword",
+ "hidden": "1"
}
],
- "/opt/traffic_ops/install/data/json/profiles.json":[
- {
- "Traffic Ops url":"https://localhost",
- "config_var":"tm.url"
- },
+ "/opt/traffic_ops/install/data/json/profiles.json": [
{
- "Human-readable CDN Name. (No whitespace, please)":"kabletown_cdn",
- "config_var":"cdn_name"
+ "Traffic Ops url": "https://[::]",
+ "config_var": "tm.url"
},
{
- "Health Polling Interval (milliseconds)":"8000",
- "config_var":"health_polling_int"
+ "Human-readable CDN Name. (No whitespace, please)": "kabletown_cdn",
+ "config_var": "cdn_name"
},
{
- "DNS sub-domain for which your CDN is authoritative":"cdn1.kabletown.net",
- "config_var":"dns_subdomain"
- },
- {
- "TLD SOA admin":"traffic_ops",
- "config_var":"soa_admin"
- },
- {
- "TrafficServer Drive Prefix":"/dev/sd",
- "config_var":"driver_prefix"
- },
- {
- "TrafficServer RAM Drive Prefix":"/dev/ram",
- "config_var":"ram_drive_prefix"
- },
- {
- "TrafficServer RAM Drive Letters (comma separated)":"0,1,2,3,4,5,6,7",
- "config_var":"ram_drive_letters"
- },
+ "DNS sub-domain for which your CDN is authoritative": "cdn1.kabletown.net",
+ "config_var": "dns_subdomain"
+ }
+ ],
+ "/opt/traffic_ops/install/data/json/users.json": [
{
- "Health Threshold Load Average":"25",
- "config_var":"health_thresh_load_avg"
+ "Administration username for Traffic Ops": "admin",
+ "config_var": "tmAdminUser"
},
{
- "Health Threshold Available Bandwidth in Kbps":"1750000",
- "config_var":"health_thresh_kbps"
- },
+ "Password for the admin user": "twelve",
+ "config_var": "tmAdminPw",
+ "hidden": "1"
+ }
+ ],
+ "/opt/traffic_ops/install/data/profiles/": [
{
- "Traffic Server Health Connection Timeout (milliseconds)":"2000",
- "config_var":"health_connect_timeout"
+ "Add custom profiles?": "no",
+ "config_var": "custom_profiles"
}
]
}
diff --git a/traffic_ops/install/bin/postinstall b/traffic_ops/install/bin/postinstall
index ebaf512..a121852 100755
--- a/traffic_ops/install/bin/postinstall
+++ b/traffic_ops/install/bin/postinstall
@@ -19,10 +19,9 @@ set -o errexit -o nounset
for arg in "$@"; do
case $arg in
- -h*) action=bypass
- ;;
- -defaults*) action=bypass
- ;;
+ -h*) action=bypass;;
+ -defaults*) action=bypass;;
+ -n*) action=bypass;;
esac
done
@@ -41,19 +40,9 @@ if [[ ! $(su - postgres psql -w -c 'show is_superuser' </dev/null 2>/dev/null) =
exit 1
fi
-# install carton first
-cpanm Carton
-
-# carton installs all the perl dependencies in cpanfile
-cd /opt/traffic_ops/app
-
-export POSTGRES_HOME=${POSTGRES_HOME:-/usr/pgsql-13}
-perl -MCarton::CLI -e'Carton::CLI->new->run(@ARGV)'
-
# Install go and goose
/opt/traffic_ops/install/bin/install_goose.sh
-export PERL5LIB=/opt/traffic_ops/app/lib:/opt/traffic_ops/app/local/lib/perl5
/opt/traffic_ops/install/bin/_postinstall "$@"
# should all be owned by trafops user
diff --git a/traffic_ops/install/bin/postinstall.py b/traffic_ops/install/bin/postinstall.py
deleted file mode 100755
index ef13458..0000000
--- a/traffic_ops/install/bin/postinstall.py
+++ /dev/null
@@ -1,1522 +0,0 @@
-#!/usr/bin/env bash
-"exec" "bash" "-c" "PATH+=:/usr/libexec/; exec \$(type -p python38 python3.8 python36 python3.8 python3 python python27 python2.7 python2 platform-python | head -n1) \"$0\" $*"
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# There's a bug in asteroid with Python 3.9's NamedTuple being
-# recognized for the dynamically generated class that it is. Should be fixed
-# with the next release, but 'til then...
-#pylint:disable=inherit-non-class
-from __future__ import print_function
-
-"""
-This script is meant as a drop-in replacement for the old _postinstall Perl script.
-
-It does, however, offer several more command-line flags not present in the original, to aid in
-testing.
-
--a, --automatic If there are questions in the config file which do not have answers,
- the script will look to the defaults for the answer. If the answer is
- not in the defaults the script will exit.
---cfile [FILE] An input config file used to ask and answer questions.
---debug Enables verbose logging output.
---defaults [FILE] Writes out a configuration file with defaults which can be used as
- input. If no FILE is given, writes to stdout.
--n, --no-root Enable running as a non-root user (may cause failure).
--r DIR, --root-directory DIR Set the directory to be treated as the system's root directory (e.g.
- for testing). Default: /
--u USER, --ops-user USER Specify a username to own Traffic Ops files and processes.
- Default: trafops
--g GROUP, --ops-group GROUP Specify the group to own Traffic Ops files and processes.
- Default: trafops
---no-restart-to Skip restarting Traffic Ops after configuration and database changes
- are applied.
---no-database Skip all database operations.
-
->>> [c for c in [[a for a in b if not a.config_var] for b in DEFAULTS.values()] if c]
-[]
-"""
-import argparse
-import base64
-import errno
-import getpass
-import grp
-import hashlib
-import json
-import logging
-import os
-import pwd
-import random
-import re
-import shutil
-import stat
-import string
-import subprocess
-import sys
-
-from collections import namedtuple
-from struct import unpack, pack
-
-# Paths for output configuration files
-DATABASE_CONF_FILE = "/opt/traffic_ops/app/conf/production/database.conf"
-DB_CONF_FILE = "/opt/traffic_ops/app/db/dbconf.yml"
-CDN_CONF_FILE = "/opt/traffic_ops/app/conf/cdn.conf"
-LDAP_CONF_FILE = "/opt/traffic_ops/app/conf/ldap.conf"
-USERS_CONF_FILE = "/opt/traffic_ops/install/data/json/users.json"
-PROFILES_CONF_FILE = "/opt/traffic_ops/install/data/profiles/"
-OPENSSL_CONF_FILE = "/opt/traffic_ops/install/data/json/openssl_configuration.json"
-PARAM_CONF_FILE = "/opt/traffic_ops/install/data/json/profiles.json"
-
-
-POST_INSTALL_CFG = "/opt/traffic_ops/install/data/json/post_install.json"
-
-# Log file for the installer
-# TODO: determine if logging to a file should be directly supported.
-# LOG_FILE = "/var/log/traffic_ops/postinstall.log"
-
-# Log file for CPAN output
-# TODO: The Perl used to "rotate" this file on every run, for some reason. Should we?
-# CPAN_LOG_FILE = "/var/log/traffic_ops/cpan.log"
-
-# Configuration file output with answers which can be used as input to postinstall
-# TODO: Perl used to always write its defaults out to this file when requested.
-# Python, instead, outputs to stdout. This is breaking, but more flexible. Change it?
-# OUTPUT_CONFIG_FILE = "/opt/traffic_ops/install/bin/configuration_file.json"
-
-if sys.version_info.major >= 3:
- # Accepting a string for json.dump()'s `indent` keyword argument is a Python 3 feature
- indent = "\t" # type: str
-else:
- indent = 4 # type: int
- str = unicode # type: type[unicode]
-
-class Question(object):
- """
- Question represents a single question to be asked of the user, to determine a configuration
- value.
-
- >>> Question("question", "answer", "var")
- Question(question='question', default='answer', config_var='var', hidden=False)
- """
-
- def __init__(self, question, default, config_var, hidden = False): # type: (str, str, str, bool) -> None
- self.question = question
- self.default = default
- self.config_var = config_var
- self.hidden = hidden
-
- def __str__(self): # type: () -> str
- if self.default:
- return "{question} [{default}]: ".format(question=self.question, default=self.default)
- return "{question}: ".format(question=self.question)
-
- def __repr__(self): # type: () -> str
- qstn = self.question
- ans = self.default
- cfgvr = self.config_var
- hddn = self.hidden
- return "Question(question='{qstn}', default='{ans}', config_var='{cfgvr}', hidden={hddn})".format(qstn=qstn, ans=ans, cfgvr=cfgvr, hddn=hddn)
-
- def ask(self): # type: () -> str
- """
- Asks the user the Question interactively.
-
- If 'hidden' is true, output will not be echoed.
- """
- if self.hidden:
- while True:
- passwd = getpass.getpass(str(self))
- if not passwd:
- continue
- if passwd == getpass.getpass("Re-Enter {question}: ".format(question=self.question)):
- return passwd
- print("Error: passwords do not match, try again")
- ipt = input(self)
- return ipt if ipt else self.default
-
- def to_json(self): # type: () -> str
- """
- Converts a question to JSON encoding.
-
- >>> Question("Do the thing?", "yes", "cfg_var", True).to_json()
- '{"Do the thing?": "yes", "config_var": "cfg_var", "hidden": true}'
- >>> Question("Do the other thing?", "no", "other cfg_var").to_json()
- '{"Do the other thing?": "no", "config_var": "other cfg_var"}'
- """
- qstn = self.question
- ans = self.default
- cfgvr = self.config_var
- if self.hidden:
- return '{{"{qstn}": "{ans}", "config_var": "{cfgvr}", "hidden": true}}'.format(qstn=qstn, ans=ans, cfgvr=cfgvr)
- return '{{"{qstn}": "{ans}", "config_var": "{cfgvr}"}}'.format(qstn=qstn, ans=ans, cfgvr=cfgvr)
-
- def serialize(self): # type: () -> object
- """Returns a serializable dictionary, suitable for converting to JSON."""
- return {self.question: self.default, "config_var": self.config_var, "hidden": self.hidden}
-
-class User(namedtuple('User', ['username', 'password'])):
- """Users represents a user that will be inserted into the Traffic Ops database.
-
- Attributes
- ----------
- self.username: str
- The user's username.
- self.password: str
- The user's password - IN PLAINTEXT.
- """
-
-class SSLConfig:
- """SSLConfig bundles the options for generating new (self-signed) SSL certificates"""
-
- def __init__(self, gen_cert, cfg_map): # type: (bool, dict[str, str]) -> None
-
- self.gen_cert = gen_cert
- self.rsa_password = cfg_map["rsaPassword"]
- self.params = "/C={country}/ST={state}/L={locality}/O={company}/OU={org_unit}/CN={common_name}/"
- self.params = self.params.format(**cfg_map)
-
-class CDNConfig(namedtuple('CDNConfig', ['gen_secret', 'num_secrets', 'port', 'num_workers', 'url', 'ldap_conf_location'])):
- """CDNConfig holds all of the options needed to format a cdn.conf file."""
-
- def generate_secret(self, conf):
- """
- Generates new secrets - if configured to do so - and adds them to the passed cdn.conf
- configuration.
- """
- if not self.gen_secret:
- return
-
- if isinstance(conf, dict) and "secrets" in conf and isinstance(conf["secrets"], list):
- logging.debug("Secrets found in cdn.conf file")
- else:
- conf["secrets"] = []
- logging.debug("No secrets found in cdn.conf file")
-
- conf["secrets"].insert(0, random_word())
-
- if self.num_secrets and len(conf["secrets"]) > self.num_secrets:
- conf["secrets"] = conf["secrets"][:self.num_secrets - 1]
-
- def insert_url(self, conf):
- """
- Inserts the configured URL - if it is not an empty string - into the passed cdn.conf
- configuration, in to.base_url.
- """
- if not self.url:
- return
-
- if "to" not in conf or not isinstance(conf["to"], dict):
- conf["to"] = {}
- conf["to"]["base_url"] = self.url
-
-# The default question/answer set
-DEFAULTS = {
- DATABASE_CONF_FILE: [
- Question("Database type", "Pg", "type"),
- Question("Database name", "traffic_ops", "dbname"),
- Question("Database server hostname IP or FQDN", "localhost", "hostname"),
- Question("Database port number", "5432", "port"),
- Question("Traffic Ops database user", "traffic_ops", "user"),
- Question("Password for Traffic Ops database user", "", "password", hidden=True)
- ],
- DB_CONF_FILE: [
- Question("Database server root (admin) user", "postgres", "pgUser"),
- Question("Password for database server admin", "", "pgPassword", hidden=True),
- Question("Download Maxmind Database?", "yes", "maxmind")
- ],
- CDN_CONF_FILE: [
- Question("Generate a new secret?", "yes", "genSecret"),
- Question("Number of secrets to keep?", "1", "keepSecrets"),
- Question("Port to serve on?", "443", "port"),
- Question("Number of workers?", "12", "workers"),
- Question("Traffic Ops url?", "http://localhost:3000", "base_url"),
- Question("ldap.conf location?", "/opt/traffic_ops/app/conf/ldap.conf", "ldap_conf_location")
- ],
- LDAP_CONF_FILE:[
- Question("Do you want to set up LDAP?", "no", "setupLdap"),
- Question("LDAP server hostname", "", "host"),
- Question("LDAP Admin DN", "", "admin_dn"),
- Question("LDAP Admin Password", "", "admin_pass", hidden=True),
- Question("LDAP Search Base", "", "search_base"),
- Question("LDAP Search Query", "", "search_query"),
- Question("LDAP Skip TLS verify", "", "insecure"),
- Question("LDAP Timeout Seconds", "", "ldap_timeout_secs")
- ],
- USERS_CONF_FILE: [
- Question("Administration username for Traffic Ops", "admin", "tmAdminUser"),
- Question("Password for the admin user", "", "tmAdminPw", hidden=True)
- ],
- PROFILES_CONF_FILE: [
- Question("Add custom profiles?", "no", "custom_profiles")
- ],
- OPENSSL_CONF_FILE: [
- Question("Do you want to generate a certificate?", "yes", "genCert"),
- Question("Country Name (2 letter code)", "", "country"),
- Question("State or Province Name (full name)", "", "state"),
- Question("Locality Name (eg, city)", "", "locality"),
- Question("Organization Name (eg, company)", "", "company"),
- Question("Organizational Unit Name (eg, section)", "", "org_unit"),
- Question("Common Name (eg, your name or your server's hostname)", "", "common_name"),
- Question("RSA Passphrase", "CHANGEME!!", "rsaPassword", hidden=True)
- ],
- PARAM_CONF_FILE: [
- Question("Traffic Ops url", "https://localhost", "tm.url"),
- Question("Human-readable CDN Name. (No whitespace, please)", "kabletown_cdn", "cdn_name"),
- Question(
- "DNS sub-domain for which your CDN is authoritative",
- "cdn1.kabletown.net",
- "dns_subdomain"
- )
- ]
-}
-
-class ConfigEncoder(json.JSONEncoder):
- """
- ConfigEncoder encodes a dictionary of filenames to configuration question lists as JSON.
-
- >>> ConfigEncoder().encode({'/test/file':[Question('question', 'default', 'cfg_var', True)]})
- '{"/test/file": [{"question": "default", "config_var": "cfg_var", "hidden": true}]}'
- """
-
- # The linter is just wrong about this
- def default(self, o): # type: (object) -> object
- """
- Returns a serializable representation of 'o'.
-
- Specifically, it does this by attempting to convert a dictionary of filenames to Question
- lists to a dictionary of filenames to lists of dictionaries of strings to strings, falling
- back on default encoding if the proper typing is not found.
- """
- if isinstance(o, Question):
- return o.serialize()
-
- return json.JSONEncoder.default(self, o)
-
-def get_config(questions, fname, automatic = False): # type: (list[Question], str, bool) -> dict[str, str]
- """Asks all provided questions, or uses their defaults in automatic mode"""
-
- logging.info("===========%s===========", fname)
-
- config = {}
-
- for question in questions:
- answer = question.default if automatic else question.ask()
-
- config[question.config_var] = answer
-
- return config
-
-def generate_db_conf(qstns, fname, automatic, root): # (list[Question], str, bool, str) -> dict
- """
- Generates the database.conf file and returns a map of its configuration.
-
- Also writes the configuration file to the file 'fname' under the directory 'root'.
- """
- db_conf = get_config(qstns, fname, automatic)
- typ = db_conf.get("type", "UNKNOWN")
- hostname = db_conf.get("hostname", "UNKNOWN")
- port = db_conf.get("port", "UNKNOWN")
-
- db_conf["description"] = "{typ} database on {hostname}:{port}".format(typ=typ, hostname=hostname, port=port)
-
- path = os.path.join(root, fname.lstrip('/'))
- with open(path, 'w+') as conf_file:
- json.dump(db_conf, conf_file, indent=indent)
- print(file=conf_file)
-
- logging.info("Database configuration has been saved")
-
- return db_conf
-
-def generate_todb_conf(qstns, fname, auto, root, conf): # (list, str, bool, str, dict) -> dict
- """
- Generates the dbconf.yml file and returns a map of its configuration.
-
- Also writes the configuration file to the file 'fname' under the directory 'root'.
- """
- todbconf = get_config(qstns, fname, auto)
-
- driver = "postgres"
- if "type" not in conf:
- logging.warning("Driver type not found in todb config; using 'postgres'")
- else:
- driver = "postgres" if conf["type"] == "Pg" else conf["type"]
-
- path = os.path.join(root, fname.lstrip('/'))
- hostname = conf.get('hostname', 'UNKNOWN')
- port = conf.get('port', 'UNKNOWN')
- user = conf.get('user', 'UNKNOWN')
- password = conf.get('password', 'UNKNOWN')
- dbname = conf.get('dbname', 'UNKNOWN')
-
- open_line = "host={hostname} port={port} user={user} password={password} dbname={dbname}".format(hostname=hostname, port=port, user=user, password=password, dbname=dbname)
- with open(path, 'w+') as conf_file:
- print("production:", file=conf_file)
- print(" driver:", driver, file=conf_file)
- print(" open: {open_line} sslmode=disable".format(open_line=open_line), file=conf_file)
-
- return todbconf
-
-def generate_ldap_conf(questions, fname, automatic, root): # type: (list[Question], str, bool, str) -> None
- """
- Generates the ldap.conf file by asking the questions or using default answers in auto mode.
-
- Also writes the configuration to the file 'fname' under the directory 'root'
- """
- use_ldap_question = [q for q in questions if q.question == "Do you want to set up LDAP?"]
- if not use_ldap_question:
- logging.warning("Couldn't find question asking if LDAP should be set up, using default: no")
- return
- use_ldap = use_ldap_question[0].default if automatic else use_ldap_question[0].ask()
-
- if use_ldap.lower() not in {'y', 'yes'}:
- logging.info("Not setting up ldap")
- return
-
- ldap_conf = get_config([q for q in questions if q is not use_ldap_question[0]], fname, automatic)
- keys = (
- 'host',
- 'admin_dn',
- 'admin_pass',
- 'search_base',
- 'search_query',
- 'insecure',
- 'ldap_timeout_secs'
- )
-
- for key in keys:
- if key not in ldap_conf:
- raise ValueError("{key} is a required key in {fname}".format(key=key, fname=fname))
-
- if not re.match(r"^\S+:\d+$", ldap_conf["host"]):
- raise ValueError("host in {fname} must be of form 'hostname:port'".format(fname=fname))
-
- path = os.path.join(root, fname.lstrip('/'))
- try:
- os.makedirs(os.path.dirname(path))
- except OSError as e:
- if e.errno == errno.EEXIST:
- pass
- with open(path, 'w+') as conf_file:
- json.dump(ldap_conf, conf_file, indent=indent)
- print(file=conf_file)
-
-def hash_pass(passwd): # type: (str) -> str
- """
- Generates a Scrypt-based hash of the given password in a Perl-compatible format.
- It's hard-coded - like the Perl - to use 64 random bytes for the salt, n=16384,
- r=8, p=1 and dklen=64.
- """
- n = 2 ** 14
- r_val = 8
- p_val = 1
- dklen = 64
- salt = os.urandom(dklen)
- hashed = Scrypt(password=passwd.encode(), salt=salt, cost_factor=n, block_size_factor=r_val, parallelization_factor=p_val, key_length=dklen).derive()
- hashed_b64 = base64.standard_b64encode(hashed).decode()
- salt_b64 = base64.standard_b64encode(salt).decode()
-
- return "SCRYPT:{n}:{r_val}:{p_val}:{salt_b64}:{hashed_b64}".format(n=n, r_val=r_val, p_val=p_val, salt_b64=salt_b64, hashed_b64=hashed_b64)
-
-
-class Scrypt:
- def __init__(self, password, salt, cost_factor, block_size_factor, parallelization_factor, key_length): # type: (bytes, bytes, int, int, int, int) -> None
- self.password = password # type: bytes
- self.salt = salt # type: bytes
- self.cost_factor = cost_factor # type: int
- self.block_size_factor = block_size_factor # type: int
- self.parallelization_factor = parallelization_factor # type: int
- self.key_length = key_length
- self.block_unit = 32 * self.block_size_factor # 1 block unit = 32 * block_size_factor 32-bit ints
-
- def derive(self): # type: () -> bytes
- salt_length = 2 ** 7 * self.block_size_factor * self.parallelization_factor # type: int
- pack_format = '<' + 'L' * int(salt_length / 4) # `<` means `little-endian` and `L` means `unsigned long`
- salt = hashlib.pbkdf2_hmac('sha256', password=self.password, salt=self.salt, iterations=1, dklen=salt_length) # type: bytes
- block = list(unpack(pack_format, salt)) # type: list[int]
- block = self.ROMix(block)
- salt = pack(pack_format, *block)
- key = hashlib.pbkdf2_hmac('sha256', password=self.password, salt=salt, iterations=1, dklen=self.key_length) # type: bytes
- return key
-
- def ROMix(self, block): # type: (list[int]) -> list[int]
- xored_block = [0] * len(block) # type: list[int]
- variations = [list()] * self.cost_factor # type: list[list[int]]
- variations[0] = block
- index = 1
- while index < self.cost_factor:
- variations[index] = self.block_mix(variations[index - 1])
- index += 1
- block = self.block_mix(variations[-1])
- for unused in variations:
- variation_index = block[self.block_unit - 16] % self.cost_factor # type: int
- variation = variations[variation_index]
- for index, unused in enumerate(xored_block):
- xored_block[index] = block[index] ^ variation[index]
- block = self.block_mix(xored_block)
- return block
-
- def block_mix(self, previous_block): # type: (list[int]) -> list[int]
- block = previous_block[:] # type: list[int]
- X_length = 16 # X is the list of numbers within `block` that we mix
- copy_index = self.block_unit - X_length
- X = previous_block[copy_index:copy_index + X_length] # type: list[int]
- octet_index = 0 # type: int
- block_xor_index = 0
- while octet_index < 2 * self.block_size_factor:
- for index, unused in enumerate(X):
- X[index] ^= previous_block[block_xor_index + index]
- block_xor_index += X_length
- self.salsa20(X)
- block_offset = (int(octet_index / 2) + octet_index % 2 * self.block_size_factor) * X_length
- block[block_offset:block_offset + X_length] = X
- octet_index += 1
- return block
-
- def salsa20(self, block): # type: (list[int]) -> None
- X = block[:] # make a copy (list.copy() is Python 3-only)
- for i in range(0, 4):
- # These bit shifting operations could be condensed into a single line of list comprehensions,
- # but there is a >3x performance benefit from writing it out explicitly.
- bits = X[0] + X[12] & 0xffffffff
- X[4] ^= bits << 7 | bits >> 32 - 7
- bits = X[4] + X[0] & 0xffffffff
- X[8] ^= bits << 9 | bits >> 32 - 9
- bits = X[8] + X[4] & 0xffffffff
- X[12] ^= bits << 13 | bits >> 32 - 13
- bits = X[12] + X[8] & 0xffffffff
- X[0] ^= bits << 18 | bits >> 32 - 18
- bits = X[5] + X[1] & 0xffffffff
- X[9] ^= bits << 7 | bits >> 32 - 7
- bits = X[9] + X[5] & 0xffffffff
- X[13] ^= bits << 9 | bits >> 32 - 9
- bits = X[13] + X[9] & 0xffffffff
- X[1] ^= bits << 13 | bits >> 32 - 13
- bits = X[1] + X[13] & 0xffffffff
- X[5] ^= bits << 18 | bits >> 32 - 18
- bits = X[10] + X[6] & 0xffffffff
- X[14] ^= bits << 7 | bits >> 32 - 7
- bits = X[14] + X[10] & 0xffffffff
- X[2] ^= bits << 9 | bits >> 32 - 9
- bits = X[2] + X[14] & 0xffffffff
- X[6] ^= bits << 13 | bits >> 32 - 13
- bits = X[6] + X[2] & 0xffffffff
- X[10] ^= bits << 18 | bits >> 32 - 18
- bits = X[15] + X[11] & 0xffffffff
- X[3] ^= bits << 7 | bits >> 32 - 7
- bits = X[3] + X[15] & 0xffffffff
- X[7] ^= bits << 9 | bits >> 32 - 9
- bits = X[7] + X[3] & 0xffffffff
- X[11] ^= bits << 13 | bits >> 32 - 13
- bits = X[11] + X[7] & 0xffffffff
- X[15] ^= bits << 18 | bits >> 32 - 18
- bits = X[0] + X[3] & 0xffffffff
- X[1] ^= bits << 7 | bits >> 32 - 7
- bits = X[1] + X[0] & 0xffffffff
- X[2] ^= bits << 9 | bits >> 32 - 9
- bits = X[2] + X[1] & 0xffffffff
- X[3] ^= bits << 13 | bits >> 32 - 13
- bits = X[3] + X[2] & 0xffffffff
- X[0] ^= bits << 18 | bits >> 32 - 18
- bits = X[5] + X[4] & 0xffffffff
- X[6] ^= bits << 7 | bits >> 32 - 7
- bits = X[6] + X[5] & 0xffffffff
- X[7] ^= bits << 9 | bits >> 32 - 9
- bits = X[7] + X[6] & 0xffffffff
- X[4] ^= bits << 13 | bits >> 32 - 13
- bits = X[4] + X[7] & 0xffffffff
- X[5] ^= bits << 18 | bits >> 32 - 18
- bits = X[10] + X[9] & 0xffffffff
- X[11] ^= bits << 7 | bits >> 32 - 7
- bits = X[11] + X[10] & 0xffffffff
- X[8] ^= bits << 9 | bits >> 32 - 9
- bits = X[8] + X[11] & 0xffffffff
- X[9] ^= bits << 13 | bits >> 32 - 13
- bits = X[9] + X[8] & 0xffffffff
- X[10] ^= bits << 18 | bits >> 32 - 18
- bits = X[15] + X[14] & 0xffffffff
- X[12] ^= bits << 7 | bits >> 32 - 7
- bits = X[12] + X[15] & 0xffffffff
- X[13] ^= bits << 9 | bits >> 32 - 9
- bits = X[13] + X[12] & 0xffffffff
- X[14] ^= bits << 13 | bits >> 32 - 13
- bits = X[14] + X[13] & 0xffffffff
- X[15] ^= bits << 18 | bits >> 32 - 18
-
- for index in range(0, 16):
- block[index] = block[index] + X[index] & 0xffffffff
-
-
-def generate_users_conf(qstns, fname, auto, root): # type: (list[Question], str, bool, str) -> User
- """
- Generates a users.json file from the given questions and returns a User containing the same
- information.
- """
- config = get_config(qstns, fname, auto)
-
- if "tmAdminUser" not in config or "tmAdminPw" not in config:
- raise ValueError("{fname} must include 'tmAdminUser' and 'tmAdminPw'".format(fname=fname))
-
- hashed_pass = hash_pass(config["tmAdminPw"])
-
- path = os.path.join(root, fname.lstrip('/'))
- with open(path, 'w+') as conf_file:
- json.dump({"username": config["tmAdminUser"], "password": hashed_pass}, conf_file, indent=indent)
- print(file=conf_file)
-
- return User(config["tmAdminUser"], config["tmAdminPw"])
-
-def generate_profiles_dir(questions): # type: (list[Question]) -> None
- """
- I truly have no idea what's going on here. This is what the Perl did, so I
- copied it. It does nothing. Literally nothing.
- """
- #pylint:disable=unused-variable
- user_in = questions
- #pylint:enable=unused-variable
-
-def generate_openssl_conf(questions, fname, auto): # type: (list[Question], str, bool) -> SSLConfig
- """
- Constructs an SSLConfig by asking the passed questions, or using their default answers if in
- auto mode.
- """
- cfg_map = get_config(questions, fname, auto)
- if "genCert" not in cfg_map:
- raise ValueError("missing 'genCert' key")
-
- gen_cert = cfg_map["genCert"].lower() in {"y", "yes"}
-
- return SSLConfig(gen_cert, cfg_map)
-
-def generate_param_conf(qstns, fname, auto, root): # type: (list[Question], str, bool, str) -> dict
- """
- Generates a profiles.json by asking the passed questions, or using their default answers in auto
- mode.
-
- Also writes the file to 'fname' in the directory 'root'.
- """
- conf = get_config(qstns, fname, auto)
-
- path = os.path.join(root, fname.lstrip('/'))
- with open(path, 'w+') as conf_file:
- json.dump(conf, conf_file, indent=indent)
- print(file=conf_file)
-
- return conf
-
-def sanity_check_config(cfg, automatic): # type: (dict[str, list[Question]], bool) -> int
- """
- Checks a user-input configuration file, and outputs the number of files in the
- default question set that did not appear in the input.
-
- :param cfg: The user's parsed input questions.
- :param automatic: If :keyword:`True` all missing questions will use their default answers.
- Otherwise, the user will be prompted for answers.
- """
- diffs = 0
-
- for fname, file in DEFAULTS.items():
- if fname not in cfg:
- logging.warning("File '%s' found in defaults but not config file", fname)
- cfg[fname] = []
-
- for default_value in file:
- for config_value in cfg[fname]:
- if default_value.config_var == config_value.config_var:
- break
- else:
- question = default_value.question
- answer = default_value.default
-
- if not automatic:
- logging.info("Prompting user for answer")
- if default_value.hidden:
- answer = default_value.ask()
- elif default_value.hidden:
- logging.info("Adding question '%s' with default answer", question)
- else:
- logging.info("Adding question '%s' with default answer %s", question, answer)
-
- # The Perl here would ask questions, but those would just get asked later
- # anyway, so I'm not sure why.
- cfg[fname].append(Question(question, answer, default_value.config_var, default_value.hidden))
- diffs += 1
-
- return diffs
-
-def unmarshal_config(dct): # type: (dict) -> dict[str, list[Question]]
- """
- Reads in a raw parsed configuration file and returns the resulting configuration.
-
- >>> unmarshal_config({"test": [{"Do the thing?": "yes", "config_var": "thing"}]})
- {'test': [Question(question='Do the thing?', default='yes', config_var='thing', hidden=False)]}
- >>> unmarshal_config({"test": [{"foo": "", "config_var": "bar", "hidden": True}]})
- {'test': [Question(question='foo', default='', config_var='bar', hidden=True)]}
- """
- ret = {}
- for file, questions in dct.items():
- if not isinstance(questions, list):
- raise ValueError("file '{file}' has malformed questions".format(file=file))
-
- qstns = []
- for qstn in questions:
- if not isinstance(qstn, dict):
- raise ValueError("file '{file}' has a malformed question ({qstn})".format(file=file, qstn=qstn))
- try:
- question = next(key for key in qstn.keys() if key not in ("hidden", "config_var"))
- except StopIteration:
- raise ValueError("question in '{file}' has no question/answer properties ({qstn})".format(file=file, qstn=qstn))
-
- answer = qstn[question]
- if not isinstance(question, str) or not isinstance(answer, str):
- errstr = "question in '{file}' has malformed question/answer property ({question}: {answer})".format(file=file, question=question, answer=answer)
- raise ValueError(errstr)
-
- del qstn[question]
- hidden = False
- if "hidden" in qstn:
- hidden = bool(qstn["hidden"])
- del qstn["hidden"]
-
- if "config_var" not in qstn:
- raise ValueError("question in '{file}' has no 'config_var' property".format(file=file))
- cfg_var = qstn["config_var"]
- if not isinstance(cfg_var, str):
- raise ValueError("question in '{file}' has malformed 'config_var' property ({cfg_var})".format(file=file, cfg_var=cfg_var))
- del qstn["config_var"]
-
- if qstn:
- logging.warning("Found unknown extra properties in question in '%s' (%r)", file, qstn.keys())
-
- qstns.append(Question(question, answer, cfg_var, hidden=hidden))
- ret[file] = qstns
-
- return ret
-
-def setup_maxmind(maxmind_answer, root): # type: (str, str) -> None
- """
- If 'maxmind_answer' is a truthy response ('y' or 'yes' (case-insensitive), sets up a Maxmind
- database using `wget`.
- """
- if maxmind_answer.lower() not in {'y', 'yes'}:
- logging.info("Not downloading Maxmind data")
- return
-
- os.chdir(os.path.join(root, 'opt/traffic_ops/app/public/routing'))
-
- def failed_download(e, ip_version): # type: (Exception, int) -> None
- logging.error("Failed to download MaxMind data")
- logging.debug("(ipv%d) Exception: %s", ip_version, e)
-
- wget = "/usr/bin/wget"
- cmd = [wget, "https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz"]
- # Perl ignored errors downloading the databases, so we do too
- try:
- subprocess.check_call(
- cmd,
- stderr=subprocess.PIPE,
- stdout=subprocess.PIPE,
- universal_newlines=True
- )
- except subprocess.CalledProcessError as e:
- failed_download(e, 4)
- except subprocess.SubprocessError as e:
- failed_download(e, 4)
-
- cmd[1] = (
- "https://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz"
- )
- try:
- subprocess.check_call(
- cmd,
- stderr=subprocess.PIPE,
- stdout=subprocess.PIPE,
- universal_newlines=True
- )
- except subprocess.CalledProcessError as e:
- failed_download(e, 6)
- except subprocess.SubprocessError as e:
- failed_download(e, 6)
-
-def exec_openssl(description, *cmd_args): # type: (str, ...) -> bool
- """
- Executes openssl with the supplied command-line arguments.
-
- :param description: Describes the operation taking place for logging purposes.
- :returns: Whether or not the execution succeeded, success being defined by an exit code of zero
- """
- logging.info(description)
-
- cmd = ("/usr/bin/openssl",) + cmd_args
-
- while True:
- proc = subprocess.Popen(
- cmd,
- stderr=subprocess.PIPE,
- stdout=subprocess.PIPE,
- universal_newlines=True,
- )
- proc.wait()
- if proc.returncode == 0:
- return True
-
- logging.debug("openssl exec failed with code %s; stderr: %s", proc.returncode, proc.stderr)
- while True:
- ans = input("{description} failed. Try again (y/n) [y]: ".format(description=description))
- if not ans or ans.lower().startswith('n'):
- return False
- if ans.lower().startswith('y'):
- break
-
-def setup_certificates(conf, root, ops_user, ops_group): # type: (SSLConfig, str, str, str) -> int
- """
- Generates self-signed SSL certificates from the given configuration.
- :returns: For whatever reason this subroutine needs to dictate the return code of the script, so that's what it returns.
- """
- if not conf.gen_cert:
- logging.info("Not generating openssl certification")
- return 0
-
- if not os.path.isfile('/usr/bin/openssl') or not os.access('/usr/bin/openssl', os.X_OK):
- logging.error("Unable to install SSL certificates as openssl is not installed")
- cmd = os.path.join(root, "opt/traffic_ops/install/bin/generateCert")
- logging.error("Install openssl and then run %s to install SSL certificates", cmd)
- return 4
-
- logging.info("Installing SSL Certificates")
- logging.info("\n\tWe're now running a script to generate a self signed X509 SSL certificate")
- logging.info("Postinstall SSL Certificate Creation")
-
- # Perl logs this before actually generating a key. So we do too.
- logging.info("The server key has been generated")
-
- args = (
- "genrsa",
- "-des3",
- "-out",
- "server.key",
- "-passout",
- "pass:{rsa_password}".format(rsa_password=conf.rsa_password),
- "1024"
- )
- if not exec_openssl("Generating an RSA Private Server Key", *args):
- return 1
-
- args = (
- "req",
- "-new",
- "-key",
- "server.key",
- "-out",
- "server.csr",
- "-passin",
- "pass:{rsa_password}".format(rsa_password=conf.rsa_password),
- "-subj",
- conf.params
- )
- if not exec_openssl("Creating a Certificate Signing Request (CSR)", *args):
- return 1
-
- logging.info("The Certificate Signing Request has been generated")
- os.rename("server.key", "server.key.orig")
-
- args = (
- "rsa",
- "-in",
- "server.key.orig",
- "-out",
- "server.key",
- "-passin",
- "pass:{rsa_password}".format(rsa_password=conf.rsa_password)
- )
- if not exec_openssl("Removing the pass phrase from the server key", *args):
- return 1
-
- logging.info("The pass phrase has been removed from the server key")
-
- args = (
- "x509",
- "-req",
- "-days",
- "365",
- "-in",
- "server.csr",
- "-signkey",
- "server.key",
- "-out",
- "server.crt"
- )
- if not exec_openssl("Generating a Self-signed certificate", *args):
- return 1
-
- logging.info("A server key and self signed certificate has been generated")
- logging.info("Installing a server key and certificate")
-
- keypath = os.path.join(root, 'etc/pki/tls/private/localhost.key')
- shutil.copy("server.key", keypath)
- os.chmod(keypath, stat.S_IRUSR | stat.S_IWUSR)
- os.chown(keypath, pwd.getpwnam(ops_user).pw_uid, grp.getgrnam(ops_group).gr_gid)
-
- logging.info("The private key has been installed")
- logging.info("Installing self signed certificate")
-
- certpath = os.path.join(root, 'etc/pki/tls/certs/localhost.crt')
- shutil.copy("server.crt", certpath)
- os.chmod(certpath, stat.S_IRUSR | stat.S_IWUSR)
- os.chown(certpath, pwd.getpwnam(ops_user).pw_uid, grp.getgrnam(ops_group).gr_gid)
-
- logging.info("Saving the self signed csr")
-
- csrpath = os.path.join(root, 'etc/pki/tls/certs/localhost.csr')
- shutil.copy("server.csr", csrpath)
- os.chmod(csrpath, stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IWGRP | stat.S_IROTH)
- os.chown(csrpath, pwd.getpwnam(ops_user).pw_uid, grp.getgrnam(ops_group).gr_gid)
-
- log_msg = """
- The self signed certificate has now been installed.
-
- You may obtain a certificate signed by a Certificate Authority using the
- server.csr file saved in the current directory. Once you have obtained
- a signed certificate, copy it to %s and
- restart Traffic Ops."""
- logging.info(log_msg, certpath)
-
- cdn_conf_path = os.path.join(root, "opt/traffic_ops/app/conf/cdn.conf")
-
- try:
- with open(cdn_conf_path) as conf_file:
- cdn_conf = json.load(conf_file)
- except (OSError, ValueError) as e:
- exception = OSError("reading {cdn_conf_path}: {e}".format(cdn_conf_path=cdn_conf_path, e=e))
- exception.__cause__ = e
- raise exception
-
- if (
- not isinstance(cdn_conf, dict) or
- "hypnotoad" not in cdn_conf or
- not isinstance(cdn_conf["hypnotoad"], dict)
- ):
- logging.critical("Malformed %s; improper object and/or missing 'hypnotoad' key", cdn_conf_path)
- return 1
-
- hypnotoad = cdn_conf["hypnotoad"]
- if (
- "listen" not in hypnotoad or
- not isinstance(hypnotoad["listen"], list) or
- not hypnotoad["listen"] or
- not isinstance(hypnotoad["listen"][0], str)
- ):
- log_msg = """ The "listen" portion of %s is missing from %s
- Please ensure it contains the same structure as the one originally installed"""
- logging.error(log_msg, cdn_conf_path, cdn_conf_path)
- return 1
-
- listen = hypnotoad["listen"][0]
-
- if "cert={certpath}".format(certpath=certpath) not in listen or "key={keypath}".format(keypath=keypath) not in listen:
- log_msg = """ The "listen" portion of %s is:
- %s
- and does not reference the same "cert=" and "key=" values as are created here.
- Please modify %s to add the following as parameters:
- ?cert=%s&key=%s"""
- logging.error(log_msg, cdn_conf_path, listen, cdn_conf_path, certpath, keypath)
- return 1
-
- return 0
-
-def random_word(length = 12): # type: (int) -> str
- """
- Returns a randomly generated string 'length' characters long containing only word
- characters ([a-zA-Z0-9_]).
- """
- word_chars = string.ascii_letters + string.digits + '_'
- return ''.join(random.choice(word_chars) for _ in range(length))
-
-def generate_cdn_conf(questions, fname, automatic, root): # type: (list[Question], str, bool, str) -> None
- """
- Generates some properties of a cdn.conf file based on the passed questions.
-
- This modifies or writes the file 'fname' under the directory 'root'.
- """
- cdn_conf = get_config(questions, fname, automatic)
-
- if "genSecret" not in cdn_conf:
- raise ValueError("missing 'genSecret' config_var")
-
- gen_secret = cdn_conf["genSecret"].lower() in {'y', 'yes'}
-
- try:
- num_secrets = int(cdn_conf["keepSecrets"])
- except KeyError as e:
- exception = ValueError("missing 'keepSecrets' config_var")
- exception.__cause__ = e
- raise exception
- except ValueError as e:
- exception = ValueError("invalid 'keepSecrets' config_var value: {e}".format(e=e))
- exception.__cause__ = e
- raise exception
-
- try:
- port = int(cdn_conf["port"])
- except KeyError as e:
- exception = ValueError("missing 'port' config_var")
- exception.__cause__ = e
- raise exception
- except ValueError as e:
- exception = ValueError("invalid 'port' config_var value: {e}".format(e=e))
- exception.__cause__ = e
- raise exception
-
- try:
- workers = int(cdn_conf["workers"])
- except KeyError as e:
- exception = ValueError("missing 'workers' config_var")
- exception.__cause__ = e
- raise exception
- except ValueError as e:
- exception = ValueError("invalid 'workers' config_var value: {e}".format(e=e))
- exception.__cause__ = e
- raise exception
-
- try:
- url = cdn_conf["base_url"]
- except KeyError as e:
- exception = ValueError("missing 'base_url' config_var")
- exception.__cause__ = e
- raise exception
-
- try:
- ldap_loc = cdn_conf["ldap_conf_location"]
- except KeyError as e:
- exception = ValueError("missing 'ldap_conf_location' config_var")
- exception.__cause__ = e
- raise exception
-
- conf = CDNConfig(gen_secret, num_secrets, port, workers, url, ldap_loc)
-
- path = os.path.join(root, fname.lstrip('/'))
- existing_conf = {}
- if os.path.isfile(path):
- with open(path) as conf_file:
- try:
- existing_conf = json.load(conf_file)
- except ValueError as e:
- exception = ValueError("invalid existing cdn.config at {path}: {e}".format(path=path, e=e))
- exception.__cause__ = e
- raise exception
-
- if not isinstance(existing_conf, dict):
- logging.warning("Existing cdn.conf (at '%s') is not an object - overwriting", path)
- existing_conf = {}
-
- conf.generate_secret(existing_conf)
- conf.insert_url(existing_conf)
-
- if (
- "traffic_ops_golang" not in existing_conf or
- not isinstance(existing_conf["traffic_ops_golang"], dict)
- ):
- existing_conf["traffic_ops_golang"] = {}
-
- existing_conf["traffic_ops_golang"]["port"] = conf.port
- err_log = os.path.join(root, "var/log/traffic_ops/error.log")
- existing_conf["traffic_ops_golang"]["log_location_error"] = err_log
- access_log = os.path.join(root, "var/log/traffic_ops/access.log")
- existing_conf["traffic_ops_golang"]["log_location_event"] = access_log
-
- if "hypnotoad" not in existing_conf or not isinstance(existing_conf["hypnotoad"], dict):
- existing_conf["hypnotoad"]["workers"] = conf.num_workers
-
- with open(path, "w+") as conf_file:
- json.dump(existing_conf, conf_file, indent=indent)
- print(file=conf_file)
- logging.info("CDN configuration has been saved")
-
-def db_connection_string(dbconf): # type: (dict) -> str
- """
- Constructs a database connection string from the passed configuration object.
- """
- user = dbconf["user"]
- password = dbconf["password"]
- db_name = "traffic_ops" if dbconf["type"] == "Pg" else dbconf["type"]
- hostname = dbconf["hostname"]
- port = dbconf["port"]
- return "postgresql://{user}:{password}@{hostname}:{port}/{db_name}".format(user=user, password=password, hostname=hostname, port=port, db_name=db_name)
-
-def exec_psql(conn_str, query): # type: (str, str) -> str
- """
- Executes SQL queries by forking and exec-ing '/usr/bin/psql'.
-
- :param conn_str: A "connection string" that defines the postgresql resource in the format
- {schema}://{user}:{password}@{host or IP}:{port}/{database}
- :param query: The query to be run. It can actually be a script containing multiple queries.
- :returns: The comma-separated columns of each line-delimited row of the results of the query.
- """
- cmd = ["/usr/bin/psql", "--tuples-only", "-d", conn_str, "-c", query]
- proc = subprocess.Popen(
- cmd,
- stderr=subprocess.PIPE,
- stdout=subprocess.PIPE,
- universal_newlines=True,
- )
- proc.wait()
- if proc.returncode != 0:
- logging.debug("psql exec failed; stderr: %s\n\tstdout: %s", proc.stderr, proc.stdout)
- raise OSError("failed to execute database query")
- if sys.version_info.major >= 3:
- return proc.stdout.strip()
- else:
- return string.strip(proc.stdout)
-
-def invoke_db_admin_pl(action, root): # type: (str, str) -> None
- """
- Exectues admin with the given action, and looks for it from the given root directory.
- """
- path = os.path.join(root, "opt/traffic_ops/app")
- # This is a workaround for admin using hard-coded relative paths. That
- # should be fixed at some point, IMO, but for now this works.
- os.chdir(path)
- cmd = [os.path.join(path, "db/admin"), "--env=production", action]
- proc = subprocess.Popen(
- cmd,
- stderr=subprocess.PIPE,
- stdout=subprocess.PIPE,
- universal_newlines=True,
- )
- proc.wait()
- if proc.returncode != 0:
- logging.debug("admin exec failed; stderr: %s\n\tstdout: %s", proc.stderr, proc.stdout)
- raise OSError("Database {action} failed".format(action=action))
- logging.info("Database %s succeeded", action)
-
-def setup_database_data(conn_str, user, param_conf, root): # type: (str, User, dict, str) -> None
- """
- Sets up all necessary initial database data using `/usr/bin/sql`
- """
- logging.info("paramconf %s", param_conf)
- logging.info("Setting up the database data")
-
- tables_found_query = '''
- SELECT EXISTS(
- SELECT 1
- FROM pg_tables
- WHERE schemaname = 'public'
- AND tablename = 'tm_user'
- );'''
- if exec_psql(conn_str, tables_found_query) == "t":
- logging.info("Found existing tables skipping table creation")
- else:
- invoke_db_admin_pl("load_schema", root)
-
- invoke_db_admin_pl("migrate", root)
- invoke_db_admin_pl("seed", root)
- invoke_db_admin_pl("patch", root)
-
- hashed_pass = hash_pass(user.password)
- insert_admin_query = '''
- INSERT INTO tm_user (username, tenant_id, role, local_passwd, confirm_local_passwd)
- VALUES (
- '{}',
- (SELECT id FROM tenant WHERE name = 'root'),
- (SELECT id FROM role WHERE name = 'admin'),
- '{hashed_pass}',
- '{hashed_pass}'
- )
- ON CONFLICT (username) DO NOTHING;
- '''.format(user.username, hashed_pass=hashed_pass)
- _ = exec_psql(conn_str, insert_admin_query)
-
- logging.info("=========== Setting up cdn")
- insert_cdn_query = "\n\t-- global parameters" + '''
- INSERT INTO cdn (name, domain_name, dnssec_enabled)
- VALUES ('{cdn_name}', '{dns_subdomain}', false)
- ON CONFLICT DO NOTHING;
- '''.format(**param_conf)
- logging.info("\n%s", insert_cdn_query)
- _ = exec_psql(conn_str, insert_cdn_query)
-
- tm_url = param_conf["tm.url"]
-
- logging.info("=========== Setting up parameters")
- insert_parameters_query = "\n\t-- global parameters" + '''
- INSERT INTO parameter (name, config_file, value)
- VALUES ('tm.url', 'global', '{tm_url}'),
- ('tm.infourl', 'global', '{tm_url}/doc'),
- -- CRConfic.json parameters
- ('geolocation.polling.url', 'CRConfig.json', '{tm_url}/routing/GeoLite2-City.mmdb.gz'),
- ('geolocation6.polling.url', 'CRConfig.json', '{tm_url}/routing/GeoLiteCityv6.dat.jz')
- ON CONFLICT (name, config_file, value) DO NOTHING;
- '''.format(tm_url=tm_url)
- logging.info("\n%s", insert_parameters_query)
- _ = exec_psql(conn_str, insert_parameters_query)
-
- logging.info("\n=========== Setting up profiles")
- insert_profiles_query = "\n\t-- global parameters" + '''
- INSERT INTO profile (name, description, type, cdn)
- VALUES ('GLOBAL' 'Global Traffic Ops profile, DO NOT DELETE', 'UNK_PROFILE', (SELECT id FROM cdn WHERE name='ALL'))
- ON CONFLICT DO NOTHING;
-
- INSERT INTO profile_parameter (profile, parameter)
- VALUES
- (
- (SELECT id FROM profile WHERE name = 'GLOBAL'),
- (
- SELECT id
- FROM parameter
- WHERE name = 'tm.url'
- AND config_file = 'global'
- AND value = '{tm_url}'
- )
- ),
- (
- (SELECT id FROM profile WHERE name = 'GLOBAL'),
- (
- SELECT id
- FROM parameter
- WHERE name = 'tm.infourl'
- AND config_file = 'global'
- AND value = '{tm_url}/doc'
- )
- ),
- (
- (SELECT id FROM profile WHERE name = 'GLOBAL'),
- (
- SELECT id
- FROM parameter
- WHERE name = 'geolocation.polling.url'
- AND config_file = 'CRConfig.json'
- AND value = '{tm_url}/routing/GeoLite2-City.mmdb.gz'
- )
- ),
- (
- (SELECT id FROM profile WHERE name = 'GLOBAL'),
- (
- SELECT id
- FROM parameter
- WHERE name = 'geolocation6.polling.url'
- AND config_file = 'CRConfig.json'
- AND value = '{tm_url}/routing/GeoLiteCityv6.mmdb.gz'
- )
- )
- ON CONFLICT (profile, parameter) DO NOTHING;
- '''.format(tm_url=tm_url)
- logging.info("\n%s", insert_profiles_query)
- _ = exec_psql(conn_str, insert_cdn_query)
-
-def main(
-automatic, # type: bool
-debug, # type: bool
-defaults, # type: str
-cfile, # type: str
-root_dir, # type: str
-ops_user, # type: str
-ops_group, # type: str
-no_restart_to, # type: bool
-no_database, # type: bool
-):
- """
- Runs the main routine given the parsed arguments as input.
- :rtype: int
- """
- if debug:
- logging.getLogger().setLevel(logging.DEBUG)
- else:
- logging.getLogger().setLevel(logging.INFO)
-
- # At this point, the Perl script... unzipped its own logfile?
-
- logging.info("Starting postinstall")
- # The Perl printed this whether or not the logger was actually at the debug level
- # so we do too
- logging.info("Debug is on")
-
- if automatic:
- logging.info("Running in automatic mode")
-
- if defaults is not None:
- try:
- if defaults:
- try:
- with open(defaults, "w") as dump_file:
- json.dump(DEFAULTS, dump_file, indent=indent)
- except OSError as e:
- logging.critical("Writing output: %s", e)
- return 1
- else:
- json.dump(DEFAULTS, sys.stdout, cls=ConfigEncoder, indent=indent)
- print()
- except ValueError as e:
- logging.critical("Converting defaults to JSON: %s", e)
- return 1
- return 0
-
- if not cfile:
- logging.info("No input file given - using defaults")
- user_input = DEFAULTS
- else:
- logging.info("Using input file %s", cfile)
- try:
- with open(cfile) as conf_file:
- user_input = unmarshal_config(json.load(conf_file))
- diffs = sanity_check_config(user_input, automatic)
- logging.info(
- "File sanity check complete - found %s difference%s",
- diffs,
- '' if diffs == 1 else 's'
- )
- except (OSError, ValueError) as e:
- logging.critical("Reading in input file '%s': %s", cfile, e)
- return 1
-
- try:
- dbconf = generate_db_conf(user_input[DATABASE_CONF_FILE], DATABASE_CONF_FILE, automatic, root_dir)
- todbconf = generate_todb_conf(user_input[DB_CONF_FILE], DB_CONF_FILE, automatic, root_dir, dbconf)
- generate_ldap_conf(user_input[LDAP_CONF_FILE], LDAP_CONF_FILE, automatic, root_dir)
- admin_conf = generate_users_conf(
- user_input[USERS_CONF_FILE],
- USERS_CONF_FILE,
- automatic,
- root_dir
- )
- generate_profiles_dir(user_input[PROFILES_CONF_FILE])
- opensslconf = generate_openssl_conf(user_input[OPENSSL_CONF_FILE], OPENSSL_CONF_FILE, automatic)
- paramconf = generate_param_conf(user_input[PARAM_CONF_FILE], PARAM_CONF_FILE, automatic, root_dir)
- postinstall_cfg = os.path.join(root_dir, POST_INSTALL_CFG.lstrip('/'))
- if not os.path.isfile(postinstall_cfg):
- with open(postinstall_cfg, 'w+') as conf_file:
- print("{}", file=conf_file)
- except OSError as e:
- logging.critical("Writing configuration: %s", e)
- return 1
- except ValueError as e:
- logging.critical("Generating configuration: %s", e)
- return 1
-
- try:
- setup_maxmind(todbconf.get("maxmind", "no"), root_dir)
- except OSError as e:
- logging.critical("Setting up MaxMind: %s", e)
- return 1
-
- try:
- cert_code = setup_certificates(opensslconf, root_dir, ops_user, ops_group)
- if cert_code:
- return cert_code
- except OSError as e:
- logging.critical("Setting up SSL Certificates: %s", e)
- return 1
-
- try:
- generate_cdn_conf(user_input[CDN_CONF_FILE], CDN_CONF_FILE, automatic, root_dir)
- except OSError as e:
- logging.critical("Generating cdn.conf: %s", e)
- return 1
-
- if not no_database:
- try:
- conn_str = db_connection_string(dbconf)
- except KeyError as e:
- logging.error("Missing database connection variable: %s", e)
- logging.error(
- "Can't connect to the database. " \
- "Use the script `/opt/traffic_ops/install/bin/todb_bootstrap.sh` " \
- "on the db server to create it and run `postinstall` again."
- )
- return 1
-
- if not os.path.isfile("/usr/bin/psql") or not os.access("/usr/bin/psql", os.X_OK):
- logging.critical("psql is not installed, please install it to continue with database setup")
- return 1
-
- def db_connect_failed():
- logging.error("Failed to set up database: %s", e)
- logging.error(
- "Can't connect to the database. "
- "Use the script `/opt/traffic_ops/install/bin/todb_bootstrap.sh` "
- "on the db server to create it and run `postinstall` again."
- )
-
- try:
- setup_database_data(conn_str, admin_conf, paramconf, root_dir)
- except (OSError, subprocess.CalledProcessError)as e:
- db_connect_failed()
- return 1
- except subprocess.SubprocessError as e:
- db_connect_failed()
- return 1
-
-
- if not no_restart_to:
- logging.info("Starting Traffic Ops")
- try:
- cmd = ["/sbin/service", "traffic_ops", "restart"]
- proc = subprocess.Popen(
- cmd,
- stderr=subprocess.PIPE,
- stdout=subprocess.PIPE,
- universal_newlines=True,
- )
- if proc.wait():
- raise subprocess.CalledProcessError(proc.returncode, cmd)
- except subprocess.CalledProcessError as e:
- logging.critical("Failed to restart Traffic Ops, return code %s: %s", e.returncode, e)
- logging.debug("stderr: %s\n\tstdout: %s", proc.stderr, proc.stdout)
- return 1
- except OSError as e:
- logging.critical("Failed to restart Traffic Ops: unknown error occurred: %s", e)
- return 1
- # Perl didn't actually do any "waiting" before reporting success, so
- # neither do we
- logging.info("Waiting for Traffic Ops to restart")
- else:
- logging.info("Skipping Traffic Ops restart")
- logging.info("Success! Postinstall complete.")
-
- return 0
-
-if __name__ == '__main__':
- PARSER = argparse.ArgumentParser()
- PARSER.add_argument(
- "-a",
- "--automatic",
- help="If there are questions in the config file which do not have answers, the script " +
- "will look to the defaults for the answer. If the answer is not in the defaults the " +
- "script will exit",
- action="store_true"
- )
- PARSER.add_argument(
- "--cfile",
- help="An input config file used to ask and answer questions",
- type=str,
- default=None
- )
- PARSER.add_argument(
- "-cfile",
- help=argparse.SUPPRESS,
- type=str,
- default=None,
- dest="legacy_cfile"
- )
- PARSER.add_argument("--debug", help="Enables verbose output", action="store_true")
- PARSER.add_argument("-debug", help=argparse.SUPPRESS, dest="legacy_debug", action="store_true")
- PARSER.add_argument(
- "--defaults",
- help="Writes out a configuration file with defaults which can be used as input",
- type=str,
- nargs="?",
- default=None,
- const=""
- )
- PARSER.add_argument(
- "-defaults",
- help=argparse.SUPPRESS,
- type=str,
- nargs="?",
- default=None,
- const="",
- dest="legacy_defaults"
- )
- PARSER.add_argument(
- "-n",
- "--no-root",
- help="Enable running as a non-root user (may cause failure)",
- action="store_true"
- )
- PARSER.add_argument(
- "-r",
- "--root-directory",
- help="Set the directory to be treated as the system's root directory (e.g. for testing)",
- type=str,
- default="/"
- )
- PARSER.add_argument(
- "-u",
- "--ops-user",
- help="Specify a username to own Traffic Ops files and processes",
- type=str,
- default="trafops"
- )
- PARSER.add_argument(
- "-g",
- "--ops-group",
- help="Specify the group to own Traffic Ops files and processes",
- type=str,
- default="trafops"
- )
- PARSER.add_argument(
- "--no-restart-to",
- help="Skip restarting Traffic Ops after configuration and database changes are applied",
- action="store_true"
- )
- PARSER.add_argument("--no-database", help="Skip all database operations", action="store_true")
-
- ARGS = PARSER.parse_args()
-
- USED_LEGACY_ARGS = False
- DEFAULTS_ARG = None
- if ARGS.legacy_defaults:
- if ARGS.defaults:
- logging.error("cannot specify both '--defaults' and '-defaults'")
- sys.exit(1)
- USED_LEGACY_ARGS = True
- DEFAULTS_ARG = ARGS.legacy_defaults
- else:
- DEFAULTS_ARG = ARGS.defaults
-
- DEBUG = False
- if ARGS.legacy_debug:
- if ARGS.debug:
- logging.error("cannot specify both '--debug' and '-debug'")
- sys.exit(1)
- USED_LEGACY_ARGS = True
- DEBUG = ARGS.legacy_debug
- else:
- DEBUG = ARGS.debug
-
- CFILE = None
- if ARGS.legacy_cfile:
- if ARGS.cfile:
- logging.error("cannot specify both '--cfile' and '-cfile'")
- sys.exit(1)
- USED_LEGACY_ARGS = True
- CFILE = ARGS.legacy_cfile
- else:
- CFILE = ARGS.cfile
-
- if not ARGS.no_root and os.getuid() != 0:
- logging.error("You must run this script as the root user")
- logging.shutdown()
- sys.exit(1)
-
- if USED_LEGACY_ARGS:
- logging.warning(
- "passing long options with a single '-' is deprecated, please use '--' in the future"
- )
-
- try:
- EXIT_CODE = main(
- ARGS.automatic,
- DEBUG,
- DEFAULTS_ARG,
- CFILE,
- os.path.abspath(ARGS.root_directory),
- ARGS.ops_user,
- ARGS.ops_group,
- ARGS.no_restart_to,
- ARGS.no_database
- )
- sys.exit(EXIT_CODE)
- except KeyboardInterrupt:
- sys.exit(1)
- finally:
- logging.shutdown()
diff --git a/traffic_ops/install/bin/postinstall.test.sh b/traffic_ops/install/bin/postinstall.test.sh
index b0c44fe..aa5d30a 100755
--- a/traffic_ops/install/bin/postinstall.test.sh
+++ b/traffic_ops/install/bin/postinstall.test.sh
@@ -46,10 +46,10 @@ python_version="${python_version:-3}";
python_bin="${python_bin:-/usr/bin/python${python_version}}";
if [[ ! -x "$python_bin" && "$python_version" -ge 3 ]]; then
- echo "Python 3.6+ is required to run - or test - postinstall.py" >&2;
+ echo "Python 3.6+ is required to run - or test - _postinstall" >&2;
exit 1;
elif [[ ! -x "$python_bin" && "$python_version" == 2 ]]; then
- echo "Python ${python_version} is required to run - or test - postinstall.py against Python 2" >&2;
+ echo "Python ${python_version} is required to run - or test - _postinstall against Python 2" >&2;
fi
readonly TO_PASSWORD=twelve;
@@ -59,8 +59,16 @@ trap 'rm -rf $ROOT_DIR' EXIT;
"$python_bin" <<EOF;
from __future__ import print_function
+import importlib
import sys
-from postinstall import Scrypt
+from os.path import dirname, join
+module_name = '_postinstall'
+if sys.version_info.major >= 3:
+ from importlib.machinery import SourceFileLoader
+ Scrypt = SourceFileLoader(module_name, join(dirname(__file__), module_name)).load_module(module_name).Scrypt
+else:
+ import imp
+ Scrypt = imp.load_source(module_name, join(dirname(__file__), module_name)).Scrypt
passwd = '${TO_PASSWORD}'
n = 2 ** 10
@@ -110,7 +118,7 @@ EOF
mkdir -p "$ROOT_DIR/opt/traffic_ops/install/data/json";
mkdir "$ROOT_DIR/opt/traffic_ops/install/bin";
-# defaults.json is used as input into the `--cfile` option of postinstall.py
+# defaults.json is used as input into the `--cfile` option of _postinstall
# for testing purposes
cat <<- EOF > "$ROOT_DIR/defaults.json"
{
@@ -318,7 +326,7 @@ cat <<- EOF > "$ROOT_DIR/defaults.json"
}
EOF
-"$python_bin" "$MY_DIR/postinstall.py" --no-root --root-directory="$ROOT_DIR" --no-restart-to --no-database --ops-user="$(whoami)" --ops-group="$(id -gn)" --automatic --cfile="$ROOT_DIR/defaults.json" --debug 2>"$ROOT_DIR/stderr" | tee "$ROOT_DIR/stdout"
+"$python_bin" "$MY_DIR/_postinstall" --no-root --root-directory="$ROOT_DIR" --no-restart-to --no-database --ops-user="$(whoami)" --ops-group="$(id -gn)" --automatic --cfile="$ROOT_DIR/defaults.json" --debug 2>"$ROOT_DIR/stderr" | tee "$ROOT_DIR/stdout"
if grep -q 'ERROR' $ROOT_DIR/stderr; then
echo "Errors found in script logs" >&2;
@@ -449,7 +457,7 @@ if not isinstance(conf['traffic_ops_golang'], dict) or len(conf['traffic_ops_gol
print('Malformed traffic_ops_golang object in cdn.conf:', conf['traffic_ops_golang'], sys.stderr)
exit(1)
-if conf['traffic_ops_golang']['port'] != 443:
+if conf['traffic_ops_golang']['port'] != '443':
print('Incorrect traffic_ops_golang.port, expected: 443, got:', conf['traffic_ops_golang']['port'], file=sys.stderr)
exit(1)