You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficcontrol.apache.org by GitBox <gi...@apache.org> on 2020/07/09 15:53:28 UTC
[GitHub] [trafficcontrol] jrushford opened a new pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
jrushford opened a new pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872
This PR is a transliteration of the traffic_ops_ort.pl perl script to go. It built command is 'traffic_ops_t3c' and is designed to replace the perl script entirely. 'traffic_ops_t3c' depends on and uses 'atstccfg'. The go version here implements all the functionality in the perl version. unit and integration tests will be added. Looking for feedback from the community on this Draft PR.
## Which Traffic Control components are affected by this PR?
<!-- Please delete all components from this list that are NOT affected by this
Pull Request. Also, feel free to add the name of a tool or script that is
affected but not on the list.
Additionally, if this Pull Request does NOT affect documentation, please
explain why documentation is not required. -->
- Traffic Ops ORT
## What is the best way to verify this PR?
Using a test environment, compare the run results from traffic_ops_ort.pl to the run results from 'traffic_ops_t3c'
## The following criteria are ALL met by this PR
- [ ] This PR includes tests OR I have explained why tests are unnecessary
- [ ] This PR includes documentation OR I have explained why documentation is unnecessary
- [ ] This PR includes an update to CHANGELOG.md OR such an update is not necessary
- [X] This PR includes any and all required license headers
- [ ] This PR ensures that database migration sequence is correct OR this PR does not include a database migration
- [X] This PR **DOES NOT FIX A SERIOUS SECURITY VULNERABILITY** (see [the Apache Software Foundation's security guidelines](https://www.apache.org/security/) for details)
## Additional Information
<!-- If you would like to include any additional information on the PR for
potential reviewers please put it here.
command usage:
`
./traffic_ops_t3c --help
Usage: traffic_ops_t3c [options] <Traffic_Ops_URL> <Traffic_Ops_Login>
<Traffic_Ops_URL> = URL to Traffic Ops host. Example: https://trafficops.company.net
<Traffic_Ops_Login> Example: 'username:password'
[options]:
--dispersion=[time in seconds] | -D, [time in seconds] wait a random number between 0 and <time in seconds> before starting, default = 300s
--login-dispersion=[time in seconds] | -l, [time in seconds] wait a random number between 0 and <time in seconds> befor login, default = 0
--log-location-debug=[value] | -d [value], Where to log debugs. May be a file path, stdout, stderr, or null, default stdout
--log-location-error=[value] | -e [value], Where to log errors. May be a file path, stdout, stderr, or null, default stdout
--log-location-info=[value] | -i [value], Where to log info. May be a file path, stdout, stderr, or null, default stdout
--log-location-warning=[value] | -w [value], Where to log warnings. May be a file path, stdout, stderr, or null, default stdout
--run-mode=[mode] | -m [mode] where mode is one of [interactive | report | badass | syncds | revalidate ], default = report
--cache-hostname=[hostname] | -H [hostname], Host name of the cache to generate config for. Must be the server host name in Traffic Ops, not a URL, and not the FQDN
--num-retries=[number] | -r [number], retry connection to Traffic Ops URL [number] times, default is 3
--reval-wait-time=[seconds] | -T [seconds] wait a random number of seconds between 0 and [seconds] before revlidation, default is 60
--rev-proxy-disable=[true|false] | -P [true|false] bypass the reverse proxy even if one has been configured, default = false
--skip-os-check=[true|false] | -s [true | false] bypass the check for a supported CentOS version. default = false
--traffic-ops-timeout-milliseconds=[milliseconds] | -t [milliseconds] the Traffic Ops request timeout in milliseconds. Default = 30000 (30 seconds)
--traffic-ops-url=[url] | -u [url], Traffic Ops URL. Must be the full URL, including the scheme. Required. May also be set with the environment variable TO_URL
--traffic-ops-user=[username] | -U [username], Traffic Ops username. Required. May also be set with the environment variable TO_USER
--traffic-ops-password=[password] | -P [password], Traffic Ops password. Required. May also be set with the environment variable TO_PASS
--wait-for-parents | -w [true | false] do not update if parent_pending = 1 in the update json. default = true, wait for parents
--help | -h, Print usage information and exit
`
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r454555279
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
Review comment:
vendored
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#issuecomment-656266631
yeah the systemd stuff is easily removed or bypassed but, the initial goal was to transliterate the perl.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453933247
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
+ if rc == 1 { // file is not part of any package.
+ return nil, nil
+ } else if rc == 0 { // add the package name the file belongs to.
+ log.Debugf("output from file-query: %s\n", string(output))
+ result = append(result, string(strings.TrimSpace(string(output))))
+ log.Debugf("result length: %d, result: %s\n", len(result), string(output))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-provides": // returns the package name that provides 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatprovides", name)
+ log.Debugf("pkg-provides - name: %s, output: %s\n", name, output)
+ if rc == 1 { // no package provides 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-query": // returns the package name for 'name'.
+ output, rc, err := ExecCommand("/bin/rpm", "-q", name)
+ if rc == 1 { // the package is not installed.
+ return nil, nil
+ } else if rc == 0 { // add the rpm name
+ result = append(result, string(strings.TrimSpace(string(output))))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-requires": // returns a list of packages that requires package 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatrequires", name)
+ if rc == 1 { // no package reuires package 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ }
+ return result, nil
+}
+
+func RandomDuration(max time.Duration) time.Duration {
+ rand.Seed(time.Now().UnixNano())
+ return time.Duration(rand.Int63n(int64(max)))
+}
+
+func CheckUser(cfg config.Cfg) bool {
+ result := true
+ user, _ := user.Current()
+
+ switch cfg.RunMode {
+ case config.BADASS:
+ fallthrough
+ case config.INTERACTIVE:
+ fallthrough
+ case config.SYNCDS:
+ if user.Username != "root" {
+ log.Errorf("Only the root user may run in BADASS, INTERACTIVE, or SYNCDS mode, current user: %s\n",
+ user.Username)
+ result = false
+ }
+ default:
+ log.Infof("current mode: %s, run user: %s\n", cfg.RunMode, user.Username)
+ }
+ return result
+}
+
+func CleanTmpDir() bool {
+ now := time.Now().Unix()
+
+ files, err := ioutil.ReadDir(config.TmpBase)
+ if err != nil {
+ log.Errorf("opening %s: %v\n", config.TmpBase, err)
+ return false
+ }
+ for _, f := range files {
+ // remove any directory and its contents under the config.TmpBase (/tmp/ort) that
+ // is more than a week old.
+ if f.IsDir() && (now-f.ModTime().Unix()) > 604800 {
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453893598
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
Review comment:
ok, it's changed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453895452
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453852871
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
+ "net/url"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+)
+
+const (
+ StatusDir = "/opt/ort/status"
+ AtsTcConfig = "/opt/ort/atstccfg"
+ Chkconfig = "/sbin/chkconfig"
+ Service = "/sbin/service"
+ SystemCtl = "/bin/systemctl"
+ TmpBase = "/tmp/ort"
+ TSHome = "/opt/trafficserver"
+ TrafficCtl = TSHome + "/bin/traffic_ctl"
+ TrafficServerOwner = "ats"
+)
+
+type Mode int
+
+const (
+ BADASS Mode = 0
+ INTERACTIVE = 1
+ REPORT = 2
+ REVALIDATE = 3
+ SYNCDS = 4
+)
+
+func (m Mode) String() string {
+ switch m {
+ case 0:
+ return "BADASS"
+ case 1:
+ return "INTERACTIVE"
+ case 2:
+ return "REPORT"
+ case 3:
+ return "REVALIDATE"
+ case 4:
+ return "SYNCDS"
+ }
+ return ""
+}
+
+type SvcManagement int
+
+const (
+ UNKNOWN = 0
+ SYSTEMD = 1
+ SYSTEMV = 2 // legacy System V Init.
+)
+
+func (s SvcManagement) String() string {
+ switch s {
+ case UNKNOWN:
+ return "UNKNOWN"
+ case SYSTEMD:
+ return "SYSTEMD"
+ case SYSTEMV:
+ return "SYSTEMV"
+ }
+ return "UNKNOWN"
+}
+
+type Cfg struct {
+ Dispersion time.Duration
+ LogLocationDebug string
+ LogLocationErr string
+ LogLocationInfo string
+ LogLocationWarn string
+ LoginDispersion time.Duration
+ CacheHostName string
+ SvcManagement SvcManagement
+ Retries int
+ RevalWaitTime time.Duration
+ ReverseProxyDisable bool
+ RunMode Mode
+ SkipOSCheck bool
+ TOTimeoutMS time.Duration
+ TOUser string
+ TOPass string
+ TOURL string
+ WaitForParents bool
+ YumOptions string
+}
+
+func (cfg Cfg) ErrorLog() log.LogLocation { return log.LogLocation(cfg.LogLocationErr) }
+func (cfg Cfg) WarningLog() log.LogLocation { return log.LogLocation(cfg.LogLocationWarn) }
+func (cfg Cfg) InfoLog() log.LogLocation { return log.LogLocation(cfg.LogLocationInfo) }
+func (cfg Cfg) DebugLog() log.LogLocation { return log.LogLocation(cfg.LogLocationDebug) }
+func (cfg Cfg) EventLog() log.LogLocation { return log.LogLocation(log.LogLocationNull) } // event logging is not used.
+
+func GetCfg() (Cfg, error) {
+ var err error
+
+ dispersionPtr := getopt.IntLong("dispersion", 'D', 300, "[seconds] wait a random number of seconds between 0 and [seconds] before starting, default 300")
+ loginDispersionPtr := getopt.IntLong("login-dispersion", 'l', 0, "[seconds] wait a random number of seconds betwee 0 and [seconds] befor login to traffic ops, default 0")
+ logLocationDebugPtr := getopt.StringLong("log-location-debug", 'd', "", "Where to log debugs. May be a file path, stdout, stderr, or null, default ''")
+ logLocationErrorPtr := getopt.StringLong("log-location-error", 'e', "stderr", "Where to log errors. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationInfoPtr := getopt.StringLong("log-location-info", 'i', "stderr", "Where to log info. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationWarnPtr := getopt.StringLong("log-location-warning", 'w', "stderr", "Where to log warnings. May be a file path, stdout, stderr, or null, default stderr")
+ cacheHostNamePtr := getopt.StringLong("cache-host-name", 'H', "", "Host name of the cache to generate config for. Must be the server host name in Traffic Ops, not a URL, and not the FQDN")
+ retriesPtr := getopt.IntLong("num-retries", 'r', 3, "[number] retry connection to Traffic Ops URL [number] times, default is 3")
+ revalWaitTimePtr := getopt.IntLong("reval-wait-time", 'T', 60, "[seconds] wait a random number of seconds between 0 and [seconds] before revlidation, default is 60")
+ reverseProxyDisablePtr := getopt.BoolLong("reverse-proxy-disable", 'p', "[false | true] bypass the reverse proxy even if one has been configured default is false")
+ runModePtr := getopt.StringLong("run-mode", 'm', "report", "[badass | interactive | report | revalidate | syncds] run mode, default is 'report'")
+ skipOSCheckPtr := getopt.BoolLong("skip-os-check", 's', "[false | true] skip os check, default is false")
+ toTimeoutMSPtr := getopt.IntLong("traffic-ops-timeout-milliseconds", 't', 30000, "Timeout in milli-seconds for Traffic Ops requests, default is 30000")
+ toURLPtr := getopt.StringLong("traffic-ops-url", 'u', "", "Traffic Ops URL. Must be the full URL, including the scheme. Required. May also be set with the environment variable TO_URL")
+ toUserPtr := getopt.StringLong("traffic-ops-user", 'U', "", "Traffic Ops username. Required. May also be set with the environment variable TO_USER")
+ toPassPtr := getopt.StringLong("traffic-ops-password", 'P', "", "Traffic Ops password. Required. May also be set with the environment variable TO_PASS")
+ waitForParentsPtr := getopt.BoolLong("wait-for-parents", 'W', "[true | false] do not update if parent_pending = 1 in the update json. default is false, wait for parents")
+ helpPtr := getopt.BoolLong("help", 'h', "Print usage information and exit")
+ getopt.Parse()
+
+ dispersion := time.Second * time.Duration(*dispersionPtr)
+ loginDispersion := time.Second * time.Duration(*loginDispersionPtr)
+ logLocationDebug := *logLocationDebugPtr
+ logLocationError := *logLocationErrorPtr
+ logLocationInfo := *logLocationInfoPtr
+ logLocationWarn := *logLocationWarnPtr
+
+ var cacheHostName string
+ if len(*cacheHostNamePtr) > 0 {
+ cacheHostName = *cacheHostNamePtr
+ } else {
+ cacheHostName, _ = os.Hostname()
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453916470
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
+ cfg, ok := r.configFiles[name]
+ return cfg, ok
+}
+
+// fetches and parses the multipart config files for a cache from traffic ops
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453853019
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
+ "net/url"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+)
+
+const (
+ StatusDir = "/opt/ort/status"
+ AtsTcConfig = "/opt/ort/atstccfg"
+ Chkconfig = "/sbin/chkconfig"
+ Service = "/sbin/service"
+ SystemCtl = "/bin/systemctl"
+ TmpBase = "/tmp/ort"
+ TSHome = "/opt/trafficserver"
+ TrafficCtl = TSHome + "/bin/traffic_ctl"
+ TrafficServerOwner = "ats"
+)
+
+type Mode int
+
+const (
+ BADASS Mode = 0
+ INTERACTIVE = 1
+ REPORT = 2
+ REVALIDATE = 3
+ SYNCDS = 4
+)
+
+func (m Mode) String() string {
+ switch m {
+ case 0:
+ return "BADASS"
+ case 1:
+ return "INTERACTIVE"
+ case 2:
+ return "REPORT"
+ case 3:
+ return "REVALIDATE"
+ case 4:
+ return "SYNCDS"
+ }
+ return ""
+}
+
+type SvcManagement int
+
+const (
+ UNKNOWN = 0
+ SYSTEMD = 1
+ SYSTEMV = 2 // legacy System V Init.
+)
+
+func (s SvcManagement) String() string {
+ switch s {
+ case UNKNOWN:
+ return "UNKNOWN"
+ case SYSTEMD:
+ return "SYSTEMD"
+ case SYSTEMV:
+ return "SYSTEMV"
+ }
+ return "UNKNOWN"
+}
+
+type Cfg struct {
+ Dispersion time.Duration
+ LogLocationDebug string
+ LogLocationErr string
+ LogLocationInfo string
+ LogLocationWarn string
+ LoginDispersion time.Duration
+ CacheHostName string
+ SvcManagement SvcManagement
+ Retries int
+ RevalWaitTime time.Duration
+ ReverseProxyDisable bool
+ RunMode Mode
+ SkipOSCheck bool
+ TOTimeoutMS time.Duration
+ TOUser string
+ TOPass string
+ TOURL string
+ WaitForParents bool
+ YumOptions string
+}
+
+func (cfg Cfg) ErrorLog() log.LogLocation { return log.LogLocation(cfg.LogLocationErr) }
+func (cfg Cfg) WarningLog() log.LogLocation { return log.LogLocation(cfg.LogLocationWarn) }
+func (cfg Cfg) InfoLog() log.LogLocation { return log.LogLocation(cfg.LogLocationInfo) }
+func (cfg Cfg) DebugLog() log.LogLocation { return log.LogLocation(cfg.LogLocationDebug) }
+func (cfg Cfg) EventLog() log.LogLocation { return log.LogLocation(log.LogLocationNull) } // event logging is not used.
+
+func GetCfg() (Cfg, error) {
+ var err error
+
+ dispersionPtr := getopt.IntLong("dispersion", 'D', 300, "[seconds] wait a random number of seconds between 0 and [seconds] before starting, default 300")
+ loginDispersionPtr := getopt.IntLong("login-dispersion", 'l', 0, "[seconds] wait a random number of seconds betwee 0 and [seconds] befor login to traffic ops, default 0")
+ logLocationDebugPtr := getopt.StringLong("log-location-debug", 'd', "", "Where to log debugs. May be a file path, stdout, stderr, or null, default ''")
+ logLocationErrorPtr := getopt.StringLong("log-location-error", 'e', "stderr", "Where to log errors. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationInfoPtr := getopt.StringLong("log-location-info", 'i', "stderr", "Where to log info. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationWarnPtr := getopt.StringLong("log-location-warning", 'w', "stderr", "Where to log warnings. May be a file path, stdout, stderr, or null, default stderr")
+ cacheHostNamePtr := getopt.StringLong("cache-host-name", 'H', "", "Host name of the cache to generate config for. Must be the server host name in Traffic Ops, not a URL, and not the FQDN")
+ retriesPtr := getopt.IntLong("num-retries", 'r', 3, "[number] retry connection to Traffic Ops URL [number] times, default is 3")
+ revalWaitTimePtr := getopt.IntLong("reval-wait-time", 'T', 60, "[seconds] wait a random number of seconds between 0 and [seconds] before revlidation, default is 60")
+ reverseProxyDisablePtr := getopt.BoolLong("reverse-proxy-disable", 'p', "[false | true] bypass the reverse proxy even if one has been configured default is false")
+ runModePtr := getopt.StringLong("run-mode", 'm', "report", "[badass | interactive | report | revalidate | syncds] run mode, default is 'report'")
+ skipOSCheckPtr := getopt.BoolLong("skip-os-check", 's', "[false | true] skip os check, default is false")
+ toTimeoutMSPtr := getopt.IntLong("traffic-ops-timeout-milliseconds", 't', 30000, "Timeout in milli-seconds for Traffic Ops requests, default is 30000")
+ toURLPtr := getopt.StringLong("traffic-ops-url", 'u', "", "Traffic Ops URL. Must be the full URL, including the scheme. Required. May also be set with the environment variable TO_URL")
+ toUserPtr := getopt.StringLong("traffic-ops-user", 'U', "", "Traffic Ops username. Required. May also be set with the environment variable TO_USER")
+ toPassPtr := getopt.StringLong("traffic-ops-password", 'P', "", "Traffic Ops password. Required. May also be set with the environment variable TO_PASS")
+ waitForParentsPtr := getopt.BoolLong("wait-for-parents", 'W', "[true | false] do not update if parent_pending = 1 in the update json. default is false, wait for parents")
+ helpPtr := getopt.BoolLong("help", 'h', "Print usage information and exit")
+ getopt.Parse()
+
+ dispersion := time.Second * time.Duration(*dispersionPtr)
+ loginDispersion := time.Second * time.Duration(*loginDispersionPtr)
+ logLocationDebug := *logLocationDebugPtr
+ logLocationError := *logLocationErrorPtr
+ logLocationInfo := *logLocationInfoPtr
+ logLocationWarn := *logLocationWarnPtr
+
+ var cacheHostName string
+ if len(*cacheHostNamePtr) > 0 {
+ cacheHostName = *cacheHostNamePtr
+ } else {
+ cacheHostName, _ = os.Hostname()
+ }
+
+ retries := *retriesPtr
+ revalWaitTime := time.Second * time.Duration(*revalWaitTimePtr)
+ reverseProxyDisable := *reverseProxyDisablePtr
+ runMode := *runModePtr
+ skipOsCheck := *skipOSCheckPtr
+ toTimeoutMS := time.Millisecond * time.Duration(*toTimeoutMSPtr)
+ toURL := *toURLPtr
+ toUser := *toUserPtr
+ toPass := *toPassPtr
+ waitForParents := *waitForParentsPtr
+ help := *helpPtr
+
+ if help {
+ Usage()
+ os.Exit(0)
+ }
+
+ runModeStr := strings.ToUpper(runMode)
+ var _runMode Mode
Review comment:
changed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rob05c commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
rob05c commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r452941642
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
+ "net/url"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+)
+
+const (
+ StatusDir = "/opt/ort/status"
+ AtsTcConfig = "/opt/ort/atstccfg"
+ Chkconfig = "/sbin/chkconfig"
+ Service = "/sbin/service"
+ SystemCtl = "/bin/systemctl"
+ TmpBase = "/tmp/ort"
+ TSHome = "/opt/trafficserver"
+ TrafficCtl = TSHome + "/bin/traffic_ctl"
+ TrafficServerOwner = "ats"
+)
+
+type Mode int
+
+const (
+ BADASS Mode = 0
+ INTERACTIVE = 1
+ REPORT = 2
+ REVALIDATE = 3
+ SYNCDS = 4
+)
+
+func (m Mode) String() string {
+ switch m {
+ case 0:
+ return "BADASS"
+ case 1:
+ return "INTERACTIVE"
+ case 2:
+ return "REPORT"
+ case 3:
+ return "REVALIDATE"
+ case 4:
+ return "SYNCDS"
+ }
+ return ""
+}
+
+type SvcManagement int
+
+const (
+ UNKNOWN = 0
+ SYSTEMD = 1
+ SYSTEMV = 2 // legacy System V Init.
+)
+
+func (s SvcManagement) String() string {
+ switch s {
+ case UNKNOWN:
+ return "UNKNOWN"
+ case SYSTEMD:
+ return "SYSTEMD"
+ case SYSTEMV:
+ return "SYSTEMV"
+ }
+ return "UNKNOWN"
+}
+
+type Cfg struct {
+ Dispersion time.Duration
+ LogLocationDebug string
+ LogLocationErr string
+ LogLocationInfo string
+ LogLocationWarn string
+ LoginDispersion time.Duration
+ CacheHostName string
+ SvcManagement SvcManagement
+ Retries int
+ RevalWaitTime time.Duration
+ ReverseProxyDisable bool
+ RunMode Mode
+ SkipOSCheck bool
+ TOTimeoutMS time.Duration
+ TOUser string
+ TOPass string
+ TOURL string
+ WaitForParents bool
+ YumOptions string
+}
+
+func (cfg Cfg) ErrorLog() log.LogLocation { return log.LogLocation(cfg.LogLocationErr) }
+func (cfg Cfg) WarningLog() log.LogLocation { return log.LogLocation(cfg.LogLocationWarn) }
+func (cfg Cfg) InfoLog() log.LogLocation { return log.LogLocation(cfg.LogLocationInfo) }
+func (cfg Cfg) DebugLog() log.LogLocation { return log.LogLocation(cfg.LogLocationDebug) }
+func (cfg Cfg) EventLog() log.LogLocation { return log.LogLocation(log.LogLocationNull) } // event logging is not used.
+
+func GetCfg() (Cfg, error) {
+ var err error
+
+ dispersionPtr := getopt.IntLong("dispersion", 'D', 300, "[seconds] wait a random number of seconds between 0 and [seconds] before starting, default 300")
+ loginDispersionPtr := getopt.IntLong("login-dispersion", 'l', 0, "[seconds] wait a random number of seconds betwee 0 and [seconds] befor login to traffic ops, default 0")
+ logLocationDebugPtr := getopt.StringLong("log-location-debug", 'd', "", "Where to log debugs. May be a file path, stdout, stderr, or null, default ''")
+ logLocationErrorPtr := getopt.StringLong("log-location-error", 'e', "stderr", "Where to log errors. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationInfoPtr := getopt.StringLong("log-location-info", 'i', "stderr", "Where to log info. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationWarnPtr := getopt.StringLong("log-location-warning", 'w', "stderr", "Where to log warnings. May be a file path, stdout, stderr, or null, default stderr")
+ cacheHostNamePtr := getopt.StringLong("cache-host-name", 'H', "", "Host name of the cache to generate config for. Must be the server host name in Traffic Ops, not a URL, and not the FQDN")
+ retriesPtr := getopt.IntLong("num-retries", 'r', 3, "[number] retry connection to Traffic Ops URL [number] times, default is 3")
+ revalWaitTimePtr := getopt.IntLong("reval-wait-time", 'T', 60, "[seconds] wait a random number of seconds between 0 and [seconds] before revlidation, default is 60")
+ reverseProxyDisablePtr := getopt.BoolLong("reverse-proxy-disable", 'p', "[false | true] bypass the reverse proxy even if one has been configured default is false")
+ runModePtr := getopt.StringLong("run-mode", 'm', "report", "[badass | interactive | report | revalidate | syncds] run mode, default is 'report'")
+ skipOSCheckPtr := getopt.BoolLong("skip-os-check", 's', "[false | true] skip os check, default is false")
+ toTimeoutMSPtr := getopt.IntLong("traffic-ops-timeout-milliseconds", 't', 30000, "Timeout in milli-seconds for Traffic Ops requests, default is 30000")
+ toURLPtr := getopt.StringLong("traffic-ops-url", 'u', "", "Traffic Ops URL. Must be the full URL, including the scheme. Required. May also be set with the environment variable TO_URL")
+ toUserPtr := getopt.StringLong("traffic-ops-user", 'U', "", "Traffic Ops username. Required. May also be set with the environment variable TO_USER")
+ toPassPtr := getopt.StringLong("traffic-ops-password", 'P', "", "Traffic Ops password. Required. May also be set with the environment variable TO_PASS")
+ waitForParentsPtr := getopt.BoolLong("wait-for-parents", 'W', "[true | false] do not update if parent_pending = 1 in the update json. default is false, wait for parents")
+ helpPtr := getopt.BoolLong("help", 'h', "Print usage information and exit")
+ getopt.Parse()
+
+ dispersion := time.Second * time.Duration(*dispersionPtr)
+ loginDispersion := time.Second * time.Duration(*loginDispersionPtr)
+ logLocationDebug := *logLocationDebugPtr
+ logLocationError := *logLocationErrorPtr
+ logLocationInfo := *logLocationInfoPtr
+ logLocationWarn := *logLocationWarnPtr
+
+ var cacheHostName string
+ if len(*cacheHostNamePtr) > 0 {
+ cacheHostName = *cacheHostNamePtr
+ } else {
+ cacheHostName, _ = os.Hostname()
Review comment:
This needs to check the error. I'd say if there's no override arg, and this returns an err, to just exit.
But it needs to at least log an error.
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
+ "net/url"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+)
+
+const (
+ StatusDir = "/opt/ort/status"
+ AtsTcConfig = "/opt/ort/atstccfg"
+ Chkconfig = "/sbin/chkconfig"
+ Service = "/sbin/service"
+ SystemCtl = "/bin/systemctl"
+ TmpBase = "/tmp/ort"
+ TSHome = "/opt/trafficserver"
+ TrafficCtl = TSHome + "/bin/traffic_ctl"
+ TrafficServerOwner = "ats"
+)
+
+type Mode int
+
+const (
+ BADASS Mode = 0
+ INTERACTIVE = 1
+ REPORT = 2
+ REVALIDATE = 3
+ SYNCDS = 4
+)
+
+func (m Mode) String() string {
+ switch m {
+ case 0:
+ return "BADASS"
+ case 1:
+ return "INTERACTIVE"
+ case 2:
+ return "REPORT"
+ case 3:
+ return "REVALIDATE"
+ case 4:
+ return "SYNCDS"
+ }
+ return ""
+}
+
+type SvcManagement int
+
+const (
+ UNKNOWN = 0
+ SYSTEMD = 1
+ SYSTEMV = 2 // legacy System V Init.
+)
+
+func (s SvcManagement) String() string {
+ switch s {
+ case UNKNOWN:
+ return "UNKNOWN"
+ case SYSTEMD:
+ return "SYSTEMD"
+ case SYSTEMV:
+ return "SYSTEMV"
+ }
+ return "UNKNOWN"
+}
+
+type Cfg struct {
+ Dispersion time.Duration
+ LogLocationDebug string
+ LogLocationErr string
+ LogLocationInfo string
+ LogLocationWarn string
+ LoginDispersion time.Duration
+ CacheHostName string
+ SvcManagement SvcManagement
+ Retries int
+ RevalWaitTime time.Duration
+ ReverseProxyDisable bool
+ RunMode Mode
+ SkipOSCheck bool
+ TOTimeoutMS time.Duration
+ TOUser string
+ TOPass string
+ TOURL string
+ WaitForParents bool
+ YumOptions string
+}
+
+func (cfg Cfg) ErrorLog() log.LogLocation { return log.LogLocation(cfg.LogLocationErr) }
+func (cfg Cfg) WarningLog() log.LogLocation { return log.LogLocation(cfg.LogLocationWarn) }
+func (cfg Cfg) InfoLog() log.LogLocation { return log.LogLocation(cfg.LogLocationInfo) }
+func (cfg Cfg) DebugLog() log.LogLocation { return log.LogLocation(cfg.LogLocationDebug) }
+func (cfg Cfg) EventLog() log.LogLocation { return log.LogLocation(log.LogLocationNull) } // event logging is not used.
+
+func GetCfg() (Cfg, error) {
+ var err error
+
+ dispersionPtr := getopt.IntLong("dispersion", 'D', 300, "[seconds] wait a random number of seconds between 0 and [seconds] before starting, default 300")
+ loginDispersionPtr := getopt.IntLong("login-dispersion", 'l', 0, "[seconds] wait a random number of seconds betwee 0 and [seconds] befor login to traffic ops, default 0")
+ logLocationDebugPtr := getopt.StringLong("log-location-debug", 'd', "", "Where to log debugs. May be a file path, stdout, stderr, or null, default ''")
+ logLocationErrorPtr := getopt.StringLong("log-location-error", 'e', "stderr", "Where to log errors. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationInfoPtr := getopt.StringLong("log-location-info", 'i', "stderr", "Where to log info. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationWarnPtr := getopt.StringLong("log-location-warning", 'w', "stderr", "Where to log warnings. May be a file path, stdout, stderr, or null, default stderr")
+ cacheHostNamePtr := getopt.StringLong("cache-host-name", 'H', "", "Host name of the cache to generate config for. Must be the server host name in Traffic Ops, not a URL, and not the FQDN")
+ retriesPtr := getopt.IntLong("num-retries", 'r', 3, "[number] retry connection to Traffic Ops URL [number] times, default is 3")
+ revalWaitTimePtr := getopt.IntLong("reval-wait-time", 'T', 60, "[seconds] wait a random number of seconds between 0 and [seconds] before revlidation, default is 60")
+ reverseProxyDisablePtr := getopt.BoolLong("reverse-proxy-disable", 'p', "[false | true] bypass the reverse proxy even if one has been configured default is false")
+ runModePtr := getopt.StringLong("run-mode", 'm', "report", "[badass | interactive | report | revalidate | syncds] run mode, default is 'report'")
+ skipOSCheckPtr := getopt.BoolLong("skip-os-check", 's', "[false | true] skip os check, default is false")
+ toTimeoutMSPtr := getopt.IntLong("traffic-ops-timeout-milliseconds", 't', 30000, "Timeout in milli-seconds for Traffic Ops requests, default is 30000")
+ toURLPtr := getopt.StringLong("traffic-ops-url", 'u', "", "Traffic Ops URL. Must be the full URL, including the scheme. Required. May also be set with the environment variable TO_URL")
+ toUserPtr := getopt.StringLong("traffic-ops-user", 'U', "", "Traffic Ops username. Required. May also be set with the environment variable TO_USER")
+ toPassPtr := getopt.StringLong("traffic-ops-password", 'P', "", "Traffic Ops password. Required. May also be set with the environment variable TO_PASS")
+ waitForParentsPtr := getopt.BoolLong("wait-for-parents", 'W', "[true | false] do not update if parent_pending = 1 in the update json. default is false, wait for parents")
+ helpPtr := getopt.BoolLong("help", 'h', "Print usage information and exit")
+ getopt.Parse()
+
+ dispersion := time.Second * time.Duration(*dispersionPtr)
+ loginDispersion := time.Second * time.Duration(*loginDispersionPtr)
+ logLocationDebug := *logLocationDebugPtr
+ logLocationError := *logLocationErrorPtr
+ logLocationInfo := *logLocationInfoPtr
+ logLocationWarn := *logLocationWarnPtr
+
+ var cacheHostName string
+ if len(*cacheHostNamePtr) > 0 {
+ cacheHostName = *cacheHostNamePtr
+ } else {
+ cacheHostName, _ = os.Hostname()
+ }
+
+ retries := *retriesPtr
+ revalWaitTime := time.Second * time.Duration(*revalWaitTimePtr)
+ reverseProxyDisable := *reverseProxyDisablePtr
+ runMode := *runModePtr
+ skipOsCheck := *skipOSCheckPtr
+ toTimeoutMS := time.Millisecond * time.Duration(*toTimeoutMSPtr)
+ toURL := *toURLPtr
+ toUser := *toUserPtr
+ toPass := *toPassPtr
+ waitForParents := *waitForParentsPtr
+ help := *helpPtr
+
+ if help {
+ Usage()
+ os.Exit(0)
+ }
+
+ runModeStr := strings.ToUpper(runMode)
+ var _runMode Mode
Review comment:
Go local variables should be `camelCase`, shouldn't start with an underscore.
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
+ "net/url"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+)
+
+const (
+ StatusDir = "/opt/ort/status"
+ AtsTcConfig = "/opt/ort/atstccfg"
+ Chkconfig = "/sbin/chkconfig"
+ Service = "/sbin/service"
+ SystemCtl = "/bin/systemctl"
+ TmpBase = "/tmp/ort"
+ TSHome = "/opt/trafficserver"
+ TrafficCtl = TSHome + "/bin/traffic_ctl"
+ TrafficServerOwner = "ats"
+)
+
+type Mode int
+
+const (
+ BADASS Mode = 0
+ INTERACTIVE = 1
+ REPORT = 2
+ REVALIDATE = 3
+ SYNCDS = 4
+)
+
+func (m Mode) String() string {
+ switch m {
+ case 0:
+ return "BADASS"
+ case 1:
+ return "INTERACTIVE"
+ case 2:
+ return "REPORT"
+ case 3:
+ return "REVALIDATE"
+ case 4:
+ return "SYNCDS"
+ }
+ return ""
+}
+
+type SvcManagement int
+
+const (
+ UNKNOWN = 0
+ SYSTEMD = 1
+ SYSTEMV = 2 // legacy System V Init.
+)
+
+func (s SvcManagement) String() string {
+ switch s {
+ case UNKNOWN:
+ return "UNKNOWN"
+ case SYSTEMD:
+ return "SYSTEMD"
+ case SYSTEMV:
+ return "SYSTEMV"
+ }
+ return "UNKNOWN"
+}
+
+type Cfg struct {
+ Dispersion time.Duration
+ LogLocationDebug string
+ LogLocationErr string
+ LogLocationInfo string
+ LogLocationWarn string
+ LoginDispersion time.Duration
+ CacheHostName string
+ SvcManagement SvcManagement
+ Retries int
+ RevalWaitTime time.Duration
+ ReverseProxyDisable bool
+ RunMode Mode
+ SkipOSCheck bool
+ TOTimeoutMS time.Duration
+ TOUser string
+ TOPass string
+ TOURL string
+ WaitForParents bool
+ YumOptions string
+}
+
+func (cfg Cfg) ErrorLog() log.LogLocation { return log.LogLocation(cfg.LogLocationErr) }
+func (cfg Cfg) WarningLog() log.LogLocation { return log.LogLocation(cfg.LogLocationWarn) }
+func (cfg Cfg) InfoLog() log.LogLocation { return log.LogLocation(cfg.LogLocationInfo) }
+func (cfg Cfg) DebugLog() log.LogLocation { return log.LogLocation(cfg.LogLocationDebug) }
+func (cfg Cfg) EventLog() log.LogLocation { return log.LogLocation(log.LogLocationNull) } // event logging is not used.
+
+func GetCfg() (Cfg, error) {
+ var err error
+
+ dispersionPtr := getopt.IntLong("dispersion", 'D', 300, "[seconds] wait a random number of seconds between 0 and [seconds] before starting, default 300")
+ loginDispersionPtr := getopt.IntLong("login-dispersion", 'l', 0, "[seconds] wait a random number of seconds betwee 0 and [seconds] befor login to traffic ops, default 0")
+ logLocationDebugPtr := getopt.StringLong("log-location-debug", 'd', "", "Where to log debugs. May be a file path, stdout, stderr, or null, default ''")
+ logLocationErrorPtr := getopt.StringLong("log-location-error", 'e', "stderr", "Where to log errors. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationInfoPtr := getopt.StringLong("log-location-info", 'i', "stderr", "Where to log info. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationWarnPtr := getopt.StringLong("log-location-warning", 'w', "stderr", "Where to log warnings. May be a file path, stdout, stderr, or null, default stderr")
+ cacheHostNamePtr := getopt.StringLong("cache-host-name", 'H', "", "Host name of the cache to generate config for. Must be the server host name in Traffic Ops, not a URL, and not the FQDN")
+ retriesPtr := getopt.IntLong("num-retries", 'r', 3, "[number] retry connection to Traffic Ops URL [number] times, default is 3")
+ revalWaitTimePtr := getopt.IntLong("reval-wait-time", 'T', 60, "[seconds] wait a random number of seconds between 0 and [seconds] before revlidation, default is 60")
+ reverseProxyDisablePtr := getopt.BoolLong("reverse-proxy-disable", 'p', "[false | true] bypass the reverse proxy even if one has been configured default is false")
+ runModePtr := getopt.StringLong("run-mode", 'm', "report", "[badass | interactive | report | revalidate | syncds] run mode, default is 'report'")
+ skipOSCheckPtr := getopt.BoolLong("skip-os-check", 's', "[false | true] skip os check, default is false")
+ toTimeoutMSPtr := getopt.IntLong("traffic-ops-timeout-milliseconds", 't', 30000, "Timeout in milli-seconds for Traffic Ops requests, default is 30000")
+ toURLPtr := getopt.StringLong("traffic-ops-url", 'u', "", "Traffic Ops URL. Must be the full URL, including the scheme. Required. May also be set with the environment variable TO_URL")
+ toUserPtr := getopt.StringLong("traffic-ops-user", 'U', "", "Traffic Ops username. Required. May also be set with the environment variable TO_USER")
+ toPassPtr := getopt.StringLong("traffic-ops-password", 'P', "", "Traffic Ops password. Required. May also be set with the environment variable TO_PASS")
+ waitForParentsPtr := getopt.BoolLong("wait-for-parents", 'W', "[true | false] do not update if parent_pending = 1 in the update json. default is false, wait for parents")
+ helpPtr := getopt.BoolLong("help", 'h', "Print usage information and exit")
+ getopt.Parse()
+
+ dispersion := time.Second * time.Duration(*dispersionPtr)
+ loginDispersion := time.Second * time.Duration(*loginDispersionPtr)
+ logLocationDebug := *logLocationDebugPtr
+ logLocationError := *logLocationErrorPtr
+ logLocationInfo := *logLocationInfoPtr
+ logLocationWarn := *logLocationWarnPtr
+
+ var cacheHostName string
+ if len(*cacheHostNamePtr) > 0 {
+ cacheHostName = *cacheHostNamePtr
+ } else {
+ cacheHostName, _ = os.Hostname()
+ }
+
+ retries := *retriesPtr
+ revalWaitTime := time.Second * time.Duration(*revalWaitTimePtr)
+ reverseProxyDisable := *reverseProxyDisablePtr
+ runMode := *runModePtr
+ skipOsCheck := *skipOSCheckPtr
+ toTimeoutMS := time.Millisecond * time.Duration(*toTimeoutMSPtr)
+ toURL := *toURLPtr
+ toUser := *toUserPtr
+ toPass := *toPassPtr
+ waitForParents := *waitForParentsPtr
+ help := *helpPtr
+
+ if help {
+ Usage()
+ os.Exit(0)
+ }
+
+ runModeStr := strings.ToUpper(runMode)
+ var _runMode Mode
Review comment:
Personally, I prefer to always use the := syntax, runMode := Mode(0) because
(1) It lets you use a consistent declaration syntax, instead of having 2 different declaration syntaxes. It's always better to have One Right Way to do things.
(2) It makes the initialization explicit. It isn't always obvious what the zero-value is with var, but with := it's immediately visible.
But not everyone feels that way, so I won't push for := and avoiding var if you don't agree.
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
+ "net/url"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+)
+
+const (
+ StatusDir = "/opt/ort/status"
+ AtsTcConfig = "/opt/ort/atstccfg"
+ Chkconfig = "/sbin/chkconfig"
+ Service = "/sbin/service"
+ SystemCtl = "/bin/systemctl"
+ TmpBase = "/tmp/ort"
+ TSHome = "/opt/trafficserver"
+ TrafficCtl = TSHome + "/bin/traffic_ctl"
+ TrafficServerOwner = "ats"
+)
+
+type Mode int
+
+const (
+ BADASS Mode = 0
+ INTERACTIVE = 1
+ REPORT = 2
+ REVALIDATE = 3
+ SYNCDS = 4
+)
+
+func (m Mode) String() string {
+ switch m {
+ case 0:
+ return "BADASS"
+ case 1:
+ return "INTERACTIVE"
+ case 2:
+ return "REPORT"
+ case 3:
+ return "REVALIDATE"
+ case 4:
+ return "SYNCDS"
+ }
+ return ""
+}
+
+type SvcManagement int
+
+const (
+ UNKNOWN = 0
+ SYSTEMD = 1
+ SYSTEMV = 2 // legacy System V Init.
+)
+
+func (s SvcManagement) String() string {
+ switch s {
+ case UNKNOWN:
+ return "UNKNOWN"
+ case SYSTEMD:
+ return "SYSTEMD"
+ case SYSTEMV:
+ return "SYSTEMV"
+ }
+ return "UNKNOWN"
+}
+
+type Cfg struct {
+ Dispersion time.Duration
+ LogLocationDebug string
+ LogLocationErr string
+ LogLocationInfo string
+ LogLocationWarn string
+ LoginDispersion time.Duration
+ CacheHostName string
+ SvcManagement SvcManagement
+ Retries int
+ RevalWaitTime time.Duration
+ ReverseProxyDisable bool
+ RunMode Mode
+ SkipOSCheck bool
+ TOTimeoutMS time.Duration
+ TOUser string
+ TOPass string
+ TOURL string
+ WaitForParents bool
+ YumOptions string
+}
+
+func (cfg Cfg) ErrorLog() log.LogLocation { return log.LogLocation(cfg.LogLocationErr) }
+func (cfg Cfg) WarningLog() log.LogLocation { return log.LogLocation(cfg.LogLocationWarn) }
+func (cfg Cfg) InfoLog() log.LogLocation { return log.LogLocation(cfg.LogLocationInfo) }
+func (cfg Cfg) DebugLog() log.LogLocation { return log.LogLocation(cfg.LogLocationDebug) }
+func (cfg Cfg) EventLog() log.LogLocation { return log.LogLocation(log.LogLocationNull) } // event logging is not used.
+
+func GetCfg() (Cfg, error) {
+ var err error
+
+ dispersionPtr := getopt.IntLong("dispersion", 'D', 300, "[seconds] wait a random number of seconds between 0 and [seconds] before starting, default 300")
+ loginDispersionPtr := getopt.IntLong("login-dispersion", 'l', 0, "[seconds] wait a random number of seconds betwee 0 and [seconds] befor login to traffic ops, default 0")
+ logLocationDebugPtr := getopt.StringLong("log-location-debug", 'd', "", "Where to log debugs. May be a file path, stdout, stderr, or null, default ''")
+ logLocationErrorPtr := getopt.StringLong("log-location-error", 'e', "stderr", "Where to log errors. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationInfoPtr := getopt.StringLong("log-location-info", 'i', "stderr", "Where to log info. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationWarnPtr := getopt.StringLong("log-location-warning", 'w', "stderr", "Where to log warnings. May be a file path, stdout, stderr, or null, default stderr")
+ cacheHostNamePtr := getopt.StringLong("cache-host-name", 'H', "", "Host name of the cache to generate config for. Must be the server host name in Traffic Ops, not a URL, and not the FQDN")
+ retriesPtr := getopt.IntLong("num-retries", 'r', 3, "[number] retry connection to Traffic Ops URL [number] times, default is 3")
+ revalWaitTimePtr := getopt.IntLong("reval-wait-time", 'T', 60, "[seconds] wait a random number of seconds between 0 and [seconds] before revlidation, default is 60")
+ reverseProxyDisablePtr := getopt.BoolLong("reverse-proxy-disable", 'p', "[false | true] bypass the reverse proxy even if one has been configured default is false")
+ runModePtr := getopt.StringLong("run-mode", 'm', "report", "[badass | interactive | report | revalidate | syncds] run mode, default is 'report'")
+ skipOSCheckPtr := getopt.BoolLong("skip-os-check", 's', "[false | true] skip os check, default is false")
+ toTimeoutMSPtr := getopt.IntLong("traffic-ops-timeout-milliseconds", 't', 30000, "Timeout in milli-seconds for Traffic Ops requests, default is 30000")
+ toURLPtr := getopt.StringLong("traffic-ops-url", 'u', "", "Traffic Ops URL. Must be the full URL, including the scheme. Required. May also be set with the environment variable TO_URL")
+ toUserPtr := getopt.StringLong("traffic-ops-user", 'U', "", "Traffic Ops username. Required. May also be set with the environment variable TO_USER")
+ toPassPtr := getopt.StringLong("traffic-ops-password", 'P', "", "Traffic Ops password. Required. May also be set with the environment variable TO_PASS")
+ waitForParentsPtr := getopt.BoolLong("wait-for-parents", 'W', "[true | false] do not update if parent_pending = 1 in the update json. default is false, wait for parents")
+ helpPtr := getopt.BoolLong("help", 'h', "Print usage information and exit")
+ getopt.Parse()
+
+ dispersion := time.Second * time.Duration(*dispersionPtr)
+ loginDispersion := time.Second * time.Duration(*loginDispersionPtr)
+ logLocationDebug := *logLocationDebugPtr
+ logLocationError := *logLocationErrorPtr
+ logLocationInfo := *logLocationInfoPtr
+ logLocationWarn := *logLocationWarnPtr
+
+ var cacheHostName string
+ if len(*cacheHostNamePtr) > 0 {
+ cacheHostName = *cacheHostNamePtr
+ } else {
+ cacheHostName, _ = os.Hostname()
+ }
+
+ retries := *retriesPtr
+ revalWaitTime := time.Second * time.Duration(*revalWaitTimePtr)
+ reverseProxyDisable := *reverseProxyDisablePtr
+ runMode := *runModePtr
+ skipOsCheck := *skipOSCheckPtr
+ toTimeoutMS := time.Millisecond * time.Duration(*toTimeoutMSPtr)
+ toURL := *toURLPtr
+ toUser := *toUserPtr
+ toPass := *toPassPtr
+ waitForParents := *waitForParentsPtr
+ help := *helpPtr
+
+ if help {
+ Usage()
+ os.Exit(0)
+ }
+
+ runModeStr := strings.ToUpper(runMode)
+ var _runMode Mode
+ switch runModeStr {
+ case "INTERACTIVE":
+ _runMode = INTERACTIVE
+ case "REPORT":
+ _runMode = REPORT
+ case "BADASS":
+ _runMode = BADASS
+ case "SYNCDS":
+ _runMode = SYNCDS
+ case "REVALIDATE":
+ _runMode = REVALIDATE
+ default:
+ fmt.Fprintf(os.Stderr, "%s is an invalid mode.\n", runModeStr)
Review comment:
Is there a reason this prints and exits, but other errors later return? Should this likewise return and let the caller exit?
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
+ "net/url"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+)
+
+const (
+ StatusDir = "/opt/ort/status"
+ AtsTcConfig = "/opt/ort/atstccfg"
+ Chkconfig = "/sbin/chkconfig"
+ Service = "/sbin/service"
+ SystemCtl = "/bin/systemctl"
+ TmpBase = "/tmp/ort"
+ TSHome = "/opt/trafficserver"
+ TrafficCtl = TSHome + "/bin/traffic_ctl"
+ TrafficServerOwner = "ats"
+)
+
+type Mode int
+
+const (
+ BADASS Mode = 0
+ INTERACTIVE = 1
+ REPORT = 2
+ REVALIDATE = 3
+ SYNCDS = 4
+)
+
+func (m Mode) String() string {
+ switch m {
+ case 0:
+ return "BADASS"
+ case 1:
+ return "INTERACTIVE"
+ case 2:
+ return "REPORT"
+ case 3:
+ return "REVALIDATE"
+ case 4:
+ return "SYNCDS"
+ }
+ return ""
+}
+
+type SvcManagement int
+
+const (
+ UNKNOWN = 0
+ SYSTEMD = 1
+ SYSTEMV = 2 // legacy System V Init.
+)
+
+func (s SvcManagement) String() string {
+ switch s {
+ case UNKNOWN:
+ return "UNKNOWN"
+ case SYSTEMD:
+ return "SYSTEMD"
+ case SYSTEMV:
+ return "SYSTEMV"
+ }
+ return "UNKNOWN"
+}
+
+type Cfg struct {
+ Dispersion time.Duration
+ LogLocationDebug string
+ LogLocationErr string
+ LogLocationInfo string
+ LogLocationWarn string
+ LoginDispersion time.Duration
+ CacheHostName string
+ SvcManagement SvcManagement
+ Retries int
+ RevalWaitTime time.Duration
+ ReverseProxyDisable bool
+ RunMode Mode
+ SkipOSCheck bool
+ TOTimeoutMS time.Duration
+ TOUser string
+ TOPass string
+ TOURL string
+ WaitForParents bool
+ YumOptions string
+}
+
+func (cfg Cfg) ErrorLog() log.LogLocation { return log.LogLocation(cfg.LogLocationErr) }
+func (cfg Cfg) WarningLog() log.LogLocation { return log.LogLocation(cfg.LogLocationWarn) }
+func (cfg Cfg) InfoLog() log.LogLocation { return log.LogLocation(cfg.LogLocationInfo) }
+func (cfg Cfg) DebugLog() log.LogLocation { return log.LogLocation(cfg.LogLocationDebug) }
+func (cfg Cfg) EventLog() log.LogLocation { return log.LogLocation(log.LogLocationNull) } // event logging is not used.
+
+func GetCfg() (Cfg, error) {
+ var err error
+
+ dispersionPtr := getopt.IntLong("dispersion", 'D', 300, "[seconds] wait a random number of seconds between 0 and [seconds] before starting, default 300")
+ loginDispersionPtr := getopt.IntLong("login-dispersion", 'l', 0, "[seconds] wait a random number of seconds betwee 0 and [seconds] befor login to traffic ops, default 0")
+ logLocationDebugPtr := getopt.StringLong("log-location-debug", 'd', "", "Where to log debugs. May be a file path, stdout, stderr, or null, default ''")
+ logLocationErrorPtr := getopt.StringLong("log-location-error", 'e', "stderr", "Where to log errors. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationInfoPtr := getopt.StringLong("log-location-info", 'i', "stderr", "Where to log info. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationWarnPtr := getopt.StringLong("log-location-warning", 'w', "stderr", "Where to log warnings. May be a file path, stdout, stderr, or null, default stderr")
+ cacheHostNamePtr := getopt.StringLong("cache-host-name", 'H', "", "Host name of the cache to generate config for. Must be the server host name in Traffic Ops, not a URL, and not the FQDN")
+ retriesPtr := getopt.IntLong("num-retries", 'r', 3, "[number] retry connection to Traffic Ops URL [number] times, default is 3")
+ revalWaitTimePtr := getopt.IntLong("reval-wait-time", 'T', 60, "[seconds] wait a random number of seconds between 0 and [seconds] before revlidation, default is 60")
+ reverseProxyDisablePtr := getopt.BoolLong("reverse-proxy-disable", 'p', "[false | true] bypass the reverse proxy even if one has been configured default is false")
+ runModePtr := getopt.StringLong("run-mode", 'm', "report", "[badass | interactive | report | revalidate | syncds] run mode, default is 'report'")
+ skipOSCheckPtr := getopt.BoolLong("skip-os-check", 's', "[false | true] skip os check, default is false")
+ toTimeoutMSPtr := getopt.IntLong("traffic-ops-timeout-milliseconds", 't', 30000, "Timeout in milli-seconds for Traffic Ops requests, default is 30000")
+ toURLPtr := getopt.StringLong("traffic-ops-url", 'u', "", "Traffic Ops URL. Must be the full URL, including the scheme. Required. May also be set with the environment variable TO_URL")
+ toUserPtr := getopt.StringLong("traffic-ops-user", 'U', "", "Traffic Ops username. Required. May also be set with the environment variable TO_USER")
+ toPassPtr := getopt.StringLong("traffic-ops-password", 'P', "", "Traffic Ops password. Required. May also be set with the environment variable TO_PASS")
+ waitForParentsPtr := getopt.BoolLong("wait-for-parents", 'W', "[true | false] do not update if parent_pending = 1 in the update json. default is false, wait for parents")
+ helpPtr := getopt.BoolLong("help", 'h', "Print usage information and exit")
+ getopt.Parse()
+
+ dispersion := time.Second * time.Duration(*dispersionPtr)
+ loginDispersion := time.Second * time.Duration(*loginDispersionPtr)
+ logLocationDebug := *logLocationDebugPtr
+ logLocationError := *logLocationErrorPtr
+ logLocationInfo := *logLocationInfoPtr
+ logLocationWarn := *logLocationWarnPtr
+
+ var cacheHostName string
+ if len(*cacheHostNamePtr) > 0 {
+ cacheHostName = *cacheHostNamePtr
+ } else {
+ cacheHostName, _ = os.Hostname()
+ }
+
+ retries := *retriesPtr
+ revalWaitTime := time.Second * time.Duration(*revalWaitTimePtr)
+ reverseProxyDisable := *reverseProxyDisablePtr
+ runMode := *runModePtr
+ skipOsCheck := *skipOSCheckPtr
+ toTimeoutMS := time.Millisecond * time.Duration(*toTimeoutMSPtr)
+ toURL := *toURLPtr
+ toUser := *toUserPtr
+ toPass := *toPassPtr
+ waitForParents := *waitForParentsPtr
+ help := *helpPtr
+
+ if help {
+ Usage()
+ os.Exit(0)
+ }
+
+ runModeStr := strings.ToUpper(runMode)
+ var _runMode Mode
+ switch runModeStr {
+ case "INTERACTIVE":
+ _runMode = INTERACTIVE
+ case "REPORT":
+ _runMode = REPORT
+ case "BADASS":
+ _runMode = BADASS
+ case "SYNCDS":
+ _runMode = SYNCDS
+ case "REVALIDATE":
+ _runMode = REVALIDATE
+ default:
+ fmt.Fprintf(os.Stderr, "%s is an invalid mode.\n", runModeStr)
+ Usage()
+ os.Exit(-1)
Review comment:
It's easier to debug and automate scripts with unique error codes. Would you object to putting this in a const, and making it something unique instead of `-1`? I'd suggest starting at `132` since lower codes are used by Linux and Bash.
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
Review comment:
Nitpick: path/filepath.Join
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
Review comment:
Nitpick: `else` could be removed, reduce indentation and make it easier to read
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
Review comment:
Nitpick: `sl := []string{}` is terser
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
Review comment:
Nitpick: since the above returns, this `else` isn't necessary, can reduce indentation to make it easier to read
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
Review comment:
Go local vars shouldn't be camelCase, not start with `_`. I'd also vote more than 1 letter, for variables with a scope this long.
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
Review comment:
This needs to be vendored. Traffic Control vendors all libraries, to prevent things from breaking if they change on the internet, and to make building without the internet possible.
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
+ if rc == 1 { // file is not part of any package.
+ return nil, nil
+ } else if rc == 0 { // add the package name the file belongs to.
+ log.Debugf("output from file-query: %s\n", string(output))
+ result = append(result, string(strings.TrimSpace(string(output))))
+ log.Debugf("result length: %d, result: %s\n", len(result), string(output))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-provides": // returns the package name that provides 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatprovides", name)
+ log.Debugf("pkg-provides - name: %s, output: %s\n", name, output)
+ if rc == 1 { // no package provides 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-query": // returns the package name for 'name'.
+ output, rc, err := ExecCommand("/bin/rpm", "-q", name)
+ if rc == 1 { // the package is not installed.
+ return nil, nil
+ } else if rc == 0 { // add the rpm name
+ result = append(result, string(strings.TrimSpace(string(output))))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-requires": // returns a list of packages that requires package 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatrequires", name)
+ if rc == 1 { // no package reuires package 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ }
+ return result, nil
+}
+
+func RandomDuration(max time.Duration) time.Duration {
+ rand.Seed(time.Now().UnixNano())
+ return time.Duration(rand.Int63n(int64(max)))
+}
+
+func CheckUser(cfg config.Cfg) bool {
+ result := true
+ user, _ := user.Current()
+
+ switch cfg.RunMode {
+ case config.BADASS:
+ fallthrough
+ case config.INTERACTIVE:
+ fallthrough
+ case config.SYNCDS:
+ if user.Username != "root" {
+ log.Errorf("Only the root user may run in BADASS, INTERACTIVE, or SYNCDS mode, current user: %s\n",
+ user.Username)
+ result = false
+ }
+ default:
+ log.Infof("current mode: %s, run user: %s\n", cfg.RunMode, user.Username)
+ }
+ return result
+}
+
+func CleanTmpDir() bool {
+ now := time.Now().Unix()
+
+ files, err := ioutil.ReadDir(config.TmpBase)
+ if err != nil {
+ log.Errorf("opening %s: %v\n", config.TmpBase, err)
+ return false
+ }
+ for _, f := range files {
+ // remove any directory and its contents under the config.TmpBase (/tmp/ort) that
+ // is more than a week old.
+ if f.IsDir() && (now-f.ModTime().Unix()) > 604800 {
+ dir := config.TmpBase + "/" + f.Name()
+ if f.Name() == "." || f.Name() == ".." {
+ continue
+ }
+ err = os.RemoveAll(dir)
Review comment:
Would you mind adding a safety check that `dir != "/"` here? I'm worried if `config.TmpBase` and `f.Name()` were both `/`, this could end up doing an `rm /`.
Maybe I'm paranoid, but that'd be super-bad, I'd feel better if we at least checked for it
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
+ "net/url"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+)
+
+const (
+ StatusDir = "/opt/ort/status"
+ AtsTcConfig = "/opt/ort/atstccfg"
+ Chkconfig = "/sbin/chkconfig"
+ Service = "/sbin/service"
+ SystemCtl = "/bin/systemctl"
+ TmpBase = "/tmp/ort"
+ TSHome = "/opt/trafficserver"
+ TrafficCtl = TSHome + "/bin/traffic_ctl"
+ TrafficServerOwner = "ats"
+)
+
+type Mode int
+
+const (
+ BADASS Mode = 0
Review comment:
Go consts should be CamelCase, not ALL_CAPS
https://github.com/golang/go/wiki/CodeReviewComments#mixed-caps
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
Review comment:
This should check if `status.Name == nil`. It'll panic if it gets a bad object from TO.
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
+ "net/url"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+)
+
+const (
+ StatusDir = "/opt/ort/status"
+ AtsTcConfig = "/opt/ort/atstccfg"
+ Chkconfig = "/sbin/chkconfig"
+ Service = "/sbin/service"
+ SystemCtl = "/bin/systemctl"
+ TmpBase = "/tmp/ort"
+ TSHome = "/opt/trafficserver"
+ TrafficCtl = TSHome + "/bin/traffic_ctl"
+ TrafficServerOwner = "ats"
+)
+
+type Mode int
+
+const (
+ BADASS Mode = 0
+ INTERACTIVE = 1
+ REPORT = 2
+ REVALIDATE = 3
+ SYNCDS = 4
+)
+
+func (m Mode) String() string {
+ switch m {
+ case 0:
+ return "BADASS"
+ case 1:
+ return "INTERACTIVE"
+ case 2:
+ return "REPORT"
+ case 3:
+ return "REVALIDATE"
+ case 4:
+ return "SYNCDS"
+ }
+ return ""
+}
+
+type SvcManagement int
+
+const (
+ UNKNOWN = 0
+ SYSTEMD = 1
+ SYSTEMV = 2 // legacy System V Init.
+)
+
+func (s SvcManagement) String() string {
+ switch s {
+ case UNKNOWN:
+ return "UNKNOWN"
+ case SYSTEMD:
+ return "SYSTEMD"
+ case SYSTEMV:
+ return "SYSTEMV"
+ }
+ return "UNKNOWN"
+}
+
+type Cfg struct {
+ Dispersion time.Duration
+ LogLocationDebug string
+ LogLocationErr string
+ LogLocationInfo string
+ LogLocationWarn string
+ LoginDispersion time.Duration
+ CacheHostName string
+ SvcManagement SvcManagement
+ Retries int
+ RevalWaitTime time.Duration
+ ReverseProxyDisable bool
+ RunMode Mode
+ SkipOSCheck bool
+ TOTimeoutMS time.Duration
+ TOUser string
+ TOPass string
+ TOURL string
+ WaitForParents bool
+ YumOptions string
+}
+
+func (cfg Cfg) ErrorLog() log.LogLocation { return log.LogLocation(cfg.LogLocationErr) }
+func (cfg Cfg) WarningLog() log.LogLocation { return log.LogLocation(cfg.LogLocationWarn) }
+func (cfg Cfg) InfoLog() log.LogLocation { return log.LogLocation(cfg.LogLocationInfo) }
+func (cfg Cfg) DebugLog() log.LogLocation { return log.LogLocation(cfg.LogLocationDebug) }
+func (cfg Cfg) EventLog() log.LogLocation { return log.LogLocation(log.LogLocationNull) } // event logging is not used.
+
+func GetCfg() (Cfg, error) {
+ var err error
+
+ dispersionPtr := getopt.IntLong("dispersion", 'D', 300, "[seconds] wait a random number of seconds between 0 and [seconds] before starting, default 300")
+ loginDispersionPtr := getopt.IntLong("login-dispersion", 'l', 0, "[seconds] wait a random number of seconds betwee 0 and [seconds] befor login to traffic ops, default 0")
+ logLocationDebugPtr := getopt.StringLong("log-location-debug", 'd', "", "Where to log debugs. May be a file path, stdout, stderr, or null, default ''")
+ logLocationErrorPtr := getopt.StringLong("log-location-error", 'e', "stderr", "Where to log errors. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationInfoPtr := getopt.StringLong("log-location-info", 'i', "stderr", "Where to log info. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationWarnPtr := getopt.StringLong("log-location-warning", 'w', "stderr", "Where to log warnings. May be a file path, stdout, stderr, or null, default stderr")
+ cacheHostNamePtr := getopt.StringLong("cache-host-name", 'H', "", "Host name of the cache to generate config for. Must be the server host name in Traffic Ops, not a URL, and not the FQDN")
+ retriesPtr := getopt.IntLong("num-retries", 'r', 3, "[number] retry connection to Traffic Ops URL [number] times, default is 3")
+ revalWaitTimePtr := getopt.IntLong("reval-wait-time", 'T', 60, "[seconds] wait a random number of seconds between 0 and [seconds] before revlidation, default is 60")
+ reverseProxyDisablePtr := getopt.BoolLong("reverse-proxy-disable", 'p', "[false | true] bypass the reverse proxy even if one has been configured default is false")
+ runModePtr := getopt.StringLong("run-mode", 'm', "report", "[badass | interactive | report | revalidate | syncds] run mode, default is 'report'")
+ skipOSCheckPtr := getopt.BoolLong("skip-os-check", 's', "[false | true] skip os check, default is false")
+ toTimeoutMSPtr := getopt.IntLong("traffic-ops-timeout-milliseconds", 't', 30000, "Timeout in milli-seconds for Traffic Ops requests, default is 30000")
+ toURLPtr := getopt.StringLong("traffic-ops-url", 'u', "", "Traffic Ops URL. Must be the full URL, including the scheme. Required. May also be set with the environment variable TO_URL")
+ toUserPtr := getopt.StringLong("traffic-ops-user", 'U', "", "Traffic Ops username. Required. May also be set with the environment variable TO_USER")
+ toPassPtr := getopt.StringLong("traffic-ops-password", 'P', "", "Traffic Ops password. Required. May also be set with the environment variable TO_PASS")
+ waitForParentsPtr := getopt.BoolLong("wait-for-parents", 'W', "[true | false] do not update if parent_pending = 1 in the update json. default is false, wait for parents")
+ helpPtr := getopt.BoolLong("help", 'h', "Print usage information and exit")
+ getopt.Parse()
+
+ dispersion := time.Second * time.Duration(*dispersionPtr)
+ loginDispersion := time.Second * time.Duration(*loginDispersionPtr)
+ logLocationDebug := *logLocationDebugPtr
+ logLocationError := *logLocationErrorPtr
+ logLocationInfo := *logLocationInfoPtr
+ logLocationWarn := *logLocationWarnPtr
+
+ var cacheHostName string
+ if len(*cacheHostNamePtr) > 0 {
+ cacheHostName = *cacheHostNamePtr
+ } else {
+ cacheHostName, _ = os.Hostname()
+ }
+
+ retries := *retriesPtr
+ revalWaitTime := time.Second * time.Duration(*revalWaitTimePtr)
+ reverseProxyDisable := *reverseProxyDisablePtr
+ runMode := *runModePtr
+ skipOsCheck := *skipOSCheckPtr
+ toTimeoutMS := time.Millisecond * time.Duration(*toTimeoutMSPtr)
+ toURL := *toURLPtr
+ toUser := *toUserPtr
+ toPass := *toPassPtr
+ waitForParents := *waitForParentsPtr
+ help := *helpPtr
+
+ if help {
+ Usage()
+ os.Exit(0)
+ }
+
+ runModeStr := strings.ToUpper(runMode)
+ var _runMode Mode
+ switch runModeStr {
+ case "INTERACTIVE":
+ _runMode = INTERACTIVE
+ case "REPORT":
+ _runMode = REPORT
+ case "BADASS":
+ _runMode = BADASS
+ case "SYNCDS":
+ _runMode = SYNCDS
+ case "REVALIDATE":
+ _runMode = REVALIDATE
+ default:
+ fmt.Fprintf(os.Stderr, "%s is an invalid mode.\n", runModeStr)
+ Usage()
+ os.Exit(-1)
+ }
+
+ urlSourceStr := "argument" // for error messages
+ if toURL == "" {
+ urlSourceStr = "environment variable"
+ toURL = os.Getenv("TO_URL")
+ }
+ if toUser == "" {
+ toUser = os.Getenv("TO_USER")
+ }
+ if toPass == "" {
+ toPass = os.Getenv("TO_PASS")
+ }
+
+ usageStr := "basic usage: traffic_ops_t3c --traffic-ops-url=myurl --traffic-ops-user=myuser --traffic-ops-password=mypass --cache-host-name=my-cache"
+ if strings.TrimSpace(toURL) == "" {
+ return Cfg{}, errors.New("Missing required argument --traffic-ops-url or TO_URL environment variable. " + usageStr)
+ }
+ if strings.TrimSpace(toUser) == "" {
+ return Cfg{}, errors.New("Missing required argument --traffic-ops-user or TO_USER environment variable. " + usageStr)
+ }
+ if strings.TrimSpace(toPass) == "" {
+ return Cfg{}, errors.New("Missing required argument --traffic-ops-password or TO_PASS environment variable. " + usageStr)
+ }
+ if strings.TrimSpace(cacheHostName) == "" {
+ return Cfg{}, errors.New("Missing required argument --cache-host-name. " + usageStr)
+ }
+
+ toURLParsed, err := url.Parse(toURL)
+ if err != nil {
+ return Cfg{}, errors.New("parsing Traffic Ops URL from " + urlSourceStr + " '" + toURL + "': " + err.Error())
+ } else if err = validateURL(toURLParsed); err != nil {
+ return Cfg{}, errors.New("invalid Traffic Ops URL from " + urlSourceStr + " '" + toURL + "': " + err.Error())
+ }
+
+ svcManagement := getOSSvcManagement()
+ yumOptions := os.Getenv("YUM_OPTIONS")
+
+ cfg := Cfg{
+ Dispersion: dispersion,
+ LogLocationDebug: logLocationDebug,
+ LogLocationErr: logLocationError,
+ LogLocationInfo: logLocationInfo,
+ LogLocationWarn: logLocationWarn,
+ LoginDispersion: time.Second * time.Duration(loginDispersion),
+ CacheHostName: cacheHostName,
+ SvcManagement: svcManagement,
+ Retries: retries,
+ RevalWaitTime: revalWaitTime,
+ ReverseProxyDisable: reverseProxyDisable,
+ RunMode: _runMode,
+ SkipOSCheck: skipOsCheck,
+ TOTimeoutMS: toTimeoutMS,
+ TOUser: toUser,
+ TOPass: toPass,
+ TOURL: toURL,
+ WaitForParents: waitForParents,
+ YumOptions: yumOptions,
+ }
+
+ if err = log.InitCfg(cfg); err != nil {
+ return Cfg{}, errors.New("Initializing loggers: " + err.Error() + "\n")
+ }
+
+ printConfig(cfg)
+
+ return cfg, nil
+}
+
+func validateURL(u *url.URL) error {
+ if u == nil {
+ return errors.New("nil url")
+ }
+ if u.Scheme != "http" && u.Scheme != "https" {
+ return errors.New("scheme expected 'http' or 'https', actual '" + u.Scheme + "'")
+ }
+ if strings.TrimSpace(u.Host) == "" {
+ return errors.New("no host")
+ }
+ return nil
+}
+
+func isCommandAvailable(name string) bool {
+ cmd := exec.Command("/bin/sh", "-c", name, "--version")
+ if err := cmd.Run(); err != nil {
+ return false
+ }
+
+ return true
+}
+
+func getOSSvcManagement() SvcManagement {
+ var _svcManager SvcManagement
+
+ if isCommandAvailable(SystemCtl) {
+ _svcManager = SYSTEMD
+ } else if isCommandAvailable(Service) {
+ _svcManager = SYSTEMV
+ }
+ if !isCommandAvailable(Chkconfig) {
+ return UNKNOWN
+ }
+
+ // we have what we need
+ return _svcManager
+}
+
+func printConfig(cfg Cfg) {
+ log.Debugf("Dispersion: %d\n", cfg.Dispersion)
+ log.Debugf("LogLocationDebug: %s\n", cfg.LogLocationDebug)
+ log.Debugf("LogLocationErr: %s\n", cfg.LogLocationErr)
+ log.Debugf("LogLocationInfo: %s\n", cfg.LogLocationInfo)
+ log.Debugf("LogLocationWarn: %s\n", cfg.LogLocationWarn)
+ log.Debugf("LoginDispersion: %d\n", cfg.LoginDispersion)
+ log.Debugf("CacheHostName: %s\n", cfg.CacheHostName)
+ log.Debugf("SvcManagement: %s\n", cfg.SvcManagement)
+ log.Debugf("Retries: %d\n", cfg.Retries)
+ log.Debugf("RevalWaitTime: %d\n", cfg.RevalWaitTime)
+ log.Debugf("ReverseProxyDisable: %t\n", cfg.ReverseProxyDisable)
+ log.Debugf("RunMode: %s\n", cfg.RunMode)
+ log.Debugf("SkipOSCheck: %t\n", cfg.SkipOSCheck)
+ log.Debugf("TOTimeoutMS: %d\n", cfg.TOTimeoutMS)
+ log.Debugf("TOUser: %s\n", cfg.TOUser)
+ log.Debugf("TOPass: %s\n", cfg.TOPass)
+ log.Debugf("TOURL: %s\n", cfg.TOURL)
+ log.Debugf("WaitForParents: %t\n", cfg.WaitForParents)
+ log.Debugf("YumOptions: %s\n", cfg.YumOptions)
+}
+
+func Usage() {
Review comment:
Could this use https://godoc.org/github.com/pborman/getopt#PrintUsage ?
##########
File path: traffic_ops_ort/traffic_ops_t3c/traffic_ops_t3c.go
##########
@@ -0,0 +1,169 @@
+package main
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/torequest"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "os"
+ "time"
+)
+
+func runSysctl(cfg config.Cfg) {
+ var ans bool
+
+ switch cfg.RunMode {
+ case config.INTERACTIVE:
+ ans, _ = util.AskYesNo("sysctl configuration has changed. 'sysctl -p' needs to be run. Should I do that now? (Y/[n])")
+ case config.BADASS:
+ ans = true
+ }
+
+ if ans == true {
+ _, rc, err := util.ExecCommand("/usr/sbin/sysctl", "-p")
Review comment:
This should be in the path, right? Better to use just `sysctl` or `/usr/bin/env sysctl`, in case some Distro puts it somewhere else?
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
Review comment:
Should be `len(pkgs) == 0` for safety.
`pkgs` could be length 0 but not nil, and `[0]` is dereferenced below.
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
Review comment:
I don't see a check for `len(r.pkgs)` anywhere. Is it ever possible for `len(r.pkgs) < pkgs[0]`? If so, this would panic.
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
Review comment:
Nitpick: `path/filepath.Join`
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
+ cfg, ok := r.configFiles[name]
+ return cfg, ok
+}
+
+// fetches and parses the multipart config files for a cache from traffic ops
+// and loads them into the configFiles map.
+func (r *TrafficOpsReq) GetConfigFileList() error {
+ var atsUid int = 0
+ var atsGid int = 0
+
+ atsUser, err := user.Lookup(config.TrafficServerOwner)
+ if err != nil {
+ log.Errorf("could not lookup the trafficserver, '%s', owner uid, using uid/gid 0")
+ } else {
+ atsUid, err = strconv.Atoi(atsUser.Uid)
+ if err != nil {
+ log.Errorf("could not parse the ats UID.")
+ atsUid = 0
+ }
+ atsGid, err = strconv.Atoi(atsUser.Gid)
+ if err != nil {
+ log.Errorf("could not parse the ats GID.")
+ atsUid = 0
+ }
+ }
+
+ fBytes, err := r.atsTcExec("get-config-files")
+ if err != nil {
+ return err
+ }
+ buf := bytes.NewBuffer(fBytes)
+ var hdr string
+ for {
+ hdr, err = buf.ReadString('\n')
+ if err != nil {
+ return err
+ } else {
+ if strings.HasPrefix(hdr, "Content-Type: multipart/mixed") {
+ break
+ }
+ }
+ }
+ // split on the ": " to trim off the leading space
+ har := strings.Split(hdr, ": ")
+ if har[0] != "Content-Type" || !strings.HasPrefix(har[1], "multipart/mixed") {
+ return errors.New("invalid config file data received from Traffic Ops, not in multipart format.")
+ }
+ msg := &mail.Message{
+ Header: map[string][]string{har[0]: {har[1]}},
+ Body: bytes.NewReader(buf.Bytes()),
+ }
+ mediaType, params, err := mime.ParseMediaType(msg.Header.Get("Content-Type"))
+ if err != nil {
+ return err
+ }
+ if strings.HasPrefix(mediaType, "multipart/") {
+ mr := multipart.NewReader(msg.Body, params["boundary"])
+ // if the configFiles map is not empty, create a new map
+ // and the old map will be garbage collected.
+ if len(r.configFiles) > 0 {
+ log.Infoln("intializing a new configFiles map")
+ r.configFiles = make(map[string]*ConfigFile)
+ }
+ for {
+ var cf *ConfigFile
+
+ p, err := mr.NextPart()
+ if err == io.EOF {
+ return nil
+ }
+ if err != nil {
+ return err
+ }
+ dataBytes, err := ioutil.ReadAll(p)
+ if err != nil {
+ return err
+ }
+ path := p.Header.Get("Path")
+ if path != "" {
+ cf = new(ConfigFile)
Review comment:
Nitpick: could be created in a single call and without `new`:
```
r.configFiles[cf.Name] = &ConfigFile{
Header: p.Header,
Name: filepath.Base(path),
Path: path,
Dir: filepath.Dir(path),
Body: dataBytes,
Uid: atsUid,
Gid: atsGid,
Perm: 0644,
}
```
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
Review comment:
I haven't tested yet, but one of the things I had to fix in the Perl, was that calling `rpm` for every file and/or remap line was super-slow, it made ORT take like 20mins instead of 20s. This will probably need to cache the results for the same name.
I'll try to verify when I run the code.
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
Review comment:
Nitpick: https://golang.org/pkg/path/filepath/#Join would be better.
It guarantees the right slashes are used for the OS, and multiple duplicate slashes are removed.
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
+ cfg, ok := r.configFiles[name]
+ return cfg, ok
+}
+
+// fetches and parses the multipart config files for a cache from traffic ops
Review comment:
GoDoc, `GetConfigFileList fetches and parses the multipart config files for a cache from Traffic Ops.`
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
Review comment:
Should this trim spaces first, in case it's " # foo" ?
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
+ if rc == 1 { // file is not part of any package.
+ return nil, nil
+ } else if rc == 0 { // add the package name the file belongs to.
+ log.Debugf("output from file-query: %s\n", string(output))
+ result = append(result, string(strings.TrimSpace(string(output))))
+ log.Debugf("result length: %d, result: %s\n", len(result), string(output))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-provides": // returns the package name that provides 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatprovides", name)
+ log.Debugf("pkg-provides - name: %s, output: %s\n", name, output)
+ if rc == 1 { // no package provides 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-query": // returns the package name for 'name'.
+ output, rc, err := ExecCommand("/bin/rpm", "-q", name)
+ if rc == 1 { // the package is not installed.
+ return nil, nil
+ } else if rc == 0 { // add the rpm name
+ result = append(result, string(strings.TrimSpace(string(output))))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-requires": // returns a list of packages that requires package 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatrequires", name)
+ if rc == 1 { // no package reuires package 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ }
+ return result, nil
+}
+
+func RandomDuration(max time.Duration) time.Duration {
+ rand.Seed(time.Now().UnixNano())
+ return time.Duration(rand.Int63n(int64(max)))
+}
+
+func CheckUser(cfg config.Cfg) bool {
+ result := true
+ user, _ := user.Current()
+
+ switch cfg.RunMode {
+ case config.BADASS:
+ fallthrough
+ case config.INTERACTIVE:
+ fallthrough
+ case config.SYNCDS:
+ if user.Username != "root" {
+ log.Errorf("Only the root user may run in BADASS, INTERACTIVE, or SYNCDS mode, current user: %s\n",
+ user.Username)
+ result = false
+ }
+ default:
+ log.Infof("current mode: %s, run user: %s\n", cfg.RunMode, user.Username)
+ }
+ return result
+}
+
+func CleanTmpDir() bool {
+ now := time.Now().Unix()
+
+ files, err := ioutil.ReadDir(config.TmpBase)
+ if err != nil {
+ log.Errorf("opening %s: %v\n", config.TmpBase, err)
+ return false
+ }
+ for _, f := range files {
+ // remove any directory and its contents under the config.TmpBase (/tmp/ort) that
+ // is more than a week old.
+ if f.IsDir() && (now-f.ModTime().Unix()) > 604800 {
+ dir := config.TmpBase + "/" + f.Name()
Review comment:
Nitpick: `filepath.Join`
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
Review comment:
Just to let you know, I asked on the Mailing List, and nobody ever uses Interactive Mode, so we agreed to remove it.
Not a huge deal if you want to leave it in for now. But you could remove it too if you wanted.
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
Review comment:
Is this necessary? It seems like anything calling it can just call `string.Contains`?
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
Review comment:
Nitpick: all of these can be terser with `:=`, and they can avoid the separate map/slice initialization:
`lineCount := 0`, `overrides := map[string]int{}`, `newLines := []string{}`
##########
File path: traffic_ops_ort/traffic_ops_t3c/traffic_ops_t3c.go
##########
@@ -0,0 +1,169 @@
+package main
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/torequest"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "os"
+ "time"
+)
+
+func runSysctl(cfg config.Cfg) {
+ var ans bool
+
+ switch cfg.RunMode {
+ case config.INTERACTIVE:
+ ans, _ = util.AskYesNo("sysctl configuration has changed. 'sysctl -p' needs to be run. Should I do that now? (Y/[n])")
+ case config.BADASS:
+ ans = true
+ }
+
+ if ans == true {
+ _, rc, err := util.ExecCommand("/usr/sbin/sysctl", "-p")
+ if err != nil {
+ log.Errorln("sysctl -p failed")
+ } else if rc == 0 {
+ log.Debugf("sysctl -p ran succesfully.")
+ }
+ }
+}
+
+func main() {
+ var syncdsUpdate torequest.UpdateStatus
+ var lock util.FileLock
+ cfg, err := config.GetCfg()
+ if err != nil {
+ fmt.Println(err)
+ os.Exit(-1)
+ }
+
+ trops := torequest.NewTrafficOpsReq(cfg)
+
+ // if doing os checks, insure there is a 'systemctl' or 'service' and 'chkconfig' commands.
+ if !cfg.SkipOSCheck && cfg.SvcManagement == config.UNKNOWN {
+ log.Errorln("OS checks are enabled and unable to find any know service management tools.")
+ }
+
+ // create and clean the config.TmpBase (/tmp/ort)
+ if !util.MkDir(config.TmpBase, cfg) {
+ os.Exit(1)
+ } else if !util.CleanTmpDir() {
+ os.Exit(1)
+ }
+ if cfg.RunMode != config.REPORT {
+ if !lock.GetLock(config.TmpBase + "/to_ort.lock") {
+ os.Exit(1)
+ }
+ }
+
+ fmt.Println(time.Now().Format(time.UnixDate))
+
+ if !util.CheckUser(cfg) {
+ lock.UnlockAndExit(-1)
+ }
+
+ toolName := trops.GetHeaderComment()
+ log.Debugf("toolname: %s\n", toolName)
+
+ // if running in REVALIDATE mode, check to see if it's
+ // necessary to continue
+ if cfg.RunMode == config.REVALIDATE {
+ syncdsUpdate, err = trops.CheckRevalidateState(false)
+ if err != nil || syncdsUpdate == torequest.UPDATE_TROPS_NOTNEEDED {
+ if err != nil {
+ log.Errorln(err)
+ }
+ os.Exit(1)
+ }
+ } else {
+ syncdsUpdate, err = trops.CheckSyncDSState()
+ if err != nil {
+ log.Errorln(err)
+ os.Exit(1)
Review comment:
Custom/const error codes?
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
+ if rc == 1 { // file is not part of any package.
+ return nil, nil
+ } else if rc == 0 { // add the package name the file belongs to.
+ log.Debugf("output from file-query: %s\n", string(output))
+ result = append(result, string(strings.TrimSpace(string(output))))
+ log.Debugf("result length: %d, result: %s\n", len(result), string(output))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-provides": // returns the package name that provides 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatprovides", name)
+ log.Debugf("pkg-provides - name: %s, output: %s\n", name, output)
+ if rc == 1 { // no package provides 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-query": // returns the package name for 'name'.
+ output, rc, err := ExecCommand("/bin/rpm", "-q", name)
+ if rc == 1 { // the package is not installed.
+ return nil, nil
+ } else if rc == 0 { // add the rpm name
+ result = append(result, string(strings.TrimSpace(string(output))))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-requires": // returns a list of packages that requires package 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatrequires", name)
+ if rc == 1 { // no package reuires package 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ }
+ return result, nil
+}
+
+func RandomDuration(max time.Duration) time.Duration {
+ rand.Seed(time.Now().UnixNano())
+ return time.Duration(rand.Int63n(int64(max)))
+}
+
+func CheckUser(cfg config.Cfg) bool {
+ result := true
+ user, _ := user.Current()
+
+ switch cfg.RunMode {
+ case config.BADASS:
+ fallthrough
+ case config.INTERACTIVE:
+ fallthrough
+ case config.SYNCDS:
+ if user.Username != "root" {
+ log.Errorf("Only the root user may run in BADASS, INTERACTIVE, or SYNCDS mode, current user: %s\n",
+ user.Username)
+ result = false
+ }
+ default:
+ log.Infof("current mode: %s, run user: %s\n", cfg.RunMode, user.Username)
+ }
+ return result
+}
+
+func CleanTmpDir() bool {
+ now := time.Now().Unix()
+
+ files, err := ioutil.ReadDir(config.TmpBase)
+ if err != nil {
+ log.Errorf("opening %s: %v\n", config.TmpBase, err)
+ return false
+ }
+ for _, f := range files {
+ // remove any directory and its contents under the config.TmpBase (/tmp/ort) that
+ // is more than a week old.
+ if f.IsDir() && (now-f.ModTime().Unix()) > 604800 {
Review comment:
Mind putting `604800` in a constant, just so it's obvious what it is? It's not immediately obvious that it's a week.
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
+ if rc == 1 { // file is not part of any package.
+ return nil, nil
+ } else if rc == 0 { // add the package name the file belongs to.
+ log.Debugf("output from file-query: %s\n", string(output))
+ result = append(result, string(strings.TrimSpace(string(output))))
+ log.Debugf("result length: %d, result: %s\n", len(result), string(output))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-provides": // returns the package name that provides 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatprovides", name)
+ log.Debugf("pkg-provides - name: %s, output: %s\n", name, output)
+ if rc == 1 { // no package provides 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-query": // returns the package name for 'name'.
+ output, rc, err := ExecCommand("/bin/rpm", "-q", name)
+ if rc == 1 { // the package is not installed.
+ return nil, nil
+ } else if rc == 0 { // add the rpm name
+ result = append(result, string(strings.TrimSpace(string(output))))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-requires": // returns a list of packages that requires package 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatrequires", name)
+ if rc == 1 { // no package reuires package 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ }
+ return result, nil
+}
+
+func RandomDuration(max time.Duration) time.Duration {
+ rand.Seed(time.Now().UnixNano())
+ return time.Duration(rand.Int63n(int64(max)))
+}
+
+func CheckUser(cfg config.Cfg) bool {
+ result := true
+ user, _ := user.Current()
Review comment:
The error here needs handled
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
Review comment:
GoDoc comments should be full sentences, starting with the name of the symbol and ending in a period. `isPackageInstalled returns true/false if the named rpm package is installed.`
This doesn't technically break GoDoc because lowercase private funcs aren't documented. But it's still a really good idea to conform to GoDoc so it's easier for people to read.
Typo, `fale`.
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
Review comment:
Needs to be vendored
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
Review comment:
GoDoc, should be `PackageInfo runs the rpm command.`.
Sentences below need capitalized too.
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
+ if rc == 1 { // file is not part of any package.
+ return nil, nil
+ } else if rc == 0 { // add the package name the file belongs to.
+ log.Debugf("output from file-query: %s\n", string(output))
+ result = append(result, string(strings.TrimSpace(string(output))))
+ log.Debugf("result length: %d, result: %s\n", len(result), string(output))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-provides": // returns the package name that provides 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatprovides", name)
+ log.Debugf("pkg-provides - name: %s, output: %s\n", name, output)
+ if rc == 1 { // no package provides 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-query": // returns the package name for 'name'.
+ output, rc, err := ExecCommand("/bin/rpm", "-q", name)
+ if rc == 1 { // the package is not installed.
+ return nil, nil
+ } else if rc == 0 { // add the rpm name
+ result = append(result, string(strings.TrimSpace(string(output))))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-requires": // returns a list of packages that requires package 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatrequires", name)
+ if rc == 1 { // no package reuires package 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ }
+ return result, nil
+}
+
+func RandomDuration(max time.Duration) time.Duration {
+ rand.Seed(time.Now().UnixNano())
+ return time.Duration(rand.Int63n(int64(max)))
+}
+
+func CheckUser(cfg config.Cfg) bool {
+ result := true
+ user, _ := user.Current()
Review comment:
This variable shadows the package name. Mind renaming it, just to avoid confusion?
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
Review comment:
GoDoc, `GetConfigFile fetches a 'Configfile' by file name.`
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
+ cfg, ok := r.configFiles[name]
+ return cfg, ok
+}
+
+// fetches and parses the multipart config files for a cache from traffic ops
+// and loads them into the configFiles map.
+func (r *TrafficOpsReq) GetConfigFileList() error {
+ var atsUid int = 0
+ var atsGid int = 0
+
+ atsUser, err := user.Lookup(config.TrafficServerOwner)
+ if err != nil {
+ log.Errorf("could not lookup the trafficserver, '%s', owner uid, using uid/gid 0")
+ } else {
+ atsUid, err = strconv.Atoi(atsUser.Uid)
+ if err != nil {
+ log.Errorf("could not parse the ats UID.")
+ atsUid = 0
+ }
+ atsGid, err = strconv.Atoi(atsUser.Gid)
+ if err != nil {
+ log.Errorf("could not parse the ats GID.")
+ atsUid = 0
+ }
+ }
+
+ fBytes, err := r.atsTcExec("get-config-files")
+ if err != nil {
+ return err
+ }
+ buf := bytes.NewBuffer(fBytes)
+ var hdr string
+ for {
+ hdr, err = buf.ReadString('\n')
+ if err != nil {
+ return err
+ } else {
+ if strings.HasPrefix(hdr, "Content-Type: multipart/mixed") {
+ break
+ }
+ }
+ }
+ // split on the ": " to trim off the leading space
+ har := strings.Split(hdr, ": ")
+ if har[0] != "Content-Type" || !strings.HasPrefix(har[1], "multipart/mixed") {
+ return errors.New("invalid config file data received from Traffic Ops, not in multipart format.")
+ }
+ msg := &mail.Message{
+ Header: map[string][]string{har[0]: {har[1]}},
+ Body: bytes.NewReader(buf.Bytes()),
+ }
+ mediaType, params, err := mime.ParseMediaType(msg.Header.Get("Content-Type"))
+ if err != nil {
+ return err
+ }
+ if strings.HasPrefix(mediaType, "multipart/") {
+ mr := multipart.NewReader(msg.Body, params["boundary"])
+ // if the configFiles map is not empty, create a new map
+ // and the old map will be garbage collected.
+ if len(r.configFiles) > 0 {
+ log.Infoln("intializing a new configFiles map")
+ r.configFiles = make(map[string]*ConfigFile)
+ }
+ for {
+ var cf *ConfigFile
+
+ p, err := mr.NextPart()
+ if err == io.EOF {
+ return nil
+ }
+ if err != nil {
+ return err
+ }
+ dataBytes, err := ioutil.ReadAll(p)
+ if err != nil {
+ return err
+ }
+ path := p.Header.Get("Path")
+ if path != "" {
+ cf = new(ConfigFile)
+ cf.Header = p.Header
+ cf.Name = filepath.Base(path)
+ cf.Path = path
+ cf.Dir = filepath.Dir(path)
+ cf.Body = dataBytes
+ cf.Uid = atsUid
+ cf.Gid = atsGid
+ cf.Perm = 0644
+ }
+ r.configFiles[cf.Name] = cf
+ }
+ }
+ return nil
+}
+
+// looks up the tm.toolname parameter from traffic ops.
+func (r *TrafficOpsReq) GetHeaderComment() string {
+ var toolname string = ""
Review comment:
Go naming convention: `toolName := ""`
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
+ cfg, ok := r.configFiles[name]
+ return cfg, ok
+}
+
+// fetches and parses the multipart config files for a cache from traffic ops
+// and loads them into the configFiles map.
+func (r *TrafficOpsReq) GetConfigFileList() error {
+ var atsUid int = 0
+ var atsGid int = 0
+
+ atsUser, err := user.Lookup(config.TrafficServerOwner)
+ if err != nil {
+ log.Errorf("could not lookup the trafficserver, '%s', owner uid, using uid/gid 0")
+ } else {
+ atsUid, err = strconv.Atoi(atsUser.Uid)
+ if err != nil {
+ log.Errorf("could not parse the ats UID.")
+ atsUid = 0
+ }
+ atsGid, err = strconv.Atoi(atsUser.Gid)
+ if err != nil {
+ log.Errorf("could not parse the ats GID.")
+ atsUid = 0
+ }
+ }
+
+ fBytes, err := r.atsTcExec("get-config-files")
+ if err != nil {
+ return err
+ }
+ buf := bytes.NewBuffer(fBytes)
+ var hdr string
+ for {
+ hdr, err = buf.ReadString('\n')
+ if err != nil {
+ return err
+ } else {
+ if strings.HasPrefix(hdr, "Content-Type: multipart/mixed") {
+ break
+ }
+ }
+ }
+ // split on the ": " to trim off the leading space
+ har := strings.Split(hdr, ": ")
+ if har[0] != "Content-Type" || !strings.HasPrefix(har[1], "multipart/mixed") {
+ return errors.New("invalid config file data received from Traffic Ops, not in multipart format.")
+ }
+ msg := &mail.Message{
+ Header: map[string][]string{har[0]: {har[1]}},
+ Body: bytes.NewReader(buf.Bytes()),
+ }
+ mediaType, params, err := mime.ParseMediaType(msg.Header.Get("Content-Type"))
+ if err != nil {
+ return err
+ }
+ if strings.HasPrefix(mediaType, "multipart/") {
+ mr := multipart.NewReader(msg.Body, params["boundary"])
+ // if the configFiles map is not empty, create a new map
+ // and the old map will be garbage collected.
+ if len(r.configFiles) > 0 {
+ log.Infoln("intializing a new configFiles map")
+ r.configFiles = make(map[string]*ConfigFile)
+ }
+ for {
+ var cf *ConfigFile
+
+ p, err := mr.NextPart()
+ if err == io.EOF {
+ return nil
+ }
+ if err != nil {
+ return err
+ }
+ dataBytes, err := ioutil.ReadAll(p)
+ if err != nil {
+ return err
+ }
+ path := p.Header.Get("Path")
+ if path != "" {
+ cf = new(ConfigFile)
+ cf.Header = p.Header
+ cf.Name = filepath.Base(path)
+ cf.Path = path
+ cf.Dir = filepath.Dir(path)
+ cf.Body = dataBytes
+ cf.Uid = atsUid
+ cf.Gid = atsGid
+ cf.Perm = 0644
+ }
+ r.configFiles[cf.Name] = cf
+ }
+ }
+ return nil
+}
+
+// looks up the tm.toolname parameter from traffic ops.
+func (r *TrafficOpsReq) GetHeaderComment() string {
+ var toolname string = ""
+ out, err := r.atsTcExec("system-info")
+ if err != nil {
+ log.Errorln(err)
+ } else {
+ var result map[string]interface{}
+ if err = json.Unmarshal(out, &result); err != nil {
+ log.Errorln(err)
+ } else {
+ _tn := result["tm.toolname"]
+ if _tn, ok := _tn.(string); ok {
+ toolname = _tn
+ log.Infof("Found tm.toolname: %v\n", toolname)
+ } else {
+ log.Errorln("Did not find tm.toolname!")
+ }
+ }
+ }
+ return toolname
+}
+
+func (r *TrafficOpsReq) CheckRevalidateState(sleepOverride bool) (UpdateStatus, error) {
+ updateStatus := UPDATE_TROPS_NOTNEEDED
+ log.Infoln("Checking revalidate state.")
+
+ if r.Cfg.RunMode == config.REVALIDATE || sleepOverride {
+ serverStatus, err := r.getUpdateStatus()
+ log.Infof("my status: %s\n", serverStatus.Status)
+ if err != nil {
+ log.Errorln(err)
+ return updateStatus, err
+ } else {
+ if serverStatus.UseRevalPending == false {
+ log.Errorln("Update URL: Instant invalidate is not enabled. Separated revalidation requires upgrading to Traffic Ops version 2.2 and enabling this feature.")
+ return UPDATE_TROPS_NOTNEEDED, nil
+ }
+ if serverStatus.RevalPending == true {
+ log.Errorln("Traffic Ops is signaling that a revalidation is waiting to be applied.")
+ updateStatus = UPDATE_TROPS_NEEDED
+ if serverStatus.ParentRevalPending == true {
+ log.Errorln("Traffic Ops is signaling that my parents need to revalidate.")
+ // no update needed until my parents are updated.
+ updateStatus = UPDATE_TROPS_NOTNEEDED
+ }
+ } else if serverStatus.RevalPending == false {
+ log.Errorln("In revalidate mode, but no update needs to be applied. I'm outta here.")
+ return UPDATE_TROPS_NOTNEEDED, nil
+ } else {
+ log.Errorln("Traffic Ops is signaling that no revalidations are waiting to be applied.")
+ }
+ }
+
+ err = r.checkStatusFiles(serverStatus.Status)
+ if err != nil {
+ log.Errorln(err)
+ }
+ }
+
+ return updateStatus, nil
+}
+
+func (r *TrafficOpsReq) CheckSyncDSState() (UpdateStatus, error) {
+ updateStatus := UPDATE_TROPS_NOTNEEDED
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion)
+ log.Debugln("Checking syncds state.")
+ if r.Cfg.RunMode == config.SYNCDS || r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.REPORT {
+ serverStatus, err := r.getUpdateStatus()
+ if err != nil {
+ log.Errorln(err)
+ return updateStatus, err
+ }
+
+ if serverStatus.UpdatePending {
+ if r.Cfg.Dispersion > 0 {
+ log.Infof("Sleeping for %ds (dispersion) before proceeding with updates.\n\n", (randDispSec / time.Second))
+ r.sleepTimer(serverStatus)
+ }
+ log.Errorln("Traffic Ops is signaling that an update is waiting to be applied")
+ updateStatus = UPDATE_TROPS_NEEDED
+
+ if serverStatus.ParentPending && r.Cfg.WaitForParents && !serverStatus.UseRevalPending {
+ log.Errorln("Traffic Ops is signaling that my parents need an update.")
+ if r.Cfg.RunMode == config.SYNCDS {
+ log.Infof("In syncds mode, sleeping for %ds to see if the update my parents need is cleared.", randDispSec/time.Second)
+ r.sleepTimer(serverStatus)
+ serverStatus, err = r.getUpdateStatus()
+ if err != nil {
+ return updateStatus, err
+ }
+ if serverStatus.ParentPending || serverStatus.ParentRevalPending {
+ log.Errorln("My parents still need an update, bailing.")
+ return UPDATE_TROPS_NOTNEEDED, nil
+ } else {
+ log.Debugln("The update on my parents cleared; continuing.")
+ }
+ }
+ } else {
+ log.Debugf("Traffic Ops is signaling that my parents do not need an update, or wait_for_parents is false.")
+ }
+ } else if r.Cfg.RunMode == config.SYNCDS {
+ log.Errorln("In syncds mode, but no syncds update needs to be applied. Running revalidation befor exiting.")
+ r.RevalidateWhileSleeping()
+ return UPDATE_TROPS_NOTNEEDED, nil
+ } else {
+ log.Errorln("Traffic Ops is signaling that no update is waiting to be applied.")
+ }
+
+ // check local status files.
+ err = r.checkStatusFiles(serverStatus.Status)
+ if err != nil {
+ log.Errorln(err)
+ }
+ }
+ return updateStatus, nil
+}
+
+// process all config files retrieved from Traffic Ops. perl version is process_config_files
+func (r *TrafficOpsReq) ProcessConfigFiles() (UpdateStatus, error) {
+ var updateStatus UpdateStatus = UPDATE_TROPS_NOTNEEDED
+
+ log.Infoln(" ======== Start processing config files ========")
+
+ for _, cfg := range r.configFiles {
+ // add service metadata
+ if util.FileMatch(cfg.Path, "/opt/trafficserver/") || util.FileMatch(cfg.Dir, "udev") {
+ arr, err := util.PackageInfo("pkg-query", "trafficserver")
+ if err != nil {
+ return UPDATE_TROPS_FAILED, errors.New("rpm query error: " + err.Error())
+ } else if len(arr) > 0 && strings.HasPrefix(arr[0], "trafficserver") {
+ cfg.Service = "trafficserver"
+ } else if r.Cfg.RunMode == config.SYNCDS {
+ return UPDATE_TROPS_FAILED, errors.New("In syncds mode, but trafficserver isn't installed. Bailing.")
+ }
+ } else if util.FileMatch(cfg.Path, "/opt/ort") && util.FileMatch(cfg.Name, "12M_facts") {
+ cfg.Service = "puppet"
+ } else if util.FileMatch(cfg.Path, "cron") || util.FileMatch(cfg.Name, "sysctl.conf") || util.FileMatch(cfg.Name, "50-ats.rules") || util.FileMatch(cfg.Name, "cron") {
+ cfg.Service = "system"
+ } else if util.FileMatch(cfg.Path, "ntp.conf") {
+ cfg.Service = "ntpd"
+ } else {
+ cfg.Service = "unknown"
+ }
+
+ log.Debugf("In %s mode, I'm about to process config file: %s, service: %s\n", r.Cfg.RunMode, cfg.Path, cfg.Service)
+
+ err := r.checkConfigFile(cfg)
+ if err != nil {
+ log.Errorln(err)
+ }
+ }
+
+ for _, cfg := range r.configFiles {
+ if cfg.ChangeNeeded &&
+ !cfg.ChangeApplied &&
+ cfg.AuditComplete &&
+ !cfg.PreReqFailed &&
+ !cfg.AuditFailed {
+ if cfg.Name == "plugin.config" && r.configFiles["remap.config"].PreReqFailed == true {
+ updateStatus = UPDATE_TROPS_FAILED
+ log.Errorln("plugin.config changed however, prereqs failed for remap.config so I am skipping updates for plugin.config")
+ continue
+ } else if cfg.Name == "remap.config" && r.configFiles["plugin.config"].PreReqFailed == true {
+ updateStatus = UPDATE_TROPS_FAILED
+ log.Errorln("remap.config changed however, prereqs failed for plugin.config so I am skipping updates for remap.config")
+ continue
+ } else {
+ log.Debugf("All Prereqs passed for replacing %s on disk with that in Traffic Ops.\n", cfg.Name)
+ err := r.replaceCfgFile(cfg)
+ if err != nil {
+ log.Errorf("failed to replace the config file, '%s', on disk with data in Traffic Ops.\n", cfg.Name)
+ }
+ }
+ }
+ }
+ return updateStatus, nil
+}
+
+func (r *TrafficOpsReq) ProcessPackages() error {
+ var pkgs []Package
+ var install []string // install package list.
+ var uninstall []string // uninstall package list
+
+ // get the package list for this cache from Traffic Ops.
+ out, err := r.atsTcExec("packages")
+ if err != nil {
+ return err
+ }
+
+ if err = json.Unmarshal(out, &pkgs); err != nil {
+ return err
+ }
+
+ // loop through the package list to build an install and uninstall list.
+ for ii := range pkgs {
+ var instpkg string // installed package
+ var reqpkg string // required package
+ log.Infof("Processing package %s-%s\n", pkgs[ii].Name, pkgs[ii].Version)
+ // check to see if any package by name is installed.
+ arr, err := util.PackageInfo("pkg-query", pkgs[ii].Name)
+ if err != nil {
+ return err
+ }
+ // go needs the ternary operator :)
Review comment:
Yes, yes it does ☹️
Expressions instead of statements would also be nice,
```
instpkg = if len(arr) == 1 {
arr[0]
} else {
""
}
```
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
+ cfg, ok := r.configFiles[name]
+ return cfg, ok
+}
+
+// fetches and parses the multipart config files for a cache from traffic ops
+// and loads them into the configFiles map.
+func (r *TrafficOpsReq) GetConfigFileList() error {
+ var atsUid int = 0
+ var atsGid int = 0
+
+ atsUser, err := user.Lookup(config.TrafficServerOwner)
+ if err != nil {
+ log.Errorf("could not lookup the trafficserver, '%s', owner uid, using uid/gid 0")
+ } else {
+ atsUid, err = strconv.Atoi(atsUser.Uid)
+ if err != nil {
+ log.Errorf("could not parse the ats UID.")
+ atsUid = 0
+ }
+ atsGid, err = strconv.Atoi(atsUser.Gid)
+ if err != nil {
+ log.Errorf("could not parse the ats GID.")
+ atsUid = 0
+ }
+ }
+
+ fBytes, err := r.atsTcExec("get-config-files")
+ if err != nil {
+ return err
+ }
+ buf := bytes.NewBuffer(fBytes)
+ var hdr string
+ for {
+ hdr, err = buf.ReadString('\n')
+ if err != nil {
+ return err
+ } else {
+ if strings.HasPrefix(hdr, "Content-Type: multipart/mixed") {
+ break
+ }
+ }
+ }
+ // split on the ": " to trim off the leading space
+ har := strings.Split(hdr, ": ")
+ if har[0] != "Content-Type" || !strings.HasPrefix(har[1], "multipart/mixed") {
+ return errors.New("invalid config file data received from Traffic Ops, not in multipart format.")
+ }
+ msg := &mail.Message{
+ Header: map[string][]string{har[0]: {har[1]}},
+ Body: bytes.NewReader(buf.Bytes()),
+ }
+ mediaType, params, err := mime.ParseMediaType(msg.Header.Get("Content-Type"))
+ if err != nil {
+ return err
+ }
+ if strings.HasPrefix(mediaType, "multipart/") {
+ mr := multipart.NewReader(msg.Body, params["boundary"])
+ // if the configFiles map is not empty, create a new map
+ // and the old map will be garbage collected.
+ if len(r.configFiles) > 0 {
+ log.Infoln("intializing a new configFiles map")
+ r.configFiles = make(map[string]*ConfigFile)
+ }
+ for {
+ var cf *ConfigFile
+
+ p, err := mr.NextPart()
+ if err == io.EOF {
+ return nil
+ }
+ if err != nil {
+ return err
+ }
+ dataBytes, err := ioutil.ReadAll(p)
+ if err != nil {
+ return err
+ }
+ path := p.Header.Get("Path")
+ if path != "" {
+ cf = new(ConfigFile)
+ cf.Header = p.Header
+ cf.Name = filepath.Base(path)
+ cf.Path = path
+ cf.Dir = filepath.Dir(path)
+ cf.Body = dataBytes
+ cf.Uid = atsUid
+ cf.Gid = atsGid
+ cf.Perm = 0644
+ }
+ r.configFiles[cf.Name] = cf
+ }
+ }
+ return nil
+}
+
+// looks up the tm.toolname parameter from traffic ops.
+func (r *TrafficOpsReq) GetHeaderComment() string {
+ var toolname string = ""
+ out, err := r.atsTcExec("system-info")
+ if err != nil {
+ log.Errorln(err)
+ } else {
+ var result map[string]interface{}
+ if err = json.Unmarshal(out, &result); err != nil {
+ log.Errorln(err)
+ } else {
+ _tn := result["tm.toolname"]
Review comment:
Go naming convention, `tn`
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
+ cfg, ok := r.configFiles[name]
+ return cfg, ok
+}
+
+// fetches and parses the multipart config files for a cache from traffic ops
+// and loads them into the configFiles map.
+func (r *TrafficOpsReq) GetConfigFileList() error {
+ var atsUid int = 0
+ var atsGid int = 0
+
+ atsUser, err := user.Lookup(config.TrafficServerOwner)
+ if err != nil {
+ log.Errorf("could not lookup the trafficserver, '%s', owner uid, using uid/gid 0")
+ } else {
+ atsUid, err = strconv.Atoi(atsUser.Uid)
+ if err != nil {
+ log.Errorf("could not parse the ats UID.")
+ atsUid = 0
+ }
+ atsGid, err = strconv.Atoi(atsUser.Gid)
+ if err != nil {
+ log.Errorf("could not parse the ats GID.")
+ atsUid = 0
+ }
+ }
+
+ fBytes, err := r.atsTcExec("get-config-files")
+ if err != nil {
+ return err
+ }
+ buf := bytes.NewBuffer(fBytes)
+ var hdr string
+ for {
+ hdr, err = buf.ReadString('\n')
+ if err != nil {
+ return err
+ } else {
+ if strings.HasPrefix(hdr, "Content-Type: multipart/mixed") {
+ break
+ }
+ }
+ }
+ // split on the ": " to trim off the leading space
+ har := strings.Split(hdr, ": ")
+ if har[0] != "Content-Type" || !strings.HasPrefix(har[1], "multipart/mixed") {
Review comment:
This needs to check `len(har) > 1` before dereferencing `har[1]`, or it'll panic on malformed data
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
+ cfg, ok := r.configFiles[name]
+ return cfg, ok
+}
+
+// fetches and parses the multipart config files for a cache from traffic ops
+// and loads them into the configFiles map.
+func (r *TrafficOpsReq) GetConfigFileList() error {
+ var atsUid int = 0
+ var atsGid int = 0
+
+ atsUser, err := user.Lookup(config.TrafficServerOwner)
+ if err != nil {
+ log.Errorf("could not lookup the trafficserver, '%s', owner uid, using uid/gid 0")
+ } else {
+ atsUid, err = strconv.Atoi(atsUser.Uid)
+ if err != nil {
+ log.Errorf("could not parse the ats UID.")
+ atsUid = 0
+ }
+ atsGid, err = strconv.Atoi(atsUser.Gid)
+ if err != nil {
+ log.Errorf("could not parse the ats GID.")
+ atsUid = 0
+ }
+ }
+
+ fBytes, err := r.atsTcExec("get-config-files")
+ if err != nil {
+ return err
+ }
+ buf := bytes.NewBuffer(fBytes)
+ var hdr string
+ for {
+ hdr, err = buf.ReadString('\n')
+ if err != nil {
+ return err
+ } else {
+ if strings.HasPrefix(hdr, "Content-Type: multipart/mixed") {
+ break
+ }
+ }
+ }
+ // split on the ": " to trim off the leading space
+ har := strings.Split(hdr, ": ")
+ if har[0] != "Content-Type" || !strings.HasPrefix(har[1], "multipart/mixed") {
+ return errors.New("invalid config file data received from Traffic Ops, not in multipart format.")
+ }
+ msg := &mail.Message{
+ Header: map[string][]string{har[0]: {har[1]}},
+ Body: bytes.NewReader(buf.Bytes()),
+ }
+ mediaType, params, err := mime.ParseMediaType(msg.Header.Get("Content-Type"))
Review comment:
Nitpick: `rfc.ContentType`
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
+ cfg, ok := r.configFiles[name]
+ return cfg, ok
+}
+
+// fetches and parses the multipart config files for a cache from traffic ops
+// and loads them into the configFiles map.
+func (r *TrafficOpsReq) GetConfigFileList() error {
+ var atsUid int = 0
+ var atsGid int = 0
+
+ atsUser, err := user.Lookup(config.TrafficServerOwner)
+ if err != nil {
+ log.Errorf("could not lookup the trafficserver, '%s', owner uid, using uid/gid 0")
+ } else {
+ atsUid, err = strconv.Atoi(atsUser.Uid)
+ if err != nil {
+ log.Errorf("could not parse the ats UID.")
+ atsUid = 0
+ }
+ atsGid, err = strconv.Atoi(atsUser.Gid)
+ if err != nil {
+ log.Errorf("could not parse the ats GID.")
+ atsUid = 0
+ }
+ }
+
+ fBytes, err := r.atsTcExec("get-config-files")
+ if err != nil {
+ return err
+ }
+ buf := bytes.NewBuffer(fBytes)
+ var hdr string
+ for {
+ hdr, err = buf.ReadString('\n')
+ if err != nil {
+ return err
+ } else {
+ if strings.HasPrefix(hdr, "Content-Type: multipart/mixed") {
+ break
+ }
+ }
+ }
+ // split on the ": " to trim off the leading space
+ har := strings.Split(hdr, ": ")
+ if har[0] != "Content-Type" || !strings.HasPrefix(har[1], "multipart/mixed") {
Review comment:
Nitpick: we try to keep RFC constants in `lib/go-rfc`. Can be `rfc.ContentType`
https://github.com/apache/trafficcontrol/blob/master/lib/go-rfc/http.go#L32
It'd also be good if you wanted to add `multipart/mixed` there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453878403
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453920635
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
+ cfg, ok := r.configFiles[name]
+ return cfg, ok
+}
+
+// fetches and parses the multipart config files for a cache from traffic ops
+// and loads them into the configFiles map.
+func (r *TrafficOpsReq) GetConfigFileList() error {
+ var atsUid int = 0
+ var atsGid int = 0
+
+ atsUser, err := user.Lookup(config.TrafficServerOwner)
+ if err != nil {
+ log.Errorf("could not lookup the trafficserver, '%s', owner uid, using uid/gid 0")
+ } else {
+ atsUid, err = strconv.Atoi(atsUser.Uid)
+ if err != nil {
+ log.Errorf("could not parse the ats UID.")
+ atsUid = 0
+ }
+ atsGid, err = strconv.Atoi(atsUser.Gid)
+ if err != nil {
+ log.Errorf("could not parse the ats GID.")
+ atsUid = 0
+ }
+ }
+
+ fBytes, err := r.atsTcExec("get-config-files")
+ if err != nil {
+ return err
+ }
+ buf := bytes.NewBuffer(fBytes)
+ var hdr string
+ for {
+ hdr, err = buf.ReadString('\n')
+ if err != nil {
+ return err
+ } else {
+ if strings.HasPrefix(hdr, "Content-Type: multipart/mixed") {
+ break
+ }
+ }
+ }
+ // split on the ": " to trim off the leading space
+ har := strings.Split(hdr, ": ")
+ if har[0] != "Content-Type" || !strings.HasPrefix(har[1], "multipart/mixed") {
+ return errors.New("invalid config file data received from Traffic Ops, not in multipart format.")
+ }
+ msg := &mail.Message{
+ Header: map[string][]string{har[0]: {har[1]}},
+ Body: bytes.NewReader(buf.Bytes()),
+ }
+ mediaType, params, err := mime.ParseMediaType(msg.Header.Get("Content-Type"))
+ if err != nil {
+ return err
+ }
+ if strings.HasPrefix(mediaType, "multipart/") {
+ mr := multipart.NewReader(msg.Body, params["boundary"])
+ // if the configFiles map is not empty, create a new map
+ // and the old map will be garbage collected.
+ if len(r.configFiles) > 0 {
+ log.Infoln("intializing a new configFiles map")
+ r.configFiles = make(map[string]*ConfigFile)
+ }
+ for {
+ var cf *ConfigFile
+
+ p, err := mr.NextPart()
+ if err == io.EOF {
+ return nil
+ }
+ if err != nil {
+ return err
+ }
+ dataBytes, err := ioutil.ReadAll(p)
+ if err != nil {
+ return err
+ }
+ path := p.Header.Get("Path")
+ if path != "" {
+ cf = new(ConfigFile)
+ cf.Header = p.Header
+ cf.Name = filepath.Base(path)
+ cf.Path = path
+ cf.Dir = filepath.Dir(path)
+ cf.Body = dataBytes
+ cf.Uid = atsUid
+ cf.Gid = atsGid
+ cf.Perm = 0644
+ }
+ r.configFiles[cf.Name] = cf
+ }
+ }
+ return nil
+}
+
+// looks up the tm.toolname parameter from traffic ops.
+func (r *TrafficOpsReq) GetHeaderComment() string {
+ var toolname string = ""
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#issuecomment-656242765
@ocket8888 As a team, caching proxies, decided this work is part of the overall ORT rework as outlined in the blueprint for the reasons @rob05c has mentioned. It was assigned to me in the CDN Q2/Q3 work plan and I'll continue with the ORT blueprint work. I'm not arguing wether one is better than the other (go vs python) but all the remaining blueprint work will be in go as well.
This go version does manage rpm installation and does require /sbin/service and /bin/systemctl to insure that systemd loads/reloads 'service' files
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453993717
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
Review comment:
Okay, this is fixed. Not doing an rpm query on every file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453853749
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
+ if rc == 1 { // file is not part of any package.
+ return nil, nil
+ } else if rc == 0 { // add the package name the file belongs to.
+ log.Debugf("output from file-query: %s\n", string(output))
+ result = append(result, string(strings.TrimSpace(string(output))))
+ log.Debugf("result length: %d, result: %s\n", len(result), string(output))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-provides": // returns the package name that provides 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatprovides", name)
+ log.Debugf("pkg-provides - name: %s, output: %s\n", name, output)
+ if rc == 1 { // no package provides 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-query": // returns the package name for 'name'.
+ output, rc, err := ExecCommand("/bin/rpm", "-q", name)
+ if rc == 1 { // the package is not installed.
+ return nil, nil
+ } else if rc == 0 { // add the rpm name
+ result = append(result, string(strings.TrimSpace(string(output))))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-requires": // returns a list of packages that requires package 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatrequires", name)
+ if rc == 1 { // no package reuires package 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ }
+ return result, nil
+}
+
+func RandomDuration(max time.Duration) time.Duration {
+ rand.Seed(time.Now().UnixNano())
+ return time.Duration(rand.Int63n(int64(max)))
+}
+
+func CheckUser(cfg config.Cfg) bool {
+ result := true
+ user, _ := user.Current()
+
+ switch cfg.RunMode {
+ case config.BADASS:
+ fallthrough
+ case config.INTERACTIVE:
+ fallthrough
+ case config.SYNCDS:
+ if user.Username != "root" {
+ log.Errorf("Only the root user may run in BADASS, INTERACTIVE, or SYNCDS mode, current user: %s\n",
+ user.Username)
+ result = false
+ }
+ default:
+ log.Infof("current mode: %s, run user: %s\n", cfg.RunMode, user.Username)
+ }
+ return result
+}
+
+func CleanTmpDir() bool {
+ now := time.Now().Unix()
+
+ files, err := ioutil.ReadDir(config.TmpBase)
+ if err != nil {
+ log.Errorf("opening %s: %v\n", config.TmpBase, err)
+ return false
+ }
+ for _, f := range files {
+ // remove any directory and its contents under the config.TmpBase (/tmp/ort) that
+ // is more than a week old.
+ if f.IsDir() && (now-f.ModTime().Unix()) > 604800 {
Review comment:
added a constant, OneWeek
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453913913
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] dneuman64 commented on pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
dneuman64 commented on pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#issuecomment-656259339
I think it's fine to eventually include a flag to bypass the SystemD calls if it can be done and make it so we don't have to maintain two version of ORT (which we never should have anyway).
We decided, even during the blueprint discussions, that we would be moving towards re-implementing ORT in golang. For the most part the programming language shouldn't matter; we should embrace community over code and be thankful that we are addressing the issues with ORT and working to make it better.
I understand that there is a python version of ORT but IMO that is not a production piece of code and we should not continue to maintain it if we do not have to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rob05c commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
rob05c commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453886080
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
Review comment:
Maybe you're thinking of Report Mode? They did ask that we keep a dedicated Report Mode.
But nobody objected to removing Interactive, and everyone I asked specifically said they've never used it.
But again, we can leave it for now if you want, it's easy to remove later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453915762
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
Review comment:
ok :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453916131
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453932887
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
+ if rc == 1 { // file is not part of any package.
+ return nil, nil
+ } else if rc == 0 { // add the package name the file belongs to.
+ log.Debugf("output from file-query: %s\n", string(output))
+ result = append(result, string(strings.TrimSpace(string(output))))
+ log.Debugf("result length: %d, result: %s\n", len(result), string(output))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-provides": // returns the package name that provides 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatprovides", name)
+ log.Debugf("pkg-provides - name: %s, output: %s\n", name, output)
+ if rc == 1 { // no package provides 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-query": // returns the package name for 'name'.
+ output, rc, err := ExecCommand("/bin/rpm", "-q", name)
+ if rc == 1 { // the package is not installed.
+ return nil, nil
+ } else if rc == 0 { // add the rpm name
+ result = append(result, string(strings.TrimSpace(string(output))))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-requires": // returns a list of packages that requires package 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatrequires", name)
+ if rc == 1 { // no package reuires package 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ }
+ return result, nil
+}
+
+func RandomDuration(max time.Duration) time.Duration {
+ rand.Seed(time.Now().UnixNano())
+ return time.Duration(rand.Int63n(int64(max)))
+}
+
+func CheckUser(cfg config.Cfg) bool {
+ result := true
+ user, _ := user.Current()
+
+ switch cfg.RunMode {
+ case config.BADASS:
+ fallthrough
+ case config.INTERACTIVE:
+ fallthrough
+ case config.SYNCDS:
+ if user.Username != "root" {
+ log.Errorf("Only the root user may run in BADASS, INTERACTIVE, or SYNCDS mode, current user: %s\n",
+ user.Username)
+ result = false
+ }
+ default:
+ log.Infof("current mode: %s, run user: %s\n", cfg.RunMode, user.Username)
+ }
+ return result
+}
+
+func CleanTmpDir() bool {
+ now := time.Now().Unix()
+
+ files, err := ioutil.ReadDir(config.TmpBase)
+ if err != nil {
+ log.Errorf("opening %s: %v\n", config.TmpBase, err)
+ return false
+ }
+ for _, f := range files {
+ // remove any directory and its contents under the config.TmpBase (/tmp/ort) that
+ // is more than a week old.
+ if f.IsDir() && (now-f.ModTime().Unix()) > 604800 {
+ dir := config.TmpBase + "/" + f.Name()
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453859877
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
+ if rc == 1 { // file is not part of any package.
+ return nil, nil
+ } else if rc == 0 { // add the package name the file belongs to.
+ log.Debugf("output from file-query: %s\n", string(output))
+ result = append(result, string(strings.TrimSpace(string(output))))
+ log.Debugf("result length: %d, result: %s\n", len(result), string(output))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-provides": // returns the package name that provides 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatprovides", name)
+ log.Debugf("pkg-provides - name: %s, output: %s\n", name, output)
+ if rc == 1 { // no package provides 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-query": // returns the package name for 'name'.
+ output, rc, err := ExecCommand("/bin/rpm", "-q", name)
+ if rc == 1 { // the package is not installed.
+ return nil, nil
+ } else if rc == 0 { // add the rpm name
+ result = append(result, string(strings.TrimSpace(string(output))))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-requires": // returns a list of packages that requires package 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatrequires", name)
+ if rc == 1 { // no package reuires package 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ }
+ return result, nil
+}
+
+func RandomDuration(max time.Duration) time.Duration {
+ rand.Seed(time.Now().UnixNano())
+ return time.Duration(rand.Int63n(int64(max)))
+}
+
+func CheckUser(cfg config.Cfg) bool {
+ result := true
+ user, _ := user.Current()
+
+ switch cfg.RunMode {
+ case config.BADASS:
+ fallthrough
+ case config.INTERACTIVE:
+ fallthrough
+ case config.SYNCDS:
+ if user.Username != "root" {
+ log.Errorf("Only the root user may run in BADASS, INTERACTIVE, or SYNCDS mode, current user: %s\n",
+ user.Username)
+ result = false
+ }
+ default:
+ log.Infof("current mode: %s, run user: %s\n", cfg.RunMode, user.Username)
+ }
+ return result
+}
+
+func CleanTmpDir() bool {
+ now := time.Now().Unix()
+
+ files, err := ioutil.ReadDir(config.TmpBase)
+ if err != nil {
+ log.Errorf("opening %s: %v\n", config.TmpBase, err)
+ return false
+ }
+ for _, f := range files {
+ // remove any directory and its contents under the config.TmpBase (/tmp/ort) that
+ // is more than a week old.
+ if f.IsDir() && (now-f.ModTime().Unix()) > 604800 {
+ dir := config.TmpBase + "/" + f.Name()
+ if f.Name() == "." || f.Name() == ".." {
+ continue
+ }
+ err = os.RemoveAll(dir)
Review comment:
safety check added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] ocket8888 commented on pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
ocket8888 commented on pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#issuecomment-656231025
Why is learning and working with the Python more work than learning and working with the Go? They're both not the Perl script.
FWIW, the Python script has the advantage of already being tested as good enough for a CiaB environment, and it was also tested once a while back on a production server. It wasn't much of a test, but it showed that with actual production data it can output correct configs (that was also before `atstccfg`, so means a bit less).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] ocket8888 commented on pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
ocket8888 commented on pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#issuecomment-656210808
I think the current plan for the future of ORT is in [the related blueprint](https://github.com/apache/trafficcontrol/blob/master/blueprints/ort-rewrite-unix-style.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453936197
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
Review comment:
no it's not. I've forgotten why I bothered with it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on pull request #4872: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#issuecomment-659710958
@rob05c I fixed the issue where Traffic Ops was not getting updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453962891
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
+ cfg, ok := r.configFiles[name]
+ return cfg, ok
+}
+
+// fetches and parses the multipart config files for a cache from traffic ops
+// and loads them into the configFiles map.
+func (r *TrafficOpsReq) GetConfigFileList() error {
+ var atsUid int = 0
+ var atsGid int = 0
+
+ atsUser, err := user.Lookup(config.TrafficServerOwner)
+ if err != nil {
+ log.Errorf("could not lookup the trafficserver, '%s', owner uid, using uid/gid 0")
+ } else {
+ atsUid, err = strconv.Atoi(atsUser.Uid)
+ if err != nil {
+ log.Errorf("could not parse the ats UID.")
+ atsUid = 0
+ }
+ atsGid, err = strconv.Atoi(atsUser.Gid)
+ if err != nil {
+ log.Errorf("could not parse the ats GID.")
+ atsUid = 0
+ }
+ }
+
+ fBytes, err := r.atsTcExec("get-config-files")
+ if err != nil {
+ return err
+ }
+ buf := bytes.NewBuffer(fBytes)
+ var hdr string
+ for {
+ hdr, err = buf.ReadString('\n')
+ if err != nil {
+ return err
+ } else {
+ if strings.HasPrefix(hdr, "Content-Type: multipart/mixed") {
+ break
+ }
+ }
+ }
+ // split on the ": " to trim off the leading space
+ har := strings.Split(hdr, ": ")
+ if har[0] != "Content-Type" || !strings.HasPrefix(har[1], "multipart/mixed") {
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#issuecomment-656212944
Command Usage:
`Usage: traffic_ops_t3c [options] <Traffic_Ops_URL> <Traffic_Ops_Login>
<Traffic_Ops_URL> = URL to Traffic Ops host. Example: https://trafficops.company.net
<Traffic_Ops_Login> Example: 'username:password'
[options]:
--dispersion=[time in seconds] | -D, [time in seconds] wait a random number between 0 and <time in seconds> before starting, default = 300s
--login-dispersion=[time in seconds] | -l, [time in seconds] wait a random number between 0 and <time in seconds> befor login, default = 0
--log-location-debug=[value] | -d [value], Where to log debugs. May be a file path, stdout, stderr, or null, default stdout
--log-location-error=[value] | -e [value], Where to log errors. May be a file path, stdout, stderr, or null, default stdout
--log-location-info=[value] | -i [value], Where to log info. May be a file path, stdout, stderr, or null, default stdout
--log-location-warning=[value] | -w [value], Where to log warnings. May be a file path, stdout, stderr, or null, default stdout
--run-mode=[mode] | -m [mode] where mode is one of [interactive | report | badass | syncds | revalidate ], default = report
--cache-hostname=[hostname] | -H [hostname], Host name of the cache to generate config for. Must be the server host name in Traffic Ops, not a URL, and not the FQDN
--num-retries=[number] | -r [number], retry connection to Traffic Ops URL [number] times, default is 3
--reval-wait-time=[seconds] | -T [seconds] wait a random number of seconds between 0 and [seconds] before revlidation, default is 60
--rev-proxy-disable=[true|false] | -P [true|false] bypass the reverse proxy even if one has been configured, default = false
--skip-os-check=[true|false] | -s [true | false] bypass the check for a supported CentOS version. default = false
--traffic-ops-timeout-milliseconds=[milliseconds] | -t [milliseconds] the Traffic Ops request timeout in milliseconds. Default = 30000 (30 seconds)
--traffic-ops-url=[url] | -u [url], Traffic Ops URL. Must be the full URL, including the scheme. Required. May also be set with the environment variable TO_URL
--traffic-ops-user=[username] | -U [username], Traffic Ops username. Required. May also be set with the environment variable TO_USER
--traffic-ops-password=[password] | -P [password], Traffic Ops password. Required. May also be set with the environment variable TO_PASS
--wait-for-parents | -w [true | false] do not update if parent_pending = 1 in the update json. default = true, wait for parents
--help | -h, Print usage information and exit`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453970461
##########
File path: traffic_ops_ort/traffic_ops_t3c/traffic_ops_t3c.go
##########
@@ -0,0 +1,169 @@
+package main
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/torequest"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "os"
+ "time"
+)
+
+func runSysctl(cfg config.Cfg) {
+ var ans bool
+
+ switch cfg.RunMode {
+ case config.INTERACTIVE:
+ ans, _ = util.AskYesNo("sysctl configuration has changed. 'sysctl -p' needs to be run. Should I do that now? (Y/[n])")
+ case config.BADASS:
+ ans = true
+ }
+
+ if ans == true {
+ _, rc, err := util.ExecCommand("/usr/sbin/sysctl", "-p")
Review comment:
This suggestion goes against security practices that I've used. This command is run by root and the user environment could be compromised and who knows what 'sysctl' does when 'env' or your PATH points to some compromised 'sysctl' say it /tmp/sysctl. I think it's unlikely that sysctl would get moved by some Distro but if it does, deal with it when it happens.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453889260
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
+ if rc == 1 { // file is not part of any package.
+ return nil, nil
+ } else if rc == 0 { // add the package name the file belongs to.
+ log.Debugf("output from file-query: %s\n", string(output))
+ result = append(result, string(strings.TrimSpace(string(output))))
+ log.Debugf("result length: %d, result: %s\n", len(result), string(output))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-provides": // returns the package name that provides 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatprovides", name)
+ log.Debugf("pkg-provides - name: %s, output: %s\n", name, output)
+ if rc == 1 { // no package provides 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-query": // returns the package name for 'name'.
+ output, rc, err := ExecCommand("/bin/rpm", "-q", name)
+ if rc == 1 { // the package is not installed.
+ return nil, nil
+ } else if rc == 0 { // add the rpm name
+ result = append(result, string(strings.TrimSpace(string(output))))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-requires": // returns a list of packages that requires package 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatrequires", name)
+ if rc == 1 { // no package reuires package 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ }
+ return result, nil
+}
+
+func RandomDuration(max time.Duration) time.Duration {
+ rand.Seed(time.Now().UnixNano())
+ return time.Duration(rand.Int63n(int64(max)))
+}
+
+func CheckUser(cfg config.Cfg) bool {
+ result := true
+ user, _ := user.Current()
Review comment:
changed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453977318
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
Review comment:
changed to using short declarations.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453984996
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
Review comment:
Yeah, I'm not really using the 'file-query' but, I am using the 'pkg-query' on plugin .so files only, not every file. But, I should cache those lookups. I'll make the change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453853532
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
+ "net/url"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+)
+
+const (
+ StatusDir = "/opt/ort/status"
+ AtsTcConfig = "/opt/ort/atstccfg"
+ Chkconfig = "/sbin/chkconfig"
+ Service = "/sbin/service"
+ SystemCtl = "/bin/systemctl"
+ TmpBase = "/tmp/ort"
+ TSHome = "/opt/trafficserver"
+ TrafficCtl = TSHome + "/bin/traffic_ctl"
+ TrafficServerOwner = "ats"
+)
+
+type Mode int
+
+const (
+ BADASS Mode = 0
+ INTERACTIVE = 1
+ REPORT = 2
+ REVALIDATE = 3
+ SYNCDS = 4
+)
+
+func (m Mode) String() string {
+ switch m {
+ case 0:
+ return "BADASS"
+ case 1:
+ return "INTERACTIVE"
+ case 2:
+ return "REPORT"
+ case 3:
+ return "REVALIDATE"
+ case 4:
+ return "SYNCDS"
+ }
+ return ""
+}
+
+type SvcManagement int
+
+const (
+ UNKNOWN = 0
+ SYSTEMD = 1
+ SYSTEMV = 2 // legacy System V Init.
+)
+
+func (s SvcManagement) String() string {
+ switch s {
+ case UNKNOWN:
+ return "UNKNOWN"
+ case SYSTEMD:
+ return "SYSTEMD"
+ case SYSTEMV:
+ return "SYSTEMV"
+ }
+ return "UNKNOWN"
+}
+
+type Cfg struct {
+ Dispersion time.Duration
+ LogLocationDebug string
+ LogLocationErr string
+ LogLocationInfo string
+ LogLocationWarn string
+ LoginDispersion time.Duration
+ CacheHostName string
+ SvcManagement SvcManagement
+ Retries int
+ RevalWaitTime time.Duration
+ ReverseProxyDisable bool
+ RunMode Mode
+ SkipOSCheck bool
+ TOTimeoutMS time.Duration
+ TOUser string
+ TOPass string
+ TOURL string
+ WaitForParents bool
+ YumOptions string
+}
+
+func (cfg Cfg) ErrorLog() log.LogLocation { return log.LogLocation(cfg.LogLocationErr) }
+func (cfg Cfg) WarningLog() log.LogLocation { return log.LogLocation(cfg.LogLocationWarn) }
+func (cfg Cfg) InfoLog() log.LogLocation { return log.LogLocation(cfg.LogLocationInfo) }
+func (cfg Cfg) DebugLog() log.LogLocation { return log.LogLocation(cfg.LogLocationDebug) }
+func (cfg Cfg) EventLog() log.LogLocation { return log.LogLocation(log.LogLocationNull) } // event logging is not used.
+
+func GetCfg() (Cfg, error) {
+ var err error
+
+ dispersionPtr := getopt.IntLong("dispersion", 'D', 300, "[seconds] wait a random number of seconds between 0 and [seconds] before starting, default 300")
+ loginDispersionPtr := getopt.IntLong("login-dispersion", 'l', 0, "[seconds] wait a random number of seconds betwee 0 and [seconds] befor login to traffic ops, default 0")
+ logLocationDebugPtr := getopt.StringLong("log-location-debug", 'd', "", "Where to log debugs. May be a file path, stdout, stderr, or null, default ''")
+ logLocationErrorPtr := getopt.StringLong("log-location-error", 'e', "stderr", "Where to log errors. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationInfoPtr := getopt.StringLong("log-location-info", 'i', "stderr", "Where to log info. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationWarnPtr := getopt.StringLong("log-location-warning", 'w', "stderr", "Where to log warnings. May be a file path, stdout, stderr, or null, default stderr")
+ cacheHostNamePtr := getopt.StringLong("cache-host-name", 'H', "", "Host name of the cache to generate config for. Must be the server host name in Traffic Ops, not a URL, and not the FQDN")
+ retriesPtr := getopt.IntLong("num-retries", 'r', 3, "[number] retry connection to Traffic Ops URL [number] times, default is 3")
+ revalWaitTimePtr := getopt.IntLong("reval-wait-time", 'T', 60, "[seconds] wait a random number of seconds between 0 and [seconds] before revlidation, default is 60")
+ reverseProxyDisablePtr := getopt.BoolLong("reverse-proxy-disable", 'p', "[false | true] bypass the reverse proxy even if one has been configured default is false")
+ runModePtr := getopt.StringLong("run-mode", 'm', "report", "[badass | interactive | report | revalidate | syncds] run mode, default is 'report'")
+ skipOSCheckPtr := getopt.BoolLong("skip-os-check", 's', "[false | true] skip os check, default is false")
+ toTimeoutMSPtr := getopt.IntLong("traffic-ops-timeout-milliseconds", 't', 30000, "Timeout in milli-seconds for Traffic Ops requests, default is 30000")
+ toURLPtr := getopt.StringLong("traffic-ops-url", 'u', "", "Traffic Ops URL. Must be the full URL, including the scheme. Required. May also be set with the environment variable TO_URL")
+ toUserPtr := getopt.StringLong("traffic-ops-user", 'U', "", "Traffic Ops username. Required. May also be set with the environment variable TO_USER")
+ toPassPtr := getopt.StringLong("traffic-ops-password", 'P', "", "Traffic Ops password. Required. May also be set with the environment variable TO_PASS")
+ waitForParentsPtr := getopt.BoolLong("wait-for-parents", 'W', "[true | false] do not update if parent_pending = 1 in the update json. default is false, wait for parents")
+ helpPtr := getopt.BoolLong("help", 'h', "Print usage information and exit")
+ getopt.Parse()
+
+ dispersion := time.Second * time.Duration(*dispersionPtr)
+ loginDispersion := time.Second * time.Duration(*loginDispersionPtr)
+ logLocationDebug := *logLocationDebugPtr
+ logLocationError := *logLocationErrorPtr
+ logLocationInfo := *logLocationInfoPtr
+ logLocationWarn := *logLocationWarnPtr
+
+ var cacheHostName string
+ if len(*cacheHostNamePtr) > 0 {
+ cacheHostName = *cacheHostNamePtr
+ } else {
+ cacheHostName, _ = os.Hostname()
+ }
+
+ retries := *retriesPtr
+ revalWaitTime := time.Second * time.Duration(*revalWaitTimePtr)
+ reverseProxyDisable := *reverseProxyDisablePtr
+ runMode := *runModePtr
+ skipOsCheck := *skipOSCheckPtr
+ toTimeoutMS := time.Millisecond * time.Duration(*toTimeoutMSPtr)
+ toURL := *toURLPtr
+ toUser := *toUserPtr
+ toPass := *toPassPtr
+ waitForParents := *waitForParentsPtr
+ help := *helpPtr
+
+ if help {
+ Usage()
+ os.Exit(0)
+ }
+
+ runModeStr := strings.ToUpper(runMode)
+ var _runMode Mode
+ switch runModeStr {
+ case "INTERACTIVE":
+ _runMode = INTERACTIVE
+ case "REPORT":
+ _runMode = REPORT
+ case "BADASS":
+ _runMode = BADASS
+ case "SYNCDS":
+ _runMode = SYNCDS
+ case "REVALIDATE":
+ _runMode = REVALIDATE
+ default:
+ fmt.Fprintf(os.Stderr, "%s is an invalid mode.\n", runModeStr)
+ Usage()
+ os.Exit(-1)
Review comment:
okay, done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453892127
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453938072
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
Review comment:
changed it and removed the function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453797611
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
+ "net/url"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+)
+
+const (
+ StatusDir = "/opt/ort/status"
+ AtsTcConfig = "/opt/ort/atstccfg"
+ Chkconfig = "/sbin/chkconfig"
+ Service = "/sbin/service"
+ SystemCtl = "/bin/systemctl"
+ TmpBase = "/tmp/ort"
+ TSHome = "/opt/trafficserver"
+ TrafficCtl = TSHome + "/bin/traffic_ctl"
+ TrafficServerOwner = "ats"
+)
+
+type Mode int
+
+const (
+ BADASS Mode = 0
Review comment:
Updated all const's such that they are in CamelCase.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford edited a comment on pull request #4872: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford edited a comment on pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#issuecomment-656212944
Command Usage is explained in the traffic_ops_ort/t3c/README.md
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rob05c commented on pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
rob05c commented on pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#issuecomment-656214197
>I think the current plan for the future of ORT is in the related blueprint.
Right, the goal here is to enable that. The Perl is extremely difficult to maintain and ensure correctness while modifying. Doing the blueprint wholesale would be a very large project, and difficult to ensure the end is correct and has parity. Doing the blueprint one piece at a time while continually modifying the Perl would likewise be difficult and dangerous.
This lets us do the Blueprint one piece at a time, carving out pieces of the now-Go script, and modifying the main/old script in Go to use each new, small piece will be much easier and safer than Perl.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] ocket8888 commented on pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
ocket8888 commented on pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#issuecomment-656214546
Okay, but why not use the transliteration from Perl to Python that already exists under `infrastructure/cdn-in-a-box/ort`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453933751
##########
File path: traffic_ops_ort/traffic_ops_t3c/traffic_ops_t3c.go
##########
@@ -0,0 +1,169 @@
+package main
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/torequest"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "os"
+ "time"
+)
+
+func runSysctl(cfg config.Cfg) {
+ var ans bool
+
+ switch cfg.RunMode {
+ case config.INTERACTIVE:
+ ans, _ = util.AskYesNo("sysctl configuration has changed. 'sysctl -p' needs to be run. Should I do that now? (Y/[n])")
+ case config.BADASS:
+ ans = true
+ }
+
+ if ans == true {
+ _, rc, err := util.ExecCommand("/usr/sbin/sysctl", "-p")
+ if err != nil {
+ log.Errorln("sysctl -p failed")
+ } else if rc == 0 {
+ log.Debugf("sysctl -p ran succesfully.")
+ }
+ }
+}
+
+func main() {
+ var syncdsUpdate torequest.UpdateStatus
+ var lock util.FileLock
+ cfg, err := config.GetCfg()
+ if err != nil {
+ fmt.Println(err)
+ os.Exit(-1)
+ }
+
+ trops := torequest.NewTrafficOpsReq(cfg)
+
+ // if doing os checks, insure there is a 'systemctl' or 'service' and 'chkconfig' commands.
+ if !cfg.SkipOSCheck && cfg.SvcManagement == config.UNKNOWN {
+ log.Errorln("OS checks are enabled and unable to find any know service management tools.")
+ }
+
+ // create and clean the config.TmpBase (/tmp/ort)
+ if !util.MkDir(config.TmpBase, cfg) {
+ os.Exit(1)
+ } else if !util.CleanTmpDir() {
+ os.Exit(1)
+ }
+ if cfg.RunMode != config.REPORT {
+ if !lock.GetLock(config.TmpBase + "/to_ort.lock") {
+ os.Exit(1)
+ }
+ }
+
+ fmt.Println(time.Now().Format(time.UnixDate))
+
+ if !util.CheckUser(cfg) {
+ lock.UnlockAndExit(-1)
+ }
+
+ toolName := trops.GetHeaderComment()
+ log.Debugf("toolname: %s\n", toolName)
+
+ // if running in REVALIDATE mode, check to see if it's
+ // necessary to continue
+ if cfg.RunMode == config.REVALIDATE {
+ syncdsUpdate, err = trops.CheckRevalidateState(false)
+ if err != nil || syncdsUpdate == torequest.UPDATE_TROPS_NOTNEEDED {
+ if err != nil {
+ log.Errorln(err)
+ }
+ os.Exit(1)
+ }
+ } else {
+ syncdsUpdate, err = trops.CheckSyncDSState()
+ if err != nil {
+ log.Errorln(err)
+ os.Exit(1)
Review comment:
ok, I added them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rob05c commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
rob05c commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453886080
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
Review comment:
Maybe you're thinking of Report Mode? They did ask that we keep a dedicated Report Mode.
But I asked and nobody object to removing Interactive, everyone I asked specifically said they've never used it.
But again, we can leave it for now if you want, it's easy to remove later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rob05c commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
rob05c commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r452945998
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
+ "net/url"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+)
+
+const (
+ StatusDir = "/opt/ort/status"
+ AtsTcConfig = "/opt/ort/atstccfg"
+ Chkconfig = "/sbin/chkconfig"
+ Service = "/sbin/service"
+ SystemCtl = "/bin/systemctl"
+ TmpBase = "/tmp/ort"
+ TSHome = "/opt/trafficserver"
+ TrafficCtl = TSHome + "/bin/traffic_ctl"
+ TrafficServerOwner = "ats"
+)
+
+type Mode int
+
+const (
+ BADASS Mode = 0
+ INTERACTIVE = 1
+ REPORT = 2
+ REVALIDATE = 3
+ SYNCDS = 4
+)
+
+func (m Mode) String() string {
+ switch m {
+ case 0:
+ return "BADASS"
+ case 1:
+ return "INTERACTIVE"
+ case 2:
+ return "REPORT"
+ case 3:
+ return "REVALIDATE"
+ case 4:
+ return "SYNCDS"
+ }
+ return ""
+}
+
+type SvcManagement int
+
+const (
+ UNKNOWN = 0
+ SYSTEMD = 1
+ SYSTEMV = 2 // legacy System V Init.
+)
+
+func (s SvcManagement) String() string {
+ switch s {
+ case UNKNOWN:
+ return "UNKNOWN"
+ case SYSTEMD:
+ return "SYSTEMD"
+ case SYSTEMV:
+ return "SYSTEMV"
+ }
+ return "UNKNOWN"
+}
+
+type Cfg struct {
+ Dispersion time.Duration
+ LogLocationDebug string
+ LogLocationErr string
+ LogLocationInfo string
+ LogLocationWarn string
+ LoginDispersion time.Duration
+ CacheHostName string
+ SvcManagement SvcManagement
+ Retries int
+ RevalWaitTime time.Duration
+ ReverseProxyDisable bool
+ RunMode Mode
+ SkipOSCheck bool
+ TOTimeoutMS time.Duration
+ TOUser string
+ TOPass string
+ TOURL string
+ WaitForParents bool
+ YumOptions string
+}
+
+func (cfg Cfg) ErrorLog() log.LogLocation { return log.LogLocation(cfg.LogLocationErr) }
+func (cfg Cfg) WarningLog() log.LogLocation { return log.LogLocation(cfg.LogLocationWarn) }
+func (cfg Cfg) InfoLog() log.LogLocation { return log.LogLocation(cfg.LogLocationInfo) }
+func (cfg Cfg) DebugLog() log.LogLocation { return log.LogLocation(cfg.LogLocationDebug) }
+func (cfg Cfg) EventLog() log.LogLocation { return log.LogLocation(log.LogLocationNull) } // event logging is not used.
+
+func GetCfg() (Cfg, error) {
+ var err error
+
+ dispersionPtr := getopt.IntLong("dispersion", 'D', 300, "[seconds] wait a random number of seconds between 0 and [seconds] before starting, default 300")
+ loginDispersionPtr := getopt.IntLong("login-dispersion", 'l', 0, "[seconds] wait a random number of seconds betwee 0 and [seconds] befor login to traffic ops, default 0")
+ logLocationDebugPtr := getopt.StringLong("log-location-debug", 'd', "", "Where to log debugs. May be a file path, stdout, stderr, or null, default ''")
+ logLocationErrorPtr := getopt.StringLong("log-location-error", 'e', "stderr", "Where to log errors. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationInfoPtr := getopt.StringLong("log-location-info", 'i', "stderr", "Where to log info. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationWarnPtr := getopt.StringLong("log-location-warning", 'w', "stderr", "Where to log warnings. May be a file path, stdout, stderr, or null, default stderr")
+ cacheHostNamePtr := getopt.StringLong("cache-host-name", 'H', "", "Host name of the cache to generate config for. Must be the server host name in Traffic Ops, not a URL, and not the FQDN")
+ retriesPtr := getopt.IntLong("num-retries", 'r', 3, "[number] retry connection to Traffic Ops URL [number] times, default is 3")
+ revalWaitTimePtr := getopt.IntLong("reval-wait-time", 'T', 60, "[seconds] wait a random number of seconds between 0 and [seconds] before revlidation, default is 60")
+ reverseProxyDisablePtr := getopt.BoolLong("reverse-proxy-disable", 'p', "[false | true] bypass the reverse proxy even if one has been configured default is false")
+ runModePtr := getopt.StringLong("run-mode", 'm', "report", "[badass | interactive | report | revalidate | syncds] run mode, default is 'report'")
+ skipOSCheckPtr := getopt.BoolLong("skip-os-check", 's', "[false | true] skip os check, default is false")
+ toTimeoutMSPtr := getopt.IntLong("traffic-ops-timeout-milliseconds", 't', 30000, "Timeout in milli-seconds for Traffic Ops requests, default is 30000")
+ toURLPtr := getopt.StringLong("traffic-ops-url", 'u', "", "Traffic Ops URL. Must be the full URL, including the scheme. Required. May also be set with the environment variable TO_URL")
+ toUserPtr := getopt.StringLong("traffic-ops-user", 'U', "", "Traffic Ops username. Required. May also be set with the environment variable TO_USER")
+ toPassPtr := getopt.StringLong("traffic-ops-password", 'P', "", "Traffic Ops password. Required. May also be set with the environment variable TO_PASS")
+ waitForParentsPtr := getopt.BoolLong("wait-for-parents", 'W', "[true | false] do not update if parent_pending = 1 in the update json. default is false, wait for parents")
+ helpPtr := getopt.BoolLong("help", 'h', "Print usage information and exit")
+ getopt.Parse()
+
+ dispersion := time.Second * time.Duration(*dispersionPtr)
+ loginDispersion := time.Second * time.Duration(*loginDispersionPtr)
+ logLocationDebug := *logLocationDebugPtr
+ logLocationError := *logLocationErrorPtr
+ logLocationInfo := *logLocationInfoPtr
+ logLocationWarn := *logLocationWarnPtr
+
+ var cacheHostName string
+ if len(*cacheHostNamePtr) > 0 {
+ cacheHostName = *cacheHostNamePtr
+ } else {
+ cacheHostName, _ = os.Hostname()
+ }
+
+ retries := *retriesPtr
+ revalWaitTime := time.Second * time.Duration(*revalWaitTimePtr)
+ reverseProxyDisable := *reverseProxyDisablePtr
+ runMode := *runModePtr
+ skipOsCheck := *skipOSCheckPtr
+ toTimeoutMS := time.Millisecond * time.Duration(*toTimeoutMSPtr)
+ toURL := *toURLPtr
+ toUser := *toUserPtr
+ toPass := *toPassPtr
+ waitForParents := *waitForParentsPtr
+ help := *helpPtr
+
+ if help {
+ Usage()
+ os.Exit(0)
+ }
+
+ runModeStr := strings.ToUpper(runMode)
+ var _runMode Mode
Review comment:
Personally, I prefer to always use the := syntax, `runMode := Mode(0)` because
(1) It lets you use a consistent declaration syntax, instead of having 2 different declaration syntaxes. It's always better to have One Right Way to do things.
(2) It makes the initialization explicit. It isn't always obvious what the zero-value is with var, but with := it's immediately visible.
But not everyone feels that way, so I won't push for := and avoiding var if you don't agree.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rob05c commented on pull request #4872: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
rob05c commented on pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#issuecomment-670628098
I'm merging this: it's a step toward the ORT Rewrite Blueprint, and I've tested and it seems to work as expected.
It's also fairly safe: it doesn't remove the old Perl, so users can keep using that while testing this, until we're confident enough to deprecate & remove the Perl. The only Production thing this changes is the RPM, adding t3c. I tested that thoroughly, the RPM builds and installs correctly for me, should cause no issues for anyone in production.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453853166
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
+ "net/url"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+)
+
+const (
+ StatusDir = "/opt/ort/status"
+ AtsTcConfig = "/opt/ort/atstccfg"
+ Chkconfig = "/sbin/chkconfig"
+ Service = "/sbin/service"
+ SystemCtl = "/bin/systemctl"
+ TmpBase = "/tmp/ort"
+ TSHome = "/opt/trafficserver"
+ TrafficCtl = TSHome + "/bin/traffic_ctl"
+ TrafficServerOwner = "ats"
+)
+
+type Mode int
+
+const (
+ BADASS Mode = 0
+ INTERACTIVE = 1
+ REPORT = 2
+ REVALIDATE = 3
+ SYNCDS = 4
+)
+
+func (m Mode) String() string {
+ switch m {
+ case 0:
+ return "BADASS"
+ case 1:
+ return "INTERACTIVE"
+ case 2:
+ return "REPORT"
+ case 3:
+ return "REVALIDATE"
+ case 4:
+ return "SYNCDS"
+ }
+ return ""
+}
+
+type SvcManagement int
+
+const (
+ UNKNOWN = 0
+ SYSTEMD = 1
+ SYSTEMV = 2 // legacy System V Init.
+)
+
+func (s SvcManagement) String() string {
+ switch s {
+ case UNKNOWN:
+ return "UNKNOWN"
+ case SYSTEMD:
+ return "SYSTEMD"
+ case SYSTEMV:
+ return "SYSTEMV"
+ }
+ return "UNKNOWN"
+}
+
+type Cfg struct {
+ Dispersion time.Duration
+ LogLocationDebug string
+ LogLocationErr string
+ LogLocationInfo string
+ LogLocationWarn string
+ LoginDispersion time.Duration
+ CacheHostName string
+ SvcManagement SvcManagement
+ Retries int
+ RevalWaitTime time.Duration
+ ReverseProxyDisable bool
+ RunMode Mode
+ SkipOSCheck bool
+ TOTimeoutMS time.Duration
+ TOUser string
+ TOPass string
+ TOURL string
+ WaitForParents bool
+ YumOptions string
+}
+
+func (cfg Cfg) ErrorLog() log.LogLocation { return log.LogLocation(cfg.LogLocationErr) }
+func (cfg Cfg) WarningLog() log.LogLocation { return log.LogLocation(cfg.LogLocationWarn) }
+func (cfg Cfg) InfoLog() log.LogLocation { return log.LogLocation(cfg.LogLocationInfo) }
+func (cfg Cfg) DebugLog() log.LogLocation { return log.LogLocation(cfg.LogLocationDebug) }
+func (cfg Cfg) EventLog() log.LogLocation { return log.LogLocation(log.LogLocationNull) } // event logging is not used.
+
+func GetCfg() (Cfg, error) {
+ var err error
+
+ dispersionPtr := getopt.IntLong("dispersion", 'D', 300, "[seconds] wait a random number of seconds between 0 and [seconds] before starting, default 300")
+ loginDispersionPtr := getopt.IntLong("login-dispersion", 'l', 0, "[seconds] wait a random number of seconds betwee 0 and [seconds] befor login to traffic ops, default 0")
+ logLocationDebugPtr := getopt.StringLong("log-location-debug", 'd', "", "Where to log debugs. May be a file path, stdout, stderr, or null, default ''")
+ logLocationErrorPtr := getopt.StringLong("log-location-error", 'e', "stderr", "Where to log errors. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationInfoPtr := getopt.StringLong("log-location-info", 'i', "stderr", "Where to log info. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationWarnPtr := getopt.StringLong("log-location-warning", 'w', "stderr", "Where to log warnings. May be a file path, stdout, stderr, or null, default stderr")
+ cacheHostNamePtr := getopt.StringLong("cache-host-name", 'H', "", "Host name of the cache to generate config for. Must be the server host name in Traffic Ops, not a URL, and not the FQDN")
+ retriesPtr := getopt.IntLong("num-retries", 'r', 3, "[number] retry connection to Traffic Ops URL [number] times, default is 3")
+ revalWaitTimePtr := getopt.IntLong("reval-wait-time", 'T', 60, "[seconds] wait a random number of seconds between 0 and [seconds] before revlidation, default is 60")
+ reverseProxyDisablePtr := getopt.BoolLong("reverse-proxy-disable", 'p', "[false | true] bypass the reverse proxy even if one has been configured default is false")
+ runModePtr := getopt.StringLong("run-mode", 'm', "report", "[badass | interactive | report | revalidate | syncds] run mode, default is 'report'")
+ skipOSCheckPtr := getopt.BoolLong("skip-os-check", 's', "[false | true] skip os check, default is false")
+ toTimeoutMSPtr := getopt.IntLong("traffic-ops-timeout-milliseconds", 't', 30000, "Timeout in milli-seconds for Traffic Ops requests, default is 30000")
+ toURLPtr := getopt.StringLong("traffic-ops-url", 'u', "", "Traffic Ops URL. Must be the full URL, including the scheme. Required. May also be set with the environment variable TO_URL")
+ toUserPtr := getopt.StringLong("traffic-ops-user", 'U', "", "Traffic Ops username. Required. May also be set with the environment variable TO_USER")
+ toPassPtr := getopt.StringLong("traffic-ops-password", 'P', "", "Traffic Ops password. Required. May also be set with the environment variable TO_PASS")
+ waitForParentsPtr := getopt.BoolLong("wait-for-parents", 'W', "[true | false] do not update if parent_pending = 1 in the update json. default is false, wait for parents")
+ helpPtr := getopt.BoolLong("help", 'h', "Print usage information and exit")
+ getopt.Parse()
+
+ dispersion := time.Second * time.Duration(*dispersionPtr)
+ loginDispersion := time.Second * time.Duration(*loginDispersionPtr)
+ logLocationDebug := *logLocationDebugPtr
+ logLocationError := *logLocationErrorPtr
+ logLocationInfo := *logLocationInfoPtr
+ logLocationWarn := *logLocationWarnPtr
+
+ var cacheHostName string
+ if len(*cacheHostNamePtr) > 0 {
+ cacheHostName = *cacheHostNamePtr
+ } else {
+ cacheHostName, _ = os.Hostname()
+ }
+
+ retries := *retriesPtr
+ revalWaitTime := time.Second * time.Duration(*revalWaitTimePtr)
+ reverseProxyDisable := *reverseProxyDisablePtr
+ runMode := *runModePtr
+ skipOsCheck := *skipOSCheckPtr
+ toTimeoutMS := time.Millisecond * time.Duration(*toTimeoutMSPtr)
+ toURL := *toURLPtr
+ toUser := *toUserPtr
+ toPass := *toPassPtr
+ waitForParents := *waitForParentsPtr
+ help := *helpPtr
+
+ if help {
+ Usage()
+ os.Exit(0)
+ }
+
+ runModeStr := strings.ToUpper(runMode)
+ var _runMode Mode
+ switch runModeStr {
+ case "INTERACTIVE":
+ _runMode = INTERACTIVE
+ case "REPORT":
+ _runMode = REPORT
+ case "BADASS":
+ _runMode = BADASS
+ case "SYNCDS":
+ _runMode = SYNCDS
+ case "REVALIDATE":
+ _runMode = REVALIDATE
+ default:
+ fmt.Fprintf(os.Stderr, "%s is an invalid mode.\n", runModeStr)
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r455189915
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,383 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "strings"
+ "time"
+)
+
+const OneWeek = 604800
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func DirectoryExists(dir string) (bool, os.FileInfo) {
+ info, err := os.Stat(dir)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return info.IsDir(), info
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ // the service command returns an error and return code of '3'
+ // if a service is loaded and not running. So if rc == 3, ignore
+ // the error
+ output, rc, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil && rc != 3 {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "cfg-files": // returns a list of the package configuration files.
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-c", name)
+ if rc == 1 { // rpm package for 'name' was not found.
+ return nil, nil
+ } else if rc == 0 { // add the package name the file belongs to.
+ log.Debugf("output from cfg-files query: %s\n", string(output))
+ files := strings.Split(string(output), "\n")
+ for ii := range files {
+ result = append(result, strings.TrimSpace(files[ii]))
+ }
+ log.Debugf("result length: %d, result: %s\n", len(result), string(output))
+ } else if err != nil {
+ return nil, err
+ }
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
+ if rc == 1 { // file is not part of any package.
+ return nil, nil
+ } else if rc == 0 { // add the package name the file belongs to.
+ log.Debugf("output from file-query: %s\n", string(output))
+ result = append(result, string(strings.TrimSpace(string(output))))
+ log.Debugf("result length: %d, result: %s\n", len(result), string(output))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-provides": // returns the package name that provides 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatprovides", name)
+ log.Debugf("pkg-provides - name: %s, output: %s\n", name, output)
+ if rc == 1 { // no package provides 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-query": // returns the package name for 'name'.
+ output, rc, err := ExecCommand("/bin/rpm", "-q", name)
+ if rc == 1 { // the package is not installed.
+ return nil, nil
+ } else if rc == 0 { // add the rpm name
+ result = append(result, string(strings.TrimSpace(string(output))))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-requires": // returns a list of packages that requires package 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatrequires", name)
+ if rc == 1 { // no package reuires package 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ }
+ return result, nil
+}
+
+func RandomDuration(max time.Duration) time.Duration {
+ rand.Seed(time.Now().UnixNano())
+ return time.Duration(rand.Int63n(int64(max)))
Review comment:
I've fixed this. Had a math issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453941797
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
+ cfg, ok := r.configFiles[name]
+ return cfg, ok
+}
+
+// fetches and parses the multipart config files for a cache from traffic ops
+// and loads them into the configFiles map.
+func (r *TrafficOpsReq) GetConfigFileList() error {
+ var atsUid int = 0
+ var atsGid int = 0
+
+ atsUser, err := user.Lookup(config.TrafficServerOwner)
+ if err != nil {
+ log.Errorf("could not lookup the trafficserver, '%s', owner uid, using uid/gid 0")
+ } else {
+ atsUid, err = strconv.Atoi(atsUser.Uid)
+ if err != nil {
+ log.Errorf("could not parse the ats UID.")
+ atsUid = 0
+ }
+ atsGid, err = strconv.Atoi(atsUser.Gid)
+ if err != nil {
+ log.Errorf("could not parse the ats GID.")
+ atsUid = 0
+ }
+ }
+
+ fBytes, err := r.atsTcExec("get-config-files")
+ if err != nil {
+ return err
+ }
+ buf := bytes.NewBuffer(fBytes)
+ var hdr string
+ for {
+ hdr, err = buf.ReadString('\n')
+ if err != nil {
+ return err
+ } else {
+ if strings.HasPrefix(hdr, "Content-Type: multipart/mixed") {
+ break
+ }
+ }
+ }
+ // split on the ": " to trim off the leading space
+ har := strings.Split(hdr, ": ")
+ if har[0] != "Content-Type" || !strings.HasPrefix(har[1], "multipart/mixed") {
+ return errors.New("invalid config file data received from Traffic Ops, not in multipart format.")
+ }
+ msg := &mail.Message{
+ Header: map[string][]string{har[0]: {har[1]}},
+ Body: bytes.NewReader(buf.Bytes()),
+ }
+ mediaType, params, err := mime.ParseMediaType(msg.Header.Get("Content-Type"))
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453915944
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453881534
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
Review comment:
I thought I read that Cox wants to keep it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r454555122
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
Review comment:
vendored
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#issuecomment-656263264
> I understand that there is a python version of ORT but IMO that is not a production piece of code
Right, Go blows Python out of the water in terms of performance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453896794
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
Review comment:
added space trimming.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r454000886
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
+ cfg, ok := r.configFiles[name]
+ return cfg, ok
+}
+
+// fetches and parses the multipart config files for a cache from traffic ops
+// and loads them into the configFiles map.
+func (r *TrafficOpsReq) GetConfigFileList() error {
+ var atsUid int = 0
+ var atsGid int = 0
+
+ atsUser, err := user.Lookup(config.TrafficServerOwner)
+ if err != nil {
+ log.Errorf("could not lookup the trafficserver, '%s', owner uid, using uid/gid 0")
+ } else {
+ atsUid, err = strconv.Atoi(atsUser.Uid)
+ if err != nil {
+ log.Errorf("could not parse the ats UID.")
+ atsUid = 0
+ }
+ atsGid, err = strconv.Atoi(atsUser.Gid)
+ if err != nil {
+ log.Errorf("could not parse the ats GID.")
+ atsUid = 0
+ }
+ }
+
+ fBytes, err := r.atsTcExec("get-config-files")
+ if err != nil {
+ return err
+ }
+ buf := bytes.NewBuffer(fBytes)
+ var hdr string
+ for {
+ hdr, err = buf.ReadString('\n')
+ if err != nil {
+ return err
+ } else {
+ if strings.HasPrefix(hdr, "Content-Type: multipart/mixed") {
+ break
+ }
+ }
+ }
+ // split on the ": " to trim off the leading space
+ har := strings.Split(hdr, ": ")
+ if har[0] != "Content-Type" || !strings.HasPrefix(har[1], "multipart/mixed") {
+ return errors.New("invalid config file data received from Traffic Ops, not in multipart format.")
+ }
+ msg := &mail.Message{
+ Header: map[string][]string{har[0]: {har[1]}},
+ Body: bytes.NewReader(buf.Bytes()),
+ }
+ mediaType, params, err := mime.ParseMediaType(msg.Header.Get("Content-Type"))
+ if err != nil {
+ return err
+ }
+ if strings.HasPrefix(mediaType, "multipart/") {
+ mr := multipart.NewReader(msg.Body, params["boundary"])
+ // if the configFiles map is not empty, create a new map
+ // and the old map will be garbage collected.
+ if len(r.configFiles) > 0 {
+ log.Infoln("intializing a new configFiles map")
+ r.configFiles = make(map[string]*ConfigFile)
+ }
+ for {
+ var cf *ConfigFile
+
+ p, err := mr.NextPart()
+ if err == io.EOF {
+ return nil
+ }
+ if err != nil {
+ return err
+ }
+ dataBytes, err := ioutil.ReadAll(p)
+ if err != nil {
+ return err
+ }
+ path := p.Header.Get("Path")
+ if path != "" {
+ cf = new(ConfigFile)
Review comment:
ahh, I'll make the change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453996812
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
Review comment:
checkPlugin() caches the lookup from the rpm command so that a .so in remap.config is only looked up by rpm once.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453874382
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
+ if rc == 1 { // file is not part of any package.
+ return nil, nil
+ } else if rc == 0 { // add the package name the file belongs to.
+ log.Debugf("output from file-query: %s\n", string(output))
+ result = append(result, string(strings.TrimSpace(string(output))))
+ log.Debugf("result length: %d, result: %s\n", len(result), string(output))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-provides": // returns the package name that provides 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatprovides", name)
+ log.Debugf("pkg-provides - name: %s, output: %s\n", name, output)
+ if rc == 1 { // no package provides 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-query": // returns the package name for 'name'.
+ output, rc, err := ExecCommand("/bin/rpm", "-q", name)
+ if rc == 1 { // the package is not installed.
+ return nil, nil
+ } else if rc == 0 { // add the rpm name
+ result = append(result, string(strings.TrimSpace(string(output))))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-requires": // returns a list of packages that requires package 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatrequires", name)
+ if rc == 1 { // no package reuires package 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ }
+ return result, nil
+}
+
+func RandomDuration(max time.Duration) time.Duration {
+ rand.Seed(time.Now().UnixNano())
+ return time.Duration(rand.Int63n(int64(max)))
+}
+
+func CheckUser(cfg config.Cfg) bool {
+ result := true
+ user, _ := user.Current()
+
+ switch cfg.RunMode {
+ case config.BADASS:
+ fallthrough
+ case config.INTERACTIVE:
+ fallthrough
+ case config.SYNCDS:
+ if user.Username != "root" {
+ log.Errorf("Only the root user may run in BADASS, INTERACTIVE, or SYNCDS mode, current user: %s\n",
+ user.Username)
+ result = false
+ }
+ default:
+ log.Infof("current mode: %s, run user: %s\n", cfg.RunMode, user.Username)
+ }
+ return result
+}
+
+func CleanTmpDir() bool {
+ now := time.Now().Unix()
+
+ files, err := ioutil.ReadDir(config.TmpBase)
+ if err != nil {
+ log.Errorf("opening %s: %v\n", config.TmpBase, err)
+ return false
+ }
+ for _, f := range files {
+ // remove any directory and its contents under the config.TmpBase (/tmp/ort) that
+ // is more than a week old.
+ if f.IsDir() && (now-f.ModTime().Unix()) > 604800 {
+ dir := config.TmpBase + "/" + f.Name()
Review comment:
ok, I'll use filepath.Join
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453935000
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
+ cfg, ok := r.configFiles[name]
+ return cfg, ok
+}
+
+// fetches and parses the multipart config files for a cache from traffic ops
+// and loads them into the configFiles map.
+func (r *TrafficOpsReq) GetConfigFileList() error {
+ var atsUid int = 0
+ var atsGid int = 0
+
+ atsUser, err := user.Lookup(config.TrafficServerOwner)
+ if err != nil {
+ log.Errorf("could not lookup the trafficserver, '%s', owner uid, using uid/gid 0")
+ } else {
+ atsUid, err = strconv.Atoi(atsUser.Uid)
+ if err != nil {
+ log.Errorf("could not parse the ats UID.")
+ atsUid = 0
+ }
+ atsGid, err = strconv.Atoi(atsUser.Gid)
+ if err != nil {
+ log.Errorf("could not parse the ats GID.")
+ atsUid = 0
+ }
+ }
+
+ fBytes, err := r.atsTcExec("get-config-files")
+ if err != nil {
+ return err
+ }
+ buf := bytes.NewBuffer(fBytes)
+ var hdr string
+ for {
+ hdr, err = buf.ReadString('\n')
+ if err != nil {
+ return err
+ } else {
+ if strings.HasPrefix(hdr, "Content-Type: multipart/mixed") {
+ break
+ }
+ }
+ }
+ // split on the ": " to trim off the leading space
+ har := strings.Split(hdr, ": ")
+ if har[0] != "Content-Type" || !strings.HasPrefix(har[1], "multipart/mixed") {
Review comment:
updated ad I added multipart/mixed to lib/go-rfc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453939922
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rob05c merged pull request #4872: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
rob05c merged pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453934148
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
+ "net/url"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+)
+
+const (
+ StatusDir = "/opt/ort/status"
+ AtsTcConfig = "/opt/ort/atstccfg"
+ Chkconfig = "/sbin/chkconfig"
+ Service = "/sbin/service"
+ SystemCtl = "/bin/systemctl"
+ TmpBase = "/tmp/ort"
+ TSHome = "/opt/trafficserver"
+ TrafficCtl = TSHome + "/bin/traffic_ctl"
+ TrafficServerOwner = "ats"
+)
+
+type Mode int
+
+const (
+ BADASS Mode = 0
+ INTERACTIVE = 1
+ REPORT = 2
+ REVALIDATE = 3
+ SYNCDS = 4
+)
+
+func (m Mode) String() string {
+ switch m {
+ case 0:
+ return "BADASS"
+ case 1:
+ return "INTERACTIVE"
+ case 2:
+ return "REPORT"
+ case 3:
+ return "REVALIDATE"
+ case 4:
+ return "SYNCDS"
+ }
+ return ""
+}
+
+type SvcManagement int
+
+const (
+ UNKNOWN = 0
+ SYSTEMD = 1
+ SYSTEMV = 2 // legacy System V Init.
+)
+
+func (s SvcManagement) String() string {
+ switch s {
+ case UNKNOWN:
+ return "UNKNOWN"
+ case SYSTEMD:
+ return "SYSTEMD"
+ case SYSTEMV:
+ return "SYSTEMV"
+ }
+ return "UNKNOWN"
+}
+
+type Cfg struct {
+ Dispersion time.Duration
+ LogLocationDebug string
+ LogLocationErr string
+ LogLocationInfo string
+ LogLocationWarn string
+ LoginDispersion time.Duration
+ CacheHostName string
+ SvcManagement SvcManagement
+ Retries int
+ RevalWaitTime time.Duration
+ ReverseProxyDisable bool
+ RunMode Mode
+ SkipOSCheck bool
+ TOTimeoutMS time.Duration
+ TOUser string
+ TOPass string
+ TOURL string
+ WaitForParents bool
+ YumOptions string
+}
+
+func (cfg Cfg) ErrorLog() log.LogLocation { return log.LogLocation(cfg.LogLocationErr) }
+func (cfg Cfg) WarningLog() log.LogLocation { return log.LogLocation(cfg.LogLocationWarn) }
+func (cfg Cfg) InfoLog() log.LogLocation { return log.LogLocation(cfg.LogLocationInfo) }
+func (cfg Cfg) DebugLog() log.LogLocation { return log.LogLocation(cfg.LogLocationDebug) }
+func (cfg Cfg) EventLog() log.LogLocation { return log.LogLocation(log.LogLocationNull) } // event logging is not used.
+
+func GetCfg() (Cfg, error) {
+ var err error
+
+ dispersionPtr := getopt.IntLong("dispersion", 'D', 300, "[seconds] wait a random number of seconds between 0 and [seconds] before starting, default 300")
+ loginDispersionPtr := getopt.IntLong("login-dispersion", 'l', 0, "[seconds] wait a random number of seconds betwee 0 and [seconds] befor login to traffic ops, default 0")
+ logLocationDebugPtr := getopt.StringLong("log-location-debug", 'd', "", "Where to log debugs. May be a file path, stdout, stderr, or null, default ''")
+ logLocationErrorPtr := getopt.StringLong("log-location-error", 'e', "stderr", "Where to log errors. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationInfoPtr := getopt.StringLong("log-location-info", 'i', "stderr", "Where to log info. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationWarnPtr := getopt.StringLong("log-location-warning", 'w', "stderr", "Where to log warnings. May be a file path, stdout, stderr, or null, default stderr")
+ cacheHostNamePtr := getopt.StringLong("cache-host-name", 'H', "", "Host name of the cache to generate config for. Must be the server host name in Traffic Ops, not a URL, and not the FQDN")
+ retriesPtr := getopt.IntLong("num-retries", 'r', 3, "[number] retry connection to Traffic Ops URL [number] times, default is 3")
+ revalWaitTimePtr := getopt.IntLong("reval-wait-time", 'T', 60, "[seconds] wait a random number of seconds between 0 and [seconds] before revlidation, default is 60")
+ reverseProxyDisablePtr := getopt.BoolLong("reverse-proxy-disable", 'p', "[false | true] bypass the reverse proxy even if one has been configured default is false")
+ runModePtr := getopt.StringLong("run-mode", 'm', "report", "[badass | interactive | report | revalidate | syncds] run mode, default is 'report'")
+ skipOSCheckPtr := getopt.BoolLong("skip-os-check", 's', "[false | true] skip os check, default is false")
+ toTimeoutMSPtr := getopt.IntLong("traffic-ops-timeout-milliseconds", 't', 30000, "Timeout in milli-seconds for Traffic Ops requests, default is 30000")
+ toURLPtr := getopt.StringLong("traffic-ops-url", 'u', "", "Traffic Ops URL. Must be the full URL, including the scheme. Required. May also be set with the environment variable TO_URL")
+ toUserPtr := getopt.StringLong("traffic-ops-user", 'U', "", "Traffic Ops username. Required. May also be set with the environment variable TO_USER")
+ toPassPtr := getopt.StringLong("traffic-ops-password", 'P', "", "Traffic Ops password. Required. May also be set with the environment variable TO_PASS")
+ waitForParentsPtr := getopt.BoolLong("wait-for-parents", 'W', "[true | false] do not update if parent_pending = 1 in the update json. default is false, wait for parents")
+ helpPtr := getopt.BoolLong("help", 'h', "Print usage information and exit")
+ getopt.Parse()
+
+ dispersion := time.Second * time.Duration(*dispersionPtr)
+ loginDispersion := time.Second * time.Duration(*loginDispersionPtr)
+ logLocationDebug := *logLocationDebugPtr
+ logLocationError := *logLocationErrorPtr
+ logLocationInfo := *logLocationInfoPtr
+ logLocationWarn := *logLocationWarnPtr
+
+ var cacheHostName string
+ if len(*cacheHostNamePtr) > 0 {
+ cacheHostName = *cacheHostNamePtr
+ } else {
+ cacheHostName, _ = os.Hostname()
+ }
+
+ retries := *retriesPtr
+ revalWaitTime := time.Second * time.Duration(*revalWaitTimePtr)
+ reverseProxyDisable := *reverseProxyDisablePtr
+ runMode := *runModePtr
+ skipOsCheck := *skipOSCheckPtr
+ toTimeoutMS := time.Millisecond * time.Duration(*toTimeoutMSPtr)
+ toURL := *toURLPtr
+ toUser := *toUserPtr
+ toPass := *toPassPtr
+ waitForParents := *waitForParentsPtr
+ help := *helpPtr
+
+ if help {
+ Usage()
+ os.Exit(0)
+ }
+
+ runModeStr := strings.ToUpper(runMode)
+ var _runMode Mode
+ switch runModeStr {
+ case "INTERACTIVE":
+ _runMode = INTERACTIVE
+ case "REPORT":
+ _runMode = REPORT
+ case "BADASS":
+ _runMode = BADASS
+ case "SYNCDS":
+ _runMode = SYNCDS
+ case "REVALIDATE":
+ _runMode = REVALIDATE
+ default:
+ fmt.Fprintf(os.Stderr, "%s is an invalid mode.\n", runModeStr)
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r454497493
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
Review comment:
Okay, interactive mode is history.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453897502
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r455190322
##########
File path: traffic_ops_ort/traffic_ops_t3c/config/config.go
##########
@@ -0,0 +1,343 @@
+package config
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/pborman/getopt/v2"
+ "net/url"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+)
+
+const (
+ StatusDir = "/opt/ort/status"
+ AtsTcConfig = "/opt/ort/atstccfg"
+ Chkconfig = "/sbin/chkconfig"
+ Service = "/sbin/service"
+ SystemCtl = "/bin/systemctl"
+ TmpBase = "/tmp/ort"
+ TSHome = "/opt/trafficserver"
+ TrafficCtl = TSHome + "/bin/traffic_ctl"
+ TrafficServerOwner = "ats"
+)
+
+type Mode int
+
+const (
+ BADASS Mode = 0
+ INTERACTIVE = 1
+ REPORT = 2
+ REVALIDATE = 3
+ SYNCDS = 4
+)
+
+func (m Mode) String() string {
+ switch m {
+ case 0:
+ return "BADASS"
+ case 1:
+ return "INTERACTIVE"
+ case 2:
+ return "REPORT"
+ case 3:
+ return "REVALIDATE"
+ case 4:
+ return "SYNCDS"
+ }
+ return ""
+}
+
+type SvcManagement int
+
+const (
+ UNKNOWN = 0
+ SYSTEMD = 1
+ SYSTEMV = 2 // legacy System V Init.
+)
+
+func (s SvcManagement) String() string {
+ switch s {
+ case UNKNOWN:
+ return "UNKNOWN"
+ case SYSTEMD:
+ return "SYSTEMD"
+ case SYSTEMV:
+ return "SYSTEMV"
+ }
+ return "UNKNOWN"
+}
+
+type Cfg struct {
+ Dispersion time.Duration
+ LogLocationDebug string
+ LogLocationErr string
+ LogLocationInfo string
+ LogLocationWarn string
+ LoginDispersion time.Duration
+ CacheHostName string
+ SvcManagement SvcManagement
+ Retries int
+ RevalWaitTime time.Duration
+ ReverseProxyDisable bool
+ RunMode Mode
+ SkipOSCheck bool
+ TOTimeoutMS time.Duration
+ TOUser string
+ TOPass string
+ TOURL string
+ WaitForParents bool
+ YumOptions string
+}
+
+func (cfg Cfg) ErrorLog() log.LogLocation { return log.LogLocation(cfg.LogLocationErr) }
+func (cfg Cfg) WarningLog() log.LogLocation { return log.LogLocation(cfg.LogLocationWarn) }
+func (cfg Cfg) InfoLog() log.LogLocation { return log.LogLocation(cfg.LogLocationInfo) }
+func (cfg Cfg) DebugLog() log.LogLocation { return log.LogLocation(cfg.LogLocationDebug) }
+func (cfg Cfg) EventLog() log.LogLocation { return log.LogLocation(log.LogLocationNull) } // event logging is not used.
+
+func GetCfg() (Cfg, error) {
+ var err error
+
+ dispersionPtr := getopt.IntLong("dispersion", 'D', 300, "[seconds] wait a random number of seconds between 0 and [seconds] before starting, default 300")
+ loginDispersionPtr := getopt.IntLong("login-dispersion", 'l', 0, "[seconds] wait a random number of seconds betwee 0 and [seconds] befor login to traffic ops, default 0")
+ logLocationDebugPtr := getopt.StringLong("log-location-debug", 'd', "", "Where to log debugs. May be a file path, stdout, stderr, or null, default ''")
+ logLocationErrorPtr := getopt.StringLong("log-location-error", 'e', "stderr", "Where to log errors. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationInfoPtr := getopt.StringLong("log-location-info", 'i', "stderr", "Where to log info. May be a file path, stdout, stderr, or null, default stderr")
+ logLocationWarnPtr := getopt.StringLong("log-location-warning", 'w', "stderr", "Where to log warnings. May be a file path, stdout, stderr, or null, default stderr")
+ cacheHostNamePtr := getopt.StringLong("cache-host-name", 'H', "", "Host name of the cache to generate config for. Must be the server host name in Traffic Ops, not a URL, and not the FQDN")
+ retriesPtr := getopt.IntLong("num-retries", 'r', 3, "[number] retry connection to Traffic Ops URL [number] times, default is 3")
+ revalWaitTimePtr := getopt.IntLong("reval-wait-time", 'T', 60, "[seconds] wait a random number of seconds between 0 and [seconds] before revlidation, default is 60")
+ reverseProxyDisablePtr := getopt.BoolLong("reverse-proxy-disable", 'p', "[false | true] bypass the reverse proxy even if one has been configured default is false")
+ runModePtr := getopt.StringLong("run-mode", 'm', "report", "[badass | interactive | report | revalidate | syncds] run mode, default is 'report'")
+ skipOSCheckPtr := getopt.BoolLong("skip-os-check", 's', "[false | true] skip os check, default is false")
+ toTimeoutMSPtr := getopt.IntLong("traffic-ops-timeout-milliseconds", 't', 30000, "Timeout in milli-seconds for Traffic Ops requests, default is 30000")
+ toURLPtr := getopt.StringLong("traffic-ops-url", 'u', "", "Traffic Ops URL. Must be the full URL, including the scheme. Required. May also be set with the environment variable TO_URL")
+ toUserPtr := getopt.StringLong("traffic-ops-user", 'U', "", "Traffic Ops username. Required. May also be set with the environment variable TO_USER")
+ toPassPtr := getopt.StringLong("traffic-ops-password", 'P', "", "Traffic Ops password. Required. May also be set with the environment variable TO_PASS")
+ waitForParentsPtr := getopt.BoolLong("wait-for-parents", 'W', "[true | false] do not update if parent_pending = 1 in the update json. default is false, wait for parents")
+ helpPtr := getopt.BoolLong("help", 'h', "Print usage information and exit")
+ getopt.Parse()
+
+ dispersion := time.Second * time.Duration(*dispersionPtr)
+ loginDispersion := time.Second * time.Duration(*loginDispersionPtr)
+ logLocationDebug := *logLocationDebugPtr
+ logLocationError := *logLocationErrorPtr
+ logLocationInfo := *logLocationInfoPtr
+ logLocationWarn := *logLocationWarnPtr
+
+ var cacheHostName string
+ if len(*cacheHostNamePtr) > 0 {
+ cacheHostName = *cacheHostNamePtr
+ } else {
+ cacheHostName, _ = os.Hostname()
+ }
+
+ retries := *retriesPtr
+ revalWaitTime := time.Second * time.Duration(*revalWaitTimePtr)
+ reverseProxyDisable := *reverseProxyDisablePtr
+ runMode := *runModePtr
+ skipOsCheck := *skipOSCheckPtr
+ toTimeoutMS := time.Millisecond * time.Duration(*toTimeoutMSPtr)
+ toURL := *toURLPtr
+ toUser := *toUserPtr
+ toPass := *toPassPtr
+ waitForParents := *waitForParentsPtr
+ help := *helpPtr
+
+ if help {
+ Usage()
+ os.Exit(0)
+ }
+
+ runModeStr := strings.ToUpper(runMode)
+ var _runMode Mode
+ switch runModeStr {
+ case "INTERACTIVE":
+ _runMode = INTERACTIVE
+ case "REPORT":
+ _runMode = REPORT
+ case "BADASS":
+ _runMode = BADASS
+ case "SYNCDS":
+ _runMode = SYNCDS
+ case "REVALIDATE":
+ _runMode = REVALIDATE
+ default:
+ fmt.Fprintf(os.Stderr, "%s is an invalid mode.\n", runModeStr)
+ Usage()
+ os.Exit(-1)
+ }
+
+ urlSourceStr := "argument" // for error messages
+ if toURL == "" {
+ urlSourceStr = "environment variable"
+ toURL = os.Getenv("TO_URL")
+ }
+ if toUser == "" {
+ toUser = os.Getenv("TO_USER")
+ }
+ if toPass == "" {
+ toPass = os.Getenv("TO_PASS")
+ }
+
+ usageStr := "basic usage: traffic_ops_t3c --traffic-ops-url=myurl --traffic-ops-user=myuser --traffic-ops-password=mypass --cache-host-name=my-cache"
+ if strings.TrimSpace(toURL) == "" {
+ return Cfg{}, errors.New("Missing required argument --traffic-ops-url or TO_URL environment variable. " + usageStr)
+ }
+ if strings.TrimSpace(toUser) == "" {
+ return Cfg{}, errors.New("Missing required argument --traffic-ops-user or TO_USER environment variable. " + usageStr)
+ }
+ if strings.TrimSpace(toPass) == "" {
+ return Cfg{}, errors.New("Missing required argument --traffic-ops-password or TO_PASS environment variable. " + usageStr)
+ }
+ if strings.TrimSpace(cacheHostName) == "" {
+ return Cfg{}, errors.New("Missing required argument --cache-host-name. " + usageStr)
+ }
+
+ toURLParsed, err := url.Parse(toURL)
+ if err != nil {
+ return Cfg{}, errors.New("parsing Traffic Ops URL from " + urlSourceStr + " '" + toURL + "': " + err.Error())
+ } else if err = validateURL(toURLParsed); err != nil {
+ return Cfg{}, errors.New("invalid Traffic Ops URL from " + urlSourceStr + " '" + toURL + "': " + err.Error())
+ }
+
+ svcManagement := getOSSvcManagement()
+ yumOptions := os.Getenv("YUM_OPTIONS")
+
+ cfg := Cfg{
+ Dispersion: dispersion,
+ LogLocationDebug: logLocationDebug,
+ LogLocationErr: logLocationError,
+ LogLocationInfo: logLocationInfo,
+ LogLocationWarn: logLocationWarn,
+ LoginDispersion: time.Second * time.Duration(loginDispersion),
+ CacheHostName: cacheHostName,
+ SvcManagement: svcManagement,
+ Retries: retries,
+ RevalWaitTime: revalWaitTime,
+ ReverseProxyDisable: reverseProxyDisable,
+ RunMode: _runMode,
+ SkipOSCheck: skipOsCheck,
+ TOTimeoutMS: toTimeoutMS,
+ TOUser: toUser,
+ TOPass: toPass,
+ TOURL: toURL,
+ WaitForParents: waitForParents,
+ YumOptions: yumOptions,
+ }
+
+ if err = log.InitCfg(cfg); err != nil {
+ return Cfg{}, errors.New("Initializing loggers: " + err.Error() + "\n")
+ }
+
+ printConfig(cfg)
+
+ return cfg, nil
+}
+
+func validateURL(u *url.URL) error {
+ if u == nil {
+ return errors.New("nil url")
+ }
+ if u.Scheme != "http" && u.Scheme != "https" {
+ return errors.New("scheme expected 'http' or 'https', actual '" + u.Scheme + "'")
+ }
+ if strings.TrimSpace(u.Host) == "" {
+ return errors.New("no host")
+ }
+ return nil
+}
+
+func isCommandAvailable(name string) bool {
+ cmd := exec.Command("/bin/sh", "-c", name, "--version")
+ if err := cmd.Run(); err != nil {
+ return false
+ }
+
+ return true
+}
+
+func getOSSvcManagement() SvcManagement {
+ var _svcManager SvcManagement
+
+ if isCommandAvailable(SystemCtl) {
+ _svcManager = SYSTEMD
+ } else if isCommandAvailable(Service) {
+ _svcManager = SYSTEMV
+ }
+ if !isCommandAvailable(Chkconfig) {
+ return UNKNOWN
+ }
+
+ // we have what we need
+ return _svcManager
+}
+
+func printConfig(cfg Cfg) {
+ log.Debugf("Dispersion: %d\n", cfg.Dispersion)
+ log.Debugf("LogLocationDebug: %s\n", cfg.LogLocationDebug)
+ log.Debugf("LogLocationErr: %s\n", cfg.LogLocationErr)
+ log.Debugf("LogLocationInfo: %s\n", cfg.LogLocationInfo)
+ log.Debugf("LogLocationWarn: %s\n", cfg.LogLocationWarn)
+ log.Debugf("LoginDispersion: %d\n", cfg.LoginDispersion)
+ log.Debugf("CacheHostName: %s\n", cfg.CacheHostName)
+ log.Debugf("SvcManagement: %s\n", cfg.SvcManagement)
+ log.Debugf("Retries: %d\n", cfg.Retries)
+ log.Debugf("RevalWaitTime: %d\n", cfg.RevalWaitTime)
+ log.Debugf("ReverseProxyDisable: %t\n", cfg.ReverseProxyDisable)
+ log.Debugf("RunMode: %s\n", cfg.RunMode)
+ log.Debugf("SkipOSCheck: %t\n", cfg.SkipOSCheck)
+ log.Debugf("TOTimeoutMS: %d\n", cfg.TOTimeoutMS)
+ log.Debugf("TOUser: %s\n", cfg.TOUser)
+ log.Debugf("TOPass: %s\n", cfg.TOPass)
+ log.Debugf("TOURL: %s\n", cfg.TOURL)
+ log.Debugf("WaitForParents: %t\n", cfg.WaitForParents)
+ log.Debugf("YumOptions: %s\n", cfg.YumOptions)
+}
+
+func Usage() {
Review comment:
I don't like the pborman PrintUsage, mine has more info and would like to keep it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rob05c commented on pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
rob05c commented on pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#issuecomment-656247425
>It also works without systemd. Which means it can work in CiaB. If we transliterate again, then Python will still need to stick around anyway, unless the new one can also run without systemd.
>This go version does manage rpm installation and does require /sbin/service and /bin/systemctl to insure that systemd loads/reloads 'service' files
The Blueprint breaks it up, so the part that interacts with SystemD should be one small app.
We should write the overall "ort" app that binds them together in such a way that it's easy (maybe a flag?) to omit those calls, so the whole script works without SystemD. Not an immediate concern, by-design that shouldn't be difficult to do after the fact. But something we should keep in mind.
ATS can be reloaded and restarted by calling it directly. Maybe have a flag with the option to do SystemD, ATS Directly, or none?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] ocket8888 edited a comment on pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
ocket8888 edited a comment on pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#issuecomment-656231025
Why is learning and working with the Python more work than learning and working with the Go? They're both not the Perl script.
FWIW, the Python script has the advantage of already being tested as good enough for a CiaB environment, and it was also tested once a while back on a production server. It wasn't much of a test, but it showed that with actual production data it can output correct configs (that was also before `atstccfg`, so means a bit less).
It also works without systemd. Which means it can work in CiaB. If we transliterate again, then Python will still need to stick around anyway, unless the new one can also run without systemd.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453933405
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,367 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "strings"
+ "time"
+)
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func AskYesNo(msg string) (bool, error) {
+ reader := bufio.NewReader(os.Stdin)
+ for {
+ fmt.Printf("%s [Y/n]: ", msg)
+ ans, err := reader.ReadString('\n')
+ if err != nil {
+ return false, err
+ }
+
+ if strings.ToUpper(strings.TrimSpace(ans)) == "Y" {
+ return true, nil
+ } else if strings.ToUpper(strings.TrimSpace(ans)) == "N" {
+ return false, nil
+ }
+ }
+ return false, nil
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ output, _, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func FileMatch(name string, pattern string) bool {
+ matched := strings.Contains(name, pattern)
+ return matched
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rob05c commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
rob05c commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453987451
##########
File path: traffic_ops_ort/traffic_ops_t3c/traffic_ops_t3c.go
##########
@@ -0,0 +1,169 @@
+package main
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/torequest"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "os"
+ "time"
+)
+
+func runSysctl(cfg config.Cfg) {
+ var ans bool
+
+ switch cfg.RunMode {
+ case config.INTERACTIVE:
+ ans, _ = util.AskYesNo("sysctl configuration has changed. 'sysctl -p' needs to be run. Should I do that now? (Y/[n])")
+ case config.BADASS:
+ ans = true
+ }
+
+ if ans == true {
+ _, rc, err := util.ExecCommand("/usr/sbin/sysctl", "-p")
Review comment:
That's fine; I don't feel strongly about it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rob05c commented on a change in pull request #4872: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
rob05c commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r454641054
##########
File path: traffic_ops_ort/traffic_ops_t3c/util/util.go
##########
@@ -0,0 +1,383 @@
+package util
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/gofrs/flock"
+ "io/ioutil"
+ "math/rand"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "strings"
+ "time"
+)
+
+const OneWeek = 604800
+
+type FileLock struct {
+ f_lock *flock.Flock
+ is_locked bool
+}
+
+// Try to get a file lock, non-blocking.
+func (f *FileLock) GetLock(lockFile string) bool {
+ f.f_lock = flock.New(lockFile)
+ is_locked, err := f.f_lock.TryLock()
+ f.is_locked = is_locked
+
+ if err != nil { // some OS error attempting to obtain a file lock
+ log.Errorf("unable to obtain a lock on %s\n", lockFile)
+ return false
+ }
+ if !f.is_locked { // another process is running.
+ log.Errorf("Another traffic_ops_t3c process is already running, try again later\n")
+ return false
+ }
+
+ return f.is_locked
+}
+
+// Releases a file lock and exits with the given status code.
+func (f *FileLock) UnlockAndExit(code int) {
+ if f.is_locked {
+ f.f_lock.Unlock()
+ }
+ os.Exit(code)
+}
+
+func DirectoryExists(dir string) (bool, os.FileInfo) {
+ info, err := os.Stat(dir)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return info.IsDir(), info
+}
+
+func ExecCommand(fullCommand string, arg ...string) ([]byte, int, error) {
+ var output bytes.Buffer
+ cmd := exec.Command(fullCommand, arg...)
+ cmd.Stdout = &output
+ err := cmd.Run()
+ return output.Bytes(), cmd.ProcessState.ExitCode(), err
+}
+
+func FileExists(fn string) (bool, os.FileInfo) {
+ info, err := os.Stat(fn)
+ if os.IsNotExist(err) {
+ return false, nil
+ }
+ return !info.IsDir(), info
+}
+
+func ReadFile(fn string) ([]byte, error) {
+ var data []byte
+ info, err := os.Stat(fn)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fn)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func serviceStatus(name string) (int, error) {
+ var pid int = -1
+ var result bool
+
+ // the service command returns an error and return code of '3'
+ // if a service is loaded and not running. So if rc == 3, ignore
+ // the error
+ output, rc, err := ExecCommand("/usr/sbin/service", name, "status")
+ if err != nil && rc != 3 {
+ return pid, errors.New("could not get status for service '" + name + "'\n")
+ }
+ lines := strings.Split(string(output), "\n")
+ for ii := range lines {
+ line := strings.TrimSpace(lines[ii])
+ if strings.Contains(line, "Active: active") {
+ result = true
+ }
+ if result && strings.Contains(line, "Main PID: ") {
+ fmt.Sscanf(line, "Main PID: %d", &pid)
+ }
+ }
+
+ return pid, nil
+}
+
+// start or restart the service 'service'. cmd is 'start | restart'
+func ServiceStart(service string, cmd string) (bool, error) {
+ log.Infof("ServiceStart called for '%s'\n", service)
+ pid, err := serviceStatus(service)
+ if err != nil {
+ return false, errors.New("Could not get status for '" + service + "' : " + err.Error())
+ } else if pid > 0 && cmd == "start" {
+ log.Infof("service '%s' is already running, pid: %d\n", service, pid)
+ } else {
+ _, rc, err := ExecCommand("/usr/sbin/service", service, cmd)
+ if err != nil {
+ return false, errors.New("Could not " + cmd + " the '" + service + "' service: " + err.Error())
+ } else if rc == 0 {
+ // service was sucessfully started
+ return true, nil
+ }
+ }
+ // not started, service is already running
+ return false, nil
+}
+
+func WriteFile(fn string, data []byte, perm os.FileMode) (int, error) {
+ return WriteFileWithOwner(fn, data, -1, -1, perm)
+}
+
+func WriteFileWithOwner(fn string, data []byte, uid int, gid int, perm os.FileMode) (int, error) {
+ fd, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fn + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fn + "': " + err.Error())
+ }
+ fd.Close()
+
+ if uid > -1 && gid > -1 {
+ err = os.Chown(fn, uid, gid)
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fn + "': " + err.Error())
+ }
+ }
+ return c, nil
+}
+
+func PackageAction(cmdstr string, name string) (bool, error) {
+ var rc int = -1
+ var err error = nil
+ var result bool = false
+
+ switch cmdstr {
+ case "info":
+ _, rc, err = ExecCommand("/usr/bin/yum", "info", name)
+ case "install":
+ _, rc, err = ExecCommand("/usr/bin/yum", "install", "-y", name)
+ case "remove":
+ _, rc, err = ExecCommand("/usr/bin/yum", "remove", "-y", name)
+ }
+
+ if rc == 0 {
+ result = true
+ err = nil
+ }
+ return result, err
+}
+
+// runs the rpm command.
+// if the return code from rpm == 0, then a valid package list is returned.
+//
+// if the return code is 1, the the 'name' queried for is not part of a
+// package or is not installed.
+//
+// otherwise, if the return code is not 0 or 1 and error is set, a general
+// rpm command execution error is assumed and the error is returned.
+func PackageInfo(cmdstr string, name string) ([]string, error) {
+ var result []string
+ switch cmdstr {
+ case "cfg-files": // returns a list of the package configuration files.
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-c", name)
+ if rc == 1 { // rpm package for 'name' was not found.
+ return nil, nil
+ } else if rc == 0 { // add the package name the file belongs to.
+ log.Debugf("output from cfg-files query: %s\n", string(output))
+ files := strings.Split(string(output), "\n")
+ for ii := range files {
+ result = append(result, strings.TrimSpace(files[ii]))
+ }
+ log.Debugf("result length: %d, result: %s\n", len(result), string(output))
+ } else if err != nil {
+ return nil, err
+ }
+ case "file-query": // returns the rpm name that owns the file 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "-f", name)
+ if rc == 1 { // file is not part of any package.
+ return nil, nil
+ } else if rc == 0 { // add the package name the file belongs to.
+ log.Debugf("output from file-query: %s\n", string(output))
+ result = append(result, string(strings.TrimSpace(string(output))))
+ log.Debugf("result length: %d, result: %s\n", len(result), string(output))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-provides": // returns the package name that provides 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatprovides", name)
+ log.Debugf("pkg-provides - name: %s, output: %s\n", name, output)
+ if rc == 1 { // no package provides 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-query": // returns the package name for 'name'.
+ output, rc, err := ExecCommand("/bin/rpm", "-q", name)
+ if rc == 1 { // the package is not installed.
+ return nil, nil
+ } else if rc == 0 { // add the rpm name
+ result = append(result, string(strings.TrimSpace(string(output))))
+ } else if err != nil {
+ return nil, err
+ }
+ case "pkg-requires": // returns a list of packages that requires package 'name'
+ output, rc, err := ExecCommand("/bin/rpm", "-q", "--whatrequires", name)
+ if rc == 1 { // no package reuires package 'name'
+ return nil, nil
+ } else if rc == 0 {
+ pkgs := strings.Split(string(output), "\n")
+ for ii := range pkgs {
+ result = append(result, strings.TrimSpace(pkgs[ii]))
+ }
+ } else if err != nil {
+ return nil, err
+ }
+ }
+ return result, nil
+}
+
+func RandomDuration(max time.Duration) time.Duration {
+ rand.Seed(time.Now().UnixNano())
+ return time.Duration(rand.Int63n(int64(max)))
Review comment:
This panics if `--dispersion=0` is set.
Should this just return 0 if `max` is 0?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] jrushford commented on a change in pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
jrushford commented on a change in pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#discussion_r453917225
##########
File path: traffic_ops_ort/traffic_ops_t3c/torequest/torequest.go
##########
@@ -0,0 +1,1466 @@
+package torequest
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/config"
+ "github.com/apache/trafficcontrol/traffic_ops_ort/traffic_ops_t3c/util"
+ "io"
+ "io/ioutil"
+ "mime"
+ "mime/multipart"
+ "net/mail"
+ "net/textproto"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "syscall"
+ "time"
+)
+
+type UpdateStatus int
+
+const (
+ UPDATE_TROPS_NOTNEEDED UpdateStatus = 0
+ UPDATE_TROPS_NEEDED UpdateStatus = 1
+ UPDATE_TROPS_SUCCESSFUL UpdateStatus = 2
+ UPDATE_TROPS_FAILED UpdateStatus = 3
+)
+
+type Package struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+}
+
+type TrafficOpsReq struct {
+ Cfg config.Cfg
+ pkgs map[string]bool // map of installed packages
+ plugins map[string]bool // map of verified plugins
+ configFiles map[string]*ConfigFile
+ baseBackupDir string
+ TrafficCtlReload bool // a traffic_ctl_reload is required
+ SysCtlReload bool // a reload of the sysctl.conf is required
+ NtpdRestart bool // ntpd needs restarting
+ TeakdRestart bool // a restart of teakd is required
+ TrafficServerRestart bool // a trafficserver restart is required
+ RemapConfigReload bool // remap.config should be reloaded
+}
+
+type ConfigFile struct {
+ Header textproto.MIMEHeader
+ Name string // file name
+ Dir string // install directory
+ Path string // full path
+ Service string // service assigned to
+ CfgBackup string // location to backup the config at 'Path'
+ TropsBackup string // location to backup the TrafficOps Version
+ AuditComplete bool // audit is complete
+ AuditFailed bool // audit failed
+ ChangeApplied bool // a change has been applied
+ ChangeNeeded bool // change required
+ PreReqFailed bool // failed plugin prerequiste check
+ RemapPluginConfig bool // file is a remap plugin config file
+ Body []byte
+ Perm os.FileMode // default file permissions
+ Uid int // owner uid, default is 0
+ Gid int // owner gid, default is 0
+}
+
+func (u UpdateStatus) String() string {
+ var result string
+ switch u {
+ case 0:
+ result = "UPDATE_TROPS_NOTNEEDED"
+ case 1:
+ result = "UPDATE_TROPS_NEEDED"
+ case 2:
+ result = "UPDATE_TROPS_SUCCESSFUL"
+ case 3:
+ result = "UPDATE_TROPS_FAILED"
+ }
+ return result
+}
+
+func commentsFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+
+ for ii := range body {
+ line := body[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ newlines = append(newlines, line)
+ }
+
+ return newlines
+}
+
+func newLineFilter(str string) string {
+ str = strings.ReplaceAll(str, "\r\n", "\n")
+ return strings.TrimSpace(str)
+}
+
+func unencodeFilter(body []string) []string {
+ var newlines []string
+
+ newlines = make([]string, 0)
+ sp := regexp.MustCompile(`\s+`)
+ el := regexp.MustCompile(`^\s+|\s+$`)
+ am := regexp.MustCompile(`amp;`)
+ lt := regexp.MustCompile(`<`)
+ gt := regexp.MustCompile(`>`)
+
+ for ii := range body {
+ s := body[ii]
+ s = sp.ReplaceAllString(s, " ")
+ s = el.ReplaceAllString(s, "")
+ s = am.ReplaceAllString(s, "")
+ s = lt.ReplaceAllString(s, "<")
+ s = gt.ReplaceAllString(s, ">")
+ s = strings.TrimSpace(s)
+ newlines = append(newlines, s)
+ }
+
+ return newlines
+}
+
+// for debugging
+func (r *TrafficOpsReq) DumpConfigFiles() {
+ for _, cfg := range r.configFiles {
+ fmt.Printf("Name: %s, Dir: %s, Service: %s\n",
+ cfg.Name, cfg.Dir, cfg.Service)
+ }
+}
+
+// returns a new TrafficOpsReq object.
+func NewTrafficOpsReq(cfg config.Cfg) *TrafficOpsReq {
+ unixTimeStr := strconv.FormatInt(time.Now().Unix(), 10)
+ // backup directories
+ bkupDir := config.TmpBase + "/" + unixTimeStr
+
+ return &TrafficOpsReq{
+ Cfg: cfg,
+ pkgs: make(map[string]bool),
+ plugins: make(map[string]bool),
+ configFiles: make(map[string]*ConfigFile),
+ baseBackupDir: bkupDir,
+ }
+}
+
+// wrapper to run an atstccfg command.
+func (r *TrafficOpsReq) atsTcExec(cmdstr string) ([]byte, error) {
+ log.Debugf("cmdstr: %s\n", cmdstr)
+ result, err := r.atsTcExecCommand(cmdstr, -1, -1)
+ return result, err
+}
+
+// runs an atstccfg command.
+func (r *TrafficOpsReq) atsTcExecCommand(cmdstr string, queueState int, revalState int) ([]byte, error) {
+ args := []string{
+ "--traffic-ops-timeout-milliseconds=" + strconv.FormatInt(int64(r.Cfg.TOTimeoutMS), 10),
+ "--traffic-ops-disable-proxy=" + strconv.FormatBool(r.Cfg.ReverseProxyDisable),
+ "--traffic-ops-user=" + r.Cfg.TOUser,
+ "--traffic-ops-password=" + r.Cfg.TOPass,
+ "--traffic-ops-url=" + r.Cfg.TOURL,
+ "--cache-host-name=" + r.Cfg.CacheHostName,
+ "--log-location-error=" + r.Cfg.LogLocationErr,
+ "--log-location-info=" + r.Cfg.LogLocationInfo,
+ "--log-location-warning=" + r.Cfg.LogLocationWarn,
+ }
+
+ switch cmdstr {
+ case "chkconfig":
+ args = append(args, "--get-data=chkconfig")
+ case "packages":
+ args = append(args, "--get-data=packages")
+ case "statuses":
+ args = append(args, "--get-data=statuses")
+ case "system-info":
+ args = append(args, "--get-data=system-info")
+ case "update-status":
+ args = append(args, "--get-data=update-status")
+ case "send-update":
+ var queueStatus string = "false"
+ var revalStatus string = "false"
+ if queueState > 0 {
+ queueStatus = "true"
+ }
+ if revalState > 0 {
+ revalStatus = "true"
+ }
+ args = append(args, "--set-queue-status", queueStatus, "--set-reval-status", revalStatus)
+ case "get-config-files":
+ if r.Cfg.RunMode == config.REVALIDATE {
+ args = append(args, "--revalidate-only")
+ }
+ default:
+ return nil, errors.New("invalid command '" + cmdstr + "'")
+ }
+
+ cmd := exec.Command(config.AtsTcConfig, args...)
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ err := cmd.Run()
+ return out.Bytes(), err
+}
+
+// backup the config file.
+func (r *TrafficOpsReq) backUpFile(cfg *ConfigFile) error {
+
+ // init backup directories
+ configBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_bkp"
+ cfg.CfgBackup = configBkupDir + "/" + cfg.Name
+ tropsBkupDir := r.baseBackupDir + "/" + cfg.Service + "/config_trops"
+ cfg.TropsBackup = tropsBkupDir + "/" + cfg.Name
+
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ // create config file backup directory
+ err := os.MkdirAll(configBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create config backup directory '" + configBkupDir + "': " + err.Error())
+ }
+
+ // backup the config file
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.Path + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, configBkupDir, data)
+ if err != nil {
+ return errors.New("Failed to write the config file: " + err.Error())
+ }
+
+ } else {
+ log.Debugf("Config file: %s doesn't exist. No need to back up.\n", cfg.Name)
+ }
+
+ // backup the traffic ops file.
+ err := os.MkdirAll(tropsBkupDir, 0755)
+ if err != nil {
+ return errors.New("Unable to create Trops backup directory '" + tropsBkupDir + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, tropsBkupDir, cfg.Body)
+ if err != nil {
+ return errors.New("Failed to write the config file from traffic ops: " + err.Error())
+ }
+
+ // backup the current config file.
+ return nil
+}
+
+// checks and audits config files. perl version is process_cfg_file
+func (r *TrafficOpsReq) checkConfigFile(cfg *ConfigFile) error {
+ if cfg.Name == "" {
+ cfg.AuditFailed = true
+ return errors.New("Config file name is empty is empty, skipping further checks.")
+ }
+
+ if cfg.Dir == "" {
+ return errors.New("No location information for " + cfg.Name)
+ }
+ // return if audit has already been done.
+ if cfg.AuditComplete == true {
+ return nil
+ }
+
+ if !util.MkDir(cfg.Dir, r.Cfg) {
+ return errors.New("Unable to create the directory '" + cfg.Dir + " for " + "'" + cfg.Name + "'")
+ }
+
+ log.Debugf("======== Start processing config file: %s ========\n", cfg.Name)
+
+ if cfg.Name == "remap.config" {
+ err := r.processRemapOverrides(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // perform plugin verification
+ if cfg.Name == "remap.config" || cfg.Name == "plugin.config" {
+ err := r.verifyPlugins(cfg)
+ if err != nil {
+ return err
+ }
+ }
+
+ // apply traffic ops data filters in preparation for comparison
+ // to data on disk.
+ tropsData := strings.Split(string(cfg.Body), "\n")
+ tropsData = unencodeFilter(tropsData)
+ tropsData = commentsFilter(tropsData)
+
+ var diskData []string
+ fileExists, _ := util.FileExists(cfg.Path)
+ if fileExists {
+ data, err := r.readCfgFile(cfg, "")
+ if err != nil {
+ return errors.New("reading from '" + cfg.Path + "' failed: " + err.Error())
+ }
+ diskData = strings.Split(string(data), "\n")
+ } else { // file doesn't exist on, it's new from Traffic Ops.
+ cfg.AuditComplete = true
+ cfg.ChangeNeeded = true
+ log.Infof("No such file on disk, '%s'\n", cfg.Path)
+ }
+
+ // apply disk file data filters in preparation for comparison
+ // to data from traffic ops
+ diskData = unencodeFilter(diskData)
+ diskData = commentsFilter(diskData)
+
+ // apply final new line filters disk and traffic ops data for comparison
+ disk := strings.Join(diskData, "\n")
+ disk = newLineFilter(disk)
+
+ trops := strings.Join(tropsData, "\n")
+ trops = newLineFilter(trops)
+
+ if disk != trops {
+ cfg.ChangeNeeded = true
+ log.Infof("change needed to %s\n", cfg.Name)
+ err := r.backUpFile(cfg)
+ if err != nil {
+ return errors.New("unable to back up '" + cfg.Name + "': " + err.Error())
+ }
+ } else {
+ cfg.ChangeNeeded = false
+ log.Infof("All lines match TrOps for config file: %s\n", cfg.Name)
+ }
+
+ if cfg.Name == "50-ats.rules" {
+ err := r.processUdevRules(cfg)
+ if err != nil {
+ return errors.New("unable to process udev rules in '" + cfg.Name + "': " + err.Error())
+ }
+ }
+
+ log.Infof("======== End processing config file: %s for service: %s ========\n", cfg.Name, cfg.Service)
+ cfg.AuditComplete = true
+
+ return nil
+}
+
+func (r *TrafficOpsReq) checkPlugin(plugin string) error {
+ // already verified
+ if r.plugins[plugin] == true {
+ return nil
+ }
+ pluginFile := config.TSHome + "/libexec/trafficserver/" + plugin
+ pkgs, err := util.PackageInfo("pkg-provides", pluginFile)
+ if err != nil {
+ return errors.New("unable to verify plugin " + pluginFile + ": " + err.Error())
+ }
+ if pkgs == nil { // no package is installed that provides the plugin.
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ _, ok := r.pkgs[pkgs[0]]
+ if !ok {
+ return errors.New(plugin + ": Package for plugin: " + plugin + ", is not installed.")
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) checkStatusFiles(svrStatus string) error {
+ if svrStatus == "" {
+ return errors.New("Returning; did not find status from Traffic Ops!")
+ } else {
+ log.Debugf("Found %s status from Traffic Ops.\n", svrStatus)
+ }
+ statusFile := config.StatusDir + "/" + svrStatus
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ log.Errorf("status file %s does not exist.\n", statusFile)
+ }
+ statuses, err := r.getStatuses()
+ if err != nil {
+ return fmt.Errorf("could not retrieves a statuses list from Traffic Ops: %s\n", err)
+ }
+
+ for f := range statuses {
+ otherStatus := config.StatusDir + "/" + statuses[f]
+ if otherStatus == statusFile {
+ continue
+ }
+ fileExists, _ := util.FileExists(otherStatus)
+ if r.Cfg.RunMode != config.REPORT && fileExists {
+ log.Errorf("Removing other status file %s that exists\n", otherStatus)
+ err = os.Remove(otherStatus)
+ if err != nil {
+ log.Errorf("Error removing %s: %s\n", otherStatus, err)
+ }
+ }
+ }
+
+ if r.Cfg.RunMode != config.REPORT {
+ if !util.MkDir(config.StatusDir, r.Cfg) {
+ return fmt.Errorf("unable to create '%s'\n", config.StatusDir)
+ }
+ fileExists, _ := util.FileExists(statusFile)
+ if !fileExists {
+ err = util.Touch(statusFile)
+ if err != nil {
+ return fmt.Errorf("unable to touch %s - %s\n", statusFile, err)
+ }
+ }
+ }
+ return nil
+}
+
+// fetches a list of cache statuses from traffic ops.
+func (r *TrafficOpsReq) getStatuses() ([]string, error) {
+ var statuses []tc.StatusNullable
+ sl := make([]string, 0)
+ out, err := r.atsTcExec("statuses")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ if err = json.Unmarshal(out, &statuses); err != nil {
+ log.Errorln(err)
+ return nil, err
+ } else {
+ for val := range statuses {
+ sl = append(sl, *statuses[val].Name)
+ }
+ }
+ }
+
+ return sl, nil
+}
+
+func (r *TrafficOpsReq) getUpdateStatus() (*tc.ServerUpdateStatus, error) {
+ var status tc.ServerUpdateStatus
+ out, err := r.atsTcExec("update-status")
+ if err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ if err = json.Unmarshal(out, &status); err != nil {
+ log.Errorln(err)
+ return nil, err
+ }
+ log.Debugf("ServerUpdateStatus: %#v\n", status)
+ return &status, nil
+}
+
+func (r *TrafficOpsReq) processRemapOverrides(cfg *ConfigFile) error {
+ var from string
+ var newlines []string
+ var overrides map[string]int
+ var lineCount int = 0
+ var overrideCount int = 0
+ var overridenCount int = 0
+ data := cfg.Body
+
+ overrides = make(map[string]int)
+ newlines = make([]string, 0)
+
+ if len(data) > 0 {
+ lines := strings.Split(string(data), "\n")
+ for ii := range lines {
+ str := lines[ii]
+ fields := strings.Fields(str)
+ if str == "" || len(fields) < 2 {
+ continue
+ }
+ lineCount++
+ from = fields[1]
+
+ _, ok := overrides[from]
+ if ok == true { // check if this line should be overriden
+ newstr := "##OVERRIDDEN## " + str
+ newlines = append(newlines, newstr)
+ overridenCount++
+ } else if fields[0] == "##OVERRIDE##" { // check for an override
+ from = fields[2]
+ newlines = append(newlines, "##OVERRIDE##")
+ // remove the ##OVERRIDE## comment along with the trailing space
+ newstr := strings.TrimPrefix(str, "##OVERRIDE## ")
+ // save the remap 'from field' to overrides.
+ overrides[from] = 1
+ newlines = append(newlines, newstr)
+ overrideCount++
+ } else { // no override is necessary
+ newlines = append(newlines, str)
+ }
+ }
+ } else {
+ return errors.New("The " + cfg.Name + " file is empty, nothing to process.")
+ }
+ if overrideCount > 0 {
+ log.Infof("Overrode %d old remap rule(s) with %d new remap rule(s).\n",
+ overridenCount, overrideCount)
+ newdata := strings.Join(newlines, "\n")
+ // strings.Join doesn't add a newline character to
+ // the last element in the array and we need one
+ // when the data is written out to a file.
+ if !strings.HasSuffix(newdata, "\n") {
+ newdata = newdata + "\n"
+ }
+ body := []byte(newdata)
+ cfg.Body = body
+ }
+ return nil
+}
+
+// verify disk drive device ownership and mode
+func (r *TrafficOpsReq) processUdevRules(cfg *ConfigFile) error {
+ var udevDevices map[string]string
+
+ data := string(cfg.Body)
+ lines := strings.Split(data, "\n")
+
+ udevDevices = make(map[string]string)
+ for ii := range lines {
+ var owner string
+ var device string
+ line := lines[ii]
+ if strings.HasPrefix(line, "KERNEL==") {
+ vals := strings.Split(line, "\"")
+ if len(vals) >= 3 {
+ device = vals[1]
+ owner = vals[3]
+ if owner == "root" {
+ continue
+ }
+ userInfo, err := user.Lookup(owner)
+ if err != nil {
+ log.Errorf("no such user on this system: '%s'\n", owner)
+ continue
+ } else {
+ devPath := "/dev/" + device
+ fileExists, fileInfo := util.FileExists(devPath)
+ if fileExists {
+ udevDevices[device] = devPath
+ log.Infof("Found device in 50-ats.rules: %s\n", devPath)
+ if statStruct, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
+ uid := strconv.Itoa(int(statStruct.Uid))
+ if uid != userInfo.Uid {
+ log.Errorf("Device %s is owned by uid %s, not %s (%s)\n", devPath, uid, owner, userInfo.Uid)
+ } else {
+ log.Infof("Ownership for disk device %s, is okay\n", devPath)
+ }
+ } else {
+ log.Errorf("Unable to read device owner info for %s\n", devPath)
+ }
+ }
+ }
+ }
+ }
+ }
+ fs, err := ioutil.ReadDir("/proc/fs/ext4")
+ if err != nil {
+ log.Errorln("unable to read /proc/fs/ext4, cannot audit disks for filesystem usage.")
+ } else {
+ for _, disk := range fs {
+ for k, _ := range udevDevices {
+ if strings.HasPrefix(k, disk.Name()) {
+ log.Warnf("Device %s has an active partition and filesystem!!!!\n", k)
+ }
+ }
+ }
+ }
+
+ return nil
+}
+
+// read a config file and return its contents.
+func (r *TrafficOpsReq) readCfgFile(cfg *ConfigFile, dir string) ([]byte, error) {
+ var data []byte
+ var fullFileName string
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ info, err := os.Stat(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ size := info.Size()
+
+ fd, err := os.Open(fullFileName)
+ if err != nil {
+ return nil, err
+ }
+ data = make([]byte, size)
+ c, err := fd.Read(data)
+ if err != nil || int64(c) != size {
+ return nil, errors.New("unable to completely read from '" + cfg.Name + "': " + err.Error())
+ }
+ fd.Close()
+
+ return data, nil
+}
+
+func (r *TrafficOpsReq) replaceCfgFile(cfg *ConfigFile) error {
+ var ans bool
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ fmt.Printf("\nThe '%s' file on disk needs updated with one from Traffic Ops.\n", cfg.Name)
+ ans, _ = util.AskYesNo(" [Y] override files on disk with data from Traffic Ops, [N] ignore and continue.")
+ }
+ if ans == true ||
+ r.Cfg.RunMode == config.BADASS ||
+ r.Cfg.RunMode == config.SYNCDS ||
+ r.Cfg.RunMode == config.REVALIDATE {
+
+ log.Infof("Copying '%s' to '%s'\n", cfg.TropsBackup, cfg.Path)
+ data, err := util.ReadFile(cfg.TropsBackup)
+ if err != nil {
+ return errors.New("Unable to read the config file '" + cfg.TropsBackup + "': " + err.Error())
+ }
+ _, err = r.writeCfgFile(cfg, "", data)
+ if err != nil {
+ return errors.New("Failed to write the new config file: " + err.Error())
+ } else {
+ cfg.ChangeApplied = true
+ if cfg.RemapPluginConfig == true { // trafficserver config reload is required
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ }
+ if cfg.Name == "plugin.config" { // trafficserver restart is required
+ r.TrafficServerRestart = true
+ } else if cfg.Name == "remap.config" {
+ r.TrafficCtlReload = true
+ r.RemapConfigReload = true
+ } else if cfg.Name == "sysctl.conf" { // sysctl reload is required
+ r.SysCtlReload = true
+ } else if cfg.Name == "ssl_multicert.config" {
+ r.TrafficCtlReload = true
+ } else if cfg.Name == "ntpd.conf" { // ntpd reload is required
+ r.NtpdRestart = true
+ }
+ log.Debugf("Setting change applied for '%s'\n", cfg.Name)
+ }
+ } else {
+ log.Infof("You elected not to replace %s with the version from Traffic Ops.\n")
+ cfg.ChangeApplied = false
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) sleepTimer(serverStatus *tc.ServerUpdateStatus) {
+ randDispSec := util.RandomDuration(r.Cfg.Dispersion) / time.Second
+ revalClockSec := r.Cfg.RevalWaitTime / time.Second
+
+ if serverStatus.UseRevalPending && r.Cfg.RunMode != config.BADASS {
+ log.Infoln("Performing a revalidation check before sleeping...")
+ _, err := r.RevalidateWhileSleeping()
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ }
+ if randDispSec < revalClockSec || serverStatus.UseRevalPending == false || r.Cfg.RunMode == config.BADASS {
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ } else {
+ log.Infof("%d seconds until next revalidation check.\n", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ }
+
+ for randDispSec > 0 {
+ fmt.Printf(".")
+ time.Sleep(time.Second)
+ revalClockSec--
+ if revalClockSec < 1 && r.Cfg.RunMode != config.BADASS && serverStatus.UseRevalPending {
+ fmt.Printf("\n")
+ log.Infoln("Interrupting dispersion sleep period for revalidation check.")
+ _, err := r.RevalidateWhileSleeping()
+ revalClockSec = r.Cfg.RevalWaitTime / time.Second
+ if err != nil {
+ log.Errorf("Revalidation check completed with error: %s\n", err)
+ } else {
+ log.Infoln("Revalidation check complete.")
+ }
+ if revalClockSec < randDispSec {
+ log.Infof("Revalidation check complete. %d seconds until next revalidation check.", revalClockSec)
+ log.Infof("%d seconds remaining in dispersion sleep period\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", revalClockSec)
+ } else {
+ log.Infof("Revalidation check complete. %d seconds remaining in dispersion sleep period.\n", randDispSec)
+ log.Infof("Sleeping for %d seconds: ", randDispSec)
+ }
+ }
+ randDispSec--
+ }
+ fmt.Printf("\n")
+}
+
+func (r *TrafficOpsReq) writeCfgFile(cfg *ConfigFile, dir string, data []byte) (int, error) {
+ var fullFileName string
+
+ if dir == "" {
+ fullFileName = cfg.Path
+ } else {
+ fullFileName = dir + "/" + cfg.Name
+ }
+
+ fd, err := os.OpenFile(fullFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, cfg.Perm)
+ if err != nil {
+ return 0, errors.New("unable to open '" + fullFileName + "' for writing: " + err.Error())
+ }
+
+ c, err := fd.Write(data)
+ if err != nil {
+ return 0, errors.New("error writing to '" + fullFileName + "': " + err.Error())
+ }
+ fd.Close()
+
+ if cfg.Service == "trafficserver" {
+ err = os.Chown(fullFileName, cfg.Uid, cfg.Gid)
+ } else {
+ err = os.Chown(fullFileName, 0, 0)
+ }
+ if err != nil {
+ return 0, errors.New("error changing ownership on '" + fullFileName + "': " + err.Error())
+ }
+
+ return c, nil
+}
+
+func (r *TrafficOpsReq) verifyPlugins(cfg *ConfigFile) error {
+
+ log.Debugf("Checking plugins for %s\n", cfg.Name)
+
+ str := string(cfg.Body)
+ lines := strings.Split(str, "\n")
+ if cfg.Name == "plugin.config" {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") {
+ continue
+ }
+ fields := strings.Fields(line)
+ if len(fields) > 0 {
+ plugin := fields[0]
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ }
+ } else if cfg.Name == "remap.config" && len(lines) > 0 {
+ for ii := range lines {
+ line := lines[ii]
+ if strings.HasPrefix(line, "#") || line == "" {
+ continue
+ }
+ plugins := strings.Split(line, "@plugin=")
+ if len(plugins) > 0 {
+ for jj := range plugins {
+ var plugin string = ""
+ var plugin_config string = ""
+ if jj > 0 {
+ params := strings.Split(plugins[jj], "@pparam=")
+ param_length := len(params)
+ if param_length > 0 {
+ switch param_length {
+ case 1:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = ""
+ default:
+ plugin = strings.TrimSpace(params[0])
+ plugin_config = strings.TrimSpace(params[1])
+ }
+ }
+ if strings.HasSuffix(plugin, ".so") {
+ // already verified
+ if r.plugins[plugin] == true {
+ continue
+ }
+ err := r.checkPlugin(plugin)
+ if err != nil {
+ cfg.PreReqFailed = true
+ return err
+ } else {
+ r.plugins[plugin] = true
+ log.Infof("Plugin %s has been verified.\n", plugin)
+ }
+ }
+ if plugin_config != "" {
+ if strings.HasPrefix(plugin_config, "proxy.config") || strings.HasPrefix(plugin_config, "-") {
+ continue
+ } else {
+ plugin_config = filepath.Base(plugin_config)
+ cfg, ok := r.configFiles[plugin_config]
+ if ok {
+ cfg.RemapPluginConfig = true
+ log.Debugf("%s is a remap plugin config file\n", plugin_config)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+func (r *TrafficOpsReq) CheckSystemServices() error {
+ if r.Cfg.RunMode == config.BADASS || r.Cfg.RunMode == config.INTERACTIVE {
+ if r.Cfg.RunMode == config.INTERACTIVE {
+ ans, _ := util.AskYesNo("Do you wish to enable a system service with chkconfig or systemctl?")
+ if ans == false {
+ return nil
+ }
+ }
+ out, err := r.atsTcExec("chkconfig")
+ if err != nil {
+ log.Errorln(err)
+ return err
+ } else {
+ var result []map[string]string
+ if err = json.Unmarshal(out, &result); err != nil {
+ return err
+ } else {
+ for ii := range result {
+ _n := result[ii]["name"]
+ _v := result[ii]["value"]
+ arrv := strings.Fields(_v)
+ var level []string
+ var enabled bool = false
+ for jj := range arrv {
+ nv := strings.Split(arrv[jj], ":")
+ if len(nv) == 2 && strings.Contains(nv[1], "on") {
+ level = append(level, nv[0])
+ enabled = true
+ }
+ }
+ if enabled == true {
+ if r.Cfg.SvcManagement == config.SYSTEMD {
+ out, rc, err := util.ExecCommand("/bin/systemctl", "enable", _n)
+ if err != nil {
+ log.Errorf(string(out))
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else if r.Cfg.SvcManagement == config.SYSTEMV {
+ levelValue := strings.Join(level, "")
+ _, rc, err := util.ExecCommand("/bin/chkconfig", "--level", levelValue, _n, "on")
+ if err != nil {
+ return errors.New("Unable to enable service " + _n + ": " + err.Error())
+ }
+ if rc == 0 {
+ log.Infof("The %s service has been enabled\n", _n)
+ }
+ } else {
+ log.Errorf("Unable to insure %s service is enabled, SvcMananagement type is %s\n", r.Cfg.SvcManagement)
+ }
+ }
+ }
+ }
+ }
+ }
+ return nil
+}
+
+// returns true/fale if the named rpm package is installed.
+func (r *TrafficOpsReq) isPackageInstalled(name string) bool {
+ for k, _ := range r.pkgs {
+ if strings.HasPrefix(k, name) {
+ return true
+ }
+ }
+ return false
+}
+
+// fetch a 'Configfile' by file name
+func (r *TrafficOpsReq) GetConfigFile(name string) (*ConfigFile, bool) {
+ cfg, ok := r.configFiles[name]
+ return cfg, ok
+}
+
+// fetches and parses the multipart config files for a cache from traffic ops
+// and loads them into the configFiles map.
+func (r *TrafficOpsReq) GetConfigFileList() error {
+ var atsUid int = 0
+ var atsGid int = 0
+
+ atsUser, err := user.Lookup(config.TrafficServerOwner)
+ if err != nil {
+ log.Errorf("could not lookup the trafficserver, '%s', owner uid, using uid/gid 0")
+ } else {
+ atsUid, err = strconv.Atoi(atsUser.Uid)
+ if err != nil {
+ log.Errorf("could not parse the ats UID.")
+ atsUid = 0
+ }
+ atsGid, err = strconv.Atoi(atsUser.Gid)
+ if err != nil {
+ log.Errorf("could not parse the ats GID.")
+ atsUid = 0
+ }
+ }
+
+ fBytes, err := r.atsTcExec("get-config-files")
+ if err != nil {
+ return err
+ }
+ buf := bytes.NewBuffer(fBytes)
+ var hdr string
+ for {
+ hdr, err = buf.ReadString('\n')
+ if err != nil {
+ return err
+ } else {
+ if strings.HasPrefix(hdr, "Content-Type: multipart/mixed") {
+ break
+ }
+ }
+ }
+ // split on the ": " to trim off the leading space
+ har := strings.Split(hdr, ": ")
+ if har[0] != "Content-Type" || !strings.HasPrefix(har[1], "multipart/mixed") {
+ return errors.New("invalid config file data received from Traffic Ops, not in multipart format.")
+ }
+ msg := &mail.Message{
+ Header: map[string][]string{har[0]: {har[1]}},
+ Body: bytes.NewReader(buf.Bytes()),
+ }
+ mediaType, params, err := mime.ParseMediaType(msg.Header.Get("Content-Type"))
+ if err != nil {
+ return err
+ }
+ if strings.HasPrefix(mediaType, "multipart/") {
+ mr := multipart.NewReader(msg.Body, params["boundary"])
+ // if the configFiles map is not empty, create a new map
+ // and the old map will be garbage collected.
+ if len(r.configFiles) > 0 {
+ log.Infoln("intializing a new configFiles map")
+ r.configFiles = make(map[string]*ConfigFile)
+ }
+ for {
+ var cf *ConfigFile
+
+ p, err := mr.NextPart()
+ if err == io.EOF {
+ return nil
+ }
+ if err != nil {
+ return err
+ }
+ dataBytes, err := ioutil.ReadAll(p)
+ if err != nil {
+ return err
+ }
+ path := p.Header.Get("Path")
+ if path != "" {
+ cf = new(ConfigFile)
+ cf.Header = p.Header
+ cf.Name = filepath.Base(path)
+ cf.Path = path
+ cf.Dir = filepath.Dir(path)
+ cf.Body = dataBytes
+ cf.Uid = atsUid
+ cf.Gid = atsGid
+ cf.Perm = 0644
+ }
+ r.configFiles[cf.Name] = cf
+ }
+ }
+ return nil
+}
+
+// looks up the tm.toolname parameter from traffic ops.
+func (r *TrafficOpsReq) GetHeaderComment() string {
+ var toolname string = ""
+ out, err := r.atsTcExec("system-info")
+ if err != nil {
+ log.Errorln(err)
+ } else {
+ var result map[string]interface{}
+ if err = json.Unmarshal(out, &result); err != nil {
+ log.Errorln(err)
+ } else {
+ _tn := result["tm.toolname"]
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rob05c commented on pull request #4872: DRAFT: traffic_ops_ort.pl transliteration to go
Posted by GitBox <gi...@apache.org>.
rob05c commented on pull request #4872:
URL: https://github.com/apache/trafficcontrol/pull/4872#issuecomment-656221059
I think most of us who work with ORT and ATS in Production are just more familiar with the prod Perl script. Learning and working with the Python would introduce another step of having to learn thousands of lines of code, and then we would have to verify that whatever we did with it has parity with the Perl used for Production, so we'd still have to learn and understand the Perl script. It's just more work, more to learn, and more to test.
That's how I would feel if I were doing it, anyway. I don't have any philosophical objections to working with the Python; it would just take me longer.
But at the end of the day, IMO it should be up to the implementer. Whoever is implementing it knows what's fastest and safest for them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org